If Statistics Cannot Give Us the Right Answer Every Time, Why Do We Trust It?

Understanding why imperfect predictions can still lead to better decisions.

Think back to mathematics lessons in school.

If an apple costs $2, buying five apples costs $10. Buy ten apples, and the cost doubles to $20.

The relationship is straightforward. We know the variables, we know how they relate to each other, and we can calculate the answer with certainty.

This is how many of us first learn to think about mathematics. Given the right formula and the correct inputs, we should arrive at the correct answer.

But what happens when we move from the classroom into the real world?

What if there isn't one clear answer, even when our mathematics is correct?

This is where statistics and inferential analytics become important. And it raises a question I think more people should ask:

If statistical models cannot predict outcomes with 100% accuracy, why do we continue to rely on them for important decisions?

When the rules are known, the answer is predictable

Let's return to our apples.

If every apple costs $2, the relationship between the number of apples and the total cost is perfectly predictable.

Double the quantity, and the cost doubles.

But suppose we wanted to predict how many apples a supermarket would sell tomorrow.

Now things become more complicated.

Sales might depend on price, weather, promotions, customer preferences, competing supermarkets, the day of the week and many other factors.

Even if we included all the factors we could think of, we would probably still get our prediction wrong to some extent.

Does that mean our mathematical model is useless?

Not necessarily.

It means we have moved from a problem where the outcome is determined by known rules to one where multiple factors interact, and some uncertainty remains.

And that distinction changes how we should judge the quality of our analysis.

We cannot predict every student, but we can identify useful patterns

Imagine we want to investigate whether students who study longer perform better in examinations.

We collect information from 1,000 students, recording their study hours and examination results.

We might discover that students who spend more time studying generally achieve higher scores.

But not always.

Some students who study for three hours score better than others who study for eight. Why?

Because examination performance depends on much more than study hours.

Prior knowledge, learning ability, study techniques, sleep, motivation and even how a student feels on examination day can influence the result.

We could include some of these factors in our analysis, but realistically, we are unlikely to capture everything.

Here is the important distinction:

We may not be able to predict exactly how one student will perform, but we may still be able to identify a reliable pattern across 1,000 students.

If the evidence consistently shows that students who study longer tend to perform better, that relationship is useful information.

It doesn't guarantee that studying an additional hour will improve every student's score. Nor does the relationship, by itself, prove that extra studying caused the improvement.

But it gives us a starting point for understanding what influences examination performance.

And with further investigation, we can develop a more informed view of which factors matter and how much they matter.

What does an imperfect statistical model actually tell us?

In regression analysis, one commonly used measure is R-squared, or its adjusted version.

Suppose we develop a model to explain examination results using study hours, attendance, sleep and previous academic performance.

The model produces an Adjusted R-squared of 0.70.

Put simply, the model explains a substantial proportion of the differences in examination results within our dataset, after accounting for the number of variables used.

But some variation remains unexplained.

Perhaps there are factors we haven't measured. Perhaps the relationships are more complicated than our model assumes. Or perhaps some outcomes involve an element of randomness.

Does an Adjusted R-squared of 0.70 mean our model is 70% accurate? No.

It does not mean that 70% of our predictions will be correct, or that there is a 70% chance our conclusions are true.

It tells us something about how well the model explains the observed differences in outcomes.

This is an important distinction because a model's ability to explain past observations is not necessarily the same as its ability to predict future outcomes.

We also need to test whether the findings hold up with new data.

And even where individual variables are statistically significant, that does not mean their effects are guaranteed or that they necessarily cause the outcomes we observe.

So why should we place any confidence in the findings?

Reliability is not the same as certainty

Consider tossing a fair coin.

Before each toss, we cannot know whether it will land on heads or tails.

Toss it ten times, and we might get seven heads and three tails.

But if we toss it 10,000 times, we would generally expect the proportion of heads to be much closer to 50%.

We still cannot predict the next toss.

Yet we can make increasingly reliable statements about the overall pattern as the number of tosses increases.

This illustrates an important principle in statistics: individual outcomes can be uncertain even when the underlying pattern is predictable.

Of course, real-world data is more complicated than a fair coin. Collecting more data does not automatically make our findings reliable. We must also consider whether the data is representative, whether our assumptions are reasonable and whether our analysis is appropriate.

Nevertheless, the principle remains.

Not being able to predict every individual outcome does not prevent us from learning something meaningful about the bigger picture.

We already rely on this thinking in everyday life

Consider healthcare.

Suppose carefully conducted clinical studies show that a particular treatment benefits 80% of patients with a specific condition.

Does that mean it will work for the next patient? Not necessarily.

But it gives the doctor useful evidence when considering whether to recommend that treatment, alongside its risks, alternatives and the patient's circumstances.

Or consider economics.

Economists cannot predict exactly how every household will respond to an increase in interest rates.

Some families may reduce spending. Others may have sufficient savings and continue spending as before.

Yet economists can still use data to estimate how borrowing, spending and investment might change across the wider economy.

Engineering offers another example.

An engineer cannot know the exact loads, material conditions and environmental stresses a structure will encounter throughout its entire life.

Instead, engineering design combines physical principles, empirical evidence, estimates of uncertainty and appropriate safety margins.

In all three cases, we are not eliminating uncertainty.

We are using evidence to understand uncertainty well enough to make informed decisions.

The level of confidence required also depends on the consequences of being wrong.

A supermarket forecasting tomorrow's apple sales can tolerate some error.

An engineer designing a bridge cannot accept the same degree of risk.

This is why statistical models need to be judged against their intended purpose, rather than by a single measure of how closely they fit the data.

So when should we trust a statistical model?

This is where our critical thinking becomes essential.

Just because a model produces an impressive Adjusted R-squared or identifies several statistically significant variables does not mean its conclusions should automatically be accepted.

We need to ask some basic questions.

Is the data representative of the population we want to understand?

Do the relationships make practical sense?

How large are the effects, and are they meaningful enough to matter?

How much uncertainty surrounds our estimates?

Do the results remain consistent when tested against new data?

And perhaps most importantly, are we merely observing a relationship, or do we have sufficient evidence to conclude that changing one factor will actually change the outcome?

That last question is particularly important.

If students who study longer perform better, it does not necessarily follow that requiring every student to study two additional hours will produce the same improvement.

Students who study longer may also be more motivated, better prepared or have more supportive learning environments.

Finding a relationship is not the same as proving what will happen when we intervene.

Good analytics requires us to recognise that difference.

The purpose of analytics is not to eliminate uncertainty

I think this is where we sometimes misunderstand the purpose of analytics.

In school mathematics, we are often rewarded for arriving at the correct answer.

But in business and public policy, the real objective is usually to make a better decision.

We rarely have perfect information.

We cannot know every factor influencing customer behaviour, employee performance, patient outcomes or economic growth.

And waiting until we understand everything before making a decision is rarely practical.

The more useful question is:

Do we have enough trustworthy evidence to make a better decision than we would have made without the analysis?

If our analysis helps us identify important factors, understand their likely effects, appreciate the uncertainty and evaluate alternative courses of action, it has created value.

Even if the model isn't perfect.

Indeed, a simpler model that explains the major influences on an outcome may sometimes be more useful than a complicated one that fits the historical data exceptionally well but offers little practical understanding.

The objective should never be to produce the most impressive statistical result.

It should be to develop a sufficiently reliable understanding of the problem to support a sensible decision.

Mathematics gives us answers. Statistics helps us understand uncertainty. Analytics helps us decide.

Perhaps this is the distinction we need to teach more clearly.

Mathematics allows us to calculate outcomes when the rules and inputs are known.

Inferential statistics allows us to draw conclusions from incomplete information, while recognising and measuring uncertainty.

Analytics brings that understanding into the decision-making process.

None of this removes the need for human judgement.

On the contrary, the less certain our conclusions, the more important it becomes to understand our assumptions, interpret the evidence and consider the consequences of being wrong.

A mathematically correct model can still lead to a poor decision if it addresses the wrong problem or if its limitations are ignored.

Conversely, an imperfect model can be extremely valuable if it helps us make better decisions.

Ultimately, that is what analytics should be about.

Not knowing everything. Not predicting everything. But knowing enough to make better decisions, while remaining conscious of what we do not know.