WHEN MORE INJURIES ARE ACTUALLY GOOD NEWS
Sometimes the most dangerous thing about data isn't that it is wrong.
It is that it is completely correct — but answering the wrong question.
There is a fascinating example from the First World War.
When the war began, soldiers generally went into battle wearing cloth, felt or leather headgear. In trench warfare, this offered little protection from one of the biggest dangers they faced: shell fragments, shrapnel and debris raining down from artillery fire.
Unsurprisingly, head wounds were often deadly.
So armies began introducing steel helmets. France introduced the Adrian helmet in 1915, Britain followed with the Brodie helmet, and other armies soon developed their own versions.
Then something rather strange appeared in the medical data.
The number of recorded head injuries increased.
Look at that number on its own and the conclusion seems obvious:
We introduced helmets to protect soldiers' heads.
Head injuries went up.
Therefore, the helmets were not working.
Except that this would have been almost exactly the wrong conclusion.
The missing data was on the battlefield
Before steel helmets, a soldier struck badly in the head by shell fragments might never appear in the hospital's injury statistics.
He was dead.
After helmets were introduced, some of those same impacts were no longer fatal. The soldier survived, reached a medical facility and was recorded as having a head injury.
So the number of surviving soldiers with head wounds could rise precisely because fewer soldiers were dying from them.
Historical US Army medical records describe this effect. Before helmets were introduced among French troops, roughly one in four recorded head wounds proved fatal. After their introduction, the fatality rate fell substantially, with some observations placing it at around one in seven.
The helmet had not necessarily increased the danger.
It had changed the outcome.
Death had become injury.
And ironically, that could make one particular metric look worse.
The first problem: asking the wrong question
Suppose we ask:
“Did head injuries increase after steel helmets were introduced?”
The answer may well be yes.
But that is not really the question we care about.
The purpose of the helmet was not simply to reduce the number of recorded head injuries.
It was to protect soldiers from serious and fatal head trauma.
A better question would therefore be:
“Did steel helmets reduce the likelihood that a head wound would kill a soldier?”
That is a very different question.
And once the question changes, the analysis changes with it.
The second problem: measuring the wrong metric
If we use the number of recorded head injuries as our main measure of success, the helmet may appear to have failed.
But is that really the right metric?
Perhaps we should instead look at:
fatal head wounds
survival rates after head injury
severity of injuries
deaths caused by head trauma
outcomes before and after helmets were introduced
Suddenly the picture becomes very different.
This illustrates one of the most important principles in analytics:
The metric must match the question.
Otherwise, we can perform completely accurate analysis on the wrong measure and still arrive at the wrong conclusion.
A beautifully constructed dashboard does not fix a poorly chosen KPI.
Neither does a more sophisticated statistical model.
And neither does AI.
The third problem: ignoring the context around the data
Even the right metric needs context.
Why did the number of recorded head injuries rise?
Because the introduction of helmets changed who survived long enough to become part of the dataset.
The population being measured had effectively changed.
Before helmets:
Severe head impact → death → possibly never recorded as an injury
After helmets:
Severe head impact → survival → recorded as an injury
The data wasn't lying.
The meaning of the data had changed.
And without understanding how the data was generated, an analyst could easily misinterpret the result.
The same problem appears everywhere
This isn't really a story about military helmets.
Versions of the same analytical problem appear in organisations all the time.
A hospital introduces better screening and suddenly diagnoses of a disease increase.
Has public health deteriorated?
Or has detection improved?
A company makes it easier for employees to report workplace incidents and the number of reported incidents increases.
Has the workplace become more dangerous?
Or are previously hidden problems now being captured?
A customer-service team introduces an easier complaint channel and complaints rise.
Has service quality collapsed?
Or has the organisation simply become better at hearing dissatisfied customers
A company strengthens its fraud-detection system and detected fraud increases.
Has fraud suddenly become more common?
Or has the organisation become better at finding it?
In each case, the number may be perfectly accurate.
What changes is its meaning.
Good analytics starts before the analysis
When people talk about data analytics, much of the attention goes to what happens once the data reaches the analyst:
cleaning the data,
building the model,
running the analysis,
creating the chart,
or now, asking AI to analyse it.
But some of the most important analytical decisions happen before any of that begins.
What problem are we actually trying to solve?
What question should we be asking?
What outcome are we trying to improve?
Which metric genuinely represents that outcome?
What changed in the environment?
And perhaps most importantly: Who or what is missing from the data?
These are not software questions.
They are thinking questions.
And this matters even more in the age of AI
Generative AI can already analyse large datasets, generate charts, identify patterns, summarise findings and suggest explanations at remarkable speed.
But speed does not eliminate the need for judgement.
Ask AI:
“Did head injuries increase after helmets were introduced?”
and it may correctly answer:
Yes.
Technically correct. Potentially disastrous.
The more valuable analyst asks:
“Is that actually the question we should be asking?”
And then:
“Are we measuring the right thing?”
And finally:
“What context might change how we interpret the result?”
That is the difference between processing data and reasoning with data.
The real analytical skill
The lesson from the steel helmet story isn't simply that we need more data.
More data can still give us the wrong answer if we frame the problem badly.
The deeper lesson is that good analytics requires three things:
Ask the right question.
Measure the right metric.
Interpret the result in the right context.
Get any one of those wrong and even flawless analysis can lead to a poor decision.
AI will undoubtedly make the mechanical parts of analytics faster.
But faster analysis of the wrong question simply gets us to the wrong answer faster.
The real value of a good analyst lies in knowing when to stop, challenge the obvious conclusion and ask one more question.
Sometimes even something as simple as:
Are injuries increasing because more people are being hurt — or because fewer people are dying?
Comments