Beyond the Next Shiny Thing - Building Better Decision-Making Capability in the Age of Data and AI

2 days ago

6 min read

Updated: 6 hours ago

Organisations today are surrounded by new tools.

A few years ago, the focus was dashboards, coding and analytics. Today, it is Generative AI, copilots and agents.

The tools may change, but their value ultimately depends on whether they help organisations understand a problem, make a better decision, or execute that decision more effectively.

The risk is that organisations keep chasing the next shiny thing without first asking a more basic question:

How should this decision be made?

For practical purposes, we think organisational decision-making can be viewed through three broad modes: judgement-led, evidence-led, and machine-assisted or automated.

Each has strengths. Each has limitations. And most real decisions use some combination of them.

1. Judgement-led: you know the road

A person decides based on experience, intuition and context.

This is often the fastest and least resource-intensive approach. It works particularly well when the situation is familiar, the stakes are relatively low, the issue is difficult to quantify, or a decision needs to be made quickly.

Think of an experienced manager handling an unhappy client. There may be no time to gather data and run an analysis. Judgement matters.

The weakness is that judgement can also be affected by bias, overconfidence and past experience that may not apply to the current situation.

2. Evidence-led: you check the map

A person still decides, but first tests assumptions against data and evidence.

This takes more time and effort, but it provides a clearer basis for the decision and a better opportunity to learn from what happens afterwards.

An organisation deciding whether to renew a training programme, for example, may examine participation, feedback, application at work and business outcomes before making the call.

Evidence-led decision-making does not mean judgement disappears.

It means judgement becomes more explicit, testable and informed by evidence.

Data does not eliminate bias either. Bias can enter through the question we ask, what we choose to measure and how we interpret the result.

3. Automated: you switch on cruise control

At this level, a system increasingly assists, recommends, executes or makes decisions within boundaries set by people.

This can range from simple rules-based automation to AI-assisted workflows and increasingly autonomous systems.

The cruise-control analogy is useful because most organisational automation today is not the equivalent of a fully autonomous vehicle.

The human still chooses the destination, decides when automation is appropriate, sets the boundaries and remains responsible for intervening when conditions change.

The attraction is obvious: speed, consistency and scale.

But automation changes the scale of risk too. A poor assumption embedded into a system can be repeated consistently across every case it touches.

So the more we hand over to machines, the more important it becomes to question the reasoning being embedded and scaled.

It is rarely just one

Real decisions rarely sit neatly inside one mode.

In practice, the most effective approach often combines two or all three.

Judgement + Evidence

Data informs the decision while a person weighs the context.

A hiring decision, for example, may combine assessment scores with an interview and the judgement of the hiring manager.

Evidence + Automation

A system acts based on data or predefined thresholds, while people monitor the outcomes.

Fraud detection is one example: unusual patterns may trigger an automated alert, while a person investigates the exceptions.

Judgement + Automation

The system handles routine cases while people take over when context or exceptions matter.

A chatbot may answer common questions automatically but escalate unusual or sensitive cases to an employee.

Judgement + Evidence + AI

All three can work together.

AI may prepare or process information, Analytics may provide evidence or checks, and a person may make the final trade-off.

A monthly business forecast is one example: AI may draft commentary, data supports the forecast, and management decides what assumptions or actions to accept.

The capability is not choosing one approach. It is knowing which combination fits the problem.

Critical Problem Solving is what holds the system together

Choosing between judgement, evidence and automation still requires a human capability underneath them.

We call this Critical Problem Solving.

The goal is not simply to question everything. It is to solve the right problem well.

Our working definition is:

Critical Problem Solving is the discipline of solving the right problem, questioning our assumptions, and changing course when the evidence demands it.

It starts with questions such as:

What are we actually trying to solve?
Why does it matter?
What are we assuming?
What biases might be affecting the way we see the problem?
What would success look like?
What would tell us that we are wrong?
And would we be willing to change course?

The same discipline applies across all three decision modes.

In judgement-led decisions, it challenges our experience and assumptions.

In evidence-led decisions, it challenges how we interpret the evidence.

In automated decisions, it challenges the reasoning we choose to embed and scale.

Bias does not disappear when we move from judgement to data or AI. It simply changes where it enters the decision.

Not every decision deserves the same treatment

The right approach also depends on the economics of the decision.

A low-stakes, reversible and familiar decision may not justify extensive analysis.

A consequential decision that is repeated frequently may deserve much more rigour.

And where a task is sufficiently well understood and happens at scale, automation may become attractive.

A useful way to think about this is through four questions:

Stakes: What happens if we get it wrong?
Frequency: How often does this decision occur?
Data: Do we have reliable information to support it?
Accountability: How much do we need to explain or justify what happened?

But there is another question that should come before implementation:

What would success look like — and what would tell us that this is not working?

Defining both matters. Otherwise confirmation bias, sunk costs or reluctance to admit failure can keep an ineffective decision alive long after the evidence has changed.

The approaches can evolve over time

The modes can also reinforce one another.

Imagine an organisation encounters a new problem and has little historical data.

It may initially have to rely on experience and judgement.

The organisation then observes what happens and starts collecting data.

Analytics helps test whether the original assumptions were right and improves the next decision.

As understanding increases, parts of the decision may eventually become sufficiently repeatable to automate.

One possible learning cycle is therefore:

Judgement → Evidence → Learning → Better Judgement

with automation introduced where it makes sense.

This is important: automation is not the destination for every decision.

Some decisions will always require people because they involve ambiguity, values, negotiation, context or exceptions.

The objective is not to move every decision towards AI.

It is to continually improve the way the decision is made.

Better tools do not automatically produce better decisions

We already live in a world awash with data.

Organisations have dashboards, reports, surveys and sophisticated Analytics, yet more data does not automatically create better insight.

It is entirely possible to be drowning in data while still being starved of useful insight.

AI now creates a similar challenge.

Generative AI can produce answers, recommendations and analysis in seconds. As organisations rush to become “AI-first”, the volume of plausible-looking output will only increase.

But more output does not automatically mean more value.

The challenge increasingly becomes:

Can we recognise the gems from the dirt?

Can we tell the difference between an important insight and an interesting pattern?

Between a plausible AI answer and a sound one?

Between a genuine opportunity for automation and a solution searching for a problem?

Perhaps the scarce capability in an age of abundant Data and AI is not simply the ability to produce more analysis or more answers.

It is knowing what deserves our attention, confidence and action.

Perhaps the real advantage is optionality

Organisations today are often encouraged to become “data-driven” or “AI-first”.

We think that framing may be too narrow.

There will always be decisions where human judgement is appropriate.

There will be others where evidence should play a much larger role.

And there will increasingly be tasks where automation makes economic and operational sense.

The advantage may therefore come from having all three capabilities available — and enough critical problem solving capability to know how to combine them.

Judgement provides experience and context.

Analytics provides evidence and learning.

AI and automation provide assistance, execution and scale.

Critical Problem Solving helps us decide what problem to solve, which approach fits, and when we need to change course.

Using the road analogy:

Judgement — you know the road.
Evidence — you check the map.
Automation — you switch on cruise control.
And Critical Problem Solving helps determine where we are trying to go, whether the route still makes sense, and when it is time to take back the wheel.