Approaching uncertainty

When it comes to making difficult decisions, how honest are we with ourselves about what we don’t know?

Off

Decision-makers often have to act without having all the information that they need. Identifying the correct course of action may depend on knowing how the future will unfold or understanding how some complex process works, beyond our current scientific knowledge. Decision-makers may consult expert opinion, but even experts won’t know everything. It is all too easy for false assumptions to remain unstated or unchallenged, leading to bad decisions.

How can we be confident that all the key uncertainties have been properly thought through, and a course of action has been chosen accordingly?

This problem motivated Jeremy Oakley and Tony O’Hagan to develop SHELF, the Sheffield Elicitation Framework. Jeremy and Tony, who are Professors of Statistics at the University of Sheffield, developed SHELF to combat the uncertainties that can arise when making a decision. “It’s a process for capturing uncertainty and presenting it in a form that’s useful for decision-makers,” explains Jeremy.

SHELF starts by collecting all the relevant evidence about the topic in question. Experts on the subject are then gathered and asked to make judgements on the possible outcomes based on the information, independently at first. Their judgements are shared with each other, and a carefully managed discussion follows to reach a consensus in which both the agreements and disagreements between the experts are acknowledged.

The process is supported by software that enables visualisation and comparisons of the experts’ uncertainties in real time, and provides quantitative results that can be incorporated in decision models.

The framework has a broad range of applications, “SHELF can be used anywhere people are relying on expert opinion to do things,” Jeremy says. This is how came to use SHELF. As a world top 10 pharmaceutical company, GSK turned over £17 billion in pharmaceutical products in 2017 and are leaders in drug development. But drug development is a costly process.

The average cost of a failed clinical trial varies, but it’s estimated at between £600 million and £1 billion across the pharmaceutical industry. This begs the question: how sure are we really that a trial is going to succeed? The catch-22 here is that it’s not possible to know how well a drug really works until the trial is completed, but you need to do the trial to find out if the drug really works. At a London conference in 2012, former CEO of GSK, Andrew Witty, commented on the topic of drug pricing, “If you stop failing so often you massively reduce the cost of drug development.”

This is where SHELF and expert judgement comes in. At each stage in the drug development process there’s a decision to be made about whether or not to progress. Once identified as safe, with the potential to be effective, the drug is evaluated against existing medications.

Despite companies being led to believe that clinical trials will have as high as a 90% success rate, trends in the pharmaceutical industry show the success rate of phase three to be around 60-70%. In other words, up to a third of drugs fail to make it past this stage despite being safe and potentially effective.

Following the SHELF process, the experts pool their knowledge and evidence, think through the uncertainties, and pass on their assessment of how likely a planned trial is to succeed. By using SHELF to determine the likelihood of the trial succeeding, companies can save a substantial amount of time and money. “As a result of SHELF, lots of changes can be made to trial design,” says Nicky Best, the Head of Advanced Biostatistics and Data Analytics Centre of Excellence at GSK. GSK have put SHELF at the heart of their planning for clinical trials, and have already used it in 50 trials.

SHELF is now one of the gold standard tools for expert elicitation but it’s not alchemy. “If the experts really don’t know then that’s what SHELF will show,” says Jeremy. But it can certainly give us confidence that we have properly thought through what it is that we don’t know.