What bad forecasts say about pharmaceutical and biotech Innovation
From 2005 to 2012, I was a financial analyst in the city. I forecast drug company earnings and used the forecasts to recommend drug company stocks to buy or sell. Like most investment professionals, my accuracy approached that of a monkey throwing darts at the Financial Times. However, my professional pride grew when I was told by a former colleague – a management consult with drug industry clients – that analysts like me were, in aggregate, better at forecasting the success or failure of drugs than some of the companies producing them.
I asked him how drug companies came up with such inferior forecasts. I can’t remember his exact words, but the answer might have been like this: “They get all the top global experts, the ‘Key Opinion Leaders’, in that disease area – maybe 10 or 15 top professors in basic science and in clinical medicine – and they fly them to a workshop meeting where they also get their own scientists, a bunch of health economists, specialist market researchers, regulatory experts, etc., etc. The workshop kicks off a project involving tens of people doing intense research, and 12 months later they have their forecasts.”
He asked me how, as a stock market analyst, I worked out my superior forecasts. Again, I can’t remember my precise answer, but it may have been along the lines of: “For the really important drugs, I do a very cheap, very quick, and very scaled-down approximation of the process you just described. However, some of the time I just guess. Guesses often happen late at night, when I am tired. I need fresh numbers before the stock markets open the next morning, otherwise I will look stupid and lazy compared with the other analysts. I try to make sure that the guess is not obviously mad and I sometimes avoid round numbers, so the guess looks less like a guess.”
I had more or less forgotten this exchange until October 2013, when I came across a report written by a firm of management consultants (McKinsey) and published in Nature Reviews Drug Discovery . The report tended to both verify and quantify the preceding conversation. The report showed that financial analysts are often wildly inaccurate (i.e. monkeys throwing darts). The analysts’ efforts are shown in the figure below, which is taken from the report. The report was coy about the details of the drug companies’ own forecasting efforts, perhaps not wishing to embarrass corporate clients. However it strongly hinted that drug companies were often even worse than the analysts.
Figure Legend. a | The majority of consensus analyst forecasts are off by more than 40%. ‘Consensus’ is an average of the analysts’ forecasts, so is more precise than the individual analysts’ forecasts, which would be even more spread out. The graph shows the percentage difference in estimated versus actual peak sales (calculated as the consensus estimate of peak sales minus the actual peak sales divided by the actual peak sales) for 260 drugs forecasted in the year before launch. b | Variance in consensus estimate versus actual peak sales. Although forecast error decreases over time, it remains as high as 45% even 6 years after launch.
I think that such bad forecasts say something important about pharmaceutical innovation. Bad forecasts tell us that the people who are the most expert and best informed are often hopelessly wrong about drug candidates’ real-world prospects. Even late in the day, when nearly all of the science has been done, no-one really knows whether a drug will come to market and be widely embraced by the patients and doctors who use and prescribe it: Not the drug company nor its scientists; not the “Key Opinion Leaders” in that field of medicine; not the city analysts nor the drug company shareholders; not the drug regulator; not NICE; not the NHS. The truth is that a drug’s real-world utilisation, and its real-world usefulness, only becomes clear after years of real-world experience by patients and doctors.
If, as the bad forecasts suggest, drugs’ real-world usefulness is genuinely unpredictable, then rapid pharmaceutical progress requires more drugs brought to market more cheaply. The role of the drug industry should be to provide the maximum of therapeutic variation on which real-world selection then acts. Regulation and pricing should emphasize the creation of acceptably safe variation and its subsequent real-world selection by patients and doctors. It is a mistake to insist on too much “evidence” before launch if such evidence evidently fails to support accurate predictions of drugs’ ultimate adoption and utility. At the same time, demands for relatively uninformative “evidence” increase cost and reduce the variation on which real-word selection by patients and doctors can act.
Sadly, it looks as if donation-seeking medical charities, grant-seeking academic scientists, and investment-seeking drug companies have spent so long telling everyone that new drugs are the predictable consequence of a rational scientific design process that something truly awful has happened. Policy makers have started to believe them and act as if it were true.
Dr Jack Scannell is an independent consultant