Not so Superforecasting

Last November I posted here about Superforecasting and the Good Judgement Project (GJP), run by Professor Philip Tetlock at the University of Pennsylvania. Tetlock’s book, Superforecasting: The Art and Science of Prediction, had just been published. For more details about Superforecasting and my reservations about the technique, please read the post. One of Tetlock’s conclusions was that the top 2% of forecasters merit designation as “Superforecasters” and that their prognostications should be taken seriously.

During the UK “Brexit” referendum campaign up to Thursday 23 June, the Superforecasters’ estimated likelihood of the Leave campaign being successful was made available regularly on Twitter. Never expected to be more than about 40%, their final estimate, provided on polling day, was 24%, as can be seen below:

The result was a Leave win (with 52% of the vote). A couple of experts had anticipated the outcome, for example Nigel Smith on the Reaction website. Ironically, the campaign director for Vote Leave was Dominic Cummings, an enthusiast for Superforecasting (see my November post). Tetlock tweeted after the result was known:

and Cummings (@odysseanproject) was supportive :

(SW1 is the postal district which includes Westminster and Whitehall, ie the seat of government). 

Personally, I think Cummings is letting Superforecasting off lightly – they were way off on a topic which  is almost certainly of long-term significance regionally and beyond and has had more global attention than anything in the UK since the death of Princess Diana.

I suspect that most of the Superforecasters are US-based and were relying too much on the “received wisdom” generated by the London media.  These well-paid members of the intelligentsia have little understanding of attitudes outside the capital and its associated areas of prosperity eg Oxford, Bristol and so on. The Superforecasters probably also over-discounted the opinion polling in the light of the latter’s dismal performance before the UK general election in 2015. This time the polls didn’t do so badly and were indicating for most of June that the result would be very close, ie within their inherent margin of error of a few percent at best.

UPDATE 11 JULY

In the Financial Times FT Weekend 9/10 July there was an article by Robert Armstrong in their long-running Lunch with the FT series. The paper’s guest was Philip Tetlock and most of the reporting was a largely uncritical description of Superforecasting’s powers and of some good food. The lunch date had been before the Brexit vote,so Armstrong made a follow-up phone call:

I called Tetlock after the referendum result. As a fan of his work I was quite disappointed to hear that the “supers” got Brexit wrong too. Doesn’t this stumble on such a huge issue put his aspirations at risk? The supers did nail the Scotland referendum two years ago, confidently predicting that Scots would vote to stay in the Union when the opinion polls were very close, but the stumble on Brexit makes that look more like luck. 

“If they were only making predictions on those two topics, that would be a plausible hypothesis,” Tetlock replied. But the superforecaster teams have a much longer record, in which they do measurably better than luck or individual specialists. 

That Tetlock is totally unfazed by the Brexit miss is, in a sense, the whole point. He wants to replace the model of the all-knowing policy guru with something co-operative, empiricist; something fallible, but open to systematic incremental improvement. It is a vision of knowledge as a process of slowly but meaningfully upping the chance of being right — while acknowledging that it all remains a gamble.

Again I think Superforecasting is being allowed to get away with a failure on Brexit that should be unacceptable given the claims made. Here’s another forecast tweeted today:

Having to make an adjustment from 50% to 1% in a month rather undermines the notion of “supers” having a deeper and earlier understanding of what the future holds than anyone else! (I discussed discounting of late forecasts in the earlier post). I wonder whether this forecast is going to move much more before the 27th?