Such STRANGE times

The last few weeks of the EU referendum campaign in the UK and the days since the result became clear on 24 June have been some of the strangest I’ve known, the political sphere reaching into private life in an unprecedented and uncomfortable way. Life may revert to something nearer normal when the new Prime Minister* is in place and the country becomes more settled on its future path, uncertain though it is sure to be.

By the end of the year there will be books galore attempting to explain what has been going on. The journalist Tim Shipman, for example, is promising All Out War, the inside story of Brexit. By then the strange affair of the email sent on 28 June by Sarah Gove (Michael Gove’s wife, better known as the journalist, Sarah Vine) will probably be little more than a footnote, some background to her husband’s decision to run for the party leadership and Boris Johnson’s to withdraw. For the record though, here it is:

The email originally came into the hands of Sky News, though the clearer image above comes from Guido Fawkes. According to Sky:

An email sent to Michael Gove by his wife reveals concerns about the support Boris Johnson has in the party and the media. The email, which was also sent to the Justice Secretary's aides, was passed to Sky News.

Oddly, the email refers to “you” and “your” three times each, but also to “Michael”, as if he were a third party like “Henry” and “Beth”. Or as if all those three were copy addressees, not the main one.

When Gove emerged as a candidate, Ian Leslie, quite justifiably drew attention to a long New Statesman article presciently titled Michael Gove, the polite assassin, which he wrote back in October 2015. Fascinating throughout, one passage struck me:

…[Gove’s] closest adviser, Dominic Cummings. It is impossible to understand Gove’s time at Education, or indeed Gove, without considering his relationship with the man described by Nick Clegg as “loopy” and by others as brilliant or bullying, or both. … 

Cummings, like Gove, has a love of argument, as well as a suspicion, bordering on contempt, for those who compromise, muddle through and fail to pick sides. But he doesn’t have Gove’s politesse. He cares little – or even notices – what people think of him. In a departmental meeting, Gove might make his dissatisfaction clear by his tone, but it would be Cummings who told the civil servants they were a shambles, or who shut meetings down abruptly, and Cummings who sent around hectoring emails, with liberal use of capital letters, to staff in the department.

*The new PM – see my post in 2013, The Oxford Incumbency. From the Table there (2015A was, of course, the outcome last year), it would seem almost inevitable that the next Prime Minister will be either Teresa May or Michael Gove, the other three contenders not having gone to Oxford. May currently seems the far more likely of those two. However, these are strange times when referendums overturn the status quo – so who knows?

UPDATE 11 JULY

Teresa May will become Prime Minister on 13 July – the Oxford Incumbency continues, things must be getting back to normal!

Not so Superforecasting

Last November I posted here about Superforecasting and the Good Judgement Project (GJP), run by Professor Philip Tetlock at the University of Pennsylvania. Tetlock’s book, Superforecasting: The Art and Science of Prediction, had just been published. For more details about Superforecasting and my reservations about the technique, please read the post. One of Tetlock’s conclusions was that the top 2% of forecasters merit designation as “Superforecasters” and that their prognostications should be taken seriously.

During the UK “Brexit” referendum campaign up to Thursday 23 June, the Superforecasters’ estimated likelihood of the Leave campaign being successful was made available regularly on Twitter. Never expected to be more than about 40%, their final estimate, provided on polling day, was 24%, as can be seen below:

The result was a Leave win (with 52% of the vote). A couple of experts had anticipated the outcome, for example Nigel Smith on the Reaction website. Ironically, the campaign director for Vote Leave was Dominic Cummings, an enthusiast for Superforecasting (see my November post). Tetlock tweeted after the result was known:

and Cummings (@odysseanproject) was supportive :

(SW1 is the postal district which includes Westminster and Whitehall, ie the seat of government). 

Personally, I think Cummings is letting Superforecasting off lightly – they were way off on a topic which  is almost certainly of long-term significance regionally and beyond and has had more global attention than anything in the UK since the death of Princess Diana.

I suspect that most of the Superforecasters are US-based and were relying too much on the “received wisdom” generated by the London media.  These well-paid members of the intelligentsia have little understanding of attitudes outside the capital and its associated areas of prosperity eg Oxford, Bristol and so on. The Superforecasters probably also over-discounted the opinion polling in the light of the latter’s dismal performance before the UK general election in 2015. This time the polls didn’t do so badly and were indicating for most of June that the result would be very close, ie within their inherent margin of error of a few percent at best.

UPDATE 11 JULY

In the Financial Times FT Weekend 9/10 July there was an article by Robert Armstrong in their long-running Lunch with the FT series. The paper’s guest was Philip Tetlock and most of the reporting was a largely uncritical description of Superforecasting’s powers and of some good food. The lunch date had been before the Brexit vote,so Armstrong made a follow-up phone call:

I called Tetlock after the referendum result. As a fan of his work I was quite disappointed to hear that the “supers” got Brexit wrong too. Doesn’t this stumble on such a huge issue put his aspirations at risk? The supers did nail the Scotland referendum two years ago, confidently predicting that Scots would vote to stay in the Union when the opinion polls were very close, but the stumble on Brexit makes that look more like luck. 

“If they were only making predictions on those two topics, that would be a plausible hypothesis,” Tetlock replied. But the superforecaster teams have a much longer record, in which they do measurably better than luck or individual specialists. 

That Tetlock is totally unfazed by the Brexit miss is, in a sense, the whole point. He wants to replace the model of the all-knowing policy guru with something co-operative, empiricist; something fallible, but open to systematic incremental improvement. It is a vision of knowledge as a process of slowly but meaningfully upping the chance of being right — while acknowledging that it all remains a gamble.

Again I think Superforecasting is being allowed to get away with a failure on Brexit that should be unacceptable given the claims made. Here’s another forecast tweeted today:

Having to make an adjustment from 50% to 1% in a month rather undermines the notion of “supers” having a deeper and earlier understanding of what the future holds than anyone else! (I discussed discounting of late forecasts in the earlier post). I wonder whether this forecast is going to move much more before the 27th?

Not so Little England

The December 2015 issue of Prospect magazine seems particularly exercised by the possibility that the UK could leave the European Union. Anatole Kaletsky tells readers that “Leaving Europe could be the biggest diplomatic disaster since losing America” – An ugly divorce. Peter Kellner, President of pollster YouGov, warns that “It cannot be taken for granted that voters will keep Britain in the EU” and Edward Docx (Word ® users should resist reading that as edward.docx) thinks that the “in” campaign is faltering, while the “out” campaign has the benefit of Dominic Cummings, who he describes as “ferocious, committed, unafraid, serious, passionate and utterly certain of his cause”. In his article, called The problem with the EU Debate? The “In” campaign… and the “out” campaign on the Prospect website and Nigel Farage’s Dream in the magazine*, Docx warns:

Make no mistake: in less than a year, Great Britain could be out of the EU and no longer Great or, indeed, Britain. David Cameron’s departure will surely follow Brexit, which will also be followed by Scotland’s attempted split from Britain. The splenetic strain of the Conservative Party will be left running Little England—for that is what we will be—and its business for decades to come will be the treaty-by-treaty renegotiation of our relationship with every other country in the world.

Perhaps he’s right, but just how small would this “Little England” be? To answer that, it first has to be defined and presumably for Docx it would consist of the UK less Scotland, that is to say, the countries of England, Wales and Northern Ireland.

In terms of area, he clearly has a point as the table above shows. During the discussion of Scottish independence in 2014, rUK, (r meaning rump, residual or ‘rest of’), was used rather than Little England, so for brevity I will use it here. rUK has only 68% of the area of the present UK, something which would be reflected in comparisons with the other major EU countries (and the USA):

The UK is the 8th largest EU country (of 28) in terms of area, between Italy and Romania. Curiously, rUK would still have been 9th between Romania and Greece. Scotland would be 15th between the Czech Republic (up to 14th in rUK’s absence) and Ireland. However, in population terms, because Scotland has only 8% of the UK’s people, the difference is rather different, as the next table shows:

So, within the EU, where UK is 3rd largest between France and Italy, rUK would have only dropped to 4th between Italy and Spain. Scotland on its own in the EU would be 19th between Slovakia (up to 18th in rUK’s absence) and Ireland.

Another way of presenting these conclusions is to plot population size against area as below. This shows that rUK would have remained one of the handful of EU countries with a population over 20 million, while Scotland would join the ‘shrapnel’ of over 20 smaller ones. Of course, the disproportionate influence of these countries in terms of their population is an important aspect of the EU’s democratic deficit. It might also explain in part why EU membership would be so attractive to an independent Scotland.

While on the subject of population, the UK Office for National Statitics (ONS) has recently published its projections to 2035. As the table below indicates, if Scotland were to become independent by 2025, the rUK population would be 5.6 million lower than the UK’s would have been. However, by 2035 rUK’s population has increased to the same level as the UK’s had been 10 years earlier.

Even more instructive are the equivalent Eurostat projections out to 2080. By 2050, the UK has become the most populated country in Europe. However, so would be rUK by 2060 – hardly “Little England”. (This table is calculated assuming that Scotland’s population remains at the same percentage of the UK’s from 2030 onwards, whereas the ONS figures show a downwards trend from 2015 onwards).  The quality of life in such a densely populated country as rUK (essentially England) will become is also likely to be on a downward trend.

Finally, and probably most importantly for many who will vote in the Brexit referendum: perceptions of the economic benefits of leaving the EU. The chart below, again from Eurostat, shows GDP per capita in PPS**relative to the EU’s overall 100. Many people would expect rUK to be better off than the UK, which now makes net payments to the EU while rUK is making transfers within the UK to Scotland. Conversely, Scots could be expected to be worse off. Perhaps the “in” campaign, rather than worrying about “Little England”, should concentrate on disproving this expectation – if they can get their arguments past Dominic Cummings.

* On Docx’s own website, it’s called Are we Sleepwalking to Brexit?

**According to Eurostat: “Gross domestic product (GDP) is a measure for the economic activity. It is defined as the value of all goods and services produced less the value of any goods or services used in their creation. The volume index of GDP per capita in Purchasing Power Standards (PPS) is expressed in relation to the European Union (EU28) average set to equal 100. If the index of a country is higher than 100, this country's level of GDP per head is higher than the EU average and vice versa. Basic figures are expressed in PPS, i.e. a common currency that eliminates the differences in price levels between countries allowing meaningful volume comparisons of GDP between countries. Please note that the index, calculated from PPS figures and expressed with respect to EU28 = 100, is intended for cross-country comparisons rather than for temporal comparisons."

Superforecasting

I’m going to assume that anyone who reads this post has almost certainly been landed here by Google, and therefore is likely to be familiar with the Good Judgement Project (GJP), the brainchild of Professor Philip Tetlock at the University of Pennsylvania and described in his and Dan Gardner's recently published book, Superforecasting: The Art and Science of Prediction. So I won’t provide a description of the GJP or the book.

Before going any further, I should say that I participated in the last GJP tournament and my score over some 135 questions was undistinguished – well below that of the top 2% who Tetlock defines as superforecasters (SFs). So you are welcome to dismiss anything that follows as sour grapes.

Undoubtedly, the book has been reviewed with enthusiasm in the UK by, among others , John Rentoul (not once, not twice “a terrific piece of work that deserves to be widely read”, but three times), Dominic Cummings (“Everybody reading this could do one simple thing: ask their MP whether they have done Tetlock’s training programme.”), Daniel Finkelstein (“This book shows that you can be better at forecasting. Superforecasting is an indispensable guide to this indispensable activity.”) and Dominic Lawson (“fascinating and breezily written”).

A key point to appreciate about individual SFs is that their capability as forecasters is based not so much on what they know - the forecasting questions range widely, so being a subject matter expert is not an advantage - as the way they set about it.. Nor are SFs mostly in the IQ top 1% (135+) of the population, although they are in the top 20% for intelligence and knowledge (Page 109). However, they “are almost uniformly highly numerate people” (Page 130) and “embrace[d] probabilistic thinking” (Page 152). The other factors in their success and how anyone could set about improving their own forecasting skills is the subject of the book.

SFs, being numerate, will have no problem with the Brier scoring system used by the GJP. Brier scoring may well be an eye-glazing subject for most people, but it is one that ought to be understood if an appreciation of superforecasting is to be more than superficial. Hence the Annex below for anyone interested.

Superforecasting, having more significant issues to address, does not bother the reader with much on the way the Brier score is calculated, but it does feature on pages 167/8 when the importance of objectively updating forecasts in the light of new information is being emphasised. A GJP question being asked in the first week of January 2014 was whether “the number of Syrian refugees reported by the UN Refugee Agency as of 1 April 2014” would be under 2.6 million. This chart* from page 167 shows successive forecasts made by one SF, Tim:

On page 168 the reader is told that “Tim’s final Brier score was an impressive 0.07.” The smaller the Brier Score, the better the forecast, of course. By my reckoning, that is the score which Tim would have received if he had made an initial forecast of 81% probability of the answer to the question being “Yes” and stuck with it. But, as is clear, he started with slightly better than evens in early January and then moved towards certainty in late March, thereby achieving SF status, on this as on many other questions.

But what if the UNRA had needed its forecast in January for planning purposes and a more accurate forecast in March would have been too late? The next chart attempts to break out Tim’s forecasts on a monthly basis. 

For January it looks as though Tim’s average probability was about 67% so the Brier Score for this month would have been a 0.22, rather poorer than the glittering 0.07. The GJP methodology is time-independent in the sense that it does not attach a greater value (weighting) to early forecasts as opposed to later ones. It might be interesting to see the effect of some discounting, as in (but opposite in time to) Discounted Cash Flow where early cash flows are valued more highly than late ones. This would seem appropriate if an important driver of a real-world forecast is someone’s need to make decisions well before outcomes are known.

Another time-related aspect of the GJP scoring is the termination date of the question. A scenario from the past might be helpful. On 30 September 1938 the leaders of Britain, France, Germany and Italy signed the Munich Pact allowing German occupation of the Sudetenland. The British Prime Minister, Neville Chamberlain, returned to London (above) where he was greeted by enthusiastic crowds and spoke of “peace for our time”. However, Churchill said that “England … has chosen shame, and will get war”. So a GJP question in October 1938 might well have asked “Will Britain and Germany be at war by 31 August 1939?”.

A Chamberlain supporter would have started his (or her) answer at a probability much less than 50%, a Churchill supporter at a substantially higher one. As the events of 1939 unfolded, both parties would probably have increased their estimates. However, when the question closed, the Chamberlain supporter would certainly have had the lower Brier Score. Of course, if the question had been “Will Britain and Germany be at war by 30 September 1939?”, their Brier Scores would have been reversed** and the Churchill supporter’s initial pessimism would have been vindicated. So, depending on the date of the question to within just a few days, a good (ie low) Brier Score can be obtained for questionable judgement in a broader sense. There were 135 questions in the recent GJP tournament, 23 of those ended on 31 May and another 39 between 1 and 10 June, dates which probably have more to do with Penn U’s academic year than geopolitics.

It is probably worth bearing in mind that SFs are people adept at minimising time-independent Brier Scores on questions which terminate on arbitrary dates and that is the basis on which their ability as forecasters has been assessed.

As pointed out earlier, the reviews of Superforecasting have been enthusiastic and keen to see its approach being adopted in business and government. I came across this interesting comment on Forbes/Pharma & Healthcare by an admiring reviewer, Frank David, who works in biomedical R&D:

So, companies may be able to nurture more “superforecasters” – but how can they maximize their impact within the organization? One logical strategy might be to assemble these lone-wolf prediction savants into “superteams” – and in fact, coalitions of the highest-performing predictors did outperform individual “superforecasters”. However, this was only true if the groups also had additional attributes, like a “culture of sharing” and diligent attention to avoiding “groupthink” among their members, none of which can be taken for granted, especially in a large organization.“ A busy executive might think “I want some of those” and imagine the recipe is straightforward,” Tetlock wryly observes about these “superteams”. “Sadly, it isn’t that simple.” 

A bigger question for companies is whether even individual “superforecasters” could survive the toxic trappings of modern corporate life. The GJP’s experimental bubble lacked the competitive promotion policies, dysfunctional managers, bonus-defining annual reviews and forced rankings that complicate the pure, single-minded quest for fact-based decision-making in many organizations. All too often, as Tetlock ruefully notes, “the goal of forecasting is not to see what’s coming. It is to advance the interest of the forecaster and the forecaster’s tribe,” [original emphasis] and it’s likely many would find it difficult to reconcile the key tenets of “superforecasting” with their personal and professional aspirations.

Very likely - “the toxic trappings of modern corporate life” - how true indeed.


* I may be taking the chart on page 167 too seriously, but a couple of points. The grey space above a probability of 1 is ,of course, just artistic licence. However, I don’t understand (apart from further artistic licence) why the successive forecasts have been joined by straight lines as shown. Surely a forecast remains extant until it is superseded by the next one, and the forecast line is a series of steps, as shown for January 2014 below?

** US readers, and this blog has a few, may need to be reminded that the UK declared war on Germany on 3 September 1939. Germany declared war on the USA on 11 December 1941.


Only superforecasters working in teams are likely to approach perfection - please let me know by commenting if you spot any mistakes in this post!


ANNEX

The red line is the Brier Score, S, as a function of (1-p), the dotted line for comparison is S = (1-p).


UPDATE 30 JUNE 2016

The Superforecasters' prediction for the UK "Brexit" referendum earlier this month was less than stellar - see this post.

The cleverest of the cleverest

When somebody tweets something that reinforces your own opinions and prejudices, it’s probably wise to take a closer and sceptical look. For example this from Dominic Cummings (former SPAD* for the former education secretary, Michael Gove) aka @oddyseanproject:

"DATA the cleverest of the cleverest do math/phys/engineering; the thickest of the cleverest do education & business"  

Naturally I’ve thought this for years and, of course, also believe that anything which comes from xxx.blogspot.co.uk or yyy.blogspot.com must be wisdom incarnate. Better still, the post a year ago on Psychological comments, Dr James Thompson’s blog, included this chart, well-matched to the math/phys/engineering way of looking at things:

But what exactly is going on here? Thompson took the chart from a paper in the Journal of Educational Psychology in 2009 (I’ve reproduced Figure B1 in the paper rather than Thompson’s for clarity). V, S and M stand for Verbal, Spatial and Mathematical Ability. As I understand the paper, and I am no expert, V,S and M are composites, eg V is made up of three weighted measures: Vocabulary, Reading and English, the last of these being itself a composite of items measuring capitalization, punctuation, spelling, usage, and effective expression. S and M are similarly constructed from appropriate sub-measures.

So the vertical axis, Specific Ability Level, is the measurement of V, S and M for nine disciplines spread along the horizontal axis (the blobs, ha-ha). The lines join up V, S and M scores for three levels of educational attainment in those disciplines: Bachelor, Master, Doctoral. The order in which the disciplines appear along the horizontal axis corresponds to their General Ability Level which is “the average of S+M+V” (presumably the mean of S, M and V). To dig deeper, look at Appendix B of the paper. The numbers behind some of the blobs are small eg 71 engineering doctorates, 57 maths/computer science.

The purpose of the paper was to argue the case for spatial ability as a predictor of talent in STEM subjects. It was Thompson who suggested “Draw your own conclusions about the levels of intellect required in each discipline.”, something which Cummings choose to do, surprisingly given that he has a humanities degree. However, I think there are two points to be made. Firstly, we are looking at “the cleverest” to use his words, who all have more “Ability” to get higher scores in tests than the average person – beware the vanity of small differences among a select group. Secondly, if your occupation doesn’t require spatial abilities, and many don’t, or numeracy, but does require high verbal abilities, the average arts or humanities graduate has the edge over STEM types, as the chart shows. This was the point of a post here two years ago, OK techies have their limitations.

I can’t imagine an occupation with much lower spatial ability requirements than being a SPAD (*special political adviser) – perhaps Cummings knows how many of the current 100 or so actually have “math/phys/engineering” qualifications.

But a whimper

Not many people appear twice in the same issue of Private Eye but Dominic Cummings, a former special adviser (spad) for Michael Gove has managed it this week (No 1369 27 June – 10 July), under The New Coalition Academy (page 23) and Me and My Spoon (page 26):

This feat stems from an interview he gave Alice Thomson and Rachel Sylvester which made the front page of The Times on 16 June. He is quoted as saying:

As Bismarck said about Napoleon III, Cameron is a sphinx without a riddle – he bumbles from one shambles to another without the slightest sense of purpose. … Everyone is trying to find the secret of David Cameron, but he is what he appears to be. He had a picture of Macmillan on his wall – that’s all you need to know.

Subsequently he posted on his Wordpress blog A few responses to comments, misconceptions etc about my Times interview, referring in passing to an earlier post The Hollow Men (Part 1). This starts with a selection from TS Eliot’s poem:

Close up:

On Wordpress blogs the readers’ comments are called “thoughts”, and I offered one, a very short one: “Eliot”. This wasn’t allowed through. On this blog, when I’m offered an acceptable correction I not only let it stand but respond with “Thanks, corrected” or some such. Cummings’ blog is a much grander affair than mine, and he might well have regarded Western Independent’s input as a trivial attempt at attention-seeking off his back. So, if he’d not published but had corrected his misspelling of Eliot’s name, I really wouldn’t have minded.

But at the time of this post, it’s still “Elliot”. Readers can draw their own conclusions, particularly when he and Mr G have such a reputation as advocates for academic rigour.