Coming on the heels of the U.K. general election, the Israeli general election, the Scottish referendum and the U.S. midterms, Sunday’s Greek referendum looks like the latest in a series of bad outcomes for pre-election polls across the globe. While the last few polls before the vote showed “Oxi” (“no”) ahead by just 3 to 4 percentage points, it in fact took 61 percent of the vote to 39 percent for “yes,” a margin of more than 22 percentage points. It was a landslide: “Yes” didn’t win a single parliamentary constituency.
But Greece’s pollsters have some pretty good excuses. The referendum was only called on June 27, meaning that the campaign was compressed into a period of about a week. (It’s a rough analogy, but American readers might recall how wild polls of the New Hampshire primary can be in the week after the Iowa caucus.) Furthermore, there was widespread uncertainty about what a “no” or “yes” vote meant, with voters and campaigners on both sides interpreting the referendum in ways that went well beyond the highly technical language on the ballot itself. Public opinion is liable to be volatile under these circumstances, so the polling will be volatile too if it reflects public opinion accurately.
And that’s what happened in Greece. The first several surveys after the referendum was announced showed “no” ahead by margins ranging from 10 to 25 percentage points. “Yes” then pulled ahead in a handful of polls, although never by more than a couple of percentage points. “No” rebounded late last week and led narrowly in the last few polls before election day.
Not all the failure can be explained by a last-minute swing in public opinion, however. Three polls conducted Sunday, the day of the election, also showed “no” winning by only 3 to 4 percentage points, much less than its actual margin. Another challenge is that the polls were conducted at a time of crisis in Greece. Beyond the depression-like conditions it was already facing, Greece defaulted on its debt Wednesday. Normal ways of life were disrupted, to the point that it was hard for Greeks to purchase ordinary goods and services. In other words, Greek voters may have had more pressing concerns than sitting around at home and waiting for a pollster’s phone call.1
It’s also plausible that those voters whose lives were most disrupted — those who were standing in line for hours to get cash, for example — were less likely to have the luxury of answering a poll but also more likely to vote “no” on the referendum. That could have biased the polls against “no” voters.
All of this is easier to consider after the fact, of course.2 Still, these are not arcane or abstract problems. The polls were conducted at a time of national emergency, on a referendum that was called just one week prior and that nobody was quite sure how to interpret. That seems like a recipe for a polling disaster.
And yet, rather than being appropriately cautious, the conventional wisdom was reasonably confident that the “yes” side would prevail.
When I use the term “conventional wisdom” in this article, I mostly mean the opinions of political pundits and journalists. In the case of Greece, however, the failure also extended to betting and financial markets.3 One bookmaker, Paddy Power, was so convinced that “yes” would win that it pre-emptively paid out “yes” bets. Most banks and financial institutions expected a “yes” vote. Betting markets like Betfair continued to show “yes” favored even after the polls had turned back toward “no.”
It’s not quite clear why these markets were so confident. The largest margin for “yes” in any poll during the campaign was just 3 percentage points, and even that poll showed support below 50 percent, an important consideration because undecided voters often vote “no” on referendums. Even under the best of circumstances, the polls ought not to have inspired all that much confidence in a “yes” vote, and these weren’t the best of circumstances.
There is another possible problem in Greece, however: pollster herding, which is the tendency of some polling firms to suppress (or manipulate4) results that are out of line with the consensus view of the election. In the U.K. general election in May, for example, the pollster Survation declined to publish a poll showing the Conservatives up by 6 points — about their actual margin of victory — because the company said it was “so ‘out of line’ with all the [other] polling.” And in the Virginia Senate race last year, at least two polling firms suppressed results showing a tight race between Mark Warner and Ed Gillespie because they evidently didn’t believe them. (Warner won, but by a much narrower margin than the polls predicted.)
When pollsters start herding, everything becomes a hot, sticky mess. Polls lose their independence from one another, and also from the conventional wisdom, since pollsters may be reluctant to publish “outliers” that are deemed out of line with the consensus.
There were some signs of herding in Greece. Whereas the first few polls were all over the place, the polls began to track one another closely by a few days into the campaign. And all polls conducted on election day itself showed nearly identical results, each with “no” ahead by 3 or 4 percentage points.
Considering how challenging it was to conduct polls under the conditions of the Greek referendum, we should have expected to see some healthy disagreement. Instead, we got a consensus, which proved to be quite wrong. As a practical matter, unfortunately, herding probably becomes more common the more challenging it is to conduct a poll. (And bad news: It’s becoming more challenging to conduct a poll.) Would a Greek pollster have been willing to publish a poll showing a 20-point lead for “no” when other polls — plus financial markets and commentators — expected a much tighter outcome? We don’t know, but based on the U.S. and U.K. experience, it might not.
Herding can also yield some funny outcomes when it comes to comparing the polls with the pundits. Imagine an election between two parties, the Red Party and the Blue Party. The polls show the Red Party ahead by, say, 5 percentage points. But the pundits read the “vibrations” and say the election is too close to call. A forecaster might aggregate this information and predict the Red Party to win by 2 or 3 percentage points, somewhere in between where the polls and the pundits have the race. If you prefer a diagram, it might look like this:
Sometimes this approach is a good idea. It wouldn’t have worked in the case of Greece, however, or in the U.S. elections of 2012 and 2014. Instead, in several of the recent cases in which the polls have been wrong, they’ve been wrong in the opposite direction of what the pundits anticipated. In the 2012 U.S. election, pundits saw the race as too close to call while polls showed Obama narrowly ahead. In fact, Obama not only won, but he also beat the polls (by about 3 percentage points in the average swing state). In the 2014 midterms, likewise, some mainstream and left-leaning commentators believed that polls showing Republicans ahead in key Senate races were underestimating Democratic turnout and were therefore “skewed” toward the GOP. In fact, Republicans won by wider margins than polls predicted. And in Greece, the conventional wisdom was more confident of a “yes” vote than the polls were (indeed, the polls showed “no” ahead by the end). Not only did “yes” lose — it got crushed.
Put another way, in several of these elections, the best forecast wouldn’t have come from blending polls with pundits’ opinions. Nor would it have come from using only polls (after all, the polls weren’t very good). To match the actual result, a forecast would have needed to employ a contrarian strategy of blending polls with the opposite of what pundits say:
So … the pundits are so bad that you should literally bet against whatever they say? Even I wouldn’t go that far. I’m happy to mostly ignore them instead. The problem, though, is that because of herding, it’s become harder to get what I really want — an undiluted sample of public opinion. Instead, there’s sometimes quite a bit of conventional wisdom baked into the polls. Be wary when there’s a seeming consensus among the polls, especially under circumstances that should make for difficult polling. There’s a good chance the polling will be wrong — but in the opposite direction of what most people are expecting.