Want these election updates emailed to you right when they’re published? Sign up here.
Pretty much everyone has an incentive to push the narrative that the presidential race is tightening. The television networks would like for you to keep tuning in to their horse-race coverage. Hillary Clinton’s campaign would like for you to turn out to vote, instead of getting complacent. Donald Trump’s campaign would like you to know that its candidate still has a chance.
But what do the polls say? The race probably is tightening — but perhaps not as much as the hype on the cable networks would imply. In our polls-only forecast, Trump has narrowed Clinton’s lead in the popular vote to roughly 6 percentage points from 7 points a week ago, and his chances of winning have ticked up to 17 percent from 13 percent. In our polls-plus forecast, Trump’s chances are up to 19 percent from 16 percent. Because of the high level of uncertainty in the race, we can’t say the door is closed on a narrow Trump victory. And we’re certainly a week or two removed from the period when every poll brought good news for Clinton: Plenty of polls now show negative trend lines for her (in addition to others that show a positive trend). But the race hasn’t fundamentally changed all that much, and Clinton remains in a strong position.
The data, however, offers a lot of opportunities for cherry-picking, both because there are a lot of polls and because they don’t agree all that much with one another. That’s especially true of national polls.1 Recent national surveys show everything from a 14-percentage-point Clinton lead to a 1-point lead for Trump. On average, that works out to a Clinton lead of 5 to 6 points — but there’s a lot of variation.
How unusual is it for polls to show this wide a range of outcomes? In the table below, I’ve compiled a summary of national polls conducted during the final week of the campaign in elections since 1976. (I used just one poll per polling firm — the last one each released before the election.) For 2016, I’m showing the numbers for national polls since last week’s presidential debate. Note that this is not an apples-to-apples comparison: There’s a good chance the polling range will narrow over the final week of the campaign because of herding (pollsters sometimes get scared to publish results that seem like outliers as the campaign draws to a close). But it makes for some interesting data:
| LEADING CANDIDATE POLLING MARGIN | ||||||
|---|---|---|---|---|---|---|
| YEAR | NO. POLLS | LEADING CANDIDATE | HIGH | LOW | AVERAGE | STD. DEVIATION |
| 1976 | 3 | Jimmy Carter | +4 | -1 | +1.3 | 2.5 |
| 1980 | 4 | Ronald Reagan | +5 | +1 | +2.5 | 1.9 |
| 1984 | 5 | Ronald Reagan | +25 | +10 | +17.2 | 6.2 |
| 1988 | 8 | George H.W. Bush | +19 | +4 | +9.1 | 4.8 |
| 1992 | 9 | Bill Clinton | +8 | +2 | +5.7 | 2.3 |
| 1996 | 8 | Bill Clinton | +18 | +7 | +11.8 | 3.2 |
| 2000 | 9 | George W. Bush | +5 | TIE | +2.9 | 1.8 |
| 2004 | 14 | George W. Bush | +6 | -2 | +1.6 | 2.3 |
| 2008 | 18 | Barack Obama | +11 | +5 | +7.6 | 1.8 |
| 2012 | 22 | Barack Obama | +4 | -1 | +1.2 | 1.5 |
| 2016* | 22 | Hillary Clinton | +14 | -1 | +5.3 | 3.6 |
In 2012, there were 22 polls conducted in the final week of the campaign. And they formed a very tight range: None had President Obama ahead by more than 4 percentage points or behind by more than 1 point. The standard deviation of Obama’s margin over Mitt Romney was just 1.5 points.
The 2000 and 2008 elections were similar, with only 5 or 6 percentage points between the best and worst polls for the respective candidates. No national poll in the final week of the 2000 campaign had Al Gore ahead in the popular vote, incidentally, even though he ended up winning it. The 2004 election had a slightly wider range: Some polls showed a George W. Bush lead of as large as 6 percentage points, while a couple had John Kerry ahead.
All those differences are modest compared with what we’re seeing this year. As measured by the standard deviation, the spread in polls conducted since the third presidential debate is about twice as wide as what we saw at the end of campaigns from 2000 through 2012, on average. Again, that isn’t a perfect comparison because the range of polls may narrow over the final week of this campaign. But it’s not just your imagination if you feel like there’s more variation in the polls than you’re used to.
Here’s another question, though. How much should you expect the polls to differ from one another? Even if there were no methodological differences, you’d expect some variation as a result of random sampling error. Could the seemingly huge spread in the polls this year turn out to be nothing more than statistical noise?
Probably not. For each election, I ran 10,000 simulations in which I estimated how wide the range of the polls “should” be based on the sample size of each poll.2 From these simulations, I also calculated a confidence interval for each election; in theory, the standard deviation in the polls should fall within this interval 95 percent of the time.
Well, that’s pretty interesting. The standard deviation for the 2016 polls — 3.6 percentage points — falls just outside the confidence interval, which runs from 1.8 points to 3.5 points. That suggests there are probably some real methodological differences and that the wide spread in the polls doesn’t reflect sampling error alone. For instance, live telephone polls show Clinton ahead by 6.4 points on average, a larger lead than she has in online polls (5.1 points) or automated telephone polls (2.3 points, although there are only three automated polls in the average).
What’s also interesting, though, is that there was too little variation in the polls in 2008 and, especially, in 2012. By chance alone, you’d have expected some Obama +6’s in the final polling average in 2012 or some Romney +3’s. Instead, the polls were packed within a very tight range, which probably suggests some herding.
And it wasn’t even as though the national polls did all that great in 2012. Of the 22 national polls, all but one underestimated Obama’s popular vote margin against Romney. Conversely, there was a wide spread in the polls in 1984 and 1988 — even wider than this year — but the average did a good job of forecasting the final margin in the race.
So although the wide spread in the polls this year may reflect challenges in the polling industry, you shouldn’t make a habit of berating the polls that seem to be outliers or use a somewhat unorthodox methodology. Aggregating mechanisms like polling averages and betting markets are powerful precisely because they reflect a diverse array of approaches and opinions, and they lose their power when they’re subject to herding or groupthink. You also shouldn’t cherry-pick the two or three polls that fit your narrative when there are dozens of them out there, of course. But ignoring or “debunking” the polls you don’t like is often almost as bad a sin as cherry-picking the ones you do like.