This article is part of the Debatable newsletter. You can sign up here to receive it on Tuesdays and Thursdays.
Less than 24 hours after the first ballot counts began to pour in on election night, the media and political classes had already declared a loser: polling.
“Devastating for my industry” was how one Republican pollster described the election results.
“Polling seems to be irrevocably broken, or at least our understanding of how seriously to take it is,” wrote Margaret Sullivan, a media columnist at The Washington Post.
Politico Playbook went even further: “The polling industry is a wreck, and should be blown up.”
Are they right? Here’s what people are saying.
‘A profession-wide crisis’
In the wake of the 2016 election, many pollsters resisted the narrative that their profession had taken a nosedive: National polling averages projected Hillary Clinton would win the popular vote by just under four percentage points, and she ultimately won by just over two. According to FiveThirtyEight’s founder Nate Silver, that’s about as accurate as polls have been on average since 1968.
But American elections are decided by swing states in the Electoral College, not the popular vote, and it was at the state level that polling data missed the mark. There were several proposed reasons for this, including a failure to correct for the overrepresentation of college graduates, which pollsters hoped they had fixed this time around.
That turned out not to be the case. As Dave Wasserman, an editor at the Cook Political Report, put it, polls, especially at the district level, “have rarely led us more astray & it’s going to take a long time to unpack.”
-
In the House, many analysts predicted Democrats would keep their majority or even expand it by five to 15 seats. By Thursday, Democrats were on track to keep control but lose six seats.
-
In the Senate, Lindsey Graham of South Carolina won his race, which was widely seen as competitive, by some 13 points. Susan Collins of Maine also held on to her seat, which not a single poll since July suggested would happen, according to FiveThirtyEight.
-
There were significant errors in the presidential election, too: In Florida, polls put Joe Biden ahead by an average of 2.5 points; as of Thursday’s count, he lost the state by more than three. Polls also put President Trump ahead by less than a point in Ohio, which as of Thursday he won by eight.
“No matter who wins the presidential race, it’s clear that the vast majority of the polling underestimated Trump’s support once again,” writes Nathan L. Gonzales, editor of the Inside Elections newsletter. “For months, we’ve been saying that it would take dozens of pollsters, partisan and nonpartisan, independently making the same methodological mistake for the outcome to be different than what we were projecting. And that is apparently what happened.”
A failure, or a misunderstanding?
Many pollsters believe that the general frustration with their profession owes less to a failure in polling itself than to a general misapprehension of its purpose, as well as its prominence in political discourse. As Courtney Kennedy, the director of survey research at Pew Research Center, has noted, polls can be quite useful for finding out what the entire citizenry thinks about policy issues. Even in the context of the presidential election, the national polling average may end up being off by no more than 3 to 4 percent — a mediocre margin of error, but not a terrible one by industry standards.
The problem, Zeynep Tufekci writes in The Times, is that producers and consumers of political media have come to think of polling not as a potentially blurry reflection of public opinion, but as a means of predicting the future. Presidents are not elected by a national vote in the United States, so election modelers — which are different from pollsters — like FiveThirtyEight attempt to “forecast” elections based on measurements like polling data and certain fundamental assumptions about how the political climate works (e.g., “a better economy favors incumbents”).
There’s a case to be made that this crucial distinction between polls, which are a snapshot in time, and models, which deal in probabilities, is simply one that the public doesn’t understand or that the media has done a poor job communicating.
“The apparent results of the presidential election fall within the ‘likely range’ of outcomes projected by polls and forecasts,” writes David Byler at The Washington Post. “Those ranges exist for a reason: For all that some readers treat these mathematical analyses as dispatches from the future, that’s an impossible expectation.”
But Dr. Tufekci argues the problem runs deeper than a failure of communication. In weather forecasting, both the fundamental assumptions (the science of atmospheric dynamics) and the measurements (years of detailed data from a vast number of observation stations) are highly sophisticated. But in politics, “we simply do not have anything near that kind of knowledge or data,” she writes. “While we have some theories on what influences voters, we have no fine-grained understanding of why people vote the way they do, and what polling data we have is relatively sparse.”
The upshot, as Dr. Kennedy has written, is that national polls are much better for relaying public opinion on issues and candidates writ large than for predicting the victor in the Electoral College.
Does polling have a future?
Questions about the use of modeling aside, it remains to be seen why so many pollsters independently made the same mistake of underestimating Mr. Trump’s support nationally. “Our projections in the 2018 midterm elections, using the same methods of analysis, were accurate,” Mr. Gonzales writes. “A key question moving forward is whether public opinion polling is irreparably broken or if polling is just broken in elections with Trump on the ballot.”
One theory that gained purchase after 2016 was the so-called shy Trump voter problem: “With Trump, what’s really unique was that people were reluctant to say they supported him primarily because … people call them racist, xenophobic,” Chris Kofinis, a Democratic pollster, told the Canadian Broadcasting Corporation. “You know, think of the negative that has been applied to Trump over his last four years. People didn’t want to be associated with that.”
But some believe the source of the error goes beyond the president. “The crisis we seem to be dealing with — and I’m not sure we entirely understand it yet tbh — is not a Shy Trump voter problem,” Derek Thompson of The Atlantic tweeted. “It’s an Invisible Republican problem. Up and down the ballot, from POTUS vote to district level, pollsters are whiffing on the G.O.P. share of the vote.”
Why? One explanation is that for years now, the rate of response to telephone surveys has been plummeting. (The issue hasn’t been solved by the rise of online polls, which tend to lean to the left of telephone surveys and are generally, though certainly not always, less reliable.) There’s reason to believe that voters without a college education and with lower levels of social trust, who were more likely to lean Republican, are especially loath to answer the phone, which poses a sampling problem that weighting has not been able to solve.
If conventional surveys are indeed becoming a less accurate instrument for gauging the views of an important subset of voters, regardless of who’s running for president, the polling industry will face existential questions about whether and how it can recalibrate.
“But the answer almost doesn’t matter, unless you’re a professional pollster,” David Graham argues in The Atlantic, “because after two huge presidential flops, pollsters have lost the confidence of the press and public.”
Do you have a point of view we missed? Email us at debatable@nytimes.com. Please note your name, age and location in your response, which may be included in the next newsletter.
MORE ON THE PERILS OF POLLING
“A Math Whiz on How to Stop Stressing About Election Forecasts” [The Atlantic]
“What it means to get the election wrong” [The Columbia Journalism Review]
“Never Trust the Polls Again!” [The New Republic]
“Were The Polls Wrong? A Look At The Future Of The Polling Industry” [NPR]
WHAT YOU’RE SAYING
Here’s what one reader had to say about the last edition: Five ways to ward off election anxiety.
Joan from Pennsylvania: “I agree with all the suggestions. May I add a few? Watch some old movies, especially World War II ones on Turner Classic Movies. We survived those horrible years as a united country, and we can hopefully become united again. One current movie I highly recommend is ‘Crazy Rich Asians.’ Even my husband, who isn’t a fan of comedies, enjoyed the film with its glorious scenes of Singapore.
“It may be difficult to overcome the disappointment if your candidate loses. My closest friends will also be terribly upset if Biden loses. We will console each other as best as we can, but it will take a while before our get-togethers will be fun again. And please, do take a break from the news. It’s probably the biggest anxiety producer there is. Even after this election is over, whenever that will be, you will find enough news stories to jump start the apprehension and worry, post election, all over again.”