I'm not sure I see the bimodality (or really even the "blowout"), but I'd like to point to a different culprit in the forecast errors: the "tails", specifically on the Democratic side in the 538 forecast, are given far, far too much weight. Obama in 2008 under the most favorable conditions imaginable did not break D = +200 in the EC in the post-90s realignment era of "blue states and red states". There is simply too much probability density to the left of D = +100 which accounts for pretty much the entire difference between 50/50 and 54/46 odds by my eyeball estimation.
Also, there's a distinction between the binary, yes/no issue of winning and the continuous issue of the winning margin. Consider a football match; it's absolutely normal to have a game where one side is clearly dominant but wins 1-0. Consider parliamentary seats: it's possible, even common, for an MP or a party to hold a seat for long periods without having a particularly big majority. Consider a boxing match; it's really not unusual for an evenly matched fight to end with a KO. A large but unstable margin and a small but stable one are different things.
Consider Leigh Leopards RLFC, who are currently 3rd in the table; after 26 matches they have a points difference of +153. The two clubs ahead of them (Hull KR, in first place, and Wigan, in second) +502 and +445 and the two clubs behind them (Leeds, in fourth, St Helens, in fifth) +316 and +361. Clearly Leigh have a habit of winning by narrow margins, but they whipped St Helens last night and are likely to finish third and have a favourable draw in the playoffs. Earlier in the season I thought their narrow margins were evidence that they would eventually run out of luck, but I was wrong. It would be silly to claim that all 18 wins were flukes.
First, the meaning of the surveys was clear enough to anybody who followed the matter closely, even the sorts of folks who double their mathematical powers when they kick off their shoes. The mantra during the last week of the election for those who noticed: "The most likely probability is somebody winning all seven contested states. We just don't know who would win them." It's like accounting statements. Some people just read the P&L numbers and projections, but others look at the footnotes.
Second point: I agree that surveys cannot predict winners of close horse races. But they still have two uses. First, not all horse races are close, and surveys can reliably tell you this. Second, changes in polling can be very significant, since the weighting factors are likely stable over the course of a few months.
I agree with your first paragraph. As I wrote elsewhere in this thread, the leading forecasters all emphasized that the election could go either way and that the electoral college might well not be close. We thought a lot about how to best convey uncertainty, but given the reactions of Dan and others, it does seem that we didn't do enough of that. I think one problem is that lots of people don't want to hear about uncertainty, so it's perhaps necessary to not just keep talking about uncertainty but to really scream about it.
This is very much related to a problem I have seen a lot in economics. You go to a conference and someone presents a paper with strong findings based on really terribly collected data or data from a set with no concern for the accuracy of the topic at hand. You ask about it and the reply is invariably "Well, it isn't ideal, but it is the best data we have to work with." Somehow it is a social faux pas to scream back "THEN WHY EVEN DO IT!? WHY PRESENT IT AS REAL RESEARCH WHEN YOU KNOW IT IS BASED ON NONSENSE!?"
People want to buy and sell certainty, regardless of whether it is the real thing.
I can't comment on what you've seen in economics, but, no, I don't think polling is based on nonsense. It's based on responses from real people, and in the past election the poll averages were off by a few percentage points which is regrettable but does not make them useless.
I appreciate how the polls are collected, but I don’t quite agree that they are generally useful, or perhaps more accurately that the way they are interpreted and used are useful. It seems to me that polls, at least political polls collected in the way they seem to be currently, are of somewhat negative value, as they give a misleading sense of knowledge, leading people to believe they know more than they do. I am sure there are people out there doing polling “right”, but I am not at all sure those people are doing political polling.
I am also not at all sure that polling can actually be done in a way that gives meaningful data in the ways we want it to, but that is a different question.
"I have too much of a survival instinct to get into a technical debate with [a statistician] about statistics!" is the modern-day version of George Bernard Shaw's warning about pig wrestling.
The line from Shaw is something like, "I learned long ago, never to wrestle with a pig, you get dirty; and besides, the pig likes it."
Just to be clear: No, I am not a "pig"; I am a professional who has devoted his career to the study and communication of theoretical and applied statistics. And, no, I don't "like it" when I feel the need to explain some basic principles of applied statistical measurement; I do it as part of my duty of public service when, as xkcd puts it, "someone is wrong on the internet."
Finally, no, having a statistical discussion with me will not get Dan or anyone else "dirty." Technical discussions are an opportunity to learn, and the learning goes in both directions. I make mistakes and can learn from people questioning me, and even if I happen to be correct in some dispute, I can often learn how to better communicate the relevant statistical ideas. In this particular example, I'd thought that the repeated statements about polling uncertainty coming from Elliott, Nate, me, and others had made it clear that either of the two candidates in 2024 had a good chance of winning all the swing states, but I gather from the responses of you and Dan that this was not the case, and we need to do a better job of explaining this uncertainty, also explaining why a forecast that is 50/50 is informative.
I appreciate that Andrew is fighting the good fight in the comments. I will echo two things he wrote:
A. Trump did not win in a landslide. He won 50.8% of the two-party vote — a 0.8pt deviation from "50-50," and well within the historical range of error. Yes, at the state level, this translated to a sweep of the close states, but only pundits deal with binary outcomes; Trump won most of these states by a very slim margin, again well within the historical range of error of these polls. Characterizing polls as unsuccessful because they didn't predict the "Trump blowout" is misleading. But this bring me to the second point:
B. In our editorial coverage, the forecasters highlighted the probability of a swing state sweep pretty far in advance of the election — in my case, in September! Quoth Andrew: "Nate wrote: "One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error," and Elliott's wrote: "Trump and Harris are both a normal polling error away from a blowout. The race is uncertain, but that doesn’t mean the outcome will be close." These disclaimers--given during the campaign, before the election--seem very reasonable to me."
I appreciate that people are asking questions about how precise polling can be. I wrote a whole book about exactly that! But I wonder if this is more a failure of people to read our coverage than it is a failure of our coverage.
This brings me to a final point. Ben and Lief/Dan write that the "Era of the Pollster/Data Guy" is over. The indictment of the pollster from Ben and Lief is made partially on account of pollsters not living up to Walter Lippmann's mission of using data to inform the public, on the basis of election forecasts not being clear enough. But election forecasts are not public polls; the vast majority of pollsters' work is releasing data to inform views on politics and policy, and most of that data is released publicly. I did not know Lippmann but I don't think he would have liked forecasts. But he would not have invoked the failure of forecasts to predict an outcome with 100% accuracy to dismiss all polling.
I think the polling industry is in a mess - too many polls with too many dubious questions. But presumably a specific problem here is the electoral college and FPTP magnify small differences that the polls, hit by response rates etc, can't pickup. I imagine they have a better record with a PR parliamentary election?
1. You write, "I really don’t see how you could see this as communicating that a Trump landslide was a significant probability." But Trump did not win in a landslide. He won by 2% of the vote. When Reagan won 49% of the two-party vote in 1984, that was a landslide. When Obama won 54% in 2008, that was a decisive victory, but not a landslide. Trump won less than 51% of the two-party vote. It was a close election, and that's why the Economist, Fivethirtyeight, and Nate Silver repeatedly emphasized that the election could go either way.
2. As discussed in our 1993 paper (https://sites.stat.columbia.edu/gelman/research/published/bjps1993.pdf), pre-election polling has always had problems. Historically, political scientists' fundamentals-based forecasts have been more accurate than pre-election polling averages, even though the fundamentals are based on information available before the election campaign. That's interesting! In recent years, most notably 2000, 2004, and 2008, pre-election polls have become more accurate, but the fundamentals still tell us a lot, which is why we use both sources of information in our forecasts. Anyway, my point is that, yeah, polls have low response rates, but they're as accurate as they were, many decades ago, when response rates are higher. Whether the response rate is 5% or 50%, survey researchers have to take nonresponse seriously and work on it both in design and analysis--and they do.
3. You write that, for polls, "the bias and unknown error as having grown to a level where it’s acceptable for market research, but not for important policy issues." As a political scientist, I think we can learn a lot from opinion polling, even if, as with recent elections, the polls are off by 2 percentage points. First off, the fact that the polls had Trump and Harris so close was itself informative--not every election is this close. Similarly, the fact that the polls had Obama clearly ahead in 2008 was informative, etc. To know that an election can be close is itself valuable information. Second, for most public opinion issues, you don't need 2-percentage-point accuracy or anything like that. On issues ranging from trust in institutions to attitudes toward abortion, we can get a good sense of public opinion, and trends in public opinion, and it's not so important if a particular position has 38% support or 41%.
4. You write, "The policy community has to wean itself off the use of surveys, and start using harder evidence about what is worth doing." I'd say the issue is not "harder evidence" but "different evidence." For example consider abortion policy. What is worth doing depends on politics, social values, and also outcomes (how many abortions are being done, what is the potential effect of changes in abortion attitudes, etc.). Public opinion on abortion, however measured, won't tell us "what is worth doing." But it tells us something about what people are thinking. There are other ways of measuring what people are thinking, for example by convening focus groups and doing qualitative interviewing. And survey researchers do these things. But they're not a substitute for polling. They're not "harder evidence"; they're different evidence.
Overall I disagree with your position that there should be "prohibition" of a method of data collection that produced an estimate of 51% when the outcome ended up being 49%. I think the appropriate response to a 2% error is to study nonsampling error, which we have (see our 2018 paper on bias and variance in election polls: https://sites.stat.columbia.edu/gelman/research/published/polling-errors.pdf), and do be clear about forecast uncertainty, which we at the Economist (and Elliott at 538 and Nate at his site) did. It's hard for me to see how we could've been more clear than with Nate's statement, "One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error," or with Elliott's statement, "Trump and Harris are both a normal polling error away from a blowout. The race is uncertain, but that doesn’t mean the outcome will be close"--but, sure, if you want to emphasize the uncertainty more, go for it. Also you might want to go back in time and prohibit all polling from the 1930s onward, given that polling is as accurate now as it ever has been.
Umm, minor typo: "When Reagan won 49% of the two-party vote in 1984" should be "When Reagan won 59% of the two-party vote in 1984."
Otherwise I stand by my comment. Trump did not win in a landslide, and it's hardly a serious criticism of our forecast to note that we did not forecast something that didn't happen and wasn't close to happening!
I thought that the problem was that pollsters thought they were doing "science" based on the assumption that the responders were not going to deliberately lie about their intentions. They thought that people were like lab mice, especially when using large numbers and applying statistics. Basically, "Garbage in, garbage out."
It's called social science, which by its nature is less precise than the sort of experimental science you can do on lab mice. To do an opinion poll and ask people questions and use statistical modeling to generalize to the population is not to "think that people are like lab mice." Nor does a 2% error, which is within the range anticipated by past studies on nonsampling error, invalidate the method. Finally I do not think the polls were distorted by people lying about their intentions; for more on this, see section 5 of this paper with Julia Azari: https://sites.stat.columbia.edu/gelman/research/published/what_learned_in_2016_5.pdf).
Psychology has long understood that people know about experiments and will try to mess with them. So experiments are designed to obfuscate teh real experiment. Poll takers don't do this, although they can try to cross-reference replies to other questions and data. That the polls sometimes get the results so wrong, and well outside the expected error, should not be excused. They are wrong. Full stop. For example, if 10 polls make the same prediction error, then it goes well past the individual error rate probability and shows that there was a systematic error in the polling.
I will no longer respond to phone polling, as I have experienced polling that was a "disguised" attempt to support a particular politician. That poll has "poisoned the well" of polling for me. Political organizations have used mail donation requests as requests for opinions. Choices are tailored, and designed to create a sense that the responder should donate to the organization. Whatever responses they get are probably very biased as a result.
Bottom line is that any sort of polling is now becoming so broken that it is nearly worthless. There is a good argument to ban polling in many instances, especially away from elections.
You have no obligation to respond to phone polls. I don't respond to them either! And, yes, there is nonsampling error, which can be measured (see our 2018 paper: https://sites.stat.columbia.edu/gelman/research/published/polling-errors.pdf). Serious poll-based election forecasts, including ours at the Economist, account for the possibility of nonsampling error, which is one reason that we and others emphasized before the election that our forecasts easily allowed the possibility of either candidate winning all the swing states. When you say "it goes well past the individual error rate probability and shows that there was a systematic error in the polling": yes, that's exactly what we study in our paper and that's exactly what we account for in our models.
Finally, you are entitled to your own opinion that polling is nearly worthless. I think that measures of public opinion that are off by only a few percentage points are pretty good, even if they aren't good enough to tell you ahead of time the outcome of a very close national election that is within two percentage points of a tie.
It also seems ludicrous to me to ban a social measurement technique just because it's imperfect, but maybe you'll get your wish and this will happen. The U.S. government has been cutting back on measurement of economic statistics, measurement of the weather, all sorts of things. Did you hear that vaccines are imperfect too? Well, good news, the government's discouraging vaccines too. There are lots of imperfect things in the human sciences that people want to defund or ban.
I didn't say ban polling. I said around certain events, where polling can influence an election. In multiple horse races, as in the UK, accurate polling could help to decide on strategic voting.
I'm sure you understand the concept of meta analyses that are often used for noisy medical reports to tease out the truth, although even these can be biased by selective publishing.
As regards a "close race". If polls were accurate, then you would expect polls to straddle the no difference or 50% probability. If they all err in one direction, but the result is strongly in the other, then the polls were not measuring intent correctly. That could be for any number of reasons, but clearly the polls were not just "wrong", but had systematic bias. That may not be due to the pollsters' fault, but how the polls were answered. IOW, it is the same sort of problem psychologists have to deal with.
As a biologist, I have done animal counts. I don't expect animals to try to mess up my counts. OTOH, people have agency and may have reasons to answer questions deceptively. I'm sure if the current US government were to poll left-leaning US citizens just now, they might well get answers more supportive of right-wing POVs. This is what happens in authoritarian countries. I would be very suspicious of any requests for my opinions coming from anybody right now, whoever they purported to be.
Measuring responses to vaccines need not be devoid of actual data, as well as disease outbreaks. Measuring sewage outflows is a very cheap method to gauge diseases in communities. Economic measurements are unlikely to be biased, especially when it comes from sources that report results from actual hard data. Yes, those sources could lie, but what might be their incentive, unlike the government being run by an autocrat and inexperienced, probably incompetent, sycophants, to support the autocrat's POV. We saw this in the Bush II administration with climate change reports, and we have just seen it again with teh Trump II climate report.
There are lots of things people want to ban. I am very unhappy that the UK and some US states want age verification on websites. I am also unhappy that the UK is still balking at a "right to die" bill, something we allow for our suffering pets, but apparently not for any adults. I would still maintain a ban on genetically engineered babies unless it is used for correcting genetic diseases, even though someday we will know enough to make it safe for any purpose, and it will be accepted. Yet this knowledge can be used as a weapon, and I am not sure I want the sort of intrusive surveillance to prevent malicious actors from building biological weapons. Sometimes it is better to be cautious. Think about the current issue of developing AGI. Should we do it with today's level of knowledge? Probably not. But when we can make AGIs that are fully aligned with humans, and can prevent AGIs with antagonistic POVs towards humans, then we should have AGIs, but with a way to deal with "bad" AGIs as we do with humans.
1. I don't want to ban polling, even around certain events. But, again, given what's been happening at the Bureau of Labor Statistics, the Center for Disease Control, etc., there's been a lot of attacks on data collection on the grounds that the data collected are imperfect, so it's not impossible that the government will find a way to ban polling.
2. You write, "If polls were accurate, then you would expect polls to straddle the no difference or 50% probability. If they all err in one direction, but the result is strongly in the other, then the polls were not measuring intent correctly. That could be for any number of reasons, but clearly the polls were not just "wrong", but had systematic bias." I agree completely! This is what we call "nonsampling error," but "systematic bias" is fine too. We measure this in our above-linked 2018 paper. Actually, my colleagues and I have published several papers on nonsampling error. It's an important topic in polling. Nonsampling error is not huge--it's a few percentage points--but it's not nothing, and that's why we account for it in our forecasts.
Nate wrote: "One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error," and Elliott's wrote: "Trump and Harris are both a normal polling error away from a blowout. The race is uncertain, but that doesn’t mean the outcome will be close." These disclaimers--given during the campaign, before the election--seem very reasonable to me.
Re fn 1: Over here I have heard it referred to as the doctrine of the expert client. As in, some contracts are fair and enforceable when there is expertise on both sides, but unenforceable against naive parties.
Isn't this explicable just with mean and variance? 50-50 is perfectly consistent with a high probability of a blowout. If I'm attempting to predict the outcome of a boxing match, I can certainly say "50-50, but whoever wins will do so by a knockout, not on points".
I'll bite on the probability semantics. A prediction is a reductive compression of outcomes into a binary outcome. The variance of such a random variable is proportional to its expected value.
Part of my issue with the poll aggregator industry is they play fast and loose with their statistics, representing whatever data vis suits them to tell the story they want to tell. But then claiming that they are being perfectly rigorous even though their entire enterprise is metaphysical.
As Dan says, what matters is not what they intend to communicate, but what people take away.
I am indeed rather skeptical that what people take away isn't actually pretty close to what they intend to communicate, but pollsters won't admit openly that it is what they are trying to communicate.
I can't speak for pollsters, but the election forecasters at the Economist, 538, and Silver Bulletin were all trying very hard to communicate uncertainty! If many readers did not get that point and thought of our forecasts as essentially deterministic, then, yeah, that's on us, and we didn't communicate uncertainty clearly enough--but we were trying.
To say "won't admit openly" seems to imply some secret agenda, and that wasn't happening. We were very open that the forecasts were uncertain and that either candidate could sweep all the swing states. Again, it seems that we didn't say that often enough for some people--or maybe it's just that some people didn't want to hear about the uncertainty--so maybe we should've said it louder and more often.
I can’t speak about specific pollsters, but I strongly suspect that most if not all are trying to get specific results from their polls, or at the very least applying selective rigor when it comes to how to frame and publish the results based on whether they like the outcome. Looking at that graph from the Economist I think almost everyone would take away a different message than “we are really uncertain.”
I'm surprised at that last statement of yours, because I think the graph expresses uncertainty pretty strongly! But I accept that you and others see it differently, which suggests at the very least that we need to work harder at communicating our uncertainty. So I appreciate the feedback.
Aha! I am surprised you are surprised that I was… wait I lost count :)
I see that graph and think “That is a mess but it looks like more Kamala wins likely than Trump. I expect they aren’t taking bets on that, though.” A normal person, ie non cynical/non academic, is going to look at that and say the same thing only without the last sentence. Most people don’t differentiate between “Outcome X happens 70% of the time and I am certain of that “ and “Outcome X happens 100% of the time but I am only 30% sure of that.” Much less the fact that though there is no margin of error reported on the graph what you are looking at is a massive margin of error.
Put another way, people understand “Horse A looks very likely to win, but we are not at all sure” very differently from “Out of all these outcomes Horse A wins more often than B” and even more so than “We really have no idea, but maybe Horse A is a bit more likely?” A statistician will see through the words to the numbers, but just about everyone else listens to the words (or pictures) and doesn’t discount them as noise.
I can't speak for "the poll aggregator industry" in general, but I can say that Nate Silver, Elliott Morris, and I were always very open about uncertainty, as in the quotes given above.
You write of people who are "claiming that they are being perfectly rigorous." I don't know who is claiming that polling "perfectly rigorous" or what that phrase even means. Polling can be rigorous in the sense of being transparent (which, as often is the case in statistics, can be a challenge with new data coming in real time), then there's the statistical inference, which depends on assumptions, as usual. I'm also not clear what you mean by "metaphysical" here. The most relevant dictionary definition seems to be "based on abstract (typically, excessively abstract) reasoning." I guess that statistical modeling is abstract: normal distributions and logistic curves and all the other mathematical tools that we use are abstract. These are the abstract tools used by Laplace, Gauss, Galton, and other applied statistical researchers over the centuries. Abstract reasoning in statistics has a pretty good track record: it's led to progress in fields ranging from astronomy and pharmacology to economics and sociology, and it also allows us to adjust polls and estimate public opinion to within a few percentage points.
Regarding your last point, yes, I agree that it's important what people take away from our reports, and given the reactions that you and Dan have had, I agree that we need to work even harder at communicating uncertainty. So thanks for communicating that to us: the communication was not so polite, but that's ok--maybe that's what was necessary to get our attention!
The clear communication thing is really tough. With that Economist graph example, there are definitely people with a bit more data literacy and a bit more time who read carefully, decide to just ignore the median lines (there's only one election, what does median even mean?), then do that exact eye crossing thing and say "eh, who knows." On the other hand there are people who are reading more quickly and possibly with less expertise that only notice the bigger number. Also a major dynamic has to be that a big chunk of your audience is coming to your site to *find out who is winning*, which maybe makes it different from the financial products example? Refusing to communicate at all is not a great option, you should try to figure it out.
The predictions of who will win are also a million times harder than, say, popular vote predictions because being ahead can mean "higher chance they will ultimately trounce the opponent and lower chance that they get trounced." This is not what being ahead normally means! It's a game of chess, not basketball.
Yes although I think you have to have a huge amount of expertise to really get it. I think the root of this beef is that Andrew is in the position of that xkcd cartoon that goes "you have to remember the average non geologist only knows the chemical formula for eight or nine of the feldspars", he really has a much higher baseline estimate than is realistic to hope of the understanding of ... well, of me, let alone actual laypeople
Again, consider the actual statement from Nate Silver: "One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error," and the actual statement from Elliott Morris: "Trump and Harris are both a normal polling error away from a blowout. The race is uncertain, but that doesn’t mean the outcome will be close," and similar statements of uncertainty from me.
Given your above post, I'm sure you're right that we didn't emphasize this uncertainty enough, but all of us did try, and we tried repeatedly, and we wrote it in plain English not jargon, and we wrote all of this during the campaign before the election. Contrary to your post, there's nothing we tried to "take back in a disclaimer." We were open about uncertainty all the time. And, I guess, given the reactions from you and others, we'll emphasize the uncertainty even more next time.
I'm not sure I see the bimodality (or really even the "blowout"), but I'd like to point to a different culprit in the forecast errors: the "tails", specifically on the Democratic side in the 538 forecast, are given far, far too much weight. Obama in 2008 under the most favorable conditions imaginable did not break D = +200 in the EC in the post-90s realignment era of "blue states and red states". There is simply too much probability density to the left of D = +100 which accounts for pretty much the entire difference between 50/50 and 54/46 odds by my eyeball estimation.
Also, there's a distinction between the binary, yes/no issue of winning and the continuous issue of the winning margin. Consider a football match; it's absolutely normal to have a game where one side is clearly dominant but wins 1-0. Consider parliamentary seats: it's possible, even common, for an MP or a party to hold a seat for long periods without having a particularly big majority. Consider a boxing match; it's really not unusual for an evenly matched fight to end with a KO. A large but unstable margin and a small but stable one are different things.
Consider Leigh Leopards RLFC, who are currently 3rd in the table; after 26 matches they have a points difference of +153. The two clubs ahead of them (Hull KR, in first place, and Wigan, in second) +502 and +445 and the two clubs behind them (Leeds, in fourth, St Helens, in fifth) +316 and +361. Clearly Leigh have a habit of winning by narrow margins, but they whipped St Helens last night and are likely to finish third and have a favourable draw in the playoffs. Earlier in the season I thought their narrow margins were evidence that they would eventually run out of luck, but I was wrong. It would be silly to claim that all 18 wins were flukes.
Uhh, two points.
First, the meaning of the surveys was clear enough to anybody who followed the matter closely, even the sorts of folks who double their mathematical powers when they kick off their shoes. The mantra during the last week of the election for those who noticed: "The most likely probability is somebody winning all seven contested states. We just don't know who would win them." It's like accounting statements. Some people just read the P&L numbers and projections, but others look at the footnotes.
Second point: I agree that surveys cannot predict winners of close horse races. But they still have two uses. First, not all horse races are close, and surveys can reliably tell you this. Second, changes in polling can be very significant, since the weighting factors are likely stable over the course of a few months.
Ziggy:
I agree with your first paragraph. As I wrote elsewhere in this thread, the leading forecasters all emphasized that the election could go either way and that the electoral college might well not be close. We thought a lot about how to best convey uncertainty, but given the reactions of Dan and others, it does seem that we didn't do enough of that. I think one problem is that lots of people don't want to hear about uncertainty, so it's perhaps necessary to not just keep talking about uncertainty but to really scream about it.
Regarding your second paragraph: changes in polling can sometimes be informative but it can depend on survey adjustment. In 2012 and 2016 we found that swings in the headline poll numbers were mostly coming from differential nonresponse (see here: https://sites.stat.columbia.edu/gelman/research/published/swingers.pdf or, for a quicker version, here: https://slate.com/news-and-politics/2016/08/dont-be-fooled-by-clinton-trump-polling-bounces.html). Similar issues arise when interpreting other opinion questions, and it can be important to adjust for the partisan composition of the respondents.
This is very much related to a problem I have seen a lot in economics. You go to a conference and someone presents a paper with strong findings based on really terribly collected data or data from a set with no concern for the accuracy of the topic at hand. You ask about it and the reply is invariably "Well, it isn't ideal, but it is the best data we have to work with." Somehow it is a social faux pas to scream back "THEN WHY EVEN DO IT!? WHY PRESENT IT AS REAL RESEARCH WHEN YOU KNOW IT IS BASED ON NONSENSE!?"
People want to buy and sell certainty, regardless of whether it is the real thing.
Doctor:
I can't comment on what you've seen in economics, but, no, I don't think polling is based on nonsense. It's based on responses from real people, and in the past election the poll averages were off by a few percentage points which is regrettable but does not make them useless.
I appreciate how the polls are collected, but I don’t quite agree that they are generally useful, or perhaps more accurately that the way they are interpreted and used are useful. It seems to me that polls, at least political polls collected in the way they seem to be currently, are of somewhat negative value, as they give a misleading sense of knowledge, leading people to believe they know more than they do. I am sure there are people out there doing polling “right”, but I am not at all sure those people are doing political polling.
I am also not at all sure that polling can actually be done in a way that gives meaningful data in the ways we want it to, but that is a different question.
Obviously, I endorse every word you type. :)
"I have too much of a survival instinct to get into a technical debate with [a statistician] about statistics!" is the modern-day version of George Bernard Shaw's warning about pig wrestling.
Ben:
The line from Shaw is something like, "I learned long ago, never to wrestle with a pig, you get dirty; and besides, the pig likes it."
Just to be clear: No, I am not a "pig"; I am a professional who has devoted his career to the study and communication of theoretical and applied statistics. And, no, I don't "like it" when I feel the need to explain some basic principles of applied statistical measurement; I do it as part of my duty of public service when, as xkcd puts it, "someone is wrong on the internet."
Finally, no, having a statistical discussion with me will not get Dan or anyone else "dirty." Technical discussions are an opportunity to learn, and the learning goes in both directions. I make mistakes and can learn from people questioning me, and even if I happen to be correct in some dispute, I can often learn how to better communicate the relevant statistical ideas. In this particular example, I'd thought that the repeated statements about polling uncertainty coming from Elliott, Nate, me, and others had made it clear that either of the two candidates in 2024 had a good chance of winning all the swing states, but I gather from the responses of you and Dan that this was not the case, and we need to do a better job of explaining this uncertainty, also explaining why a forecast that is 50/50 is informative.
I think the statement is better understood as a version of "I wouldn't box Mike Tyson."
I appreciate that Andrew is fighting the good fight in the comments. I will echo two things he wrote:
A. Trump did not win in a landslide. He won 50.8% of the two-party vote — a 0.8pt deviation from "50-50," and well within the historical range of error. Yes, at the state level, this translated to a sweep of the close states, but only pundits deal with binary outcomes; Trump won most of these states by a very slim margin, again well within the historical range of error of these polls. Characterizing polls as unsuccessful because they didn't predict the "Trump blowout" is misleading. But this bring me to the second point:
B. In our editorial coverage, the forecasters highlighted the probability of a swing state sweep pretty far in advance of the election — in my case, in September! Quoth Andrew: "Nate wrote: "One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error," and Elliott's wrote: "Trump and Harris are both a normal polling error away from a blowout. The race is uncertain, but that doesn’t mean the outcome will be close." These disclaimers--given during the campaign, before the election--seem very reasonable to me."
I appreciate that people are asking questions about how precise polling can be. I wrote a whole book about exactly that! But I wonder if this is more a failure of people to read our coverage than it is a failure of our coverage.
This brings me to a final point. Ben and Lief/Dan write that the "Era of the Pollster/Data Guy" is over. The indictment of the pollster from Ben and Lief is made partially on account of pollsters not living up to Walter Lippmann's mission of using data to inform the public, on the basis of election forecasts not being clear enough. But election forecasts are not public polls; the vast majority of pollsters' work is releasing data to inform views on politics and policy, and most of that data is released publicly. I did not know Lippmann but I don't think he would have liked forecasts. But he would not have invoked the failure of forecasts to predict an outcome with 100% accuracy to dismiss all polling.
https://wwnorton.com/books/9780393866971
I think the polling industry is in a mess - too many polls with too many dubious questions. But presumably a specific problem here is the electoral college and FPTP magnify small differences that the polls, hit by response rates etc, can't pickup. I imagine they have a better record with a PR parliamentary election?
Dan:
1. You write, "I really don’t see how you could see this as communicating that a Trump landslide was a significant probability." But Trump did not win in a landslide. He won by 2% of the vote. When Reagan won 49% of the two-party vote in 1984, that was a landslide. When Obama won 54% in 2008, that was a decisive victory, but not a landslide. Trump won less than 51% of the two-party vote. It was a close election, and that's why the Economist, Fivethirtyeight, and Nate Silver repeatedly emphasized that the election could go either way.
2. As discussed in our 1993 paper (https://sites.stat.columbia.edu/gelman/research/published/bjps1993.pdf), pre-election polling has always had problems. Historically, political scientists' fundamentals-based forecasts have been more accurate than pre-election polling averages, even though the fundamentals are based on information available before the election campaign. That's interesting! In recent years, most notably 2000, 2004, and 2008, pre-election polls have become more accurate, but the fundamentals still tell us a lot, which is why we use both sources of information in our forecasts. Anyway, my point is that, yeah, polls have low response rates, but they're as accurate as they were, many decades ago, when response rates are higher. Whether the response rate is 5% or 50%, survey researchers have to take nonresponse seriously and work on it both in design and analysis--and they do.
3. You write that, for polls, "the bias and unknown error as having grown to a level where it’s acceptable for market research, but not for important policy issues." As a political scientist, I think we can learn a lot from opinion polling, even if, as with recent elections, the polls are off by 2 percentage points. First off, the fact that the polls had Trump and Harris so close was itself informative--not every election is this close. Similarly, the fact that the polls had Obama clearly ahead in 2008 was informative, etc. To know that an election can be close is itself valuable information. Second, for most public opinion issues, you don't need 2-percentage-point accuracy or anything like that. On issues ranging from trust in institutions to attitudes toward abortion, we can get a good sense of public opinion, and trends in public opinion, and it's not so important if a particular position has 38% support or 41%.
4. You write, "The policy community has to wean itself off the use of surveys, and start using harder evidence about what is worth doing." I'd say the issue is not "harder evidence" but "different evidence." For example consider abortion policy. What is worth doing depends on politics, social values, and also outcomes (how many abortions are being done, what is the potential effect of changes in abortion attitudes, etc.). Public opinion on abortion, however measured, won't tell us "what is worth doing." But it tells us something about what people are thinking. There are other ways of measuring what people are thinking, for example by convening focus groups and doing qualitative interviewing. And survey researchers do these things. But they're not a substitute for polling. They're not "harder evidence"; they're different evidence.
Overall I disagree with your position that there should be "prohibition" of a method of data collection that produced an estimate of 51% when the outcome ended up being 49%. I think the appropriate response to a 2% error is to study nonsampling error, which we have (see our 2018 paper on bias and variance in election polls: https://sites.stat.columbia.edu/gelman/research/published/polling-errors.pdf), and do be clear about forecast uncertainty, which we at the Economist (and Elliott at 538 and Nate at his site) did. It's hard for me to see how we could've been more clear than with Nate's statement, "One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error," or with Elliott's statement, "Trump and Harris are both a normal polling error away from a blowout. The race is uncertain, but that doesn’t mean the outcome will be close"--but, sure, if you want to emphasize the uncertainty more, go for it. Also you might want to go back in time and prohibit all polling from the 1930s onward, given that polling is as accurate now as it ever has been.
Umm, minor typo: "When Reagan won 49% of the two-party vote in 1984" should be "When Reagan won 59% of the two-party vote in 1984."
Otherwise I stand by my comment. Trump did not win in a landslide, and it's hardly a serious criticism of our forecast to note that we did not forecast something that didn't happen and wasn't close to happening!
I thought that the problem was that pollsters thought they were doing "science" based on the assumption that the responders were not going to deliberately lie about their intentions. They thought that people were like lab mice, especially when using large numbers and applying statistics. Basically, "Garbage in, garbage out."
Alex:
It's called social science, which by its nature is less precise than the sort of experimental science you can do on lab mice. To do an opinion poll and ask people questions and use statistical modeling to generalize to the population is not to "think that people are like lab mice." Nor does a 2% error, which is within the range anticipated by past studies on nonsampling error, invalidate the method. Finally I do not think the polls were distorted by people lying about their intentions; for more on this, see section 5 of this paper with Julia Azari: https://sites.stat.columbia.edu/gelman/research/published/what_learned_in_2016_5.pdf).
Psychology has long understood that people know about experiments and will try to mess with them. So experiments are designed to obfuscate teh real experiment. Poll takers don't do this, although they can try to cross-reference replies to other questions and data. That the polls sometimes get the results so wrong, and well outside the expected error, should not be excused. They are wrong. Full stop. For example, if 10 polls make the same prediction error, then it goes well past the individual error rate probability and shows that there was a systematic error in the polling.
I will no longer respond to phone polling, as I have experienced polling that was a "disguised" attempt to support a particular politician. That poll has "poisoned the well" of polling for me. Political organizations have used mail donation requests as requests for opinions. Choices are tailored, and designed to create a sense that the responder should donate to the organization. Whatever responses they get are probably very biased as a result.
Bottom line is that any sort of polling is now becoming so broken that it is nearly worthless. There is a good argument to ban polling in many instances, especially away from elections.
Alex:
You have no obligation to respond to phone polls. I don't respond to them either! And, yes, there is nonsampling error, which can be measured (see our 2018 paper: https://sites.stat.columbia.edu/gelman/research/published/polling-errors.pdf). Serious poll-based election forecasts, including ours at the Economist, account for the possibility of nonsampling error, which is one reason that we and others emphasized before the election that our forecasts easily allowed the possibility of either candidate winning all the swing states. When you say "it goes well past the individual error rate probability and shows that there was a systematic error in the polling": yes, that's exactly what we study in our paper and that's exactly what we account for in our models.
Finally, you are entitled to your own opinion that polling is nearly worthless. I think that measures of public opinion that are off by only a few percentage points are pretty good, even if they aren't good enough to tell you ahead of time the outcome of a very close national election that is within two percentage points of a tie.
It also seems ludicrous to me to ban a social measurement technique just because it's imperfect, but maybe you'll get your wish and this will happen. The U.S. government has been cutting back on measurement of economic statistics, measurement of the weather, all sorts of things. Did you hear that vaccines are imperfect too? Well, good news, the government's discouraging vaccines too. There are lots of imperfect things in the human sciences that people want to defund or ban.
I didn't say ban polling. I said around certain events, where polling can influence an election. In multiple horse races, as in the UK, accurate polling could help to decide on strategic voting.
I'm sure you understand the concept of meta analyses that are often used for noisy medical reports to tease out the truth, although even these can be biased by selective publishing.
As regards a "close race". If polls were accurate, then you would expect polls to straddle the no difference or 50% probability. If they all err in one direction, but the result is strongly in the other, then the polls were not measuring intent correctly. That could be for any number of reasons, but clearly the polls were not just "wrong", but had systematic bias. That may not be due to the pollsters' fault, but how the polls were answered. IOW, it is the same sort of problem psychologists have to deal with.
As a biologist, I have done animal counts. I don't expect animals to try to mess up my counts. OTOH, people have agency and may have reasons to answer questions deceptively. I'm sure if the current US government were to poll left-leaning US citizens just now, they might well get answers more supportive of right-wing POVs. This is what happens in authoritarian countries. I would be very suspicious of any requests for my opinions coming from anybody right now, whoever they purported to be.
Measuring responses to vaccines need not be devoid of actual data, as well as disease outbreaks. Measuring sewage outflows is a very cheap method to gauge diseases in communities. Economic measurements are unlikely to be biased, especially when it comes from sources that report results from actual hard data. Yes, those sources could lie, but what might be their incentive, unlike the government being run by an autocrat and inexperienced, probably incompetent, sycophants, to support the autocrat's POV. We saw this in the Bush II administration with climate change reports, and we have just seen it again with teh Trump II climate report.
There are lots of things people want to ban. I am very unhappy that the UK and some US states want age verification on websites. I am also unhappy that the UK is still balking at a "right to die" bill, something we allow for our suffering pets, but apparently not for any adults. I would still maintain a ban on genetically engineered babies unless it is used for correcting genetic diseases, even though someday we will know enough to make it safe for any purpose, and it will be accepted. Yet this knowledge can be used as a weapon, and I am not sure I want the sort of intrusive surveillance to prevent malicious actors from building biological weapons. Sometimes it is better to be cautious. Think about the current issue of developing AGI. Should we do it with today's level of knowledge? Probably not. But when we can make AGIs that are fully aligned with humans, and can prevent AGIs with antagonistic POVs towards humans, then we should have AGIs, but with a way to deal with "bad" AGIs as we do with humans.
Alex:
1. I don't want to ban polling, even around certain events. But, again, given what's been happening at the Bureau of Labor Statistics, the Center for Disease Control, etc., there's been a lot of attacks on data collection on the grounds that the data collected are imperfect, so it's not impossible that the government will find a way to ban polling.
2. You write, "If polls were accurate, then you would expect polls to straddle the no difference or 50% probability. If they all err in one direction, but the result is strongly in the other, then the polls were not measuring intent correctly. That could be for any number of reasons, but clearly the polls were not just "wrong", but had systematic bias." I agree completely! This is what we call "nonsampling error," but "systematic bias" is fine too. We measure this in our above-linked 2018 paper. Actually, my colleagues and I have published several papers on nonsampling error. It's an important topic in polling. Nonsampling error is not huge--it's a few percentage points--but it's not nothing, and that's why we account for it in our forecasts.
If someone sent me that disclaimer I would immediately log onto the dark web and uber a hitman to their exact location
Worksheet:
Nate wrote: "One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error," and Elliott's wrote: "Trump and Harris are both a normal polling error away from a blowout. The race is uncertain, but that doesn’t mean the outcome will be close." These disclaimers--given during the campaign, before the election--seem very reasonable to me.
I meant Dan’s disclaimer at the top
Re fn 1: Over here I have heard it referred to as the doctrine of the expert client. As in, some contracts are fair and enforceable when there is expertise on both sides, but unenforceable against naive parties.
Isn't this explicable just with mean and variance? 50-50 is perfectly consistent with a high probability of a blowout. If I'm attempting to predict the outcome of a boxing match, I can certainly say "50-50, but whoever wins will do so by a knockout, not on points".
I'll bite on the probability semantics. A prediction is a reductive compression of outcomes into a binary outcome. The variance of such a random variable is proportional to its expected value.
Part of my issue with the poll aggregator industry is they play fast and loose with their statistics, representing whatever data vis suits them to tell the story they want to tell. But then claiming that they are being perfectly rigorous even though their entire enterprise is metaphysical.
As Dan says, what matters is not what they intend to communicate, but what people take away.
I am indeed rather skeptical that what people take away isn't actually pretty close to what they intend to communicate, but pollsters won't admit openly that it is what they are trying to communicate.
Doctor:
I can't speak for pollsters, but the election forecasters at the Economist, 538, and Silver Bulletin were all trying very hard to communicate uncertainty! If many readers did not get that point and thought of our forecasts as essentially deterministic, then, yeah, that's on us, and we didn't communicate uncertainty clearly enough--but we were trying.
To say "won't admit openly" seems to imply some secret agenda, and that wasn't happening. We were very open that the forecasts were uncertain and that either candidate could sweep all the swing states. Again, it seems that we didn't say that often enough for some people--or maybe it's just that some people didn't want to hear about the uncertainty--so maybe we should've said it louder and more often.
I can’t speak about specific pollsters, but I strongly suspect that most if not all are trying to get specific results from their polls, or at the very least applying selective rigor when it comes to how to frame and publish the results based on whether they like the outcome. Looking at that graph from the Economist I think almost everyone would take away a different message than “we are really uncertain.”
Doctor:
I'm surprised at that last statement of yours, because I think the graph expresses uncertainty pretty strongly! But I accept that you and others see it differently, which suggests at the very least that we need to work harder at communicating our uncertainty. So I appreciate the feedback.
Aha! I am surprised you are surprised that I was… wait I lost count :)
I see that graph and think “That is a mess but it looks like more Kamala wins likely than Trump. I expect they aren’t taking bets on that, though.” A normal person, ie non cynical/non academic, is going to look at that and say the same thing only without the last sentence. Most people don’t differentiate between “Outcome X happens 70% of the time and I am certain of that “ and “Outcome X happens 100% of the time but I am only 30% sure of that.” Much less the fact that though there is no margin of error reported on the graph what you are looking at is a massive margin of error.
Put another way, people understand “Horse A looks very likely to win, but we are not at all sure” very differently from “Out of all these outcomes Horse A wins more often than B” and even more so than “We really have no idea, but maybe Horse A is a bit more likely?” A statistician will see through the words to the numbers, but just about everyone else listens to the words (or pictures) and doesn’t discount them as noise.
Ben:
I can't speak for "the poll aggregator industry" in general, but I can say that Nate Silver, Elliott Morris, and I were always very open about uncertainty, as in the quotes given above.
You write of people who are "claiming that they are being perfectly rigorous." I don't know who is claiming that polling "perfectly rigorous" or what that phrase even means. Polling can be rigorous in the sense of being transparent (which, as often is the case in statistics, can be a challenge with new data coming in real time), then there's the statistical inference, which depends on assumptions, as usual. I'm also not clear what you mean by "metaphysical" here. The most relevant dictionary definition seems to be "based on abstract (typically, excessively abstract) reasoning." I guess that statistical modeling is abstract: normal distributions and logistic curves and all the other mathematical tools that we use are abstract. These are the abstract tools used by Laplace, Gauss, Galton, and other applied statistical researchers over the centuries. Abstract reasoning in statistics has a pretty good track record: it's led to progress in fields ranging from astronomy and pharmacology to economics and sociology, and it also allows us to adjust polls and estimate public opinion to within a few percentage points.
Regarding your last point, yes, I agree that it's important what people take away from our reports, and given the reactions that you and Dan have had, I agree that we need to work even harder at communicating uncertainty. So thanks for communicating that to us: the communication was not so polite, but that's ok--maybe that's what was necessary to get our attention!
The clear communication thing is really tough. With that Economist graph example, there are definitely people with a bit more data literacy and a bit more time who read carefully, decide to just ignore the median lines (there's only one election, what does median even mean?), then do that exact eye crossing thing and say "eh, who knows." On the other hand there are people who are reading more quickly and possibly with less expertise that only notice the bigger number. Also a major dynamic has to be that a big chunk of your audience is coming to your site to *find out who is winning*, which maybe makes it different from the financial products example? Refusing to communicate at all is not a great option, you should try to figure it out.
The predictions of who will win are also a million times harder than, say, popular vote predictions because being ahead can mean "higher chance they will ultimately trounce the opponent and lower chance that they get trounced." This is not what being ahead normally means! It's a game of chess, not basketball.
Yes although I think you have to have a huge amount of expertise to really get it. I think the root of this beef is that Andrew is in the position of that xkcd cartoon that goes "you have to remember the average non geologist only knows the chemical formula for eight or nine of the feldspars", he really has a much higher baseline estimate than is realistic to hope of the understanding of ... well, of me, let alone actual laypeople
Dan:
Again, consider the actual statement from Nate Silver: "One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error," and the actual statement from Elliott Morris: "Trump and Harris are both a normal polling error away from a blowout. The race is uncertain, but that doesn’t mean the outcome will be close," and similar statements of uncertainty from me.
Given your above post, I'm sure you're right that we didn't emphasize this uncertainty enough, but all of us did try, and we tried repeatedly, and we wrote it in plain English not jargon, and we wrote all of this during the campaign before the election. Contrary to your post, there's nothing we tried to "take back in a disclaimer." We were open about uncertainty all the time. And, I guess, given the reactions from you and others, we'll emphasize the uncertainty even more next time.