The Most Misleading Feature of Public Opinion Polls

The biggest inaccuracy of polls is the inbuilt assumption that respondents care about the questions they are asked

There are many ways, either through error or chicanery, that a poll can misrepresent public opinion on some issue. For example, the chosen sample can be unrepresentative, the questions can be poorly worded, or, as in this classic demonstration from Yes, Minister, respondents can be lead by the nose to give a certain answer.

Yet none of those problems is as serious as the one that afflicts almost every poll: The presumption that those polled care a whit about the issue in question. Whoever commissioned the poll of course considers it important, but that is no guarantee that respondents have ever thought about it before they were polled, or will act on their opinions in any way afterwards.

Advocacy organizations exploit this aspect of polls relentlessly. If the Antarctic Alliance polls 1000 people and asks “Would you like it if there were a law that protected penguins?”, probably 80% of people will say yes because it’s hard to hate on penguins: They are always well-dressed, they waddle in a cute way and many people are still feeling bad for them because of that egg they lost in that movie where they marched all that way in the cold — what was it called? — anyway, man that was sad, so yeah, happy to tell a pollster that we should protect those furry little guys.

Antarctic Alliance will then argue that Congress should pass the Protect the Penguins Act immediately because their new poll shows that 80% of Americans “want penguins to be protected”. But if you asked those same poll respondents if they’d be willing to donate even $10 to help the law pass, most of them would say no. And if you asked them if they would vote for the Congressional Representative on the basis of how s/he responded to the Protect the Penguins Act, most of them would say no. And if you asked them the open ended question “What are the 10 biggest challenges Congress should be addressing now?”, probably none of them would put penguin protection on their list.

To give a darker variant of this problem, gun control laws generally poll well yet don’t pass. How can we not pass something that we “support”? Easily, if the people who say they support it are not willing to do much to see it pass and the people who are against it are willing to do a lot. Polls usually miss this sort of nuance because they don’t assess how much people care about what they are being polled about.

The few polls that somewhat surmount this problem are those that assess the voting intentions only among people who intend to vote, and, those that try to assess how intensely people feel about the opinions they express (e.g., With a follow-up question of “would you be willing to have your taxes rise to make this happen?”).

The only way I can see to consistently avoid the problem of assuming respondents actually care about the issue of interest as much as do poll commissioners is to expand the usual response format of “Yes, no, or don’t know” to include the option “Don’t care”. But I doubt pollsters would ever do this because it would put them out of business to tell their clients that most people simply don’t give a fig.

Author: Keith Humphreys

Keith Humphreys is the Esther Ting Memorial Professor of Psychiatry at Stanford University and an Honorary Professor of Psychiatry at Kings College London. His research, teaching and writing have focused on addictive disorders, self-help organizations (e.g., breast cancer support groups, Alcoholics Anonymous), evaluation research methods, and public policy related to health care, mental illness, veterans, drugs, crime and correctional systems. Professor Humphreys' over 300 scholarly articles, monographs and books have been cited over thirteen thousand times by scientific colleagues. He is a regular contributor to Washington Post and has also written for the New York Times, Wall Street Journal, Washington Monthly, San Francisco Chronicle, The Guardian (UK), The Telegraph (UK), Times Higher Education (UK), Crossbow (UK) and other media outlets.

19 thoughts on “The Most Misleading Feature of Public Opinion Polls”

  1. In the specific case of gun control, it’s a safe presumption that a much greater proportion of respondents have thought about the issue before the pollster inquired, and that it’s a more visceral, acute and potent concern than the fate of critters residing at the ends of the earth & seen, only & very occasionally, on celluloid or print. So the responses to gun control can’t be assumed to be as casual and tenuous as penguin preservation.

    1. In the specific case of gun control, it’s a safe presumption that a much greater proportion of respondents have thought about the issue before the pollster inquired

      In my observation, this has not been much the case. Just the basics of “what are the existing laws?” seem to be very poorly known except among people who either have bought a gun recently, or who are heavily involved in gun law discussions. (For example, a large number of people are under the delusion that automatic weapons are generally available. A large number of people seem to think that the laws on sales/purchases of guns are different at gun shows than elsewhere. Neither of these are true, nor have they been true at any time since the 1930’s.)

      1. Automatic weapons (after passing a background check) are generally available.

        The laws on sales/purchases of guns at gun show allow sale of guns by non-dealers without any background checks. It is disingenuous (at best) to claim that this is somehow okay because the loophole is in the law and applies nationally, even outside gun shows. The biggest reason there are gun shows is to facilitate the sale of guns without background checks.

        1. From your link:

          Purchasing one requires submitting fingerprints and photographs to the Bureau of Alcohol, Tobacco, Firearms and Explosives, going through an FBI criminal background check, and paying a $200 tax, among other requirements. Only automatic weapons manufactured and registered with the federal government before 1986 can be bought, owned and sold.

          I don’t think that’s what most people mean by “generally available”; I think of “generally available” as “I could go into a gun shop and buy one.” (Note that the FBI background check is comparable to the one for a security clearance, not the quick, automated NCIS checks.)

          The biggest reason there are gun shows is to facilitate the sale of guns without background checks.

          That could only be believed by someone who has never been to a gun show. (At every gun show I’ve been to, 95% or more of the guns there were being sold by dealers.)

          Given that most guns sold at gun shows are sold by dealers (who do have to do background checks”

          1. If you can pass the very, very easy background check you will be driving away with an assault weapon–whether it be from a gun store or a gun show. Or, even if you can’t.

            As to the claims this could only be believed by someone who has never been to a gun show–well, I’ve been to plenty and saw plenty that was worrisome.

            I still remember the loons in camo roaming the aisles with assault weapons over their shoulders and for sale signs hanging from the barrel. I talked with many of them who were very certain they could sell their guns to anyone they wanted whether the buyer had a license or not. Just step outside into the parking lot was what I was told.

            Since sales by non-dealers are not registered or recorded there is no way you (or anyone else) could know what percentage of sales are made non-dealers. DO you have any real basis for your 95% claim.

          2. You seem (typically) confused about the same basic facts that I noted in my initial post that most people are ignorant of.

            The “assault weapons” are NOT AUTOMATIC. Yes, if you aren’t a dealer, and you are selling a standard non-automatic rifle, you can sell it to anyone you want. If you are a dealer, you have to run a background check through NICS–at a gun show, or elsewhere.

            And note that I made no claims about the proportion of sales–as you note, that is unknowable. The proportion of the guns that are on a dealers table, versus the ones being carried by someone with a price tag, is easy to observe; I’ve never been to a gun show where the former wasn’t the overwhelming majority of the guns on display.

          3. Sam,

            I am starting to under the basis of your argument which is apparently that the only possible basis of disagreement with you is being “confused.”

  2. The penguin intensity index changes a bit if Bernard adds the information “if the penguins go, so will Miami”.

    1. Perhaps … or maybe not.

      Immediacy and surety of negative reinforcement may be more relevant than severity. If the reinforcement is seen as being sometime in the indeterminate future, it dwindles in effectiveness. If smoking a cigarette today would give you cancer while you sleep tonight, with 100% surety, the rate of smoking would go way down.

  3. Interesting post as I recently discovered an advocacy group I wanted to learn more about, . All of their content is hidden behind a registration wall.

    “The reason we ask you to register is so that we can make sure that your voice is heard on this issue. … Once registered as a member, your are invited and enabled to ”get informed, get heard, get involved (when you can)” by checking into your personal community portal when prompted as little as 1-3 times per quarter (or as much as you want).”

    I completely understand that they can then point to the X number of actively concerned people in their organization. It’s much better than a poll or petition. But I wasn’t convinced yet that it was a big issue, or what they think the causes were, or what their proposed remedies are; and I can’t find any of that out without signing up. I know I suppose I could change my mind later and drop out, but they are not making their up-front case very well, if at all!

  4. There’s no reason to think that ensuring the availability of a “don’t care” response would actually help. A lot of people don’t feel comfortable choosing that even when it’s true, and they won’t admit that they don’t know anything about it, either.

    Who thinks we need closer ties with Fredonia?

    1. Fredonia has never done anything for us or condemned Islamic jihadism. Nor have they proved they don’t have WMD. Invade them tomorrow!

  5. I believe that opinion polls in general get an undeserved good name from election polls. For better or worse, each pollee is the world’s leading expert on whom he or she is going to vote for in the next election. There are still multiple points of uncertainty, but when there is enough professional polling data, the Nate Silvers of the world can foretell the actual election results very accurately. On almost any other subject, all the issues that James raises come to the fore to a much greater extent. While a “don’t care” category would be interesting, many pollees who ought to use that would probably take a flyer anyway.

  6. I’m sure you know this, but in case other readers don’t, it’s important to distinguish academic surveys from commercial polling. In academic surveys, there often are attempts to measure intensity of opinion.

  7. It’s easy to hate on “pollsters” here, but how about the social sciences leading by example?

    My personal response to this problem is to refuse to take part in polls which I think are badly designed (which means ALL polls where I cannot scan the entire question set before starting, which certainly includes all phone polls and most internet polls).

    However I recently received a poll in the mail from one of the UCs regarding my opinions about traffic and related matters, and it suffered from this same methodological flaw — no way to indicate the strength with which I cared about things. It also suffered the standard flaw of these documents in that it provided no way to indicate WHY I was opposed to certain things: am I opposed to building such and such a light rail line because I think it’s a boondoggle? Because I love cars? Because I love the area it will pass through? Because I think buses already serve that area well? Because it won’t personally benefit me and screw everyone else? No way to clarify that and similar points. Because it was a UC, I answered anyway, and sent it with a cover letter explaining the many flaws I saw in the poll.
    Will my letter change things? Unlikely. From my limited interaction with how Social Sciences are taught in the US, there is a massive amount of attention paid to methodological details (treated as sacred algorithm) and zero attention paid to the issue of: “does this question and do the five replies provided allow the respondent to clarify what they are thinking and why? In other words, does it allow me to answer ANY of the sort of follow-up questions a normal person might have once they have seen the brute number of 68% for, 28% against?” In other words, ALL the attention in the class is given to “how do you analyze the numbers in a way that will receive the approval of the publishing gods” and NOTHING is given to “how do you get numbers that are worth analyzing”.

    1. I think it’s worth pointing out that the literature on survey design, detailed courses on survey methodology, and so on, actually do cover these sorts of things in quite a bit of depth, at least in my experience. I think the problem lies with the fact that a nontrivial percentage of the people conducting surveys have no formal training in doing so. Graduate students often learn to design surveys via the received wisdom of their doctoral committee members (who themselves may not have been formally trained in conducting surveys), perhaps supplemented with some self-directed reading. These people then pass the same wisdom down to their graduate students, perpetuating the cycle. Just my two cents.

  8. Whenever someone cites poll results to me, I always ask to see the questions the participants were asked. And sometimes I wonder if the wording was chosen to generate a desired outcome. So this polling obsession really puts me off.

  9. Speaking of the order in which questions are asked, consider the Beck Depression Inventory, which is a 21 item self-report questionnaire for the depth of clinical depression. There are 4 options for each item, arranged in order of severity. For example:

    0 I don’t feel disappointed in myself
    1 I am disappointed in myself
    2 I am disgusted with myself
    3 I hate myself

    The first option is neutral, and each subsequent option is worse than the preceding.
    Of course, this is the structure of a hypnotic induction into a depressed state. In hypnosis, when the goal is hand levitation, the initial suggestion is that your hand is resting comfortably on your lap (which will be a true statement); then it is suggested that your hand is resting gently on your lap, then that it is resting ever so lightly, then that it is light as a feather, and finally that it does not need to continue to stay on your lap but can float as easily upwards as it can remain where it is.

    Milton Erickson himself could hardly have improved on the induction technique.

  10. Keith chose the best conservative example of this. But there’s also a great liberal example of this– abortion.

    People use abortion polls to preen against “bad” abortions. “Oh sure, I oppose abortion for sex selection”. “Yes, I think abortion on demand is wrong”. “Of course, I think abortions after 20 weeks should be illegal.”

    At the same time, Roe v. Wade, which everyone knows establishes a legal right to have an abortion for almost any reason (at least early in the pregnancy), polls very well. And every political pro I know thinks the Republicans get killed on the issue nationally, which is why every Democratic presidential candidate talks about abortion all the time and every Republican presidential candidate never mentions it unless it is brought up and talks about changing “hearts and minds” on the issue.

Comments are closed.