ERROR BANDS AROUND POLLING RESULTS
The Daily Kos is one of my favorite sites. He’s as partisan as I am, but he does his homework and tries to keep his optimism in check. And he’s diligent in reminding his readers about the margin of error in the poll figures he quotes. But that’s where I have a nit to pick with him; like most political reporters, he makes two opposite errors in reporting those error bands.
First, it’s simply wrong to say that if one candidate is leading another by some figure within the poll’s stated sampling error that the two are “statistically tied.” There’s nothing magic about those upper and lower bounds: they measure the range within which the true proportion would fall 95% of the time, if the only source of error were sampling variability. The bigger the lead, as measured, compared with the estimated sampling error, the larger the probability that the candidate who appears to be ahead is really ahead, but you’d always rather be a point up than a point down.
Moreover, if there are several polls reporting on the same race, all showing the same candidate in the lead, then the sampling error of all the polls combined is smaller than the sampling error in any one poll; the sampling error goes down as the square root of the sample size, so four polls of 1000 voters each have half the sampling error of a single poll with the same sample size. In particular, if a single poll shows some race to be within the margin of error and all the others show the gap to be larger than that, it’s confusing to say that the outlier poll shows a “statistical tie.”
So if pure sampling error — the random variation that comes in picking a group of 400 or 1000 people to ask your questions of — were the only source of error in a poll, then we could pretty well write off, say Ron Kirk in Texas.
But of course pure sampling error isn’t the only source of polling mistakes. Non-sampling error, also known as “systematic error” — the difficulty in getting a truly representative sample in a world of call screening, and the assumptions that go into classifying someone as a “likely voter” — is likely to be larger than sampling error, and much harder to measure. Sampling error you can figure with a formula, and increasing the sample size brings it down. Nonsampling error can’t be calculated that way, and increasing the sample size does absolutely nothing about it.
Guessing turnout in off-year elections is always tough, and this year it’s even tougher. So I don’t think it’s right to say that, e.g., the Dole-Bowles race in North Carolina is “statistically tied.” But it’s perfectly accurate to say that no one knows who’s going to win.