Comments on: Sample bias in the Lancet study?

By: raj

raj — Sun, 22 Oct 2006 15:34:39 +0000

Brett Bellmore at October 21, 2006 07:00 AM
a war zone, where the wrong answer might get their families murdered, why would we assume that polsters are getting accurate answers from their samples?
Oh, Brett, don’t be silly. Maybe because, in most cases in which the interviewees claimed to have had a death in the family, they asked and usually got death certificates. And in most of those cases, cause of death was indicated as being something like gunshot wounds. You really should read the report of the study before commenting on it.
It strikes me that, if there was false answers for the reason that you mention, the death count would be under-reported, not the other way around.

By: Bruce Moomaw

Bruce Moomaw — Sun, 22 Oct 2006 08:01:55 +0000

Matt Yglesias points out another factor that could very well have made the study tend to UNDERCOUNT deaths: http://www.matthewyglesias.com/archives/2006/10/fred_kaplans_take/ (last sentence)

By: Barry

Barry — Sun, 22 Oct 2006 00:54:17 +0000

The indefatigable Brett rears his ugly head again. Brett, it’s also likely that the respondents *minimized* the number of deaths – that there was suubstantial underreporting. Like the ‘didn’t see nothing’ attitude that can confound police efforts when investigating shootings in dangerous neighborhoods, where nobody wants a reputation as a ‘snitch’.

By: rilkefan

rilkefan — Sat, 21 Oct 2006 22:15:19 +0000

Lambert’s post:
scienceblogs.com/deltoid/2006/10/science_on_lancet_study.php
Seems to me he’s getting a bit out ahead of the facts of the matter, but we’ll see.

By: tc

tc — Sat, 21 Oct 2006 20:05:28 +0000

This is from the Science article:
“…Burnham counters that such streets were included and that the methods section of the published paper is oversimplified. He also told Science that he does not know exactly how the Iraqi team conducted its survey; the details about neighborhoods surveyed were destroyed ‘in case they fell into the wrong hands and could increase the risks to residents.’”
Well I guess we’ll never know.

By: Tim Lambert

Tim Lambert — Sat, 21 Oct 2006 19:45:38 +0000

My comment is here. I’m not impressed.

By: wcw

wcw — Sat, 21 Oct 2006 18:56:38 +0000

DeLong had this, too. I think the criticism ranges from false to misspecified. My comment is here: http://delong.typepad.com/sdj/2006/10/back_alleys_vs_.html#comment-24206359
——————————————————
I could see a slight bias. Did Johnson and Holloway examine the actual data set, or were they working off the description? I do not have the data and a map, but “all the households surveyed by the Lancet authors were on main roads or at intersections of smaller streets with major arteries” seems false.
The study selected a “main street” (which is not a “major artery” in my book, but again I don’t have the data and a map), then selected a residential street which intersected it (i.e., not all but none of the households sampled were “on” main roads). Somewhere on that street (not at the corner except by chance), a start household was selected, then 40 adjacent households were sampled.
Hence, as I said, I could see a slight sampling bias for corners along main roads versus the previous GPS method. However, I also see a slight sampling bias against main roads off the corners. Based on the pullquote cited, I’d say Johnson and Holloway make a misleading and factually incorrect assertion.
If they actually ran the numbers and found bias, I retract this analysis. However, I bet they didn’t.

By: Mark Kleiman

Mark Kleiman — Sat, 21 Oct 2006 16:52:10 +0000

Brett:
Of course misreporting is possible. But what reason is there to think it happened in this case? Note that each household was asked about births and deaths before and after the invasion … sorry, I meant “libeation.” The study looked at the difference. The pre-liberation births and deaths, projected onto the national population, closely matched the official counts.
So you need to come up with a story about why families decided to exaggerate the numbers of their dead after the liberation but not before.
Possible? Sure. But not very plausible.

By: la dolce vita

la dolce vita — Sat, 21 Oct 2006 16:11:27 +0000

Catowner,
I have several friends working as journalists in Baghdad and Iraq. While it’s true they don’t do as many interviews as they should, your assumption on the reason is incorrect, it has nothing to do with pools. The reason is that it’s simply too dangerous for the journalist and the interviewed for it to happen as though this were occuring in Mayberry.
Perhaps it’s time to change channels over from Fox… But then, I can’t think of a televised alternative that would help…

By: Rob

Rob — Sat, 21 Oct 2006 15:59:30 +0000

Umm, Brett thats why they checked the survey against death certificates.

By: serial catowner

serial catowner — Sat, 21 Oct 2006 14:52:16 +0000

This is just a guess from an average guy, but judging from the fact that the interviewers apparently all survived, it seems likely they were working in areas of lower than average danger.
Either that, or the press corps in Baghdad are all lounging around the hotel pool when they should be out doing interviews.

By: Brett Bellmore

Brett Bellmore — Sat, 21 Oct 2006 14:00:53 +0000

I may not be a statistician, but I did ace statistics at (Michigan) Tech, and one thing I remember that nobody defending Lancet seems to want to address:
Sampling errors aren’t the only source of error in polling. Frequently they’re not even the largest error in a poll. They’re just the easiest to calculate.
People frequently lie to polsters here in the US, to avoid nothing more than embarassment. In a war zone, where the wrong answer might get their families murdered, why would we assume that polsters are getting accurate answers from their samples?