Publishing false "facts" -- Lancet's attempt to rewrite reality

When the use of a reputable methodology is horrid, the results of a survey will be preposterous. That's the obvious point to learn from Steven E. Moore's analytical deconstruction of the recent Johns Hopkins University study that claims 600,000-655,000 Iraqis have died due to the Iraq War -- a number so far beyond any relationship to reality that it is pure propaganda. British medical journal, The Lancet, published the survey results, which follow up on a 2004 study that itself was criticized for overcalculating the death count.

Defenders of the study say that the methodology is sound -- cluster interviewing is a necessity in Third World countries because phone coverage is sparse. But what happens if you employ a proper methodology improperly? Answer: garbage in = garbage out. Here's Moore's view of HOW the surveyors performed the survey:

. . . the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, "Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey," the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points.

Neither would anyone else. For its 2004 survey of Iraq, the United Nations Development Program (UNDP) used 2,200 cluster points of 10 interviews each for a total sample of 21,688. True, interviews are expensive and not everyone has the U.N.'s bank account. However, even for a similarly sized sample, that is an extraordinarily small number of cluster points. A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711--almost three times that of the Johns Hopkins team for 93% of the sample size.

Get it? The key is not just the sample size (the ABC/Time/BBC/NHK/Der Spiegel sample is smaller than the HHU sample), it's the number of different sampling areas that is the key. If a US presidential election poll used only 30 sample areas, we'd immediately know the poll was worthless because we have 50 states and each one affects the election through the Electoral College. If you use fewer sampling areas in a Third World nation, you get additional uniformity of response due to familial, tribal and/or religious ties in the smaller area.

This is an embarrassment for The Lancet. Even the NY Times and WaPo buried the story deep inside their papers.