The following is an archived copy of a message sent to the CASI Analysis List run by Cambridge Solidarity with Iraq.

Views expressed in this archived message are those of the author, not of Cambridge Solidarity with Iraq (CASI).

[Main archive index/search] [List information] [CASI Homepage]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[casi-analysis] [1] Garfield comment re Slate/Lancet. [2] Zogby foreshadowed Lancet.

[ This message has been sent to you via the CASI-analysis mailing list ]

The Lancet survey (an estimated 100,000 excess Iraqi deaths since invasion) has
prompted a serious rejoinder by Fred Kaplan in Slate[1].  A response to Kaplan
by Professor Richard Garfield of Columbia (well known to CASI, and one of the
study's authors) is appended below [2].

Recall that a Zogby poll from mid-2003 foreshadowed the Lancet study's higher
casuality figures, finding that roughly 30% of sampled Iraqis answered 'yes' to
the question: "Were any of your own family members, neighbors, or friends killed
in this spring's war?".[3]

Remember also that the Lancet survey appears against a backdrop of suppression
of civilian causualty data.[4]

Drew Hamre

=== [1];

100,000 Dead—or 8,000
How many Iraqi civilians have died as a result of the war?
By Fred Kaplan
Posted Friday, Oct. 29, 2004, at 3:49 PM PT

The authors of a peer-reviewed study, conducted by a survey team from Johns
Hopkins University, claim that about 100,000 Iraqi civilians have died as a
result of the war. Yet a close look at the actual study, published online today
by the British medical journal the Lancet, reveals that this number is so loose
as to be meaningless.

The report's authors derive this figure by estimating how many Iraqis died in a
14-month period before the U.S. invasion, conducting surveys on how many died in
a similar period after the invasion began (more on those surveys later), and
subtracting the difference. That difference—the number of "extra" deaths in the
post-invasion period—signifies the war's toll. That number is 98,000. But read
the passage that cites the calculation more fully:

We estimate there were 98,000 extra deaths (95% CI 8000-194 000) during the
post-war period.

Readers who are accustomed to perusing statistical documents know what the set
of numbers in the parentheses means. For the other 99.9 percent of you, I'll
spell it out in plain English—which, disturbingly, the study never does. It
means that the authors are 95 percent confident that the war-caused deaths
totaled some number between 8,000 and 194,000. (The number cited in plain
language—98,000—is roughly at the halfway point in this absurdly vast range.)

This isn't an estimate. It's a dart board.

Imagine reading a poll reporting that George W. Bush will win somewhere between
4 percent and 96 percent of the votes in this Tuesday's election. You would say
that this is a useless poll and that something must have gone terribly wrong
with the sampling. The same is true of the Lancet article: It's a useless study;
something went terribly wrong with the sampling.

The problem is, ultimately, not with the scholars who conducted the study; they
did the best they could under the circumstances. The problem is the
circumstances. It's hard to conduct reliable, random surveys—and to extrapolate
meaningful data from the results of those surveys—in the chaotic, restrictive
environment of war.

However, these scholars are responsible for the hype surrounding the study.
Gilbert Burnham, one of the co-authors, told the International Herald Tribune
(for a story reprinted in today's New York Times), "We're quite sure that the
estimate of 100,000 is a conservative estimate." Yet the text of the study
reveals this is simply untrue. Burnham should have said, "We're not quite sure
what our estimate means. Assuming our model is accurate, the actual death toll
might be 100,000, or it might be somewhere between 92,000 lower and 94,000
higher than that number."

Not a meaty headline, but truer to the findings of his own study.

Here's how the Johns Hopkins team—which, for the record, was led by Dr. Les
Roberts of the university's Bloomberg School of Public Health—went about its
work. They randomly selected 33 neighborhoods across Iraq—equal-sized population
"clusters"—and, this past September, set out to interview 30 households in each.
They asked how many people in each household died, of what causes, during the 14
months before the U.S. invasion—and how many died, of what, in the 17 months
since the war began. They then took the results of their random sample and
extrapolated them to the entire country, assuming that their 33 clusters were
perfectly representative of all Iraq.

This is a time-honored technique for many epidemiological studies, but those
conducting them have to take great care that the way they select the
neighborhoods is truly random (which, as most poll-watchers of any sort know, is
difficult under the easiest of circumstances). There's a further complication
when studying the results of war, especially a war fought mainly by precision
bombs dropped from the air: The damage is not randomly distributed; it's very
heavily concentrated in a few areas.

The Johns Hopkins team had to confront this problem. One of the 33 clusters they
selected happened to be in Fallujah, one of the most heavily bombed and shelled
cities in all Iraq. Was it legitimate to extrapolate from a sample that included
such an extreme case? More awkward yet, it turned out, two-thirds of all the
violent deaths that the team recorded took place in the Fallujah cluster. They
settled the dilemma by issuing two sets of figures—one with Fallujah, the other
without. The estimate of 98,000 deaths is the extrapolation from the set that
does not include Fallujah. What's the extrapolation for the set that does
include Fallujah? They don't exactly say. Fallujah was nearly unique; it's
impossible to figure out how to extrapolate from it. A question does arise,
though: Is this difficulty a result of some peculiarity about the fighting in
Fallujah? Or is it a result of some peculiarity in the survey's methodology?

There were other problems. The survey team simply could not visit some of the
randomly chosen clusters; the roads were blocked off, in some cases by coalition
checkpoints. So the team picked other, more accessible areas that had received
similar amounts of damage. But it's unclear how they made this calculation. In
any case, the detour destroyed the survey's randomness; the results are
inherently tainted. In other cases, the team didn't find enough people in a
cluster to interview, so they expanded the survey to an adjoining cluster.
Again, at that point, the survey was no longer random, and so the results are

Beth Osborne Daponte, senior research scholar at Yale University's Institution
for Social and Policy Studies, put the point diplomatically after reading the
Lancet article this morning and discussing it with me in a phone conversation:
"It attests to the difficulty of doing this sort of survey work during a war. …
No one can come up with any credible estimates yet, at least not through the
sorts of methods used here."

The study, though, does have a fundamental flaw that has nothing to do with the
limits imposed by wartime—and this flaw suggests that, within the study's wide
range of possible casualty estimates, the real number tends more toward the
lower end of the scale. In order to gauge the risk of death brought on by the
war, the researchers first had to measure the risk of death in Iraq before the
war. Based on their survey of how many people in the sampled households died
before the war, they calculated that the mortality rate in prewar Iraq was 5
deaths per 1,000 people per year. The mortality rate after the war started—not
including Fallujah—was 7.9 deaths per 1,000 people per year. In short, the risk
of death in Iraq since the war is 58 percent higher (7.9 divided by 5 = 1.58)
than it was before the war.

But there are two problems with this calculation. First, Daponte (who has
studied Iraqi population figures for many years) questions the finding that
prewar mortality was 5 deaths per 1,000. According to quite comprehensive data
collected by the United Nations, Iraq's mortality rate from 1980-85 was 8.1 per
1,000. From 1985-90, the years leading up to the 1991 Gulf War, the rate
declined to 6.8 per 1,000. After '91, the numbers are murkier, but clearly they
went up. Whatever they were in 2002, they were almost certainly higher than 5
per 1,000. In other words, the wartime mortality rate—if it is 7.9 per
1,000—probably does not exceed the peacetime rate by as much as the Johns
Hopkins team assumes.

The second problem with the calculation goes back to the problem cited at the
top of this article—the margin of error. Here is the relevant passage from the
study: "The risk of death is 1.5-fold (1.1 – 2.3) higher after the invasion."
Those mysterious numbers in the parentheses mean the authors are 95 percent
confident that the risk of death now is between 1.1 and 2.3 times higher than it
was before the invasion—in other words, as little as 10 percent higher or as
much as 130 percent higher. Again, the math is too vague to be useful.

There is one group out there counting civilian casualties in a way that's
tangible, specific, and very useful—a team of mainly British researchers, led by
Hamit Dardagan and John Sloboda, called Iraq Body Count. They have kept a
running total of civilian deaths, derived entirely from press reports. Their
count is triple fact-checked; their database is itemized and fastidiously
sourced; and they take great pains to separate civilian from combatant
casualties (for instance, last Tuesday, the group released a report estimating
that, of the 800 Iraqis killed in last April's siege of Fallujah, 572 to 616 of
them were civilians, at least 308 of them women and children).

The IBC estimates that between 14,181 and 16,312 Iraqi civilians have died as a
result of the war—about half of them since the battlefield phase of the war
ended last May. The group also notes that these figures are probably on the low
side, since some deaths must have taken place outside the media's purview.

So, let's call it 15,000 or—allowing for deaths that the press didn't
report—20,000 or 25,000, maybe 30,000 Iraqi civilians killed in a pre-emptive
war waged (according to the latest rationale) on their behalf. That's a number
more solidly rooted in reality than the Hopkins figure—and, given that fact, no
less shocking.

=== [2]
Correspondence with one of the report's authors has yielded the following:
By Richard Garfield, one of the study's authors.

On page five of the report. second to last paragraph, the authors do give us a
margin of sampling error. They have not found a hard-and-fast 98,000 additional
deaths, but a range from 8,000 to 194,000.

That is correct. Research is more than summarizing data, it is also
interpretation. If we had just visited the 32 neighborhoods without Falluja and
did not look at the data or think about them, we would have reported 98,000
deaths, and said the measure was so imprecise that there was a 2.5% chance that
there had been less than 8,000 deaths, a 10% chance that there had been less
than about 45,000 deaths,....all of those assumptions that go with normal
distributions. But we had two other pieces of information. First, violence
accounted for only 2% of deaths before the war and was the main cause of death
after the invasion. That is something new, consistent with the dramatic rise in
mortality and reduces the likelihood that the true number was at the lower end
of the confidence range. Secondly, there is the Falluja data, which imply that
there are pockets of Anbar, or other communities like Falluja, experiencing
intense conflict, that have far more deaths than the rest of the country. We set
that aside these data in statistical analysis because the result in this cluster
was such an outlier, but it tells us that the true death toll is far more likely
to be on the high-side of our point estimate than on the low side.

Further comment would be superfluous.
[end excerpt]

=== [3]
The Zogby poll was funded (ironically) by the AEI.  See

AEI promptly misquoted their own data, in service of the (increasingly
untenable) point that this war was "comparatively gentle".  See

=== [4]
Pentagon suppresses details of civilian casualties, says expert
By Raymond Whitaker
31 October 2004

The Pentagon is collecting figures on local casualties in Iraq, contrary to its
public claims, but the results are classified, according to one of the authors
of an independent study which reported last week that the war has killed at
least 100,000 Iraqis.

"Despite the claim of the head of US Central Command at the time, General Tommy
Franks, that 'We don't do body counts', the US military does collect casualty
figures in Iraq," said Professor Richard Garfield, an expert on the effects of
conflict on civilians. "But since 1991, when Colin Powell was head of the joint
chiefs of staff, the figures have been kept secret."

Professor Garfield, who lectures at Columbia University in New York and the
London School of Hygiene and Public Health, believes the Pentagon's stance has
confused its response to the latest study. "The military is saying: 'We don't
believe it, but because we don't collect figures, we can't comment," he said.

"Mr Powell decided to keep the figures secret because of the controversy over
body counts in Vietnam, but I think democracies need this information."

The first scientific study of the human cost of the Iraq war, published last
week in The Lancet, showed a higher level of casualties than previous estimates., a website which collects accounts of Iraqi civilian deaths
reported by two separate media sources, said yesterday the toll was between
14,181 and 16,312, but admits that the spreading violence in Iraq, which has
made it all but impossible for journalists to move around safely, has undermined
its method. That did not prevent the Foreign Secretary, Jack Straw, from using
its figures to cast doubt on the academic survey.

The Government would examine the results "with very great care", Mr Straw told
BBC Radio 4's Today programme last week. "It is an estimate based on very
different methodology from standard methodology for assessing casualties, namely
on the number of people reported to have been killed at the time or around the
time." Previously the Government has dismissed the findings of the Iraqbodycount

The study by US and Iraqi researchers, led by Johns Hopkins Bloomberg School of
Public Health in Baltimore, surveyed 1,000 households in 33 randomly chosen
areas in Iraq. It found that the risk of violent death was 58 times higher in
the period since the invasion, and that most of the victims were women and

"Making conservative assumptions, about 100,000 excess deaths have happened ...
Violence accounted for most of the excess deaths, and air strikes from coalition
forces accounted for most violent deaths," said Les Roberts of the Baltimore
institution. The researchers excluded Fallujah, the most violent area of Iraq,
from their results, which would have made the toll higher. But the finding that
air strikes caused the highest casualties casts doubt on US claims that air
attacks allow pinpoint precision.

Iraq's interim government has also suppressed casualty figures. Dr Nagham
Mohsen, an official at the Iraqi Health Ministry, was compiling data from
hospital records last year. In December she was ordered by a superior to stop.
The Health Minister denied that the order was inspired by the Coalition
Provisional Authority.

Sent via the CASI-analysis mailing list
To unsubscribe, visit
All postings are archived on CASI's website at

[Campaign Against Sanctions on Iraq Homepage]