BRIEFING NOTE, 01/11/04

THE LANCET IRAQ MORTALITY SURVEY: THE UK GOVERNMENT’S RESPONSE IS INACCURATE AND MISLEADING

On Friday 29th October 2004, the Lancet medical journal published a study of post-war mortality in Iraq which estimated that at least 100,000 excess civilian deaths had occurred since the 2003 invasion; that most were caused by violence; and that most of those violent deaths were caused by coalition air strikes.

In his daily press briefing that morning, the Prime Minister's Official Spokesperson (PMOS) dismissed the study since its methodology was, he claimed, inappropriate:

“Asked if the Prime Minister was concerned about a survey published today suggesting that 100,000 Iraqi civilians had died as a result of the war in Iraq, the PMOS said that it was important to treat the figures with caution because there were a number of concerns and doubts about the methodology that had been used. Firstly, the survey appeared to be based on an extrapolation technique rather than a detailed body count. Our worries centred on the fact that the technique in question appeared to treat Iraq as if every area was one and the same. In terms of the level of conflict, that was definitely not the case. Secondly, the survey appeared to assume that bombing had taken place throughout Iraq. Again, that was not true. It had been focussed primarily on areas such as Fallujah. Consequently, we did not believe that extrapolation was an appropriate technique to use.”

These criticisms of the Lancet mortality survey were repeated again on 1st November. They are misleading, and almost entirely unfounded. This briefing note examines the claims of the Prime Minister’s Official Spokesperson.

1) "Firstly, the survey appeared to be based on an extrapolation technique rather than a detailed body count."

The fact that the survey uses an extrapolation technique does not automatically mean that it is less likely to be accurate than a body count. In fact, it is more likely to be accurate than existing attempts at body counts of Iraqi civilians. This is because in Iraq, where there are so many no-go areas, it would be impossible to count every casualty.

This is also the problem with Iraqi Ministry of Health figures obtained by counting bodies arriving at hospitals, cited by the PMOS in response to the Lancet report on 1st November: many of those who have died or been killed will never arrive at hospital in conditions of war, when access to roads and health facilities are severely disrupted, and when it might seem pointless to risk the journey for the sake of someone who is already dead.

Finally, attempts to do body counts through deaths reported in the press, like Iraq Body Count, are also necessarily underestimates, since press reports of casualties will be incomplete, not least because the areas where people are being killed are the same areas into which journalists don't dare go. As Iraq Body Count states on its website:

"Our maximum therefore refers to reported deaths - which can only be a sample of true deaths unless one assumes that every civilian death has been reported. It is likely that many if not most civilian casualties will go unreported by the media."

In these circumstances, population-based research like the Lancet study is thus arguably a much better source.

A detailed, comprehensive body count on the ground by military and medical personnel would perhaps be more accurate, but it is a task which the US and UK forces in Iraq refuse to carry out. For Downing Street to question the best available study on the grounds of preferring a methodology which they themselves refuse to operate is surprising, to say the least.

2) "Our worries centred on the fact that the technique in question appeared to treat Iraq as if every area was one and the same."

The technique does not treat Iraq as if every area was the same. The survey takes a sample in each of 33 areas throughout Iraq’s 18 governorates, precisely so that regional variation is taken into account as much as possible. All regions of the country were sampled.

The PMOS might be referring to the fact that due to the deteriorating security situation in Iraq, the survey 'paired' 12 of the 18 governorates, and took samples from only 1 of the pair in order to reduce danger to the investigators by reducing the necessary travel. In these cases, however, the process of assigning samples to one of the two governorates was randomised, and weighted according to the population of the governorates, thus 'still sampling from all regions of the country' and ensuring that 'the sample remained a random national sample' (Roberts et al p.2)

3) "Secondly, the survey appeared to assume that bombing had taken place throughout Iraq. Again, that was not true. It had been focussed primarily on areas such as Fallujah."

This is also untrue - see above. The survey was designed precisely to take variations throughout Iraq into account. Moreover, since the mortality rate due to violent deaths in the sample collected in Fallujah was so abnormally high, the Fallujah sample was excluded from the study’s final estimate. Thus far from overestimating the deaths caused elsewhere by extrapolating them from those clustered in Fallujah, as the PMOS suggests, in fact the figure of 100,000 deaths does not take into account the exceptionally high mortality probably caused in the area of the most intensive bombing, namely Fallujah. Had the Fallujah sample been included, the survey's estimate would have been of an excess of about 298,000 deaths, with 200,000 concentrated in the 3% of Iraq around Fallujah (Roberts et al p.5).

In addition, contrary to the PMOS’s claims about the tight focus of bombing, the Lancet study in fact found that in areas well away from Fallujah the most significant cause of the deaths it recorded was still violence, most attributed to coalition bombing (Roberts et al p.4).

4) "Consequently, we did not believe that extrapolation was an appropriate technique to use."

Almost all surveys except censuses involve 'extrapolation techniques' (using a sample rather than the entire population), since it is simply impossible to survey every member of a country's population. Extrapolation is a universally recognised method that governments and academics use to obtain data, and it is remarkable to see a government spokesperson rejecting it outright.

In addition, the specific sampling technique used in the Iraq mortality survey, a clustered sample survey, is widely used by government and other surveys to gather data from geographical clusters of households. See, for example, the recent report from the Home Office Research, Development and Statistics Directorate, 'Asylum seekers' experiences of the voucher scheme in the UK, March 2002', which used cluster samples in order to reduce the cost of the survey (see pp.2-3 of this report).

***

Perhaps more serious than these misrepresentations of the mortality study by the Prime Minister's Office is the UK Government's apparent disregard for efforts to ascertain the numbers of civilian deaths its actions have caused. Foreign Office Minister Baroness Symons promised the House of Lords that:

“we have done our best to count them [civilian casualties] but I am bound to say to the noble Baroness that the practicalities involved are very difficult indeed. Certainly during the early part of the engagement, it has not always been possible to know what civilian casualties there were on the ground….We do our best to count civilian casualties now."

(Lords Hansard 27 Apr 2004: Column 686)

Yet neither the US nor the UK governments have made public any aggregate data on civilian deaths in Iraq – a failure which an MOD spokesperson defended by arguing that

“there is no reliable way of estimating the number of civilian casualties caused during major combat operations.”

(BBC Online 22 September 2004)

The Lancet mortality survey argues that this is not the case:

“This survey shows that with modest funds, 4 weeks, and seven Iraqi team members willing to risk their lives, a useful measure of civilian deaths could be obtained. There seems to be little excuse for occupying forces to not be able to provide more precise tallies.”

(Roberts et al p.7)


In the last few days the Government have attempted to disparage the Lancet report on a different tack, by going back on its previous rejections of casualty estimates and instead positively citing the much lower estimate of the Iraq Body Count project, as Foreign Secretary Jack Straw did when interviewed on the Radio 4 ‘Today’ programme on 29th October. Mr Straw ignores the fact that Iraq Body Count itself makes clear that its figures cannot be considered a comprehensive estimate of civilian casualties, since ‘it is likely that many if not most civilian casualties will go unreported by the media.’ (see above). The Government’s new attention to the figures provided by this media-report-based casualty estimate is also surprising given the previous statement by an MOD spokesperson about Iraq Body Count that


“ We would caution against taking the numbers quoted in media reports and else where at face value. No source or combination of sources can produce a reliable figure."

(BBC Online 22 September 2004)

and given Defence Minister Adam Ingram’s dismissive answer to a parliamentary question in June 2003 asking “what information he has (a) given to and (b) received from the organisation called Iraq Body Count”, Mr Ingram replying simply that “Defence Ministers have had no contact with an organisation called Iraq Body Count.” (Commons Written Answers, 3 Jun 2003 Col. 37W, question 114211)

The Government must accept that despite shortcomings due to the difficulties faced by a small team of volunteer doctors collecting data in a war zone, the Lancet report is the most comprehensive study of Iraqi civilian deaths since 2003 currently available. If it is to rebuff its findings, the Government must honour its own obligations by undertaking a large-scale survey of civilian casualties. As the Lancet report states:

“ US General Tommy Franks is widely quoted as saying “we don’t do body counts”. The Geneva Conventions have clear guidance about the responsibilities of occupying armies to the civilian population they control. The fact that more than half the deaths reportedly caused by the occupying forces were women and children is cause for concern. In particular, Convention IV, Article 27 states that protected persons “. . . shall be at all times humanely treated, and shall be protected especially against acts of violence . . .”. It seems difficult to understand how a military force could monitor the extent to which civilians are protected against violence without systematically doing body counts or at least looking at the kinds of casualties they induce. In view of the political importance of this conflict, these results should be confirmed by an independent body such as the ICRC, Epicentre, or WHO.”

(Roberts et al p.7)

FURTHER INFORMATION:

  1. L. Roberts, G. Burnham, R. Lafta, J. Khudhairi, R. Garfield, ‘Mortality before and after the 2003 invasion of Iraq: cluster sample survey’, http://image.thelancet.com/extras/04art10342web.pdf
  2. Iraq Body Count, the most comprehensive indirect casualty count: http://iraqbodycount.net/ .This site includes a summary of another casualty count project, and an explanation of IBC methodology.
  3. Collation of comments on the Lancet report, by Chris Lightfoot.
  4. Critique of the report by Fred Kaplan in Slate; rebuttal by Professor Richard Garfield; and information on a recent Iraqi poll which arguably supports the Lancet figures are available here
  5. Commentary on the statistics of the report. in the Economist. The article finds the statistics valid. (Requires subscription).