We are developing the social individualist meta-context for the future. From the very serious to the extremely frivolous... lets see what is on the mind of the Samizdata people.

Samizdata, derived from Samizdat /n. - a system of clandestine publication of banned literature in the USSR [Russ.,= self-publishing house]

Inherently incredible

By now, we have surely all heard about the Lancet’s new claim that over 600,000 Iraqis are dead as a result of the US invasion of that country. Lets put that number in perspective.

It exceeds by 25% the war dead (450,000), military and civilian, suffered by Great Britain in all of World War II, including the Blitz, the African campaign, the Pacific campaign, and of course the European campaign.

It exceeds by 25% the war dead (460,000), military and civilian, suffered by Italy in all of World War II.

It exceeds the war dead(562,000), military and civilian, suffered by France in in all of World War II, including the initial battles with the Germans, the Occupation, and the reconquest by the Allies.

The death rate claimed for Iraq (around 2.6%) is approximately the same as that experienced in a number of the countries occupied by the Nazis where the Holocaust was implemented, and approaches that experienced by the Japanese in World War II (around 3.6%), which includes both the horrendous death tolls inflicted on the Japanese military during the island warfare, the virtual extermination of the Japanese navy and air force, and of course the firebombing and ultimately the nuclear bombing of Japanese cities.

Keep in mind the fact that the WWII numbers encompass a six year period, whereas the current war in Iraq dates back just over three years.

Does it seem remotely possible to you that the Iraqi war has been harder on Iraq than WWII was on a number of its major combatants, and in half the time? And doesn’t it strike you as a remarkable coincidence that the Lancet releases its studies on deaths in Iraq in the month before major US elections?

67 comments to Inherently incredible

  • odyssey

    Ummm…
    Rwanda?
    Congo?

    Completely different style of warfare than WWII.

    IIRC we’re talking 1/40 of the population. What proportion of the population are we talking with WWII? Bear in mind that populations were also smaller then…

    Straight numbers don’t give a meaningful comparison.

  • Steph

    No not remotly credable, but suspiciously well time to coincide with a US election. Remember they have tried this before.

  • madne0

    It also exceeds, by almost 300 thousand, the number of German civilians killed by Allied bombing during the entire 2nd World War. Conclusion? Carpet bombing saves lives.

  • Ummm…
    Rwanda?
    Congo?

    Ummm… your point being…?

    Straight numbers don’t give a meaningful comparison.

    Of course they do. It gives a very good indication of the intensity of the fighting.

  • There really aren’t any really accurate numbers on how many people have died. This study relies on guesswork, which I’m not inclined to trust. On the other hand, iraqbodycount.org, which keeps a careful count based on verifiable information, only includes civilians.

    That said, I expect to see three responses to this survey from Busheviks:

    1. It didn’t happen. (check)
    2. If it did happen, they deserved it.
    3. Only 600,000? That’s not nearly enough!

  • The 650k figure isn’t how many Iraqis the Americans have killed, but how many extra Iraqis have died due to the war. This includes cutting off medical supplies, the starting of the civil war, the ethnic cleansings going on, etc.

    This number isn’t surprising, since the invasion touched off a civil war based on civilian attacks. The number of deaths is pretty similar to non-disease deaths during the US War Between the States, which lasted about six months shorter than the current war.

    You’re not going to like this number, of course, because it outlines in stark detail the cost of the war in human lives, and that the Glorious Liberation of the Iraqi People you, de Havilland, and the rest of the filth envisioned is just a heap of children’s skulls.

    – Josh

  • K

    My admiration for Bush and the Rovians continues to grow. They have changed the past.

    They have managed to reduce casualty counts as far back as WW2 in order to make Lancet’s numbers look suspicous.

    But let me not be serious for a moment. From what little I have seen in the press releases the study basically asked people about deaths they were aware of and projected.

    This is a valid technique but so is looking for a gas leak with a lighted match – it may not be a good method for the circumstances.

    I suspect the problems, if any, came with using translators and intermediaries to gather the data,. And in their demeanor. And in the phrasing of the questions. Add in suspicion, among a populace where suspicion is a Darwinian plus, and you get answers that tend to please the questioner.

    Anyway. The Lancet study will itself be studied. Maybe to it’s credit, maybe not.

  • Giles

    I think the more serious point is that this is published in the lancet.

    I wouldnt have any objections if it’d been in the new statesman or whatever, but the lancet is meant to be a serious medical publication.

    This type of nonsense, I’m afraid probably undermines its credibility and by implication undermines the credibility of some serious medical research published therein. And if that means important medical research if over looked – that could cost lives.

  • Midwesterner

    Interesting twist. Apparently the MSM are so het up to get this out that they didn’t even keep their own terms. Surprised?

  • The 650k figure isn’t how many Iraqis the Americans have killed, but how many extra Iraqis have
    died due to the war.

    I am quite aware of that.

    Are you aware that the Lancet study estimates that the vast majority of “excess” deaths are deaths from violence? That the 600,000 number I use in my post is in fact the Lancet number for violent deaths?

    I think my comparisons of (claimed) deaths by violence in Iraq to military and civilian casualties in WWII is pretty much apples to apples.

  • Jordan

    Apparently the Lancet didn’t learn from the last time they tried this and were promptly laughed off the stage.

    If this continues, I eagerly await the study announcing that the entire population of planet Earth has died in Iraq a few years down the road.

    Of course the useful idiots will believe it, never mind that it disagrees with the U.N., the Iraqi government, and Iraq Body Count by an entire order of magnitude.

  • Ham

    In any case, a public argument conducted by our leaders over the number of dead in Iraq is an unedifying and quite repulsive spectacle. It makes me wonder if anyone in the White House would be capable of suggesting how many deaths would be too many. The rhetoric suggests that any number is good value, probably even 600,000. Surely not.

  • 1skeptic

    I agree

    It sounds incredible…so it can’t be true

  • The study only documents 547 actual deaths. The figure of 650,000 is a statistical extrapolation of that figure applied to the entire country.

    Make of that what you will…

  • You’re not going to like this number, of course, because it outlines in stark detail the cost of the war in human lives, and that the Glorious Liberation of the Iraqi People you, de Havilland, and the rest of the filth envisioned is just a heap of children’s skulls.

    Really? So that suggests (1) no heap of children’s skulls when your good chums the Baathists were calling the shots (2) the change of government has produced no benefits at all.

    Feel free to argue the cost to the USA and UK not being worth it to the taxpayers and/or the volunteer soldiers (the “not worth the bones of a single Pomeranian grenadier” argument and all that), or even the wisdom of how it was done, but please do not pretend you give a shit about ‘the children’ in Iraq if you prefered to leave Saddam in charge.

  • James

    The Lancet study may or may not be bunk, but this post does nothing to prove it either way. The reasoning used seems to be a kind of proof by disbelief – the same reasoning that says “The human eye is far too complex to have evolved on its own, just like the wristwatch. Therefore, evolution is not credible.”

  • Correct James, but it does show some reasoning why it seem a tad hard to just accept.

  • Really? So that suggests (1) no heap of children’s skulls when your good chums the Baathists were calling the shots (2) the change of government has produced no benefits at all.

    There were plenty of heaps of children’s skulls under Saddam’s rule – not surprising considering that the sanctions regime helped keep Saddam in power and strengthened his personal control over the economy, and the fact that he was a corrupt dictator. Of course, what Saddam took two and a half decades to accomplish, US imperialism got in 4 years. That’s progress!

    Moreover, it’s funny that your knee-jerk conservativism thinks I’m some sort of Baathist buddy because I oppose an occupation that is the proximate cause of hundreds of thousands of deaths. Those people just don’t mean a damn to you: all that matters is your messianic visions of state power. You get worked up over Tube budgets and banana weights, but when hundreds of thousands of people conflict with your glorious delusions, into the grinder they go.

    Feel free to argue the cost to the USA and UK not being worth it to the taxpayers and/or the volunteer soldiers

    There we go. 600,000 people are dead, and you want to argue tax laws. Hold on, I think someone in the local council took a 500 quid kickback!

    – Josh

  • Incorrect, James.

    The “argument from disbelief” about the evolution of the eye arises from a failure to truly take in how long the timescales involved in evolution are. In other words there is an enormous factor that people who make this argument are unable or unwilling to take in.

    The mental process involved in what you call an “argument from disbelief” in this case are quite different. To compare the contentious claimed results of War X to the uncontentious results of War Y and say that the discrepancy is most likely due to error is a perfectly rational procedure.

  • Alfred E. Neuman

    When the Lancet can provide information as to the location of 600,000+ bodies, this might be remotely credible. You see, 600,000+ bodies is a lot. If you are willing to believe a figure that, from all experience with the Lancet in the past, is politically motivated, you can’t be taken seriously.

  • James

    Yep, show us the bodies. Where they are buried, who did it, where the paperwork is.

    Given there’s allegedly 600k of them, it should be a whole heck of a lot easier than, say, locating WMDs. So it should be no trouble. No trouble at all.

    Any takers on the Left? Didn’t think so.

    It appears it’s ok to claim there are 600,000+ dead in Iraq with not a single body to show for it, but somehow not ok to claim, after 10+ years of denied inspections and gassed towns, that a small number of WMDs existed there at one time.

    The Left. You guys :)

  • I polled four of my mates and my wife as to whether they read the Lancet. None had even heard of it. From this data, i extrapolated using “Lancet methodology” that the circulation of the Lancet is zero, so noone will read the story anyway.

    ps – the error margin of my survey is +/- 5,000,000,000

  • The researcher sounds like a man with a small n.

  • Moreover, it’s funny that your knee-jerk conservativism thinks I’m some sort of Baathist buddy because I oppose an occupation that is the proximate cause of hundreds of thousands of deaths.

    And so by your logic, presumably Nazi Germany should not have been opposed as that was the cause of hundreds of thousands of deaths. People died under the Baathists and then a war happened (and is still happening) that overthrew them, but just because people die, tjhat does not make the war wrong. I would not describe you as a Baathist buddy, more what we used to call back when facing the Soviets a “useful idiot”.

    But you are no doubt just another Rothbardian idiotarian who thought communism was no big deal either and did not need to be contained with military force and peripheral wars.

  • Jim

    Funny, I seem to have missed the bit of your post where you mention the officially Samizdata-approved estimate of deaths in Iraq since the invasion, and the super-duper methodology used to arrive at it. Or were you too busy trying to dig up comparisons to feel incredulous about?

  • There were plenty of heaps of children’s skulls under Saddam’s rule – not surprising considering that the sanctions regime helped keep Saddam in power and strengthened his personal control over the economy, and the fact that he was a corrupt dictator. Of course, what Saddam took two and a half decades to accomplish, US imperialism got in 4 years. That’s progress!

    I’m sorry, I’m having a bit of trouble understanding what your position is. “We shouldn’t have invaded Iraq” seems to be part of it, and the reason seems to have a lot to do with the human suffering as a result. Fine. But here you also seem to be opposed to the prewar sanctions – on much the same grounds? Are you, in fact, in favor of no one having done anything whatever about Saddam – not sanctions not invasion? If so, how do you defend such a position? If not, please explain what it is, exactly, that you are advocating.

  • For every death they documented, they extrapolated 1196 further deaths.

    (By the way, regarding the front page post, “casualties” means “killed and wounded and missing”, not just “killed”.)

  • GF

    As a statistician [don’t all rush at once to hurt me] who used to work in a Public Health department at a well-known British university, I am not surprised at all to see such tosh published by the Lancet. The level of statistical knowledge displayed in medical journals is minimal, and in my opinion is seriously detrimental to our health. I refer you to this article. It might have been written 10 years ago, but very little has changed since then.

    Just don’t ask me what can be done about this, or the fact that so many so-called “peer-reviewed” journals are anything but the rigorous and scientific publications they like to pretend they are.

  • Jim

    “For every death they documented, they extrapolated 1196 further deaths.”

    Don’t ever try and think about what happens when someone runs an opinion poll, then, or your head will explode. Please, try and familiarise yourself with something called ‘statistics’, and then you should start to feel less alarmed by all these strange numbery things.

  • Keith

    Hmmm…a quick googling of Dr. Richard Horton produces some interesting facts–including his involvement in the MMR scandal.

  • Pete_London

    RCD asks:

    And doesn’t it strike you as a remarkable coincidence that the Lancet releases its studies on deaths in Iraq in the month before major US elections?

    The Lancet’s editor gave a speech at an ‘anti-war’ rally in Manchester recently. I wouldn’t call him the most dispassionate of observers when it comes to Iraq.

  • ian

    1. The system used to estimate ‘excess’ deaths is that designed and used by the US government to make estimates in similar events like Rwanda etc
    2. The researchers did not rely on peoples memory but asked for documentary evidence – death certificates etc.
    3. Statistical extrapolation from samples is pretty much a recognised technique these days.
    4. It isn’t a Lancet study, it is a study published in the Lancet – a small but significant difference.
    5. The suggestion that this is a political stunt is simply silly. Publications in the Lancet are peer reviewed like other scientific and academic journals.
    6. If there is a problem it is likely to be with the sampling technique. Criticism to that end would be relevant and appropriate. ‘I don’t believe it’ doesn’t hack it.

  • hovis

    I know this is purely assertion – as I am only wporking from memory – I’ll try and dig out sources later; The Lancet’s reputation for being “Whiter than white” to borrow a phrase, is far from pristine with many examples where ithas been seen to publish bad/incorrect/untrue research and not subsequently admit it was at fault and this is in the medical field not in the political one which this current survey falls into.

  • 1. The system used to estimate ‘excess’ deaths is that designed and used by the US government to make estimates in similar events like Rwanda etc
    2. The researchers did not rely on peoples memory but asked for documentary evidence – death certificates etc.

    Yes, but that leads one to wonder… If documentation is available, why didn’t they just check with the usual offices?
    Most Iraq casualty estimates come from such places – government organs or counts from verifiable reports in the media. Most of these estimates are about 1/10th of what the Johns Hopkins report found. It is unlikely in the extreme that the authorities who issue death certificates lose records of 90% of the certificates they issue.

    3. Statistical extrapolation from samples is pretty much a recognised technique these days.

    This is true. We shouldn’t question the technique as such but, as you point out later, rather the sampling techniques (or else the way they established their baseline – a much more common method of fixing numbers…)

    4. It isn’t a Lancet study, it is a study published in the Lancet – a small but significant difference.

    I don’t think it’s that significant. The Lancet is a peer-reviewed journal. By publishing something they put their stamp of approval on its research methods and at least the validity of its conclusions. Given the inevitable controversy that they must have known would surround this article, the decision to publish it rather does commit the review panel to some measure of complicity. I would also prefer that more people included the fact that the research was done at Johns Hopkins in their reports, but the Lancet isn’t off the hook for that reason.

    5. The suggestion that this is a political stunt is simply silly. Publications in the Lancet are peer reviewed like other scientific and academic journals.

    The suggestion that peer-reviewed publictions are never political stunts is simply silly.

    6. If there is a problem it is likely to be with the sampling technique. Criticism to that end would be relevant and appropriate. ‘I don’t believe it’ doesn’t hack it.

    No, the problem is with the baseline they used. The Lancet uses as its baseline the period from January 2002 to January 2003, a time when Saddam was restrained in normal killings by the no-fly zones and more-intense-than-usual international scruitiny.

    As for “I don’t believe it,” it doesn’t hack it as a complete argument, no, but it does hack it as grounds for suspicion. This is normal in science: when people report things that are at odds with the general literature on a subject, or that seem implausible based on other comparions (such as RC Dean’s comparison with deaths in WWII), it is perfectly scientific to subject that report to more scruitiny than reports that confirm what everyone already believes. The burden of proof is on the people who published the report, not on the reading public, which has come to expect estimates about 1/10th as large as those reported. I would be interested in hearing the report’s explanation (or at least speculation) of how everyone else has gotten much lower figures. Why is their report to be taken as more accurate than these others?

    Another reason to be suspicious of this report, by the way, is that this is the second time it has been rushed to publication at politically convenient moments. I find it implausible that there is nothing manipulative about it.

  • Jim

    “Most Iraq casualty estimates come from such places – government organs or counts from verifiable reports in the media. Most of these estimates are about 1/10th of what the Johns Hopkins report found. It is unlikely in the extreme that the authorities who issue death certificates lose records of 90% of the certificates they issue.”

    Death certificates are issued by hospitals, not by the central government. As the Lancet authors say, the central government doesn’t seem to have recorded more than a third (40,000 a year) of the estimated deaths (120,000 a year) in pre-war years.

    Now, you can argue that the 40,000 recorded by the government (Saddam’s government, I remind you) was the ‘real’ figure, but then that would mean that the increase in mortality since the invasion has been even more massive than the Lancet study estimates. Or you can accept that the 40,000 was a huge under-estimate, in which case you accept that passive recording systems never pick up even most of the deaths in a warzone and that a face to face survey is going to be more accurate.

    “(or else the way they established their baseline – a much more common method of fixing numbers…)”

    Please, I’d like to see your estimate of the ‘real’ baseline, so that you can prove how they’ve ‘fixed’ the numbers. If you believe Saddam’s government figures of 40,000 a year dying in 2002, then Saddam presided over a veritable paradise of good health and long life and the increase in mortality rates since then is even more appalling. If you don’t believe those figures, what exactly are you basing your objection on?

    “The suggestion that peer-reviewed publictions are never political stunts is simply silly.”

    The suggestion that it is a political stunt (and fraudulent for good measure) without having any actual valid methodological objections to make is worse than silly.

    “The Lancet uses as its baseline the period from January 2002 to January 2003, a time when Saddam was restrained in normal killings by the no-fly zones and more-intense-than-usual international scruitiny.”

    What are you actually saying here? That the baseline figure is inaccurate or that it’s accurate but ‘unfairly low’ because Saddam was meanly restrained from killing enough people as he wanted to? Please sort your argument out.

    “I would be interested in hearing the report’s explanation (or at least speculation) of how everyone else has gotten much lower figures.”

    Why don’t you actually read the damn thing, then? There’s even a plain-English version: http://web.mit.edu/CIS/pdf/Human_Cost_of_War.pdf

    They explain in pain-staking detail why passive recording systems get much lower figures, though why so many people find this so hard to believe I really don’t know – unless they’ve got some mental block because they don’t like the conclusions, of course.

  • Death certificates are issued by hospitals, not by the central government. As the Lancet authors say, the central government doesn’t seem to have recorded more than a third (40,000 a year) of the estimated deaths (120,000 a year) in pre-war years.

    That mostly answers my question about that, thanks. However, that doesn’t answer why they couldn’t have gone to the hospitals for information since the hospitals presumably have copies of the relevant data on file.

    Or you can accept that the 40,000 was a huge under-estimate, in which case you accept that passive recording systems never pick up even most of the deaths in a warzone and that a face to face survey is going to be more accurate.

    I don’t believe anything in my post even implied that surveys of this kind are inherently inaccurate. Please note that I said “this is true” and “we shouldn’t question the technique as such” in response to ian’s assertion that extrapolation is a normal technique in statistical surveys.
    The reason that I gave for questioning this particular survey is that it got quite different results from all the others and comes out at a politically convenient time.

    What are you actually saying here? That the baseline figure is inaccurate or that it’s accurate but ‘unfairly low’ because Saddam was meanly restrained from killing enough people as he wanted to? Please sort your argument out.

    I think the point of that was clear – not sure why it needs to be “sorted out.” As a period of comparison for a three-year war period, the Lancet uses a one-year period that polled some of the lowest death-rates in Saddam’s 25-year dictatorship. At the very least, one would expect them to use a similar timeframe, but they could also have included years involving unrest within Iraq under Saddam, which was not infrequent.

    The question was how this survey manages to get results that are a full order of magnitue greater than others. Monekying with the baseline, given what their baseline was, seems a rational place to look. Of course, it’s entirely possible that their results are completely accurate; the point of my post was to respond to people like ian who seem to think everyone should take it on faith that the Lancet’s results are correct simply because they appeared in a peer-reviewed journal. As someone with a lot of experience reading peer-reviewed journals, I can assure you that it is laughable to assert that politics never plays into the selections of articles, or that all articles are subjected to the kind of rigorous scruitiny their publishers claim.

    Why don’t you actually read the damn thing, then? There’s even a plain-English version: http://web.mit.edu/CIS/pdf/Human_Cost_of_War.pdf

    I will do so; thanks for the link.

  • R C Dean

    (By the way, regarding the front page post, “casualties” means “killed and wounded and missing”, not just “killed”.)

    True enough, Mr. den Beste. I was attempting brevity, but not at the cost of accuracy. Language corrected.

  • Who, Me?

    And so by your logic, presumably Nazi Germany should not have been opposed as that was the cause of hundreds of thousands of deaths.

    When Iraq invades France and starts bombing London, feel free to invade. As it was, you defended the war by saying hundreds of thousands were being killed (during Saddam’s 24 years in power), and your ‘cure’ was a typical govt screwup where hundreds of thousands (and counting) have been killed in just a few years.

    Gee, a govt cure turned out to be worse than the disease. Who would have thought that would come as a surprise to the likes of Perry and RC?

  • ian

    Joshua argues above that the problem is with the baseline date used when deaths caused by Saddam were at a low level. If this is the case then that date should surely stand as the closest to ‘normal’ conditions we are likely to get. If a comparison with the worst of Saddam is all the allies can offer that is not a very strong indicator of success is it?

  • When Iraq invades France and starts bombing London, feel free to invade.

    Doubtless a military move against Germany when they reoccupied the Rhineland in violation of treaty which would have certainly caused the Nazi government to collapse and avoided World War II would be out of the question by your “logic”.

    As it was, you defended the war by saying hundreds of thousands were being killed (during Saddam’s 24 years in power

    That was not how I defended the war, it was merely a statement of fact (and that is not what I wrote in any case). Overthrowing tyrannies and defending liberties gets people killed, that is true, but that doesn’t make it a mistake. It does take people made of sterner stuff that you though, laddie.

    Personally my support for the war was on grounds that Saddam was an intolerable threat to regional security and world oil supplies given his propensity to start wars with his neighbours. The lessening of tyranny and introduction of democracy for Iraqis was just an agreeable side effect.

  • Old Jack Tar

    That was quick. Thanks to the editor to correcting the obvious cut-and-paste mess I made of that comment. I am still rather a computer neophyte.

  • Perry E. Metzger

    First of all, there wasn’t one figure given. That’s a media simplification, because the news media is mathematically illterate. The study was statistical, and thus gives only a confidence interval. The 95% confidence interval was from something like 400k to nearly 800k dead. This was not a census, it was a statistical sample. It is remotly possible — that is, 5% possible, presuming the study was not fabricated — that the number is outside that range.

    Second of all, I’m sorry for all those who find this “inherently unbelievable” etc., but the statistical methodology used seems sound. Unless you are going to claim that the findings of statistical theory in recent centuries are a lie (and I doubt most of you are qualified to even discuss statistical methodology), or unless you are willing to claim that the survey conducted was fabricated, I’m afraid you are pretty much constrained to accept the results. Indeed, informally reviewing the paper makes one believe that if there is any systematic bias in the methodology, it would be towards undercount — any family that was completely wiped out, left the country or otherwise rendered uninterviewable could not be put into the survey, and the most seriously dangerous locations could not be surveyed. It does not appear that there was any reasonable methodological failure that could lead to upward bias in the sample. Furthermore, the survey used death certificates to verify survey reports, so, again, bias would be against undocumented deaths, again skewing the results downward, not upward.

    So, the only reasonable way to deny the numbers would be to contend that the survey was a whole cloth fabrication. I do not believe the survey was fabricated — the numbers are actually reasonably consistent with informal estimates based on morgue counts. By the way, this is also still small compared to the casualty levels of the Vietnam war.

    So, maybe some of you who are defenders of this war find these numbers unpleasant and would prefer to find a way to deny them than to believe them. That’s very human, but it is also foolish.

    The most important thing for defenders of reason and sanity to do is not to deny reality when it seems inconvenient and unpleasant. Reality cannot be fooled by fervent hopes or political rhetoric. This is as true for foolish members of the right as for foolish members of the left.

    I will repeat: the statistical methodology here seems sound. It is also consistent with earlier studies and with informal methods. If it is unpleasant to you that it is 95% likely more than 400k additional deaths have been caused in Iraq by the war, well, I don’t know what to tell you about that. However, whether you find it unpleasant or not, it seems exceptionally likely that the numbers are true.

  • Steve E

    Paul Reynolds at the Beeb…

    “The latest figures from the Iraqi health ministry (reported by the Associated Press news agency on 11 October) stated that 2,667 people were killed in Baghdad during September, 400 more than in August.

    This gives an average of about 86 per day in the capital.

    Baghdad is not the whole country of course, but AP reported the United Nations as saying that in July and August, 6,599 people were killed across the country, of which 5,106 were in Baghdad.

    This suggests that Baghdad has by far the highest number of actual and percentage dead.

    So, if the current rate in Baghdad is about 86 and the countrywide figure should be about 500 according to the Lancet report, where are the “missing” dead?

    The answer from the report’s authors would be that the dead are there, but have not been counted.”

    http://news.bbc.co.uk/1/hi/world/middle_east/6045112.stm

  • ian

    This (Link)from Columbia Journalism Review

  • Perry E. Metzger writes, “So, the only reasonable way to deny the numbers would be to contend that the survey was a whole cloth fabrication. ”

    Why are those the only two options to explain the order of magnitude discrepancy between this and other estimates, not to mention the WWII comparisons made by the author of this post? While it could be the case that some or all of the interviewers decided it was all too dangerous or irksome and just made up the results – such things have happened in the history of science – it could also be that a lot of different human factors combined. For example: hope of compensation, desire of interviewees to please the interviewers, desire of the interviewers to please the people who ran the survey, double counting of extended families, desire for sympathy, a sort of misplaced pride in suffering, desire to make the American forces look bad.

    General question: does anyone know how the numbers of people killed by the Soviets or Chinese communists have been estimated?

  • Joshua argues above that the problem is with the baseline date used when deaths caused by Saddam were at a low level. If this is the case then that date should surely stand as the closest to ‘normal’ conditions we are likely to get. If a comparison with the worst of Saddam is all the allies can offer that is not a very strong indicator of success is it?

    No, Joshua argues that if there is a problem with the data it is likely to be in the baseline since that is a common way of exaggerating statistical effects.

    I added that I tend to believe there is probably something wrong with this baseline for a couple of reasons. First, these numbers are much higher than the numbers we’ve come to expect – so it’s natural to suspect that there might be something wrong with the survey. The fact that it appeared in a peer-reviewed journal doesn’t make natural skepticism automatically go away. Now – Jim is right that we should actually read the survey (which I’m currently doing on a lunch break) before drawing definite conclusions about it, but all that is being asserted here is that the numbers seem wrong. And that much is certainly true: the numbers do seem wrong compared to results from other surveys. Second, the timeframe is different. They survey one year to establish their baseline but then do three years post-conflict. This made more sense in 2004 when the war itself was only one year long, but it still didn’t make much sense then either. Generally, if you’re going to have uneven time windows, the longer one should be the baseline (because the baseline is expected to be more general). Now again, nothing conclusive here, but it just seems wrong to use a single year to establish your baseline when you’re doing a comparison over three years of occupation. A more believable baseline would have been the average annual deathrate over the preceding 5 or 10 years. No nation establishes its general deathrate based on extrapolation from a single year, and I do think we have to question why the Johns Hopkins survey thinks it’s OK to do that for Iraq when they presumably could have easily asked for information from those surveyed extending over the last 10 years or whatever. Death certificates do not expire, after all.

  • murph

    The data collected wasn’t deliberately skewed. The entire experiment was deliberately skewed. Here’s how they did it:

    – They took a laughable base-line of 5.5 deaths per 1000 in 2002.

    – They then “discovered”, lo and behold, that the death rate over the past 3 years has been up around 12 per 1000.

    – From this they extrapolated the figure of 655,000.

    To put this in perspective, the death rate in the EU is 10 per 1000, in Hungary is 13 per 1000. In other words, The Lancet is happy for us to believe that the EU’s death rate is almost twice that of a 3rd world country.

    The Lancet are barefaced liars and fraudsters.

  • The reason I mentioned the Soviet comparison was that I understand the Soviets attempted to get as many as possible of those people killed by the Communists “credited” to the Nazis. This would have involved exaggerating their own losses in battle (huge as they genuinely were), and also, presumably, saying that people killed in the late 1930s were killed in the early 1940s. I was thinking that the ways that this was analysed might also be applied to Iraq, and thought it might be the sort of thing Samizdata commenters would know about.

    Murph,
    While I am deeply sceptical about this survey on the grounds that Iraq ought to look like Berlin 1945 and it doesn’t, I don’t think the EU death rate is an argument against it – there are a higher proportion of old people in the EU than in Iraq.

  • murph

    Point taken Natalie. However, I still believe that a death rate of 5.5, when the world average is about 8.5, is ridiculous.

  • Jordan

    Here’s another example of the Lancet’s fine work:

    The medical journal that published a controversial study linking MMR to autism says, with hindsight, it would not have published the paper.

    Richard Horton, editor of the Lancet told the BBC the researchers had a “fatal conflict of interest”.

    And this study was peer-reviewed. Yes left-wingers, peer-reviewed does not mean infallible.

    It gets better. Surely we remember the Lancet’s last feeble attempt at swinging U.S. elections?

    It seems that sometimes a desire to influence or shock public sensibilities seems to get the better of the Lancet from time to time, as it did when it claimed just prior to the 2004 elections that 8,000-194,000 (but most often trumpeted as 100,000) Iraqi civilians had been killed.

    Hell, the study’s author even admitted that they rushed to get this out before the election.

    “But they’re methodology was sound!!!11″ you say. Was it?

    And sorry, but the defense that it’s as soundly designed as can be expected for these kinds of public health surveys is a weak one. Retrospective, interview-based studies like this are poor designs. It may be the standard way of gathering data in the public health field, but that doesn’t make it the best methodology, and it certainly doesn’t make its statistics sound. For too long the field of public health has relied on these types of shotty shoddy numbers to influence public policy, whether it’s the number of people who die from second hand smoke or the number who die from eating the wrong kinds of cooking oils.

    Finally, I’ll leave with an Iraqi’s very appropriate reaction to the study:

    They shamelessly made an auction of our blood, and it didn’t make a difference if the blood was shed by a bomb or a bullet or a heart attack because the bigger the count the more useful it becomes to attack this or that policy in a political race and the more useful it becomes in cheerleading for murderous tyrannical regimes.

  • However, I still believe that a death rate of 5.5, when the world average is about 8.5, is ridiculous.

    The CIA World Factbook has it at 5.8 for 2003, though, which is not that different from the Lancet’s base of 5.5. So maybe we’re wrong about the baseline being off.

    It’s interesting – but the death rates are really low across the Arab countries. Is this because of a smaller percentage of elderly? Subsaharan Africa has the highest deathrates in the world, presumably largely due to the AIDS epidemic.

    There’s a nice graphic on the Wikipedia page based on the CIA World Factbook data.

  • This is very intersting. The World Factbook page on Iraq for 2006 lists 5.37 as the death rate. In other words, the CIA actually has a lower death rate for 2006 than 2003 (5.8). The infant mortality rate for 2006 is way up, though – 48.64 for 2006 and only 33.66 for 2003.

  • Kit Taylor

    I think the timing the report is fair, as the amount of civilian death in a war is a political issue.

    Glenn Greenwald has a good post supporting thestudy.

    (Link)

    He’s a Progressive, but his blog has a strong libertarian bent, especially anti-Patriot Act.

  • Jso

    So when the left is disproven in an argument, their response is to… point to the disproven argument again as “proof” that they aren’t wrong. Wouldn’t 600,000+ deaths be a big news story, before it could be published in a medical journal? And how about all of them being “civilian” deaths, could that even be possible that no soldiers are dying? I don’t think so.

    Glenn Greenwald has a good post supporting thestudy.

    Not a fan of sock puppets, pal.

  • Kim du Toit

    The Lancet is to truth what child molestation is to parenting.

  • Who, Me?

    Doubtless a military move against Germany when they reoccupied the Rhineland in violation of treaty which would have certainly caused the Nazi government to collapse and avoided World War II would be out of the question by your “logic”.

    Then go back in time and handle that tyrant and ignore the case of Iraq where subsequent events have proven you wrong.

    Overthrowing tyrannies and defending liberties gets people killed, that is true, but that doesn’t make it a mistake. It does take people made of sterner stuff that you though, laddie.

    Laddie, we aren’t the ones being killed. You aren’t made of anything stern by blowing off the human costs of your war. You’re a liberal, proving your moral wonderfulness at the expense of others.

  • Paul Marks

    I did not support the choice to go into Iraq (although I did accept there was a possible legal case for it if one held that the 1991 war had never finished – and certainly enemy forces continued to fire at allied aircraft and to pay money to terrorist groups).

    However, the “Lancet” account (as one would expect from an account from any “liberal” elite publication) is bullshit. The medical establishment (like so many cultural institutions) has long ago fallen to people with a political agenda. Nor is just a matter of policy overseas – we are all just to “inequality is a medical issue” (and other such – with demands more statism at the end of the stories).

    The only “data” in a body count are bodies – not what people tell you.

    “My mother died in my arms” is worthless (the mother may be in the next room making tea, “people do not lie about such things” – yes they do). Only the body of the said women is data (and one must take care not to count the same bodies twice or more times).

    However, if 600,000 plus civilians WERE killed how would this be President Bush’s fault? Did he plant the car bombs? Did he sabotage the water and power supplies (not once, but hundreds or thousands of times over the last few years)?

    “Ah but if he had not ordered the U.S. military to fight them, the resistance would not have had to do these things”.

    No one made “the resistance” do these things – they CHOOSE to them. The democratically elected government of Iraq could ask the Americans to leave at any time – is Josh (or anyone else) claiming that President Bush would refuse to pull U.S. forces out? There is no “occupation” (under an occupation the local government does not have the right to tell the occupyer to get out). And if “the resistance” really had the support of the majority of the people in Iraq they would win elections.

    It is like those “liberals” who blame the United States for what the Communists did in Indo China. What Uncle Ho, Pol Pot (and all the rest of them in Vietnam, Laos and Cambodia) did they did of their own free will – to blame the United States for the crimes of the Communists is like blaming the fireman for the fire.

    And in the Republic of Vietnam (and Laos and Cambodia) there was no democratically elected government – unlike Iraq.

  • Paul Marks

    Sorry for the errors – “not is it” not “nor is”, “we are all used to” not “we are all just to” – and so on.

  • Then go back in time and handle that tyrant and ignore the case of Iraq where subsequent events have proven you wrong.

    How has Iraq proven me wrong? Is Iraq going threaten Saudi, attack its neighbours and build nuclear weapons?

    Laddie, we aren’t the ones being killed.

    Please do not include me in your “we”. After 41 years in the Royal Navy, when British fighting men die in far off places, I regard that as “we”.

    You aren’t made of anything stern by blowing off the human costs of your war. You’re a liberal, proving your moral wonderfulness at the expense of others.

    I’m well aware of the human cost of wars but the only use your copy of the Ethics of Liberty might serve in the real world is it might stop a 7.62×39 round if you’re lucky. Wringing your hands because fighting Bad Men is an ugly business will not improve a single person’s life.

  • Ok, I’ve read the full report and I think I can make a guess as to what might have gone wrong with it. The problem isn’t with the baseline, as I thought it might be, but rather with the sampling method (as many others have said).

    They used a “clustering” method – which means that they randomly selected regions in the country in which to survey 40 houses. They claim that “each household had an equal chance of being surveyed,” but that’s not strictly true. Each house had an equal chance of being picked as the “seed” for one of these clusters, but once your house is picked then your 39 nearest neighbors have a chance close to 100% of being picked. They start with one house and then do all the other houses on the same street, etc.

    I think anyone can see what’s wrong with this. The random sampling of houses is a fair way to reproduce the baseline. Whatever neighborhoods you hit that have not experienced heavy fighting will simply reflect the prewar data. However, if you get “lucky” (on the Hopkins’ researchers’ terms) and get a neighborhood that saw fighting, then the casualties will cluster in that neighborhood, obviously. So you get a deathrate much much higher than the national average for that neighborhood. If they had truly chosen each household at random, then this would have come out in the wash, and we would have probably gotten a number close to what the passive studies report (remember, even President Bush admits that the deathrate in Iraq has been higher since the invasion than in the years immediately preceding). But the way they’ve done the study guarantees that casualty reports get amplified. That’s how they came up with a number that’s 10 times greater than what the passive studies found.

    In any case, that the report is a political hatchet job is pretty obvious from its wording. Follow Jim’s link and have a look for yourself: the people who wrote it are very clearly against the war and take every opportunity to illustrate this in the random other irrelevant facts they quote (this is especially obvious near the end).

    One other thing that stands out in the report is that they claim that in 92% of the cases the interviewee could show a death certificate to back up his claim. That may be true…but it flatly contradicts the reasoning they gave for why their report got numbers so much higher than the other “passive” studies. Essentailly – in the opening section, they claim that their methodology is superior because some deaths are not reported, so there is a danger that the authorities undercounted, etc. True, as far as it goes. But if 92% showed death certificates, then clearly these deaths were reported (or else they fabricated the certificates, whatever).

    Needless to say, it is unlikely in the extreme that the Iraqi Ministry of Health simply lost 80-90% of its files but issued mortality rate reports on the basis of the 10% it had left (and in any case, no one has reported any such thing).

    Also, the margin of error that they quote is close to 30%, which is astronomical, to say the least.

    Conclusion: the study is bogus. I will go back to believing in a number of “excess deaths” around 60,000.

  • Paul Marks

    “President Bush admits that the death rate since the invasion has been higher than the rate in the years immediately preceding” – quite correct. However, were not economic santions in the middle of causing an “economic and social breakdown that has already killed lots of people and will kill vast numbers of people”. So the death rate was due to go up anyway.

    I remember being shouted at (just outside the House of Commons) by protestors claiming this, long before the present round of fighting.

    Also (I say again) if “the resistance” sabotage water and power supplies (and so on) again and again, how is this America’s fault? The idea that any sized force can provide security for such vast networks does not seem plausable. Having lots of men walking alongside pipelines sounds like having lots of men walking along streets or standing at street corners – i.e. it provides targets for the enemy.

    American (and British) forces must not be used in a police role (that would lead to vast casualties). Donald Runsfeld was right to resist such an idea as much as he could (he could not resist it as much as he wanted to).

    Allied forces must move fast and be used to strike at enemy forces when found (this is also true in Afghanistan). They must not stand around like scare crows for the enemy to use as target practice.

    If people want a patrol in every street, and a man on every street corner, then this should be done by local forces (not by Allied forces).

    As for “the resistance are only doing this because of the occupation” (as the left say endlessly).

    Remember if the “resistance” really had majority support they would win an election and the Americans (and the few thousand British troops and few hundred other troops) would be gone.

  • Kit Taylor

    As you might expect, Tyler Cowen has a good post, skeptical of the report in the Lancet(Link)

    “A very high deaths total, taken alone, suggests (but does not prove) that the Iraqis were ready to start killing each other in great numbers the minute Saddam went away. The stronger that propensity, the less contingent it was upon the U.S. invasion, and the more likely it would have happened anyway, sooner or later. In that scenario the war greatly accelerated deaths. But short of giving Iraq an eternal dictator, that genie was already in the bottle.”

  • Jim

    Joshua, I appreciate that you’re making a genuine effort to get at the truth, but I think you’re still getting it wrong. You say that “the way they’ve done the study guarantees that casualty reports get amplified”, but this simply isn’t true. Cluster sampling is as likely to pick up areas that were unusually quiet as it is to pick up areas that saw a lot of fighting, so it is just as likely to under-estimate as it is to over-estimate the true figure. Hence the relatively wide confidence interval around the central estimate. If you think the cluster sampling method resulted in an over-estimation there is no reason to think the true number is anything less than the 400,000+ lower bound – anything else and you’re just pulling figures out of the air to suit your own agenda – and the central estimate of ~650,000 is still the most likely. But there is actually nothing wrong with this methodology – it has been used without people complaining about the results in Congo and Darfur (with the results of the survey in Darfur having been quoted by the US government and even on Samizdata posts asking why oh why nobody cares about this incredible death toll) and indeed epidemiologists have lined up to support the methodology used in this report as they did when the first one came out.

    “if 92% showed death certificates, then clearly these deaths were reported ”

    Just because the families of the deceased received death certificates from the hospital does not mean that the hospital then went to the trouble of sending records of all those deaths to the central government which then assiduously compiled them and released them without interference despite having a great political incentive to manipulate the results. But more importantly, if you now accept that the baseline mortality rate of 5.5 per 1,000 is about right, then you have to accept that the central government figures are an under-count, because before the war the government was reporting 40,000 deaths a year from its records rather than the roughly 120,000 needed to get a mortality rate of 5.5. So if the 5.5 figure is about right, which you say it is, then death certificates are not all counted by the government, but the fact that 92% of deaths in the survey were supported by death certificates shows that the surveys count of post-war deaths must have been accurate, which means that the mortality rate must have increased massively between the two periods.

  • Cluster sampling is as likely to pick up areas that were unusually quiet as it is to pick up areas that saw a lot of fighting, so it is just as likely to under-estimate as it is to over-estimate the true figure.

    Well, my reasoning on this was that samples that hit areas with low or no fighting will simply reproduce the prewar deathrate. The baseline was for a year with no fighting – so we’re talking deaths due to old age, accidents and illness – and these will show up just the same in areas that didn’t have fighting, etc. Heavy fighting will produce deaths over and above this baseline. My argument was that sampling in the way that they did would artificially inflate the effect of any fighting — by ensuring that next-door neighbors of houses that got shot up were also included. It has the effect of allowing deaths to concentrate rather than being seen as a percentage of the entire population (as mortality rates are traditionally supposed to be).

    If you think the cluster sampling method resulted in an over-estimation there is no reason to think the true number is anything less than the 400,000+ lower bound

    I disagree. We would need to see a breakdown of their actual numbers by cluster, and they pointedly do not provide this in the study. (They provide a map of the country showing number of excess violent deaths by region – that is not the same thing as actually releasing their cluster data.) But it is not at all unreasonable to expect high-concentration clusters to be able to throw off the general results on the scale I have suggested since the effect is multiplicative. A single cluster shouldn’t do this, of course, but multiple “hits” would.

    A read through Appendix A of the survey is telling on this point. They first concede the point:

    A problem with cluster surveys is that households adjacent to each other are more likely to be similar than those located farther away. In the case of localized violent events, the same event is more likely to affect households close together. This makes simple random sampling a stronger survey method where this is possible. But in war this is seldom possible.

    And then they claim to have compensated for this by surveying twice as many people as a comparable random survey would have done. (This is apparently standard procedure.) But I have three objections here. First, it is not clear to me why they couldn’t have done a random sample. They simply say “in war this is seldom possible,” which is no doubt true, but it is not at all clear that it is impossible in this case. Second, they could easily have done random sampling within their clusters but do not. Instead, they have chosen to actually go house-to-house (this is made clear on p. 3) with their survey. They havent’ even bothered to, say, skip five houses in between to minimize localization effects. They mention in Appendix A that it is common to skip houses to decrease the “design effect” they mention in the quote above, but they give no explanation for why they didn’t do so with their survey. Finally, they go into painstaking detail explaining basic statistics up to the point where they say that they tested for cluster effects and found none. It’s a bit suspicious to me that they feel the need to explain what a “sampling” is but not how they went about verifying that known problems with the method of surveying they did did not occur. Of course, I could figure it out for myself if they had provided a full datasheet, but they did not.

    It is also telling that their justification for the method at the end relies on confirmations that were done exclusively in stable environments. I have no doubt that epidemiologists swear by this method for public health surveys in nations at peace, but that does not make them appropriate for more chaotic situations. What would confirm that these methods are appropriate for war situations would be to compare cluster surveys taken during a war to random sampling surveys done after the war to see how they match. They link to an organization (it is reference 6 at the end) that they claim has validated their methods, but the link tells a completely different story. The website in question says that they are in the process of standardizing methods, and there is no indication that they are validating clustering methods specifically. In fact, all it says about particular methodologies is that it “draws from core elements of several methodologies,” which is not very helpful and certainly doesn’t back up the survey’s claim that its methods have been certified.

    it has been used without people complaining about the results in Congo and Darfur (with the results of the survey in Darfur having been quoted by the US government and even on Samizdata posts asking why oh why nobody cares about this incredible death toll)

    If it helps any, I had not heard of clustering methods before this report and I will certainly be equally wary of them when they are used in Darfur etc. in the future. I have no intention of selectively applying methodological gripes only to surveys whose results I do not like, and I frankly do not appreciate the implicit assumption that I would do so. Again, the whole point of this thread is that the numbers seem wrong. When numbers seem wrong, I think it is fair to say that it is natural to check them. I am checking these numbers because they seem wrong, not because I am trying to help any kind of propaganda campaign, nor, for that matter, are most of the other people on this thread.

    Just because the families of the deceased received death certificates from the hospital does not mean that the hospital then went to the trouble of sending records of all those deaths to the central government which then assiduously compiled them and released them without interference despite having a great political incentive to manipulate the results.

    No one implied that they went to this trouble. Check out the methodology on the other surveys and you’ll find that inspectors went to hospitals and morgues and had a look at their local records for themselves – which is standard practice in these cases, really. It is implausible that 80-90% of the records at hospitals etc. have been lost, and even more implausible that no one would have reported it if they had been.

  • If it helps any, I had not heard of clustering methods before this report and I will certainly be equally wary of them when they are used in Darfur etc. in the future. I have no intention of selectively applying methodological gripes only to surveys whose results I do not like, and I frankly do not appreciate the implicit assumption that I would do so. Again, the whole point of this thread is that the numbers seem wrong. When numbers seem wrong, I think it is fair to say that it is natural to check them. I am checking these numbers because they seem wrong, not because I am trying to help any kind of propaganda campaign, nor, for that matter, are most of the other people on this thread.

    Would like to retract this comment. I have just re-read Jim’s comment and realize that I read it incorrectly – he was not implying anything sinister about anyone’s motives for quoting earlier studies on Darfur.

    I’ll let stand the first bit: I will be equally skeptical of other such surveys in the future since it is fairly clear to me that they have a tendency toward bias.

    I think points I have raised in the comment above also make clear that this particularly study is likely to be deliberately so.

  • Jim

    Joshua, I don’t find your objections persuasive. Firstly, you say that “samples that hit areas with low or no fighting will simply reproduce the prewar deathrate.” Yes, so in a study measuring excess mortality, that would come up as zero. The fact that the number eventually arrived at was 650,000 indicates that in few if any of the randomly selected clusters was there ‘low or no fighting’.

    You say they could easily have done a random sample and you have no idea why this wouldnt’ be possible in a war-zone. But in order to know that your sample is random, you have to know the exact distribution of the population. So for a ‘purely random’ (at the household level) sample, which is what you seem to be demanding, they would have to know the exact location of every household in the population in order to be able to randomly choose who to sample. Clearly, this is not possible. This and the restrictions on movement in war-time Iraq mean that cluster sampling (where clusters are chosen at random within a region but in proportion to the population in each region) becomes the only choice. This is explained in the SMART protocol for “Measuring Mortality, Nutritional Status and Food Security in Crisis Situations(Link)“:

    “Planning the survey
    Sampling
    The sampling method is selected based upon the way in which the households are distributed and the size of the population to be surveyed.
    1. Occasionally with very small populations, every household in the population can be visited (called an exhaustive survey), but this is unusual.
    2. Where the houses of the whole population of interest are arranged in a systematic way (such as in some refugee camps), simple or systematic random sampling is used for the entire sampling process.
    3. Cluster sampling is used when households are distributed in an unstructured way that does not easily allow all the households to be listed or numbered.

    The most usual method is cluster sampling.”

    Secondly, you say they should have chosen randomly within the cluster. But by definition households clustered together are not randomly related. So surveyors take a bigger sample and test for clustering effects. This they did, and found that the effect of clustering was compensated for by the expanded sample size.

    “They link to an organization (it is reference 6 at the end) that they claim has validated their methods, but the link tells a completely different story.”

    I agree that the specific link they give isn’t helpful. But if you look around the rest of that site you will find numerous mentions of cluster sampling as a valid method, including the protocol I mentioned above which says, to repeat, “The most usual method is cluster sampling”. This method is standard, and tried and tested in crisis situations.

    As you say, “Again, the whole point of this thread is that the numbers seem wrong”. But I honestly don’t know what you or anyone else on this thread who says this is basing it on, because unlike the researchers you have not been around Iraq trying to find out. At best you see some of the media reports coming from the handful of Western media who are almost to a man holed up in one part of one city in Iraq. Our media do not even pretend to be telling us what is going on around Iraq, so I simply do not buy the argument that “these numbers seem wrong”. These numbers are evidence, and it is now your hunch that should “seem wrong”.

    “It is implausible that 80-90% of the records at hospitals etc. have been lost, and even more implausible that no one would have reported it if they had been.”

    I don’t see your point here. Nobody is claiming that 80-90% of hospital records have been lost, just that the central government never has been and certainly isn’t now very good at totting up accurate figures for deaths.

  • Firstly, you say that “samples that hit areas with low or no fighting will simply reproduce the prewar deathrate.” Yes, so in a study measuring excess mortality, that would come up as zero.

    Please correct me if I am wrong. They go to various clusters of houses and ask people in a row of 40 how many died in years w, x, y and z. They tally up the number of people in all the houses in this cluster and establish a deathrate as a percentage. They then average these rates over all 50 (actually, 47 since three got thrown out) clusters. This establishes an overall deathrate for the nation, which they multiply by the estimated population to get the total number of people killed. They then subtract off the number that the baseline would have given them, and this gives the “excess deaths.” Clearly, if you are estimating a death rate over an entire population and you hit clusters where the death rate is either (a) at or near baseline or (b) far in excess of baseline (because rather than doing a random sample of the population you are sampling areas that have similar characteristics some of which are likely to have been hit much harder than the general population) your average is in danger of being artificially higher than it would be with a random sampling. This is because few clusters are likely to actually represent the overall national deathrate and those that overestimate it will overestimate it considerably. Nothing that I am claiming here is controversial: this problem is acknowledged in the report with no explanation of how they know they fixed it (we are expected to take their word – they give us no corroborating data). It is also a generally acknowledged problem with clustering surveys, in fact.

    In the link that you send, if you scroll down to page 40 (below where you linked) there is more information on cluster sampling. It differs markedly from the method used in the Johns Hopkins survey. In the SMART protocol, individuals are selected at random from within the clusters. That is what did not happen in the Johns Hopkins survey. Out of clusters of possibly tens of thousands of households each, they have chosen 40 houses along a row. Cluster sampling in general may give an approximation to random sampling, but this kind of cluster sampling clearly is much less likely to than the kind of cluster sampling recommended in the protocol you linked.

    This method is standard, and tried and tested in crisis situations.

    But SMART prefers not to use it if possible, however, because of the inherent biases noted above. Random samples are considered superior. Nor is it clear that cluster surveys have been “tried and tested in crisis situations.” They are used in crisis situations, but I have the impression that we are waiting for definitive studies on to what extent they get the same results as more traditional methods. In stable situations with uniform distribution of what is being studied, they do. However, they seem inappropriate for situations where the effect being studied is distributed unevently over the population (as in the case of post-invasion death in Iraq), which is no doubt why they are viewed with caution and no doubt why the authors of the Johns Hopkins survey feel the need to defend themselves on their choice of method.

    The link in [6], by the way, was intended to address problems of cluster sampling in conflict situations. From p. A2:

    Validation of cluster sampling methodologies as an appropriate alternative to simple random sampling is difficult in conflict situations. … The [SMART], a collaborative network through USAID seeking to standardize and evaluate methodologies among humanitarian organizations, has established cluster sampling as an acceptable method of sampling in conflict.

    I did not, however, see any evidence that they had done so on their webpages. Rather, what they say is that cluster sampling is often used because random sampling is logistically difficult. This is not the same as having certified it as equally accurate or a valid alternative, which the Johns Hopkins link implies it establishes. The method for cluster sampling that they give in their protocol is in any case different from the method used by the Johns Hopkins survey. SMART’s protocol would avoid all of the problems I have raised with the Johns Hopkins survey.

    But I honestly don’t know what you or anyone else on this thread who says this is basing it on, because unlike the researchers you have not been around Iraq trying to find out.

    No, but we have read reports from other researchers who have been around Iraq trying to find out that come to very different conclusions. (Admittedly with passive methods, which should have underestimated the deathtoll a bit – but it still seems unlikely that the estimation would be off by 550,000.) We also have a comparison to other conflicts. It seems unlikely, for example, that Japan lost roughly the same percentage of its population (3.6% for Japan, 3.1% for Iraq if this survey is correct) during WWII. Surely, Japan would have lost much, much more after firebombings, the decimation of its army and navy and two atomic bombs. Now, I agree that such comparisons are far from conclusive, and I never said until reading the survey that I outright thought the survey was wrong. But these comparisons make it seem wrong, and so I was skeptical. Now that I have seen their methods, I am pretty certain that it’s a sampling problem and not a real reflection of the carnage in Iraq.

    don’t see your point here. Nobody is claiming that 80-90% of hospital records have been lost, just that the central government never has been and certainly isn’t now very good at totting up accurate figures for deaths.

    Look, if the survey is indeed accurate, and 92% of those surveyed also have death certificates to back up their claims, then the death certificates on file at hospitals and morgues should also fairly accurately reflect the actual death rate in the country. The Johns Hopkins survey says outright that population surveys are superior becuase they count, in addition to reported deaths, also unreported deaths. But clearly, the rate of “unreported” deaths can’t be higher than about 8% if these numbers are accurate (because 92% have death certificates, which means they reported the deaths). The passive surveys (which, by the way, were not done exclusively by the Iraqi and US governments, but also by the UN and several NGOs) get death tolls around 1/10th what the Johns Hopkins people found. If we take the Johns Hopkins survey seriously, this implies that the hospitals and morgues have lost about 80% of their records (because 8% of deaths are unreported, and the hospitals’ records show a death rate roughly 10% of that the JH survey found).
    That seems extremely unlikely.