Return of Too Many Daves
11-22-2007, 02:39 PM
In Blighty they are looking at new scientific evidence to see how it might effect previous legislature on abortion. Clearly a good thing to do. This article makes it clear how difficult it is to do so though, it that old tale of lies, damn lies, and statistics.
Some numbers in abortion debate just can't be relied on
Ben Goldacre
The Guardian
Saturday October 27 2007
Where do all those numbers in the newspapers come from? Here's a funny thing. The Commons committee on science and technology is taking evidence on "scientific developments relating to the Abortion Act 1967".
Scientific and medical expert bodies giving evidence say that survival in births below 24 weeks has not significantly improved since the 1990s, when it was only 10-20%. But one expert, a professor of neonatal medicine, says survival at 22 and 23 weeks has improved. In fact, he says survival rates in this group can be phenomenally high: 42% of children born at 23 weeks at some top specialist centres. He is quoted widely: the Independent, Telegraph, on Newsnight, by Tory MPs, and so on. The figure has a life of its own.
In the media, you get one expert saying one thing, and another saying something else. Who do you believe? The devil is in the detail. One option is to examine the messenger. John Wyatt is a member of the Christian Medical Fellowship. He didn't declare that when he went to give evidence. You don't have to. He did declare it when asked.
Prof Wyatt has relevant research experience but there were half a dozen other medics without any relevant background who submitted evidence (or their view of it) to the committee who, when asked if they had anything to declare, did mention membership of Christian or evangelical groups with an established position on abortion. I don't care for an argument that rests on competing ideologies, so let's look at Prof Wyatt's evidence, because it has been hugely reported and it goes against the evidence from a huge study called Epicure. Epicure contains all of the data for every premature birth in 1995 and 2006. It shows a modest improvement in survival for births at 24 weeks, but no significant improvement in the 10-20% rate for births at 22 and 23 weeks.
For the next bit, you need to understand one simple piece of primary school maths, which is central to medical statistics. In the sum 3/20, 3 is the numerator and 20 is the denominator: 15% survive; 3 out of 20. For Epicure, the numerator is survival to discharge from hospital, and the denominator is all births where there is a sign of life, carefully defined: 3 out of 20.
There are two ways you could get a higher survival percentage. One would be a genuine increase in the number of babies surviving, an increase in the numerator: 8 out of 20 live births survive, 40%. But you could also see an increase in the survival percentage by changing the denominator. Let's say, instead of counting as your denominator "all births where there is a sign of life in the delivery room" you counted "all babies admitted to neonatal intensive care". Now that's a different kettle of fish altogether. To be admitted to neonatal ICU, the doctors have to think you've got a chance. Often you have to be transferred from another hospital, in an ambulance, and for that you really do have to be more well. Therefore, if your denominator is "neonatal ICU admissions", your survival rates will be higher, but you are not comparing like with like. That may partly explain Prof Wyatt's figure for a very high survival rate in 23-week babies. But it's not clear.
First in his written evidence he said that the data was from a "prospectively defined" study (where they say in advance what they plan to collect). Then he was asked in the committee, when giving his oral evidence: "What was the denominator for that? Was that ... 42% survival at 23 weeks of all babies showing signs of life in the delivery room, or was it a proportion of those admitted to neonatal intensive care directly or by transfer?" Prof Wyatt replied: "The denominator was all babies born alive in the labour ward in the hospital at UCL [University College London]." This, as later became clear, turned out not to be true. Then he was asked to send the reference for the claim. He did so. It was merely an abstract for an academic conference presentation three years ago. It did not contain the figures he was quoting. He then says he has done the raw figures on a spreadsheet, especially for the committee, bespoke, if you will, and sent them in. They are entered into the record as a memo, on October 18. They show new, different, although broadly comparable figures: 50% survive at 22 weeks, then down to 46% at 23 weeks, then up to 82% at 24 weeks, then down again to 77% at 25 weeks. (That bouncing around is because the raw numbers are so small that there is a lot of random noise).
And the denominator? Prof Wyatt is clear: "I have provided the numbers and percentage of infants born alive at University College London Hospitals who survived to one year of age." The committee asked for clarification of this. Finally, October 23, another memo arrived, from Prof Wyatt, entered into the record, for all to see. For the widely quoted 42% survival rate at 23 weeks, Prof Wyatt admitted that the denominator was all babies admitted to the neonatal intensive care unit. But in his new special analysis, giving this new "46% survive at 23 weeks" figure, the figures in the previous paragraph, he claimed, the denominator was "all live births". Has he undone a prospectively designed study, and retrospectively re-designed it? Or is this now a completely different source of data to the original reference?
I don't blame Prof Wyatt, but the figure has taken on a life of its own. There may have been yet another mistake here, about the denominator. I don't know. I'm quite prepared to believe that UCL may have unusually good results. But science is about clarity and transparency, especially for public policy. You need to be very clear on things like: what do you define as a "live birth", how do you decide on what gestational age was, and so on. Even if this data stands up eventually, right now it is non-peer reviewed, non-published, utterly chaotic, personal communication of data, from 1996 to 2000, with no clear source, and with no information about how it was collected or analysed. That would be fine if it hadn't suddenly become central to the debate on abortion.
Some numbers in abortion debate just can't be relied on
Ben Goldacre
The Guardian
Saturday October 27 2007
Where do all those numbers in the newspapers come from? Here's a funny thing. The Commons committee on science and technology is taking evidence on "scientific developments relating to the Abortion Act 1967".
Scientific and medical expert bodies giving evidence say that survival in births below 24 weeks has not significantly improved since the 1990s, when it was only 10-20%. But one expert, a professor of neonatal medicine, says survival at 22 and 23 weeks has improved. In fact, he says survival rates in this group can be phenomenally high: 42% of children born at 23 weeks at some top specialist centres. He is quoted widely: the Independent, Telegraph, on Newsnight, by Tory MPs, and so on. The figure has a life of its own.
In the media, you get one expert saying one thing, and another saying something else. Who do you believe? The devil is in the detail. One option is to examine the messenger. John Wyatt is a member of the Christian Medical Fellowship. He didn't declare that when he went to give evidence. You don't have to. He did declare it when asked.
Prof Wyatt has relevant research experience but there were half a dozen other medics without any relevant background who submitted evidence (or their view of it) to the committee who, when asked if they had anything to declare, did mention membership of Christian or evangelical groups with an established position on abortion. I don't care for an argument that rests on competing ideologies, so let's look at Prof Wyatt's evidence, because it has been hugely reported and it goes against the evidence from a huge study called Epicure. Epicure contains all of the data for every premature birth in 1995 and 2006. It shows a modest improvement in survival for births at 24 weeks, but no significant improvement in the 10-20% rate for births at 22 and 23 weeks.
For the next bit, you need to understand one simple piece of primary school maths, which is central to medical statistics. In the sum 3/20, 3 is the numerator and 20 is the denominator: 15% survive; 3 out of 20. For Epicure, the numerator is survival to discharge from hospital, and the denominator is all births where there is a sign of life, carefully defined: 3 out of 20.
There are two ways you could get a higher survival percentage. One would be a genuine increase in the number of babies surviving, an increase in the numerator: 8 out of 20 live births survive, 40%. But you could also see an increase in the survival percentage by changing the denominator. Let's say, instead of counting as your denominator "all births where there is a sign of life in the delivery room" you counted "all babies admitted to neonatal intensive care". Now that's a different kettle of fish altogether. To be admitted to neonatal ICU, the doctors have to think you've got a chance. Often you have to be transferred from another hospital, in an ambulance, and for that you really do have to be more well. Therefore, if your denominator is "neonatal ICU admissions", your survival rates will be higher, but you are not comparing like with like. That may partly explain Prof Wyatt's figure for a very high survival rate in 23-week babies. But it's not clear.
First in his written evidence he said that the data was from a "prospectively defined" study (where they say in advance what they plan to collect). Then he was asked in the committee, when giving his oral evidence: "What was the denominator for that? Was that ... 42% survival at 23 weeks of all babies showing signs of life in the delivery room, or was it a proportion of those admitted to neonatal intensive care directly or by transfer?" Prof Wyatt replied: "The denominator was all babies born alive in the labour ward in the hospital at UCL [University College London]." This, as later became clear, turned out not to be true. Then he was asked to send the reference for the claim. He did so. It was merely an abstract for an academic conference presentation three years ago. It did not contain the figures he was quoting. He then says he has done the raw figures on a spreadsheet, especially for the committee, bespoke, if you will, and sent them in. They are entered into the record as a memo, on October 18. They show new, different, although broadly comparable figures: 50% survive at 22 weeks, then down to 46% at 23 weeks, then up to 82% at 24 weeks, then down again to 77% at 25 weeks. (That bouncing around is because the raw numbers are so small that there is a lot of random noise).
And the denominator? Prof Wyatt is clear: "I have provided the numbers and percentage of infants born alive at University College London Hospitals who survived to one year of age." The committee asked for clarification of this. Finally, October 23, another memo arrived, from Prof Wyatt, entered into the record, for all to see. For the widely quoted 42% survival rate at 23 weeks, Prof Wyatt admitted that the denominator was all babies admitted to the neonatal intensive care unit. But in his new special analysis, giving this new "46% survive at 23 weeks" figure, the figures in the previous paragraph, he claimed, the denominator was "all live births". Has he undone a prospectively designed study, and retrospectively re-designed it? Or is this now a completely different source of data to the original reference?
I don't blame Prof Wyatt, but the figure has taken on a life of its own. There may have been yet another mistake here, about the denominator. I don't know. I'm quite prepared to believe that UCL may have unusually good results. But science is about clarity and transparency, especially for public policy. You need to be very clear on things like: what do you define as a "live birth", how do you decide on what gestational age was, and so on. Even if this data stands up eventually, right now it is non-peer reviewed, non-published, utterly chaotic, personal communication of data, from 1996 to 2000, with no clear source, and with no information about how it was collected or analysed. That would be fine if it hadn't suddenly become central to the debate on abortion.