Sally Clark's first son died suddenly within a few weeks of his birth in September 1996, and, in December 1998, her second died in a similar manner. A month later, she was arrested and subsequently tried for the murder of both children. Clark was found guilty of the murder of two of her sons in November, 1999. The convictions were upheld at appeal in October 2000, but overturned in a second appeal in January 2003, after it emerged that Dr. Alan Williams, the prosecution forensic pathologist who examined both of her babies, had incompetently failed to disclose microbiological reports that suggested the second of her sons had died of natural causes. The conviction was overturned and she was freed from prison in 2003. The experience caused her to develop serious psychiatric problems and she died in her home in March 2007 from alcohol poisoning.
Another element--even more interesting--was the case relied on significantly flawed statistical evidence presented by pediatrician Professor Sir Roy Meadow, who testified that the chance of two children from an affluent family--Clark was a solicitor--suffering sudden infant death syndrome was 1 in 73 million.
Most of the subsequent self examination by the courts centered on Professor Meadow’s use of statistics. When he told the jury that the odds against two of Sally Clark’s children dying of unidentified natural causes (or Sudden Infant Death Syndrome) were 73 million to one, he was leaving the area of his expertise – pediatric medicine – and entering a field in whose basic principles he was completely unschooled – statistical probability. He did so without alerting the jury or the court to the fact he was no longer addressing them as an expert and thus invested a statistical fallacy with all the authority his medical status endued him with.
In giving this part of his evidence he relied upon the draft version of the latest CESDI report ( Confidential Enquiry into Stillbirths and Deaths in Infancy). This report, whose principal author was Professor Fleming, Professor of Infant Health and Development Physiology at the Institute of Child Health at Bristol University, was commissioned by the Department of Health. It was a study of factors contributing to sudden and unexpected deaths in infancy. In relaying some of the statistics from this report to the jury, Professor Meadow misconstrued them, committing three elementary statistical errors simultaneously. So serious were these errors that they moved Dr. Stephen Watkins, a trained epidemiologist with an expertise in statistics, to compose an editorial which was published in the British Medical Journal under the title ‘Conviction by Mathematical Error’. In the editorial he did not argue that Meadow’s abuse of statistics had led to the conviction but rather that the prominence of misleading statistics in Meadow’s evidence meant that these could have overshadowed the other evidence. He suggested that this necessarily rendered the conviction unsafe.
The draft version of the paper which Meadow used as his source indicated that there was one SIDS death for every 1,300 deaths. It also identified three important factors which, if present in a particular family, were capable of increasing the risk of such a death, namely the presence of a smoker, the absence of any wage-earner and the fact that the mother was below the age of 26. In families in which none of these factors applied it was said that the odds against one unexplained infant death occurring by natural causes were much higher than indicated by the general figure; instead of 1,300 to 1 they were 8543 to 1. In families in which all three risk factors were present, they were significantly lower, namely 214 to 1.
The authors of the paper made clear, however, that although the three factors they had singled out were perhaps the most important, there were also other factors which would affect the probability of a SIDS death occurring.
Meadow’s first statistical error was that he ignored this caveat. Although he was now moving from abstract generalization to concrete particular, where specific risk factors might have been identified, he assumed without inquiry that the Clark family should automatically be assigned to the lowest category of risk. Since the risk factor was subject to a large variation – from 214 to 1 to 8543 to 1, this was potentially significant.
Whatever error Meadow had introduced in this manner though, was now massively compounded by his next step. Here, in fairness to Meadow, it must be said that the paper he was relying on was itself presented in such a way that the unwary and uninformed reader might well misconstrue it.
The CESDI report contained the following passage:
Yet these two errors were relatively minor in comparison to the third statistical fallacy on which Meadow’s evidence rested. This was what Watkins has called ‘the lottery effect’ (also known as ‘the prosecutor’s fallacy’) which consists in calculating the ‘chance of a specific individual, identified in advance, having a particular experience and [using] that as the measure for the chance of some individual, out of all those at risk, suffering it.’
This fallacy is perhaps best illustrated by considering a hypothetical weekly lottery in which every member of the UK population of 60 million purchases a ticket bearing a different combination of numbers. When one particular person – let us call her Mrs. Smith – wins this lottery, the odds against her doing so in that particular week were, before the event, 60 million to one. But once Mrs. Smith has produced her ticket to show that she has won, the authorities do not invoke these odds in order to suggest that she has committed an act of fraud. Because in reality, although the odds against any one specified person winning this prize are very high, the odds against an unspecified person winning are not high at all. Indeed, in our hypothetical lottery it is clear that one person wins the lottery every week.
As the Royal Society of Statistics indicated, in a press release issued after the first trial of Sally Clark, the question which Meadow should have addressed was not the one concerned solely with the chances of two natural deaths occurring in the same family:
‘The jury needs to weigh up two competing explanations for the babies’ deaths: Sudden Infant Death Syndrome [SIDS] or murder. Two deaths by SIDS or two murders are each quite unlikely, but one has apparently happened in this case. What matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are under one explanation.’
In other words if, for the sake of argument, the chances of two unexplained natural infant deaths in a family were 1 in 500,000 and the chances of a double murder were also 1 in 500,000, then, if all other evidence was equal or equivocal, there would be a 1 in 2 chance that Sally Clark’s children died of natural causes. There is a big difference between this and the 1 in 73 million chance which Professor Meadow, in his guise as an expert, had confidently put before the jury, comparing these odds to the chance of successfully backing four different 80-1 outsiders to win the Grand National in four successive years.
The phrase "guise as an expert" is well worth remembering.
(From Richard Webster: Conviction by mathematical error?)
Another element--even more interesting--was the case relied on significantly flawed statistical evidence presented by pediatrician Professor Sir Roy Meadow, who testified that the chance of two children from an affluent family--Clark was a solicitor--suffering sudden infant death syndrome was 1 in 73 million.
Most of the subsequent self examination by the courts centered on Professor Meadow’s use of statistics. When he told the jury that the odds against two of Sally Clark’s children dying of unidentified natural causes (or Sudden Infant Death Syndrome) were 73 million to one, he was leaving the area of his expertise – pediatric medicine – and entering a field in whose basic principles he was completely unschooled – statistical probability. He did so without alerting the jury or the court to the fact he was no longer addressing them as an expert and thus invested a statistical fallacy with all the authority his medical status endued him with.
In giving this part of his evidence he relied upon the draft version of the latest CESDI report ( Confidential Enquiry into Stillbirths and Deaths in Infancy). This report, whose principal author was Professor Fleming, Professor of Infant Health and Development Physiology at the Institute of Child Health at Bristol University, was commissioned by the Department of Health. It was a study of factors contributing to sudden and unexpected deaths in infancy. In relaying some of the statistics from this report to the jury, Professor Meadow misconstrued them, committing three elementary statistical errors simultaneously. So serious were these errors that they moved Dr. Stephen Watkins, a trained epidemiologist with an expertise in statistics, to compose an editorial which was published in the British Medical Journal under the title ‘Conviction by Mathematical Error’. In the editorial he did not argue that Meadow’s abuse of statistics had led to the conviction but rather that the prominence of misleading statistics in Meadow’s evidence meant that these could have overshadowed the other evidence. He suggested that this necessarily rendered the conviction unsafe.
The draft version of the paper which Meadow used as his source indicated that there was one SIDS death for every 1,300 deaths. It also identified three important factors which, if present in a particular family, were capable of increasing the risk of such a death, namely the presence of a smoker, the absence of any wage-earner and the fact that the mother was below the age of 26. In families in which none of these factors applied it was said that the odds against one unexplained infant death occurring by natural causes were much higher than indicated by the general figure; instead of 1,300 to 1 they were 8543 to 1. In families in which all three risk factors were present, they were significantly lower, namely 214 to 1.
The authors of the paper made clear, however, that although the three factors they had singled out were perhaps the most important, there were also other factors which would affect the probability of a SIDS death occurring.
Meadow’s first statistical error was that he ignored this caveat. Although he was now moving from abstract generalization to concrete particular, where specific risk factors might have been identified, he assumed without inquiry that the Clark family should automatically be assigned to the lowest category of risk. Since the risk factor was subject to a large variation – from 214 to 1 to 8543 to 1, this was potentially significant.
Whatever error Meadow had introduced in this manner though, was now massively compounded by his next step. Here, in fairness to Meadow, it must be said that the paper he was relying on was itself presented in such a way that the unwary and uninformed reader might well misconstrue it.
The CESDI report contained the following passage:
For a family with none of these three factors, the risk of two infants dying as SIDS by chance alone will thus be 1 in (8,543 x 8,543) i.e. approximately 1 in 73m. For a family with all three factors the risk will be 1 in (214 x 214) i.e. approximately 1 in 46,000. Thus, for families with several known risk factors for SIDS, a second SIDS death, whilst uncommon, is 1,600 times more likely than for families with no such factors. Where additional adverse factors are present, the recurrence risk would correspondingly be greater still.The crucial words here are ‘by chance alone’. What these words were intended to convey was that the calculation which arrived at the figure of 1 in 73 million was an abstract mathematical exercise which could not properly be applied to any instance of recurrent infant death in a particular family. For this method of squaring the original odds could only be used legitimately if it could be shown that the causes of the two deaths were entirely independent of one another – a negative which in practice could never be established with complete certainty. Since the two children whose deaths had given rise to the allegations against Sally Clark were siblings who shared the same genetic inheritance and the same environment, there was no reason to make such an assumption. Indeed a particular set of genetic or environmental circumstances might actually mean that the death of a first child was predictive of a second death.
… When a second SIDS death occurs in the same family, in addition to careful search for inherited disorder there must always be a very thorough investigation of the circumstances – though it would be inappropriate to assume maltreatment was always the cause (quoted in General Medical Council v Meadow, 2006, paragraph 137 with italics added) .
Yet these two errors were relatively minor in comparison to the third statistical fallacy on which Meadow’s evidence rested. This was what Watkins has called ‘the lottery effect’ (also known as ‘the prosecutor’s fallacy’) which consists in calculating the ‘chance of a specific individual, identified in advance, having a particular experience and [using] that as the measure for the chance of some individual, out of all those at risk, suffering it.’
This fallacy is perhaps best illustrated by considering a hypothetical weekly lottery in which every member of the UK population of 60 million purchases a ticket bearing a different combination of numbers. When one particular person – let us call her Mrs. Smith – wins this lottery, the odds against her doing so in that particular week were, before the event, 60 million to one. But once Mrs. Smith has produced her ticket to show that she has won, the authorities do not invoke these odds in order to suggest that she has committed an act of fraud. Because in reality, although the odds against any one specified person winning this prize are very high, the odds against an unspecified person winning are not high at all. Indeed, in our hypothetical lottery it is clear that one person wins the lottery every week.
As the Royal Society of Statistics indicated, in a press release issued after the first trial of Sally Clark, the question which Meadow should have addressed was not the one concerned solely with the chances of two natural deaths occurring in the same family:
‘The jury needs to weigh up two competing explanations for the babies’ deaths: Sudden Infant Death Syndrome [SIDS] or murder. Two deaths by SIDS or two murders are each quite unlikely, but one has apparently happened in this case. What matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are under one explanation.’
In other words if, for the sake of argument, the chances of two unexplained natural infant deaths in a family were 1 in 500,000 and the chances of a double murder were also 1 in 500,000, then, if all other evidence was equal or equivocal, there would be a 1 in 2 chance that Sally Clark’s children died of natural causes. There is a big difference between this and the 1 in 73 million chance which Professor Meadow, in his guise as an expert, had confidently put before the jury, comparing these odds to the chance of successfully backing four different 80-1 outsiders to win the Grand National in four successive years.
The phrase "guise as an expert" is well worth remembering.
(From Richard Webster: Conviction by mathematical error?)
No comments:
Post a Comment