Who Says Numbers Don’t Lie?

On occasion, we here at Abnormal Use write about evidentiary issues, usually pertaining to the intersection of law and science. It is at this intersection that we find conflict, sometimes severe, regarding the standards for admissibility of expert testimony. This area of the law is usually a fertile ground for discussion, and the facts are generally pretty intriguing. And so it goes with today’s post. Let me preach on it.

Recently, friend of the blog Steve Mirsky (of Scientific American fame) alerted us a piece in The Guardian reporting on a court that had refused to allow an expert to testify in the field of mathematics. This piqued our interest, so we decided to look into the matter a little further.

The specific case that Mirsky was referring to was a murder case from England. The prosecution’s theory was that the crime was committed by a person wearing a specific type of athletic shoe whose sole had been worn down in a particular way, leaving a particular type of tread pattern behind. It was alleged that the defendant, coincidentally, had shoes whose sole matched the particular tread pattern. We understand from our friends who practice criminal defense that the legal term for such information is “a really bad fact.”

The prosecution wanted to take the really bad fact a step further. They wanted to call a statistician who would apply a mathematical proposition known as the Bayes’ theorem to the facts of the case. Basically, the Bayes’ theorem is a mathematical expression of common sense. It addresses the probability that a given event could occur given the concurrence of certain circumstantial facts. For example, let’s say that a hit-and-run fatality occurred in South Carolina and all that was known about the suspect vehicle is that it was a gray sports car. As a matter of mathematical theory, it is possible to determine the probability that the suspect car was – say – a Maserati (or any other type of car). Naturally, the more specific factual input that is provided, the less probable it is that an event consisting of all that input could occur.

And that’s probably why the English judge had such a problem with admitting expert testimony about Bayes’ theorem with regard to a criminal case. From the judge’s perspective, the statistician would testify about how improbable it would be that a defendant would have just the right pair of shoes, with just the right sole pattern, as compared to police observations of the murder’s physical evidence. Our common sense would tell us that it’s just too much of a coincidence that the defendant would have so much unusual information in common with the suspect. And that therefore, the defendant must be guilty.

But not so fast. Before we throw the book at someone based on coincidence, perhaps we should ask how many people in the relevant sample (here, England) had the same type of shoes as the suspect? If the answer to that question were 1, then the correspondence of circumstantial evidence about the defendant as compared with the suspect would become more than just a convenient coincidence. However, the further the number moves away from 1, the less relevant the statistical evidence would seem to become. In the case at hand, the number seemed to be in the order of several hundred thousand pairs of the same types of shoes had been sold around England over 10 years. Consequently, it is possible that there could have been tens, hundreds, maybe thousands of the exact same types of shoes with consistent wear patterns.

Let’s be honest about ourselves for a moment. Human nature is susceptible to drawing firm conclusions about truth and innocence based upon the coincidence of circumstances that we believe to be highly improbable based on our common sense. This is especially true when what we believe to be common sense is backed up by “science.” When it comes to statistical evidence, without knowing the relevant sample size, we really can’t evaluate in a reliable fashion how probable or improbable the circumstances are of a given event. And without having that confidence in the statistical testimony to be offered, believing that it is likely to lead to unfair extrapolations of truth and innocence, it is better to simply exclude the testimony altogether. And while we’re being real, if a criminal prosecution comes down to specious statistical evidence, the case was never that strong to begin with.

The analysis of this case reminds us of the famous quip: “There are three kinds of lies: lies; damned lies; and statistics.” Criminal convictions must be based on sterner stuff. There may be room for statistical evidence, and there often is. But statistical evidence must have the same indicia of reliability as other types of expert testimony; otherwise, there is a 100 percent chance that it must be excluded.