Judge cracks down on Bayesian stats dodginess in court

Bayes and the law

This is an interesting article but it is important not to mix up the use of Bayes theorem with poor use of probability. The DNA 1 in a million example cited is simply a classic case of the prosecutor fallacy where the 1 in a million probability of seeing the DNA match evidence in an innocent person is wrongly asumed to the same as the probability of innocence. Bayes theorem actually helps avoid this kind of probabilistic fallacy. The problem with the RvT ruling is that it will have the impact of giving jurors less information than is actually avialable. So, instead of giving some useful probability information about the likelihood of a random match experts will be reduced to vague statements like 'a random match is possible' or 'is unlikely'.

I was interviewed for an article in the Guardian about this case, see:

www.guardian.co.uk/law/2011/oct/02/formula-justice-bayes-theorem-miscarriage

Here are some links to more detailed information about the issues raised:

About the RvT ruling and the probabilistic issues raised:

www.eecs.qmul.ac.uk/~norman/papers/likelihood_ratio.pdf

The draft proposal for a project to improve the current state of practice:

https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxiYXllc2xlZ2FsfGd4OjY4MzljYzNiYjNhYjI5MDA

A report describing common legal fallacies involving probability:

/www.eecs.qmul.ac.uk/~norman/papers/fenton_neil_prob_fallacies_June2011web.pdf

A blog addressing all of these issues:

http://probabilityandlaw.blogspot.com/

Norman Fenton

Eh? Bayesian theory used in RealLife[tm]?

Spam filters, maybe, but in criminal court? With the future of actual human beings at risk? C'mon ... you have got to be kidding me.

We're turning into more of a cartoon-physics-wannabe society every day.

prosecutor's fallacy

http://en.wikipedia.org/wiki/Prosecutor%27s_fallacy

It is precisely the failure to apply Bayes' theorem that is the most glaring abuse of statistical "reasoning" in most trials. But applying it requires lots of numbers, and it you estimate them it is far too easy to put a thumb on the scale, subconsciously or otherwise.

the point made by "xlq" is also a hot topic in statistical circles. One aspect of it is the Birthday paradox: if you have a large enough population, coincidences will happen.

A related problem does not have such a catchy name. If you do a whole lot of tests for statistically significant relationships among many variables that are actually not related, you will get some that test as significant at random. See recent articles about the problem of observational studies in health and medicine.

In criminal investigation, it is one thing to get a "five point match" on a fingerprint AFTER identifying a likely suspect, it is something else entirely to have a computer spit out a five-point match from a database of millions of people's prints.

Your DNA statistics are wrong. In DNA match of 1 in 1 million probability means that the particular DNA profile is present in 1 out of every 1000 people in the general population, not one in every million. In DNA the probability is the probability that the suspect, the sample, and a random person from the population will all three have the same DNA profile. To find the DNA odds you think they are saying, take the square root of the DNA expert's probability. So, for your example, a one in a million match means that there are about 65,000 people in the population with that DNA profile.

You know what Samuel Clement's said about statistics.

@xlq

Quite - even if we take the 1 in 73 million as accurate, that still means that you'll get a double-SIDS occurrence somewhere in Europe once every few years.

Lies and Statistics

As some wise pundit once said, "There are lies, damned lies, and then there are statistics."

Adversarial vs. Inquisitorial

I don't think that the adverarial system here in the UK works at all for scientific evidence. It's too easy for both sides to put forward 'experts' who are disagreeing, and where does that leave a Jury?

The legal system should be asking the scientific world how evidence is assessed. The scientific world would rightly say 'peer review and concensous'. No consensus, no conviction. That is an inquisitorial process, which the people involved in the legal system don't like that one little bit.

The SIDS cases were appalling. A non-expert in statistics presented unchallenged 'facts' that were in fact horseshit. At no point was he required to show that he was qualified to do so. Why wasn't he charged with purgery? Why weren't the Court's officials charged with gross negligence???

Abuse of statistics, law and human rights

The more I have to do with the legal system the more appalled I am by the poor standards that seem endemic in every part of it. That is sometimes due to the inadequacy of the indivifuals but more often it is down to resources that are completely inadequate to deal with the complexity of the law.

The problem with legal aid is not that there are too many cases, or too many clamimants, it is that the law costs so much and it costs so much because it is often so complex and so poorly defined that the outcome of trials depends more on how much can be afforded than the evidence.

This is a well known example of the missuse of probability, but well worth repeating, because this error is still made, in all sorts of contexts including both court cases and politicians making policy. A DNA match is say one in a million and state that way it is enough to make most people certain. However try putting it another way. There are probably about 60 people in the UK that that DNA could belong to.

99% of serious sex offenders share a genetic trait and a previous criminal offence so the two together are an excellent indicator of criminal propensity and anybody with that genetic marker and committing that offence must be placed on thes sex offenders register. That logic, or very similar, has appeared in the supporting evidence for green papers. The fallacy is made obvious by two words. Male and speeding.

Another all too common error. 80% of people doing X will go on to commit Y. Politicians or axe grinders response is that X muist be treated very seriously because Y is so horrendous but that conclusion can not be justified. Missing information. 90% of the general population will go on to commit Y. The correct deduction is the opposite, X is actually preventative.

These and other failures of logic are endemic in policy making where children are concerned. The issues are often very emotive, and the erroneous conclusions may support popular prejudice, so nobody thinks to question them. Important aspects of the recent Bailey Review lack a verifiable evidence + logic chain, and there is strong evidence that the onconsidered consequences are highly undesirable, but the government intends to implement "in full". Our children deserve better.