back to article Software that predicts whether crims will break the law again is no better than you or me

Software that predicts how likely a criminal will reoffend – and is used by the courts to mete out punishments – is about as smart as a layperson off the street. That's according to a paper published in Science Advances on Wednesday. The research highlights the potential dangers of relying on black-box algorithms, particularly …

  1. charlieboywoof

    This software, judging from the above findings, is no better than an untrained citizen

    Not surprised in the least

    1. Anonymous Coward
      Anonymous Coward

      Not a lot of fun wading into the capital punishment datasets, whether an algorthm or not is involved, taking into account the accuracy of predictions by any group. Bias there is 90:1.

      1. AMBxx Silver badge

        It wouldn't be difficult to remove the data on race and run again. Results would be interesting to see how biased it really is.

        Tricky bit would be removing all evidence of race - school attended/area lived/previous offences etc.

        1. Santa from Exeter

          Read it again @ AMBxx

          There was no data on race used. The authors are simply stating the fact that “Black defendants are more likely to be classified as medium or high risk by COMPAS, because black defendants are more likely to have prior arrests," IMHO this could just have easily have been left out, as the same would be true if a white defendant had the same conviction profile.

          1. AMBxx Silver badge

            Re: Read it again @ AMBxx

            Sorry, missed that bit, but my second point is equally valid - you can predict someone's race pretty well based upon where they live. I'm in East Yorkshire 99.999% white. Guess my race!

            Bit like excluding someone's age, but including their year of birth.

        2. katrinab Silver badge

          The thing is that police are racist when they choose who to stop / search etc. They stop & search a lot more black people, but when they do stop & search a white person, they are more likely to find something. However the fact that black people are stoped & searched more often means that they are more likely to get caught, and therefore more likely to have a previous criminal record.

          Also, it means that black people are more likely to get caught again in future, even if they might not be more likely to commit crimes in future.

  2. Richard Jones 1
    WTF?

    Expert Systems

    The original benefits of so called expert systems was that it was claimed they could take a series of relatively simple questions and lead through a decision tree without getting side tracked or missing out important branching conditions. The tree was loaded with verified stop <> go branch choices with each the result of exhaustive study, rather than the wet finger loved by some 'experts'.

    The article does make this locked, sealed box sound more like a GiGo box, (garbage in garbage out). I did a study back in the late 1960s, this conclusively showed that factors then considered too difficult to quantify and thus termed constants, were (a) variable and (b) had far greater effects on the results than items that were studied and researched at great expense, 'the dependant variables'.

    Humans do come with some preprogramming to carry out the sort of social studies of other humans done by this magic trick box, so I am not surprised they scored pretty much as well. I think this one might need a trip back to the study room and I am surprised the call has not come sooner.

    1. jmch Silver badge

      Re: Expert Systems

      "I did a study back in the late 1960s, this conclusively showed that factors then considered too difficult to quantify and thus termed constants, were (a) variable and (b) had far greater effects on the results than items that were studied and researched at great expense, 'the dependant variables'."

      Or in other owrds, people are designing the software to look for their lost coins under the lamppost, because that's where the light is, instead of wherever they might have dropped them.

    2. Anonymous Coward
      Anonymous Coward

      Re: Expert Systems

      It probably doesn't even rise to the level of a GiGo box as technically this is an "Anything In Garbage Out" box. An AiGo if you like. I suspect the rise of the machines will bring many more of these.

      On a more serious note. How is this any different than the wand like fake bomb detectors that James McCormick was peddling to governments. It need be no more than an electronic die with a slight bias towards black "customers". Being closed source we have no way of knowing if it is simply crap AI or a deliberate con.

    3. a_yank_lurker

      Re: Expert Systems

      Claimed is the key word. In this research, people were blinding evaluating risk based on a few factors on a piece of paper. And they are about as accurate as an 'expert system'. This is without any interview or interaction with the person which might have improved their accuracy.

      The only possible advantage of the 'expert system' is it is faster.

  3. Anonymous Coward
    Anonymous Coward

    White, rich, embezzled billions that caused untold suffering - Computer says no to further crime.

    Black, poor, stole food for family - Computer says future serial killer lock them up for years.

    Welcome to the predictable future where computers take away any feelings of guilt when sentencing.

    1. TRT Silver badge

      Rich, embezzler - goes to prison, comes out, unlikely to reoffend in any capacity because they will have some honest money put away at least.

      Poor, stole food - goes to prison, comes out, likely to reoffend because their situation hasn't changed much.

      If one steals out of desperation, then expect the same response to the same situation.

      The way to stop the rich embezzler from embezzling again is to not put them in that position of financial control again. The way to stop the poor food thief from stealing is to take them out of that situation - they need employment.

      The single biggest indicator for reoffending is simply going to be the matrix product of motive and circumstance. The only situation I can think of where this would fail would be criminal psychopathy.

      NB Skin colour is irrelevant for the ACOP's example.

      How can a computer predict honesty? Not yet.

  4. DavCrav

    "Dressel reckoned the disappointing accuracy results for both man and machine stems from racial bias."

    OK, are these results disappointing? I would have thought that a 60-70% accuracy rate for a random event is pretty good.

    Here's the initial guess for a system, based on n variables:

    Given a person with variables a1,a2,...,an, look in your dataset and find all matching people. If recidivism rate is higher than 50% in that subset, click 'reoffend'. Otherwise, click 'safe'.

    You can get a bit better than that if some of the a_i are more important than others, but that's the same effect as having a larger dataset. Once you have more than, say, 20 people in each subset, you will have a good guess as to whether recidivism is higher than 50%. And once it is, you can only guess 'offend', any other system cannot perform better.

    Of course, this is entirely separate from the issue that this only tests whether you reoffend and get caught within two years. Actually, it only tests whether you are found guilty of another offence, not whether you did it either, come to think of it.

    1. AndyS

      If your freedom, or the length of your sentence for a crime, was in question, would 50% seem like a reasonable threshold to you?

      Let's say it means the difference between a 20 and 25 year sentence. Do you think a loss of 5 years of freedom should be dependent on a 50% chance? Even the 60-70% seems very low for such grave decisions.

      If accuracy cannot be improved beyond that, how about this: sentence based on the crime committed.

      1. DavCrav

        "If your freedom, or the length of your sentence for a crime, was in question, would 50% seem like a reasonable threshold to you?"

        Evidently I didn't make myself clear. I claim that you will likely never be able to do with, no matter what system you use, with more than a 60-70% accuracy. Therefore untrained people, algorithms, whatever, won't be able to guess this stuff.

        I made no judgment on whether you should or not, just that they said the results are disappointing, and I think they did fairly well for something that is random. Personally I don't like the idea of using guesswork as to whether you will reoffend to decide if you will be put away either.

        My point was, if you are trying to guess based on percentages, you need to turn percentages into yes/no, and that is always inherently inaccurate for individuals. For example, given actuarial tables, tell me when each member of a group of 100 people will die. You won't do well for each person, but will for the group. It's exactly the same thing.

    2. poohbear

      "Actually, it only tests whether you are found guilty of another offence, not whether you did it either, come to think of it."

      I love the sound of plea-bargains in the morning.

    3. katrinab Silver badge

      60% accuracy is not very accurate at all. When the vast majority of people don't commit crimes, it means that almost everyone who is identified by a system with this level of accuracy, or even a system that is 99% accurate, isn't actually a criminal.

      1. DavCrav

        "60% accuracy is not very accurate at all. When the vast majority of people don't commit crimes, it means that almost everyone who is identified by a system with this level of accuracy, or even a system that is 99% accurate, isn't actually a criminal."

        Erm, that doesn't really apply because this is a piece of software for deciding if prisoners will reoffend. Some of them won't have committed a crime, this is true, but I think 'vast majority' might be pushing it, even in the US legal system.

      2. Claptrap314 Silver badge

        Again, this 60% accuracy is likely to be the best possible for ANY predictor if you are forced into a binary decision. The first problem is that the output is binary. Imagine the weatherman being required to answer the question, "will it rain tomorrow?"

        1. jake Silver badge

          The weather bunny on KTVU (channel 2 in the Bay Area) ...

          ... just said "no".

  5. redpawn

    Just as Good as a Jury

    Show respect for your new biased overlords.

  6. John Smith 19 Gold badge
    WTF?

    Hint. In a security/law enforcement business a change of name usually means a big f**k up

    I'd suggest a search into "Northpointe'" history.

    I'll also bet this software is a lot more expensive than the cost of a few MT staff.

    Worst part of this BS is it's not AI.

    It's some sort of regression analysis using data from past offenders and then measuring which factors (probably a damm sight more than 6) affect the result (IE return to prison)

    Off the top of my head.

    Type of crime.

    "Shot her rapist" doesn't sound like much of a repeat offender risk to me. OTOH "Moved in with a women with 3 kids and molested them all" has done it 3 times already, so kind of does.

    Treatment.

    Feeding a drug habit has resulted in prolific repeat offending, but rather less if they seek, not forced onto, rehab.

    I suspect there are limits on accuracy based on the catchment area of the sample, but that maybe just my bias

    This sounds like scamware that's about as useful as that "bomb detector" that ex cop sold the Iraqis for a shedload of cash that turned out to be complete BS.

    In the name of a US cop show "this is all Bull."

  7. jake Silver badge

    From what I can recall ...

    ... of a class I took many years ago, the only thing ever proven to accurately predict the rate of recidivism is a highschool diploma. Everything else is hand waving and mumbo-jumbo, not unlike the touts who purport to know racing results and the like before the fact.

  8. This post has been deleted by its author

    1. John Smith 19 Gold badge
      Unhappy

      Re: OASys (Offender assessment and sentence management)

      Well that sounds like a class action lawsuit waiting to happen.

      The obvious ones are the "Victims of crime" because of crims who (a superficial look at) would look like poor risks for re-offending. 8 years for Jon Warboys perhaps?

      OTOH there should be a case for people held in too long as well, but that's going to be trickier to prove.

    2. mark l 2 Silver badge

      Re: OASys

      The UK OASys is based on dynamic and static risk factors. The persons, age, number of previous convictions, etc are things the crim can never change so are static risks, but it also takes into account their employment history, education, any problems with drink or drugs etc and these are things the criminal can work on by doing interventions such as drug rehab or obtaining their basic maths and English qualifications or doing a training course which will reduce their dynamic risk as supposedly it will make them more employable and less likely to reoffend.

      Of course a lot of these factors really only apply to offences where the criminal was doing it for financial gain, someone who was convicted of murder, manslaughter or sexual offences may have good education, been employed and no previous convictions so these risk factors are less likely to make any difference to whether they reoffended or not.

      1. Anonymous Coward
        Anonymous Coward

        Re: OASys

        I suspect many of the worst sexual offences could be prevented if the laws were changed to allow people more options, such as legalising and regulating brothels, removing the bans on currently banned psudoimages in conjunction with the use of VR. Every person is different, but many desires are genetic, so its not something we can prevent without any available release.

    3. druck Silver badge
      FAIL

      Re: OASys

      Perhaps the judicial review of the decision to grant John Warboys parole, will discover that OASys was used, because I can't believe any human could come to that decision.

      1. druck Silver badge
        Unhappy

        Re: OASys

        Now not happening.

  9. Wensleydale Cheese
  10. Chris G

    The eyes have it

    When I did jury service many years ago, it was easy to determine guilt.

    Eyes too close together,weasely expression? Guilty!

    1. Kane
      Joke

      Re: The eyes have it

      "Eyes too close together,weasely expression? Guilty!"

      Yes, but unfortunately you were not there to determine the guilt of the lawyer!

    2. Michael Wojcik Silver badge

      Re: The eyes have it

      Eyes too close together,weasely expression? Guilty!

      On a more depressing note, "criminal physiognomy" held on for an awfully long time. It was still being taken seriously by many people in positions of authority even in the middle of the last century, as illustrated by, for example, Josephine Tey's novels. (While Miss Pym Disposes explicitly critiques physiognomy, Tey's characters and plots remain ambivalent about it.) And it may be making a comeback.

      Of course many, many people continue to subscribe to physiognomy or at least let it affect their opinions. Indeed it may be impossible to avoid for most people, given our pre-rational evaluation of others' appearances and the many well-demonstrated limitations on rational decision-making.

  11. Anonymous Coward
    Anonymous Coward

    "How COMPAS works internally is unclear since its maker doesn’t want to give away its commercial secrets."

    WTF? The judiciary process is supposed to be transparent so that the people can see that justice has been done. Sentencing is part of that process - you can't put the decision making in a black box and cite "commercial interests". The inputs and algorithms ought to have been supplied by the judicial system to the software maker, not the other way around.

    Given that the success rate for the software is no better than the man off the street it's clearly not fit for purpose, and should be withdrawn from use. Stop wasting money and just go back to letting judges decide sentences themselves. Failing that, the researchers should put forward a competing bid for the risk assessment work using their mechanical turk method. I bet it'll be cheaper than Equivant, it's more transparent about how things are done, and has already proved to be more accurate in testing.

  12. Pete 2 Silver badge

    indemnity

    > they'd be able to guess as well as this software as to whether the criminal would break the law again.

    But that is only a small part of the process. You could equally say that the software is no worse at identifying potential reoffenders than an ordinary person.

    But that "ordinary person" comes with a lot of uncertainty regarding their own background. It would be extremely easy for a challenge to be mounted against that "ordinary person's" competency, bias or consistency. And then to repeat that challenge until a result is obtained from another "ordinary person" that suits the challenger.

    But a black-box approach, with a highly scrutinised history of over a million - sorry: MEEEEEELION - cases can demonstrate that over its history, it has shown no bias, racial leaning, random choices or inconsistency. Even if its inner workings are unknown. Its results and the analysis of them for any of those factors puts it above suspicion.

    1. Bob Wheeler
      Alert

      Re: indemnity

      Perhaps this system is attempting to solve a different problem.

      I don't know how many judges there are in the US, but I can assume a great many with all their own bias, having a system that can give a consistent result across all those jurisdictions might be what they are trying to achieve here.

    2. hplasm
      Unhappy

      Re: indemnity

      Sadly, you are correct.

  13. Bernard M. Orwell
    Holmes

    Subject to Status

    "Therefore, black defendants who don’t reoffend are predicted to be riskier than white defendants who don’t reoffend."

    Or, to put it another way, shows that black people, even those who don't reoffend, are more likely to be picked up by the police.

    1. hplasm
      Black Helicopters

      Re: Subject to Status

      "Or, to put it another way, shows that black people, even those who don't reoffend, are more likely to be picked up by the police...."

      ...or paramedics, after being shot by the police...

  14. myhandler

    So is The Register now US centric?

    The article implies "all" courts till you get down to paragraph 4.

    Even if it is US centric the Reg writers should remember the US is not the only country in the world.

  15. Raedwald Bretwalda

    This is system is probably far worse than the reported accuracy measurement. The measurement compared with software with a layperson. The alternative to using the software is not a layperson deciding whether the criminal will reoffend; the alternative would be the judge, when presumably has training and experience in such things, and so should be more accurate than a layperson.

  16. MK_E

    I dunno whether to make a reference to Psycho-Pass or the judge computer from the pilot of Blake's 7.

  17. JeffyPoooh
    Pint

    Get used to this sort of 'AI Great Disappointment' news item...

    ...You'll be seeing many of them over the next 40 years.

    1. jake Silver badge

      Re: Get used to this sort of 'AI Great Disappointment' news item...

      We've been seeing them for the past 40 years, why stop now?

  18. Claptrap314 Silver badge
    FAIL

    Race-bait much?

    Katyanna Quach, I've seen some sketchy stories here, but this article crosses over into race baiting.

    Dressel reckoned the disappointing accuracy results for both man and machine stems from racial bias.

    “Black defendants are more likely to be classified as medium or high risk by COMPAS, because black defendants are more likely to have prior arrests," she explained.

    "On the other hand, white defendants are more likely to be classified as low risk by COMPAS, because white defendants are less likely to have prior arrests. Therefore, black defendants who don’t reoffend are predicted to be riskier than white defendants who don’t reoffend.

    1) As previously mentioned, the binary classification scheme is guaranteed to have low accuracy unless the population is highly weighted towards re-offending or not. For example, suppose you have a balanced die, which you roll until a value other than four appears. You are then tasked to predict if the result will be "high" (four or more) or "low" (under four). You predict "low" with 60% accuracy and no way to improve. And if the black die is rolled until a three does not appear, there is no bias if the prediction becomes "high".

    2) The other truly bizarre aspect of this quote is that it demands actual future knowledge. "Black defendants who don't reoffend." WHAT? How in the world can you or anyone else know who is going to reoffend until the subject is dead? Maybe the author intends some sort of linguistic shorthand but this is nuts.

    3) This quote, demands that defendants be classified first by race in order to ensure that there is parity between the number of blacks classified as high-risk verses the number of whites. It is a sad fact that blacks in the US commit more crimes than whites. That the number of priors, which is intuitively one of the strongest predictors, happens to flag blacks more than whites is almost required by this fact. BTW, males will be affected in the same way. Why does the author not claim sex-based bias against men?

    Ignoring the blatant muck raking, the substance of the study is also highly problematic. That the algorithm does about as well as a random person is not super interesting. The question is whether or not the algorithm reduces judicial bias in the direction of harsher sentences for those who are more likely to reoffend. There is no way to eliminate randomness and ensure perfect fairness in a judicial system. Reducing the variability of the judge is a good thing, even if it remains far from perfect.

  19. Michael Wojcik Silver badge

    6, not 137

    "This study severely mischaracterizes the color of the COMPAS UI as green, when it fact it is blue. Thus the results are all invalid and la la la we can't hear you."

  20. Anonymous Coward
    Anonymous Coward

    Baseline

    The first question that occurred to me is "what egregious lunatic authorised the use of such a program for such a purpose?". The second was "what is the accuracy of the rule "predict YES"?" One on-line report has it that the 5-year recidivism rate for US state-level prisoners is 76.7%, so I would expect "always predict YES" to be substantially more accurate than this program.

    For that matter, the idea of punishing people for crimes they not only haven't committed yet but are quite likely to *never* commit is rather disgusting.

  21. Anonymous Coward
    Terminator

    Northpointe Compas Core ..

    Fully web-based and Windows compliant. COMPAS is applicable to offenders at all levels from non-violent misdemeanors to repeat violent felons. COMPAS offers separate norms for males, females, community and incarcerated populations.” ref

    A recent investigation by ProPublica showed that this predictive technology was operating with significant racial biasesref

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like