Voice analysis trial gives wobbly results
Tests of voice risk analysis by the Department for Work and Pensions have produced varied results. The first wave of trials involved new claimants at Jobcentre Plus, and reviews of claimants at six local authorities, being classified as low or high risk by operators with the help of a voice risk analysis system. A selection of …
"The department is committed to ensuring that decisions <snip> are supported by evidence," DWP minister Tony McNulty told Parliament on 11 March 2009.
This is a break from government as usual isn't it?
Oh dear, even the Guardian managed a more informative article about this snake oil technology:
http://www.guardian.co.uk/technology/blog/2009/mar/12/voice-risk-analysis-lie-detection-benefits-government-results
"This provided evidence that the voice risk analysis technology was helpful"
No it didn't. There was no control group. How good would the operators have been at classifying risk just by themselves, without the voice analysis software? How good would the operators have been if the caller had simply been told that voice analysis was being done, even if it wasn't actually done?
This "study" sounds like one devised by someone who wanted to produce a positive result.
On the front page of the Guardian Tech section on Thursday they had an article describing how a properly scientific study found that VRA was "no better than a coin toss". Apparently none of the users have ever asked for the results of, or commissioned, a study that checked if it was worth paying for. At a few grand a seat, I wish taxpayer funded orgs like councils, had done this first.
If you tell people you're operating a "lie detector" of some kind they're less likely to lie. According to the Guardian article, the code for doing the "Stress analysis" is 500 (count 'em 500) lines of **Visual Basic*** in the program that everyone buys.
see Charlatanry in forensic speech science:: A problem to be taken seriously.
withdrawn from http://www.diva-portal.org/su/opus/publication.xml?id=13316 owing to legal BS. more here:
http://88.80.13.160/wiki/Israeli_Nemesysco_censored_Lie_Detector_analysis_from_the_International_Journal_of_Speech_Language_and_the_Law%2C_2007
McNutty will be along with some policy based evidence shortly.
Anyone interested in the technology behind the DWP's lie detector technology should see the paper by Eriksson and Lacerda
http://www.scribd.com/doc/9673590/Eriksson-Lacerda-2007
It would be interesting to see if a voice analysis package which undertook the much simpler task of estimating age and sex would be sufficient to get the results claimed for the DWP trials.
TheMinistryofTruth blog (http://www.ministryoftruth.me.uk) has three articles that dig even further into the truth behind the technology.
A US scientific trial into a very similar system found that the 'lie detector' was actually less correct than tossing a coin. The report concluded: "In fact, roughly half of the unstressed and truthful samples were classified by LVA as stress and deceptive, respectively. A device that is, in fact, sensitive to these states should not falsely detect them if we actually failed to elicit these qualities when using our protocol."
So the evidence is that these gizmos don't work and the DWP has wasted yet more money in an attempt to crack down on unproven fraud.
Okay, I'm curious, and I'm going to ask what I'm sure will turn out to be a stupid question that I just don't see answered anywhere:
Risk of what, exactly?
so you're saying that even when they clearly did a biased test to try and make it look as positive as possible for the "technology", even the single region with the most "positive" results in favour of the conclusion they wanted to bring the results were:
out of the people it identified as "lying bastards" 60% were telling the truth, yet those it labeled as "completely honest truth telling" were lying 10% of the time (that says something else about people, but that's a whole different subject...)
if you categorized the staff in to "heads flippers" and "tails flippers" and got them to flip coins no doubt you'd find at least one region that came up with "heads flippers" getting heads 80% of the time, would this prove your "heads flipper detection technology" was working? i guess it would, as long as you selected how you listed your results to bias them in your favour
the large amount of money being paid for this is probably nothing though compared to the amount they'll need to pay once people realise there's a potential case for slander - system lied and said bad things about them and it directly cost them money (benefits reduced), sounds like exactly what those sorts of laws were made for...
The risk is of illegal asylum seeking jihadi aliens claiming 2.5p a week off the dole when the banks are getting squillions.
Some of them even claim to be living in their sister's house in order to get more benefits.
Thank you, Mr Green, I'll just pass you over to my supervisor, ED-209.
ED-209: ARE YOU CHEATING ON YOUR BENEFIT CLAIM? YOU HAVE 15 SECONDS TO REPLY!
