UK government data protection is a shambles

Just a thought....

So to the list of things the British are incompetent at we can add running a government and managing bureaucracies.

Have y'all ever thought about inviting the Swiss or the Japanese to set you up a government and then getting them to teach you how to run it without breaking it?

Y'all have a nice day now, hear?

garbage in...

The point is that the info is often wrong because the sources are tainted. For some years my NI number was wrong in 'official' records because someone mistook a 7 for a 1.

I suspect some of my NI payments were being misdirected - but its impossible to find out - well until my state pension is missing that is...

And @Avi, actually you should care. What if the police national computer has you down as a suspected paed? These days you need a CRB check even to run a cake sale at your kids' school, and imagine how /that/ CRB check would look on the headmaster's desk...

Documentation to save the day? Whoohoo!

Not likely, the main cast and crew are people, processes and technology. A documented policy is a supporting role at best.

Large scale databases [...] error rate of between five and ten per cent,

> so a government database containing 10 million records might have between 500,000 and one million errors.

Now expand that out to the population of this country and remember that the National Identity Database is supposed to collate records from multiple sources...!

10 percent error rate

"Large scale databases typically have an error rate of between five and ten per cent"... statistically, "significance" is investigated at the 2.5 and 5 percentiles. Since the number of errors expected within the base of data is greater than the significance threshholds, this volume of errors is deemed to be having a significant effect on the validity of the database - or, in lay(wo)man's terms, the database isn't worth the electricity used to display it.

Or, in spin-speak: "we have every confidence in the validity of the data in this database, but only a 10% confidence in this statement"

Unsurprising....

.. since one of the most blatant leaks is done by the DVLA for £2.50 a pop!

Wanna find out where that bank employee/ soldier/ scientist/ doctor lives, just watch them get into a car and ask the DVLA for the address. Potentially a major national security risk, but hey... they get £2.50 a spin so it must be good!

Metro this morning

As if we needed proof of that, there is this story in this morning's Metro about this guy who was bankrupt by HMRC because they kept asking him for £12,000 when in fact he owed them the princely sum of 88 pence. And then, once it's all been cleared, they have the cheek to ask him for the interest on the £0.88 and charge him several thousand pounds for the effort spent in correcting their own mistake!

And in the same paper, there's this story about a civil servant's blog being closed down for telling the (scary) truth about how it all works from the inside.

i just assume....

much like uk.gov, that the DPA doesn't apply to them. I know it's supposed to, but it quite clearly doesn't.

Re: Just a thought...

Hi Steve.

"At least we can manage to run an election properly. "

Doesn't seem to be doing y'all much good. GIGO....

"The idea of an American giving suggestions for governing a country is hilarious."

Not really. Something seems to be working well enough not to mess with.

(I'll show you my $46,00 per capita GDP if you'll show me your $35,300 per capita GDP....)

"Oh well, at least you didn't bomb us first."

Nah, we save that for people who scare us. Can't see y'all managing THAT any time soon.

"Our government may be useless at following Data Protection laws, but at least we have the laws unlike you merkins."

Can't see as it's done y'all a lot of good....

Sceptical Bastard wrote:

"Have YOU ever thought about inviting the military-industrial complex, Texan oil and the Klan to set you up a government? Oh, sorry - I see you already have."

Death-tech pays pretty good. Lots and lots of people earn pretty comfortable incomes making things to break other things (and people). It wouldn't sell so well if there wasn't a market.

If Texas Oil did run the American government then the Gulf of Mexico would be carpeted with oil rigs and the coast covered with refineries. I just peeked.... not all that many oil rigs and even fewer refineries.

If I recall, the current Bush made most of his money selling Football.

And you might want to ask Gonzales, Powell and Rice if they thought they got their jobs because they looked good in white sheets and pointed hoods.

Statistics

Model: the Eye'o'Sauron database is created by merging 6 existing databases. Everyone in the country is represented in all 6 databases. The error rate in all 6 is 10% and the errors occur independently.

Conclusion: 47% of entries in Eye'o'Sauron will be in error.

[Probability of a given database being correct: 90%. Probability of all 6 databases being correct: 0.9^6 = 53%. Probability of at least one database being in error: 1-0.9^6 = 47%.]

If each database comprises 10 separate fields, a 10% overall error rate implies an error rate of 1.1% for each field. If there are 100 fields, the per-field error rate implied is 0.11%. If you want your 100-field database to have an overall error rate of 0.1%, the error rate on individual fields has to be 0.001%: 1 error in 100,000 entries.

IOW, the overall error rate in a database is surprisingly sensitive to the number of independent fields each record contains; the more fields, the more records in error, to the point that a reasonable overall error rate is simply unobtainable. Nobody can enter data at an error rate of only 1 in 100,000. And at the end of the day, all data can be traced back to an error-prone human being entering it.

[Refutations of that last assertion welcomed.]

Reminds me of when I went out looking for a house to buy, and compared each property to the facts on file at the tax assessor's office. Fully half the places I looked at turned out to have errors of fact on those records. Most of the errors were unimportant, but at least one implied the property owner was paying $ hundreds a year more in property tax than he should have.

Has anyone studied the question of information errors, how they arise, how they can be prevented (or their number reduced), how they can be detected? ISTM that without a thoroughgoing understanding of these matters, any attempt to establish any large database is doomed to be riddled with innumerable errors large and small. Just as they are in practice!

Does DH include the NHS

I detect some economy with truth - or early institutional dementia - in the denial by the DH that they have ever been asked to correct erroneous information: even they cannot have forgotten Helen Wilkinson - it's even in Hansard!

Or does the DH consider that that was all part of the NHS and nothing to do with them?