No 10's online EU vote signup crash 'inevitable' – GDS overseer MPs have taken the Government’s digital masters to task for their inability to handle online voter registration for this month’s European Union referendum. Members grilled Cabinet Office minister for Government Policy Oliver Letwin after the Government Digital Service’s EU voter registration site crashed under "unexpected" …
If my information is still current:
- It's "overseen" by GDS
- Development and Operation of the site is provided under contract by a third party
- Hosting (as in the place the website lives) is with FCO Services (https://www.fcoservices.gov.uk/our-organisation/) which might explain why you heard it's run by the Foreign Office
Posting anonymously, for $reasons
I usually defend Gov IT as its bloody tricky and the private sector screw up just as much, you just don't hear about it.
However, in this case it's a pretty simple case of not sorting the architecture properly, and getting some short term extra horsepower to cover the spikes. I thought all these new shiny .GOV sites were created using DevOps, or whatever new buzzword methodology that's come out of Silicon Valley recently - surely that would have been anticipated?
Not many people know this but someone issued a warning exactly two years less 10 days ago about the duplicates problem. That message is presumably still on its way from the dinosaur's stubbed toe to its brain.
GDS's apply-to-register-to-vote platform is not communicative. You submit your application. And you wait. Some applications will be successful, i.e. the Electoral Registration Officer adds you to the electoral roll. And some will fail. You don't find out you've failed until you find out you can't vote.
Prediction: more newspaper headlines about all the people, whose applications have failed, being silently disenfranchised, followed by Oliver Letwin explaining that that's inevitable.
Well I do have a counter example. When registering to vote (about 3 months ago having moved countries within the UK), I received a letter asking for additional information (in my case I supplied a copy of my passport), and lo and behold a polling card arrived for the Scottish elections and more recently one for the EU referendum. So there is at least some 'catch' function.
I'm pleased to hear that there are counter-examples. But most people get no notification. We await GDS's GOV.UK Notify.
We await also their GOV.UK Verify (RIP), which is supposed to provide adequate proof that we are who we claim to be. Which is what the EROs need.
Of course, if GOV.UK Verify (RIP) worked, we wouldn't need the EROs. We wouldn't even need to register to vote. GDS could decide our entitlement to vote for us using attribute exchange and all the non-existent open data registers which support the non-existent GaaP, government as a platform.
All we would need to do is vote.
Or would we? Could GDS use data science to work out for us what we should vote?
Soon we won't be needed at all. GDS can cater to all our user needs.
There was a major campaign to get people to register to vote. When they tried to register to vote, that was unexpected. Or unprecedented, according to GDS.
Just how unexpected or unprecedented?
Cast your mind back to 21 April 2015 and the BBC's More people register to vote 'than ever before':
A record-breaking 469,000 people registered to vote online in one day for the 2015 general election - as the deadline closed on 20 April.
GDS should look at themselves.
Because GDS actively discourages transaction pricing it discourages suppliers to provide horizontal scalability for service like this. They much prefer the predictability of fixed service capacity scalable in more traditional ways.
In this instance the supplier can say it performed to capacity as can the department. As usual the poor public suffers from poor IT policy and solution design.
We now live in a world of cloud hosting everything. Scaling a site like this should be as simple as spinning up more instances for the front end and database. You should even be able to automate it "if demand reaches X, spin up Y new instances".
Ok, that's a vast over-simplification, but it is still how it should work!
This stuff shouldn't go out to AWS or any other commercial cloud provider. G-Cloud should be able to offer the same. And if it doesn't, why not?
Because the scalability of cloud services is largely a result of serving a lot of different clients with varying load requirements. A government facility isn't a cloud in this sense, it's just a web server scaled to what is thought to be the maximum likely demand.
The odd thing is that the sites always crash when they're overloaded. If overload is likely it would make sense to throttle access through a proxy that is sufficiently lightweight to have little risk of overload. It wouldn't solve the deadline problem, but it would reduce the problem where slow response makes people queue up numerous retries and open several clients, thereby increasing the overload.
I accept that the use of a commercial provider might be inappropriate, but I should have thought that the information being processed isn't super-sensitive, and it ought to be possible to devise an arrangement that respects security requirements in a commercial environment.
Recent history is littered with stories of companies (usually online retailers) who go TITSUP during peak buying periods (Christmas, Black Friday) etc.
The cause? Idiots in Suits Making Bad Economic Calculations. AKA as corporate airheads who won't listen to or seek expert advice.
They generally do this once or twice, losing millions each time. Eventually people start screaming and they hire a consultant or supplier who actually knows what they are doing. Gov IT seems to be the exception to this rule, I wonder why?.
Of course, the organizations getting it right the first time never make the headlines. Education is sorely needed, but there are always some those who will insist on learning the hard way.
You may be right about these unnamed companies but GDS are meant to be different. They're meant to have the digital future coursing through their veins. They speak it like a native. Others may make a mistake digitalwise but GDS can't, almost by definition. GDS are in Whitehall, where no-one understands the first thing about digitisation according to the current and previous executive directors of GDS, precisely because they know everything about it. And they don't wear suits.
Many government style procurement processes tend to be very strict, and want to assume that everything has a definitive cost. I can well imagine them being offered a properly scalable setup which would cope with everything, but of course the cost of that depends on usage, and the accounts department can’t cope with that. So they take a guess on usage, set a hard limit, and cross their fingers. The guys maintaining it aren’t just going to scale things up without prior authorisation, since they know there’s a good chance they won’t get reimbursed without a valid PO for the cost.
That rather sums up one of the main problems of parliament: people like Oliver Letwin who appear either not to have the capacity to understand the areas over which they have responsibility or to be intent on deceit as a means to further their agenda.
Checking out what he studied post-Eton - philosophy at Cambridge - a search brought up a speech he made this April to the UN.
https://www.gov.uk/government/speeches/we-must-redouble-our-efforts-to-meet-the-challenge-of-new-psychoactive-substances-to-bring-the-most-harmful-substances-under-international-control
His claim that the UK "is delivering a modern, balanced, evidence-based response to drugs within the UN conventions" is unconscionable. Professor David Nutt and his team were quite clear, before their forced retirement, about what a genuine evidence-based policy on recreational drugs should be.
I propose a new system to measure the incapacity of a given system (units would be "brexits", what else? - mega, giga, etc.) Inversely proportional to, say, un-expectancy of a projected incapacity event?
In lieu of more scientific methods, I estimate my own current system incapacity at 1.2 GBS (gigabrexits), cause it's Friday, totally unexpected.
PM David Cameron.. says: “I believe the creation of the Government Digital Service is one of the great unsung triumphs of the last Parliament."
Because you'd want to sing about the great triumph of the payments system for the Dartford Crossing being in alpha state over a year and a half since it went live, wouldn't you ??
Every time the latest GDS farce happens, I go off and look. Still alpha ?? Check.
I take it you're not one of the many reg commentards who would've screamed loudly about UK citizens' personal data being outsourced to a commercial entity and to servers in jurisdictions lacking our level of data protection?
There are a lot of people who care about that kind of thing, and would take a dim view if it happened. They might very well seek and get a court order declaring it illegal.
Anyway, a sufficient DoS attack can bring any server down for a while (see my post below for thoughts on who might've expected to benefit from that).
It is a relatively quiet period for HMRC, their busiest times are the deadlines of 31st January for Income Tax, 31st December / 30th September for Corporation Tax, 7th February, May, August & November for VAT, and the days that employees receive their salaries for employment taxes, usually the last Thursday or Friday of the month. The rest of the time, that is a lot of spare capacity that could be used for other things.
don't be so sure. I have just completed an online Blue Badge application
(well, to be precise I *tried* to complete an online Blue Badge application. The website crashed 5 times in exactly the same place (obviously right at the end). Then the email address I was advised to use turned out to be wrong ...)
and they don't issue a confirmation.
Neither did the Lloyds share offer website.
I suspect they are trying to recreate the snail mail service, which allowed the "didn't receive it" excuse as a starting point for any fuckups.
The whole deadline is bit silly. If you're eligible to vote, why couldn't you be able to register until right before the vote with the caveat that if you leave it too late you might not be able to exercise your right to vote as you might not be included in the lists at your polling station.
Surely if both sides are truly behind the concept of letting people decide, then enabling as many as possible eligible voters to vote should be in their interests.
Can we say one way or the other whether anyone might have deliberately DoSed the system? Cui bono?
The alacrity with which a minority of "out"ers jumped on it with cries of Judicial Review tells us someone thinks they may have something to gain from what happened: they're preparing the ground for a "vote again until you get it right" scenario. If the deadline hadn't been put back, they exclude a bunch of voters who everyone supposes to be predominantly-young, predominantly-in. A win-win for a DoS attack.
If it was regular cockup - lack of capacity - it would seem more than likely it should've gone down again before the extended deadline. As noted on Wednesday (before the event), whether it survived Thursday would provide cockup-vs-conspiracy evidence.
Or, you know, someone saw the state things were in, and just upped the number of apache processes allowed to run/threw more RAM in/split the DB across more, faster disks/whatever. IE what normally happens when a service shits itself due to load and needs more resources, and needs them at short notice.
Which is rather more likely than a shadowy cabal wanting to throw a referendum.
Steven R
I registered to vote 12 months ago and never recieved any contact to tell me if successful. I reregisted 51 weeks ago still with no confirmation and 50 weeks ago and just about every damn week since and still don't know if I am registed to vote in this cock up.
Hardly any wonder the server crashed when there seems to be no way to check if you are already registered.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
