Operators at the London Ambulance Service (LAS) were forced to use pen and paper when fielding emergency 999 calls yesterday, after a computer system upgrade went horribly wrong. The LAS said the botched upgrade, which was intended to replace a 20-year-old system, failed and caused technical havoc for its workers for more than …
Business as Usual
"But technical problems forced 999 operators to use a manual system by "recording calls on paper and passing information to ambulance crews over the radio.""
So, like they used to do for several decades before they were computerised in the first place? Hardly an outage then.
IT upgrade failures are unfortunately a common occurrence. This sort of problem is simply a day-at-the-office for many large firms, if it hadn't have been the 999 service it wouldn't have made the news.
"if it hadn't have been the 999 service it wouldn't have made the news."
yeah well it was and it did.
so balls in your court now buddy.
Can I come work in your company......... Where I work an upgrade failure of this magnitude and length of outage results in looking for a new job.
An outage no matter how low/high profile the customer, is inacceptable.
When I do upgrades test plans etc are produced and everything is tested beforehand as much as possible.
Thats not saying I get it right all the time, but the roll back plan makes sure the site is not down for 24hrs!!
"There is a risk that the working processes in the dispatch or call taking functions of the Emergency Operations Centre (EOC) do not align with those required by Command Point, causing a delay to the date of Go Live, causing a cost and time overrun."
In other words, "it might not frikkin work and it´ll all go titsup".
I now remember why I quit working for big consultancies. It´s all about Wank Word Bingo (Google it . nice little PDF - used to make meetings fun).
It's even worse than that - what they're actually saying is that they've no idea if the software is suitable for use in the EOC, or are fit for purpose, or will work with the way we do things. But sure we'll give it a go anyway and see what happens.
Did the actually scope the work?
Last time I did business analysis the idea was to provide a system that mirrored "the working processes in the dispatch or call taking functions of" the purchaser. I think some managers should be sacked.
It's 1992 all over again.. http://en.wikipedia.org/wiki/London_Ambulance_Service#1992_CAD_failure
Yeah. That monumental cock-up was required reading at one of my IT jobs as an example of how not to implement a computer system.
Yep - at uni we used this as a case study on how not to run a software engineering project.
rather than procure a system that would support the work of the ambulance service, someone took the decision to procure a system that would require the ambulance service to change the way it works? Fail number 1.
The fact that they did not sufficiently test the system prior to going live, and that they did not have a backup system in place, means that whoever signed off on this needs to get the chop now (not to mention be held accountable for any deaths that may have resulted from this).
If the software controlling the stock exchange packs up, it's not that big of a deal, just an inconvenience. If the ambulance service is unable to function properly, people die.
Sure, some of this is the suppliers' fault, but someone has to take responsibility for allowing this system to be unleashed on the ambulance service.
God, I'm literally burning with outrage at this story.
"did not have a backup system in place"
... well, apart from the pen, paper, and radio backup system.
In a way
The manual backup system works, but is much slower. ATC faces the same situation - if (when!) the system fails there's a manual reversion available, but flow rates are much reduced. This causes airport chaos as flights are heavily delayed but (hopefully) no reduction in safety.
But the LAS has no control over the rate of incoming calls. In a medical emergency, the reported 50 minutes delay might well make the difference between life and death (or a complete recovery vs severe disability).
part of the plan?
Surely switching to pen/paper/radio until they were able to fail back everybody to the existing system WAS the backup/rollback strategy for the upgrade? From the LAS press release it sounds likes that's their standing backup plan for system failure rather than the operators scrambling for paper and pens in their bags in panic.
Certainly sounds like the testing wasn't as good as it should have been though!
What about the last time?
You failed to mention the irony that the last time they upgraded (i.e. when the 20 year system they're replacing was new) there was also a monumental (and well documented) cockup
but the difference is ...
That last time it was at least their own cockup as they wrote the software "internally". They were then going from the "backup system" of pen/paper and voice radio to their current system. It's a pity they felt that they could not arrange for an organic upgrade continuing with the current IT supplier (as I believe it now is).
As it is, their C&C system has been different from all the other ambulance trusts, if they continue with their new supplier, it will still be different with yet another new C&C supplier in a vast market of IIRC now 11 ambulance trusts. One wonders how anyone is going to make any of this pay.
The big difference between the manual method / voice radio and the computerised / digital radio system is the time it takes to do a dispatch. In essence, when a call comes in normally, the address registered to the phone no (if fixed) is already displayed, the operator merely needs to confirm the location, assess the severity and press a button and an ambulance can be on its way within 30-40 seconds with the address on a screen and the satnav pointing the way. More info is then passed as it becomes available. The big time waster in the manual method is passing the address (and other) details over the radio. That can take minutes.
Bristol not like London then
In Bristol the ambulance is not always on the way in 30-40 seconds. It may still be parked at A&E with the last passenger still inside, being treated there because there is no room. For a "minor" injury like broken pelvis, a 2 hour wait or more could occur.
Then there's the issue of the satnav not being up to date, and whole housing estates not existing. I think in that case they recieved the address as Co-ords and the satnav just said it was in the middle of a large field!
There's nothing like proper testing before full deployment
. . . And this was nothing like proper testing before full deployment
Much too common to be acceptable
The NHS Trust noted (PDF, page 27) in a board meeting discussing corporate risks last month that: "There is a risk that the working processes in the dispatch or call taking functions of the Emergency Operations Centre (EOC) do not align with those required by Command Point, causing a delay to the date of Go Live, causing a cost and time overrun."
Sarcasm Alert! - Why did they think it would work? After all, it's not as if that's what the software was designed to do.
From Grumman's site-
"CommandPoint™ Computer-Aided Dispatch System (CAD)
Northrop Grumman’s CommandPoint™ Computer-Aided Dispatch System (CAD) is a mission-critical command and control system that enables communications center personnel to more appropriately manage calls for service and resources. CommandPoint CAD provides information when and where it is needed to efficiently support command and control operations of public- safety agencies. The system allows personnel to readily access information, make informed decisions, and meet demands for law enforcement, fire and emergency medical services. Enhanced operation of the emergency communications center is the most visible impact of CommandPoint CAD, but the system also provides operational benefits throughout a public-safety agency as well as benefits to the citizens of the community it serves."
I'm so sick of hearing about failed multi-million dollar/pound software implementations where the product can't do the main thing it was developed to do. It seems all too common for the vendors to give a big collective shrug and say "Oh, well, what can you do. This stuff is hard to do so failure is no surprise" Vendors need to be held accountable with clauses in the contracts that won't allow them to profit from failure. In the case of government contracts, these clauses should be required by law.
Agreed, but ...
The problem is that there are few common working practices between (for simplicity): ambulance, fire and police. As a consequence there is AFAIK no completely successful omnibus package in use in all three services in the UK. Each service has its own set of specialist software suppliers. It does not surprise me that a US system is having difficulties integrating with the Ambulance Service as well as UK supplied telecoms systems.
Curiously the Ambulance Service has the most to gain from computerisation because it has a fairly small, well defined, set of things it must do: take a call, send an ambulance, deal with the problem on arrival and either go to a hospital or back to a standby point. There is a simplish state machine and in a well run control centre, an average ambulance will need to resort to voice once or twice a day at most. The rest can all be done on a screen (or two) in the ambulance. Fire is more complex with computerisation mainly used for initial dispatch + job closedown but then anything could happen in between. Police is just "shout and holler" because they could be doing anything: there is frequently no fixed address to dispatch to, no-one knows what's going on (in any significant detail) and plod has to make his mind up when he gets there (wherever that turns out to be at the end of the car chase). You are comparing apples, potatoes and horse shit - not easy in the one computer system.
As for laying the blame in the right place, well Government is the agent of its own downfall by insisting on huge specifications and ITTs up front, spend years agreeing a contract, then the layers of contractors have to implement it (if they are all still in business waiting). Only problems? Well the spec was wrong in the first place because it was written by outside people, not consulting with the ambulance trusts adequately (or the contractors), time and tide has waited for no man and the trusts have so much more that needs to be done now anyway.
Remember: the number of distinct customers for emergency service software is very small, particularly if one specialises on one type of service. Most of the actual software houses are SMEs (usually SEs) and are sub-contracted through much bigger entities (for "safety" [of civil servants bottoms]). There could be two large organisations (with their markup) + the DoH between the SME and the Trust. There have been "consultants" involved as well. This is a long, stretchy and consequently very expensive piece of knicker elastic.
If Cameron were actually going to allow direct contact with the SMEs that physically do the work (and actually possess the domain knowledge), instead of merely talking about it, then it would all be much snappier and a whole lot cheaper.
"Curiously the Ambulance Service has the most to gain from computerisation because it has a fairly small, well defined, set of things it must do: take a call, send an ambulance, deal with the problem on arrival and either go to a hospital or back to a standby point. There is a simplish state machine and in a well run control centre, an average ambulance will need to resort to voice once or twice a day at most"
You really have no clue about Ambulance Service work, do you?
Firstly, it's very common for the reported illness / injury to be nothing like what the crew find on the scene. Also, Ambulance crews are called to assaults in pubs, houses, and in the street, just like the police, and have no idea what they may be walking into.
Secondly, as you note for the Police Service, a proportion of calls - road accidents, collapses in the street, leisure pursuit injuries - have no fixed address, and in the latter case can be some distance from accessible roads, particularly in rural areas, and may require coordination of other rescue services, such as Mountain Rescue or helicopter evacuation.
Your glib assessment of the command and control needs of the different services is sadly wrong.
Um don't blame the software
"There is a risk that the working processes in the dispatch or call taking functions of the Emergency Operations Centre (EOC) do not align with those required by Command Point,"
Look at that right there . If it's off the shelf solution why didn't they test it out first. If it's a custom build we need to look at did they not communicate what they wanted or did the vendor ignore them. Either way this should of been sorted out with a dry run .
soooo...i guess they don't run a test in a virtual environment, then IRL on a small scale, before it was implemented?. ..just sayin...
As this is a public organisation at least there's a decent chance that the investigation into this failure will be made public and we can ultimately all learn from it.
Proof the moon landings were faked!!!
The Apollo Lunar Module was built by Grumman Aircraft Engineering Corporation....