EMIS PCS, the management system for doctors' surgeries, was unavailable for use to hundreds of doctors for much of Wednesday and at least some of Thursday. A practice manager from a south London surgery who requested anonymity told us: "Our system has been down all day. "All the surgery appointments are on the clinical system …
Do none of these companies know how to design modern computer systems? It isn't difficult.
From a patient perspective
"This was an extremely rare incident for EMIS"
I use the repeat perscription service on EMIS and 2 years ago it was sometimes out over the weekend, but in the last 12/18 months it has always been available when I wanted it.
I know that fun is often poked at the NHS and IT, but EMIS is one service that works well and has a positive impact on how much time is required to request repeat medication or book a GP appointment.
On the other hand, I was in a hospital recently and they could not access my GP records or other hospital records because they were in another health authority, so EMIS may work for GP's but hospital IT still needs some work.
and if it hasn't come back in a couple of days make another appointment
"We had an outage in our data centre yesterday"
And you don't have a backup data centre that your users can be automatically switched to when the primary fails?
OK, I suppose its not important that essential information such as apppointments be available 24/365. The patients can just be told to come back some other time, and the doctors can have a day off.
But it would never happen in a properly run business. Which presumably the appointment system isn't.
properly run business
Obviously, my last company was not a properly run business then, with all field-based staff, and ALL overseas offices run on thin-clients from Citrix servers in the head office in Wiltshire. We lost connectivity for a few days. Everything went, including all phones, mobiles, internet, call-centre...
Even those with real PCs in overseas offices couldn't do anything, not even print to the printer in their own office!
Yes its a great idea in theory
Backup datacenters are a great idea in theory but in practice its adds a huge layer of complexity and hence is damn expensive. Ultimately the health managers who bought and commissioned the service would most likely have been advised of all this and quite possibly decided that the extra money involved wasn't worth it. This then leads to a problem with surgeries that have no backup procedures. Ultimately its down to the people who made the decision to purchase the system without a backup datacenter and the surgeries which didnt have proper procedures in place to deal with an outage of the system.
Moving online - they can't even crawl and now they want to sprint?
So - this EMIS that commissioned a broken by design system is now moving to a system where they can fail even more spectacularly and increase the risk of giving away everyone's PI.
How did they get a system with no local data availibility - did someone get bribed or was it sheer incompetence?
Big brother IS watching you but he buggered up the spec for his specs so its all a bit hazy.
Perfect Segway From...
Release the dogs.
If it's like most surgeries
...did the patients even notice the difference.Book for 2.30 (3 weeks in advance mind), arrive a 3.30, then you should only have a 40 minute wait.
"...chaos has ensued in large medical practices where they don’t keep paper records or paper appointment sheets..."
Personally I don't regard a situation where some eejit has completely overlooked the potential need for a fallback process in the event of failure as a computer problem.
It's a basic common sense problem of the eggs/basket variety.
The worst part of all this is that this is only a Big Fat Hairy Deal 'cos the whole kit 'n kaboodle's gone pear-shaped centrally at once. Now consider whether it makes a difference to a patient of one of these large practices whether the whole thing's down or just that some bloke outside with a jackhammer has drilled through the data cable going into their local surgery.....
"some eejit has overlooked the need for a fallback process"
Indeed, even my local dentists office has the wits to print off tomorrows appointments at the end of the preceeding day in case of a computer glitch in the morning.
This was an extremely rare incident ...
Once in a lifetime?
I hope that ...
... when GPs complained EMIS said that they couldn't be expected to deal with people turning up wanting to have their outage seen to immediately and that they should talk to the receptionist and make an appointment
So, you say your system is down?
And what makes you think that? Well, have you tried some fresh air and exercise, maybe lose some weight, then if you still think there's something wrong, come back in 3 months or so and we can think about running some tests.
I've been booking prescriptions and appointments for a couple of years without a hitch; beats ringing up and waiting ages for an answer. It is just a diary folks, not the database.
Have a pint, it is Friday :-)
Not a modern system
The problem is that EMIS PCS is NOT a modern system. So it's not too surprising that is fails.
EMIS is currently rolling out their all singing all dancing EMIS Web system. Now that is a modern system using modern design. If that fails then we can say EMIS can't design a modern system.
You may as well say Microsoft can't design a modern system because Windows XP is rubbish. Of course you can say Microsoft can't design a modern system because of issues in Windows 7...
If I read the story correctly they had a data centre issue so its not the software per se thats the problem its the DR and associated systems that have collapsed.
Doesnt bode well for the customers if the provider's current DR is that poor when they go over to a web platform.
More than appointment
The EMIS PCS system does more than appointment. It does everything from prescribing to managing blood test results. We had half a day of hand writing prescriptions when we could work out what patients were actually taking. Thank goodness our document management system runs serparately so we could at least see hospital letters etc.
Central servers were a major policy push of the late Connecting for Health back when it was NPfIT. PCS is really not a datacentre product and will sit very happily on a single server. The hosted system scale a bit by separating out the database server from the front end and attatching it all to some big storage but EMIS still runs serparate clusters of servers for areas.
This was our third half day outage this year. It is not that rare.
"EMIS is in the midst of moving surgeries onto an online system"
"EMIS is in the midst of moving surgeries onto an online system"
I find this article somewhat unclear.
If surgeries run EMIS locally, and EMIS's datacentre has a problem, how does that affect the surgery?
Or are surgeries are reliant on remote EMIS (in a non-resilient unbacked-up datacentre but that's another story) ?
Clarity is good. But it's Friday. T t f n.
"Presentation layer" people
"Do none of these companies know how to design modern computer systems? It isn't difficult."
It *is* difficult if you're a "Presentation Layer Person" and know nothing of infrastructure basics. Modern IT is largely driven by "presentation layer people".
Agreed, and a plague on all their houses
It's not just modern IT, I promise you, it's modern nearly everything.
NHS IT Still Some Way To Go or Progress
My GP has a PCT provided server with a guaranteed response time from a contract selected by the PCT. the server went down with a hardware failure on a Tuesday afternoon. On Wednesday, some hours after the response time had expired the likely lads came to fix it, they faffed about through Wednesday, Thursday, Friday came back on Monday and fixed it on Tuesday. No appointments, no repeat prescriptions, no patient records, just masses of bit of paper.
Whatever happened to having back up hardware that the data can be loaded onto to preserve a semblance of service? It was a CONTRACTED 8 hour fix service.
On second thoughts EMIS was only down for a bit over 24 hours, I guess that is progress.
Wonder if my GP uses EMIS
He's rarely available anyway - he's too busy moonlighting as consultant gynaecologist, golfer, and circumcisionist (available for weddings, parties, etc). Slots for appointments are only opened a couple of days in advance and then you play telephone lottery with the reception dragons to get a booking. If you succeed in this quest, you next have to arrange time off work to attend the appointment, which will be an hour late. Then he fobs you off with 5 boxes of whatever free samples the drug rep dropped off that week, so you end up singing soprano and growing breasts. Meanwhile, the ingrowing toenail is not getting any better.
It's easier to get an appointment with the queen to pick up an MBE, and at least Liz pretends to be interested in you (well, she'd probably notice if your hand came off when she shook it. I doubt my GP would).
Where's the resilience?
I started designing systems back in the 60s, and we were always careful to have a fallback system in place so that any failure in the main system meant that work could continue - not without disturbance, but at least nothing was stopped.
Spin forward 50(!) years, and real time medical systems are down for hours at a time!
Have we learned nothing in the past 50 years? No, don't say anything - I know the answer.
Far too many hospital administrators and executives _ought_ to be forced to watch their most-loved one die in the E.R., and know it was that administrator/executive's fault, because they didn't implement (or even consider the need for) a tested-and-working backup plan for when the computers are "down".
(I worked in hospital IT, and have some bad stories...)
Re: Plan B
I'm sure you have some horrendous stories but that's pretty vicious. Surely no one deserves that.
Remond me again
Why an appointments booking system for a surgery is holding the data outside of that surgery, in a big data centre somewhere, along with data for other surgeries?
Other than the data mining potential...
I guess DR means something different in the NHS.
Dr? Oh yes, we've got lots of those...
Cache local patients and sync them when the data centre is plugged back in.
It makes you wonder...
How many other SaaS services have single points of failure? Looks like we'll be finding out, sooner or later. The popularity of implementing them as an array of VMs increases that likelihood too, I would think.
You would be shocked
How many have barely any contingency planning for failures and are loaded with single points of failure.
The rule should be triplicate for everything but that costs money and the "it wont fail cos its good kit" mentality never takes into account the idiot unplugging the wrong cable or other organic interventions like road repairs taking out connectivity.
Any SaaS offering worth it salt should have immediate failover along with secondary and tertiary offsite backup as well so the issue these guys had would take a chain of events to achieve as opposed to one event or one kit failure.
Most local surgeries are rubbish at IT...
The fact of the matter is that most NHS local surgeries are rubbish at IT. They are small, don't have the staff and manpower, and what IT support they do get is generally of the level of "advanced hobbyist" rather than a true IT professional. I know, my brother is a GP, and I've seen and heard it all for years.
THAT is the reason they centralized the appointments database, and as much other data as they can, and while they will be moving onto a web-based system ASAP. It's a good call really - when the local staff is barely able to get a few PCs up and running and connected on a network, then that is ALL you should have them responsible for doing.
The problem is the current system really is not designed for a large centralized implementation, which is why they are at risk for these problems until the new one comes out.
3 x 1/2 day outtages
If a working day is 8 hrs Mon-Fri that 0.57% of a year down. Of course if its 24/7/365 its only 0.13%
I wonder if Trevor Potter has a comment?
Why is there a centralized system to arrange a private appointment between me and my doctor ?
Until this year, my local surgery had never had a computer
I should know, I supplied it to them, for typing letters rather than handwriting them!
I guess Cornwall has it's advantages sometimes!!
Beer, cos my doctor told me, it's not a race too see who can beat 50 units a night ;}
At least he told me that down the pub!!
And Skynet enters the medical industry...
I'm sorry, just trying to make something cheerful out of all this bad systems design.
Cheers, fellows - and you with the "don't take with alcohol" label on the empty bottle, well I guess it's all good today ><
GP's use PCT's to gain PC expertise, PCT's being demolished
With the decommissioning of PCT's across the country - this will happen more and more.
Gp surgeries just do not have the skills to manage their IT.
Its going to end in tears, trust me - insider!
"It was a CONTRACTED 8 hour fix service."
With an SLA, doubtless.
And here we have a perfect illustration of the value of most SLAs.
Unless the compensation in the SLA is significant to both sides of the deal, it'll make no difference in practice.
If it can't be fixed in time, so what, if the SLA "compensation" is a couple of days credit on the service charge, redeemable only against the same service with the same vendor (that's not a hypothetical example, that's what you get with a BT Business Broadband SLA).
There was a time when proper system design approved by clued-up techies was considered more important than a worthless SLA. Then the bean counters, PHBs, and the rest of the "B Ark" crew took over.
"Gp surgeries just do not have the skills to manage their IT."
Pretty much why the plan was for a datacentre based system which was *presumed* to offer better backup and DR capability (shared across *all* surgeries) than any one practice could afford or want.
Decent theory. Bad implementation.