HP managers are reaping the harvest of their deep cost-cutting at EDS, in the form of a massive mainframe failure that crippled some very large clients, including the taxpayer-owned bank RBS. An IBM Z10 at EDS's Stockley Park site, west of London, fell over this week after vital microcode fixes* had not been applied, because all …
IT - the most under..
valued people in the company - until it all goes pear shaped...
Reminds me of..
Reminds me of that other recent disaster with Microsoft / Danger and the T-Mobile Sidekick service. Wasn't that also a case of letting too many people go?
ha ha ha ha haaaaa
Title says it all.
I've just wet myself laughing, me and the thousands of old mainframe hacks the banks have been saying are supernumary for the past decade or soe
Anyone remember the Ingres takeover?
Similar thing happened - all "consultants" were fired at once. Then someone read the contracts realised that all the major contracts have a fixed amount of consultancy included and they had to *beg* the now embittered consultants to please come back and work for us - people had been frogmarched(some beaten!) out of thier offices by hired thugs.
to be expected
top to middle the place is full of idiots - the rest are just complacent...
Paris, because, well, she goes down much more than a mainframe
The QUEEN goes down more than a mainframe....
I only hope they had to go cap in hand to former employees, and I hope said employees took 'em to the fuckin' cleaners on contract rates.
Failing to appreciate that experienced, skilled workers who know the business are an information processing company's only asset of any real value.
Ladies & Gentlemen a round of applause to our cost cutting management consultant overlords -
Redundancies - poor Herbert...
You know he'll be the one to lose his job in the next round of "efficiency and cost cutting exercises".
Hope they've got enough capacity....
...to catch up on the back-log. If the box has been stuffed and has little spare capacity, there may not be enough left to catch-up and carry on - either with RBS or the other users of the tin.
I guess there's more to this than meets the eye? Or is it literally "nobody in EDS knows how to apply patches to the machines?"
More likely, nobody who knew that there were patches that needed to be applied, or perhaps nobody who appreciated that the consequences of cutting this particular corner would be certain catastropic failure. If someone had appreciated the need, I'm sure IBM could have supplied a man to do the work (for a price).
They'll be able to catch up on 12 hours downtime over Xmas. Worst case they pay people to work on Xmas day, though I'd guess that one of the other bank holidays will suffice. That bit of the story makes me think that they let go the man who did the capacity monitoring and planning as well, or deliberately cut everything too close to the bone to postpone the inevitable during the run up to the complete failure of RBS as a bank. Fred "the shred", remember?
Idiots hate people who are more intelligent than themselves. Most bosses are idiots. This goes a long way to explain not just this story, but the entire banking crisis of 2008-9.
Microcode fixes more like reflashing your BIOS than applying a patch.
So yes, maybe there isn't anyone left who knows how to do it; it's probably a highly specialised and infrequent procedure.
But being an IBM system
you can guarantee that the process is well documented. More likely the person receiving the bulletins about the critical firmware updates was amongst the people who was let go. Also, HP were probably doing the hardware maintenance, and you can guess how good they would be at that!
Banks, cheques, yuk!
Banks dont want you to use cheques anyway, so whilst RBS will be publicly upset so that they can claim against contract penalty clauses, the disruption to bank customers will be seen as an added bonus.
Does anything about HP/EDS surprise us any more. Bunch of muppets
Support staff are always the first people to go - the morons up at the top don't understand that things don't break because their support staff do their jobs. They see that their systems don't fall over and so obviously all those support staff aren't needed.
I remember once having to justify my team of 4 DBAs to my manager. He said that we'd only handled one call during the week so how could he justify paying for 4 DBAs. I pointed out that it was because he had 4 hard working DBAs that we'd only had one call all week.
I don't think he really understood.
You are 100% correct!
I see idiot managers make choices like this all the time.
Another example is this true story that I witnessed.
At this bank where I was just hired at I got the run down and saw a server guy let go for being an idiot. As the story goes he really did mess up things and things were well always having problems. The next day they hired a new network server guy for the bank and things slowly got better and so good that soon he just sat and read tech manuals all day and studying for more certifications to add to his expertise. Well no sooner than 3 weeks went by that a manager who was not his manager stopped and commented on that he was sick and tired of seeing him reading books all the time and not working and that he is going to be fired.
Well that manager did get the new awsome server guy fired and well I brought this up with my manager and I got that manager fired for letting go a perfectly good employee who did what he was supposed to do.
Lesson #1 If nothing is wrong with your network then nobody needs to be fired.
Lesson #2 if something is always wrong with your network and the server admins are always running around with their heads cut off then they need to be fired!
Although Lesson#1 is desired let it be known that even Scotty from the Star Trek enterprise always knows when to sass up his work duties to make it appear that he is busier than it appears to be and that is what Lesson #1 type people need to learn.
No, stop, please, my sides are aching too much. What a fantastic advert for the competency of HP and (ex) EDS management that they managed to get themselves into this position.
How can such ineptitude inspire any confidence from existing unaffected customers? How can it do anything than completely destroy their ongoing sales pitches?
As somebody who used to work for EDS I realised quite quickly that the company didn't give a jot for preserving and building on the skills of their engineering staff. All that mattered to senior (I stress senior) was meeting the next deadline and if it worked, so much the better. Middle management, good and bad, get trapped between impossible targets from on high and teams of people who know enough to realise that they're onto a loser, or know so little that thay didn't realise the job couldn't be done.
Remember, the lowest bid isn't always the best, contracts should specify minimum head count and require specific skills/qualifications/certifications to be held. No, you really shouldn't have to have this kind of thing in writing but in this day and age the bog corporates would offshore everything to a $1 a day workforce halfway around the world if they could get away with it. No ethics, no consideration for the workforce, just the end of quarter targets and absolutely nothing else matters. Your shareholders won't thank you for taking the cheapest bid when it all goes tits up like this...
Based on software called Connect Direct?
Connect direct is standard file transfer software, it's rather unlikely that the service will be based on this, rather it'll be a component. This makes as much sense as saying a service for processing payments is based on FTP or CIFS.
Brings back memories
When I worked with RBS in the late 90's the only way you could get data into or out of the "Mainframe" was via connect direct.
This would suggest that the Connect Direct service(server) fell over on the MF and no other system could feed it or drink from it.
Load Clear and...
Will the last one please turn the lights out, thank you.
Finally somebody who knows what he is talking about.... write that SMF! silly HP mainframe wannabees.
RE: Will the last one please turn the lights out, thank you.
Sorry they sacked the last person who new how to work a light switch last week.
The Reg. Irony...
Free whitepaper – A cost-benefit analysis of the IBM CoD cloud offering.
Its funny how sometimes the irony just shows up at the end of the article.
Thin Red Line
Kipling's poem "Tommy Atkins" applies, although IT staff may not be quite as mission-critical as soldiers. The operative lines are these:
'...it's Tommy this, an' Tommy that, an' "Chuck him out, the brute!"
But it's "Saviour of 'is country" when the guns begin to shoot...'
Pour encourager les autres
I really, really hope RBS sues the arse off HP - and doesn't agree to settle out of court. The useless bastards deserve to be taken to the cleaners, hung out to dry and never see another penny of public money for as long as they continue to exist.
I'm going to add, "MBA and nothing else" to my personal skills blacklist for next time I'm recruiting.
Pic of someone rifling through someone else's pockets, as we've no lawyer icon.
another company that didnt learn
never get rid of the techs that keep your systems running unless those systems are not essential
is just a file transfer system, somewhat like FTP, but with a list of trusted nodes that it can connect to. Any cheque clearing software wouldn't be based on it, at best it would be a simple source and destination for data files.
Why not having updated microcode would suddenly stop file transfer software from working is the more relevant question. More likely the version of Connect:Direct was updated and it wouldn't work with the old microcode. This kind of problem should have been spotted in testing, and at very least they should have been able to roll back quite quickly to the older version.
But no bank would allow any change (that wasn't critical) to a payment system during December, it's the highest volume time of year by a long shot.
Also, like you said, this would be covered off in pre-prod testing, one would hope. Banks are a very change averse envrionment.
Connect Direct is a file transfer package - there will be rather more to the cheque clearing system than that (although that might be the bit that failed and no doubt it is critical). Banks (in the UK at least) seem to favour Connect Direct.
I know its already been said but......
Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha
Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha
Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha Ha
I LUV IT !!!
couldn't happen to a nicer company
Ahhh :-) brought a smile to my face :-)
Couldn't happen to a nicer company
Having worked there (Stockley & Mitcheldean) in the late 90's (OVMS, not IBM), and been done over by them, this couldn't happen to a nicer company.
Security sandbox violation
"The cheque backlog was enormous" - so like, at least 9 cheques not yet processed? :-)
ps. Why do I get a security sandbox violation on the Reg's comments page when running Flash debug? :-)
wont this be fun
wont this be fun when it's EDS running the ID card database, and now your entire life is on hold... oh yes, fun indeed !
re: Banks, cheques, yuk!
That depends on if it's in-clearing or out-clearing, they want to cash the in-clearing cheques so that they get the money as soon as possible!!!
It is insane for a large IT intensive business, like a bank, to outsource its IT, it loses control over its business, as we see here! Management heads should roll, and the IT should be in-sourced again!
Banks may not like cheques, but they are the only option in many situations, particularly for large sums of money, and where people refuse to get stung by card transaction charges!
EDS & RBS really haven't a clue
EDS & RBS really haven't a clue.
It would be best if over time both companies could be run down and the business transferred to companies which understand their businesses, care for their customers, and are not driven wholly by short term gain.
P.S. Off-topic but to suggest that banking is a profession is laughable.
Pay peanuts, get monkeys.
This is what happens when you're too tight to even pay for monkeys...
They must have really went to town on cutting staff to the extent noone knew how to do it, just click a few buttons and IPL.
Not at CSC
That sort of thing would never have happened at CSC !!!
candidate for Comedy Club alert! (or Sarcasm of the Year award)
Mainly because CSC wouldn't know what a mainframe was if it bit them in the arse
I-B-M.... its a bit big to run Win95 innit?
vary net inact id=all
I somehow doubt
"Previously the updates would have been applied by the Stockley park hardware team, who have all been made redundant"
I suspect this isnt actually what happens as applying MCLs is actually done by IBM (unless of couse they don't have a support contract ) Of course the hardware teram would have been involved, in arranging the work and coordinating with IBM etc..so if they werent there then yes there would be a big hole in ownership of the process..
From everything I've seen of the way EDS have treated some loyal hardworking and essential staff at my employer who were unceremoniously sold off to EDS (now HP), the only surprise is that stuff like this doesn't happen more often. Morale must be so low that anyone with any clue and hireability (in the form of good solid experience on their CV) must be jamming the exits in their rush to abandon ship.
Pint because remember: alcohol makes the managers go away.
Once Hewlett and Packard retired...
and the bean counters took over... the downfall of HP began.
mines the one with the history book in the pocket (those who ignore history are doomed to repeat it)
What does HP know about mainframes?
PC Servers, yes. Unix (at least HP-UX), yes. Proprietary minis (HP3000) yes.
Mainframes. IE the big things with the high reliavility and the fat consultancy charges. Not much.
So let's take over EDS and fire anyone who does.
I would love to have been a fly on the wall for this one!
I can here the conference calls now, the flapping of the incident management chimps, client management on warp factor BS and all for the want of a techie who knew what they were doing!
As I have been told before, I am a COMMODITY, my skills can be sourced form anywhere! Ha ha!
Did you try switching it off and on again?
Looks good on HP, what a piece of crap company.
I wouldn't let them near a calculator let alone a PC.