Sounds fair to me ...
although I can forsee a slew of downvotes ...
The collapse of UK IT contractor 2e2 descended into farce tonight as its largest data centre customers were told to each pay £40,000 just to keep the lights on. Clients of debt-crippled 2e2 were told to cough up the cash to keep systems running until they transition to another provider, sources close to the situation have told …
although I can forsee a slew of downvotes ...
Yeah well, that STRONGLY depends on the SLA.
Does it stipulate that in case of wind-down operations, customers may have to pay to defray any running costs plus VAT, may have to wait weeks to exfil their data from servers that belong to HP and to which they have no physical access and generally be up shit creek without a paddle unless they have a second cloud on retainer? (An even then they will probably want to nuke their first cloud by wiping disks etc. but that is another matter still)
Can anyone verify?
Call it a "stupid tax" and it sounds fine to me. If keeping your servers up is that critical that you'll throw £40,000 for a possible extra week or so (nothing stopping them taking the cash and shutting it off anyway), why don't you have a backup you can switch to?
"why don't you have a backup you can switch to"
Maybe thats with 2e2 as well
Hey, you have some nice data there, it would be a pity if anything happened to it.
I guess companies that outsource such operations will have to think hard about where the data is and how to get it back in the future.
This is a 'poster child' for one of the many reasons why you do not outsource mission critical functions. It is also another horror story in my book of horror stories that gets used anytime a wet-behind-the-ears MBA type starts spouting "cloud" at WROK PALCE.
I keep telling those 20-something who head into the CEO's office trying top sell her on the "cloud" to do their homework. Most of the time, they haven't.
...usually precede rainfall. Anyone else tempted to move their data to someone elses servers and services today?
Seems to indicate that the ~400 remaining staff now officially have no hope.
if you build your business on a single point of failure, with no contingency or business continuity compensating controls, then you are the author of your own misfortune.
Exactly right. I was telling a customer of 2e2 they were at considerable risk some months ago. Not because of 2e2 specifically, or anything I knew about 2e2's situation, but because they had no, zero, zilch continuity/contigency plans that took account of a 2e2 technical, financial or corporate failure. They must be in complete panic-mode right now, this could conceivably bring their business down as well.
Try telling that to the wet-behind-the-ears MBA types that can not grok what could possibly go wrong when your cloud connection dies?
They like to think in big-vision terms, and often they are not able to see the underlying problems because they lack the ability to appreciate nuts-and-bolts issues.
>Try telling that to the wet-behind-the-ears MBA types
And that is why other than useless crappy social network companies (that flop post IPO) the upper level people making the decisions are not in their 20s.
Bring Your Own Money.
Is this Enron?
It must be Blackberry!
£10K to run the data centre for a week and £950K to keep the administrators steering the ship!
Seems odd they cannot get someone to take over that aspect of the business as it clearly has customers.
Well I did say yesterday that goodwill alone wont keep the data centres running.....
£1m to keep it running for just over a week, no wonder 2e2 made no money
Probably find a large proportion of this is for the fees of the administrators...
administrator fees > 90% (maybe??)
They have a tendency to just disappear in a shower of rain. I bet their customers are all really happy with the "savings" they've made....
Oh Mr Customer, you want what? The admin password to access your virtual servers? That will be £500K thanks
the Overpaid contractors.
From an insolvency perspective, this will have to go into a file of its own; not only in terms of how does one handle an MSP/hosting or services business Administration but also ineptitude in the lead up to Administration and beyond.
Rule is, if you outsource, you'd better be comfortable with your chosen partner and have solid terms and conditions backed by back-up alternative actions to by-pass disruption and avoid being blackmailed.
Don't ever recall this outfit handling any other IT insolvency.
Err... isnt it getting to the point where its time someone gives the public a list of who these affected customers are? considering it appears they're either about to have their budget blown to bits to the amounts of whatever the 2e2 administrators demand, or lose their data centre.
Seems the fallout from this is about to get a whole lot bigger.
That will probably drive the spinwheel in Brussels another couple of rounds.
Ready for the Cloud Directive!
Seems to me FTI Insulting are in a worse state that 2e2 were!
With no money left and saying they'll not be liable for any costs in any way why not take the money, turn the lights off and draw a line under it all? Probably the biggest Administration cock up ever!
I'm sure they can afford with all the savings they made from outsourcing.
What happens if half the big customers refuse to pay, I am guessing the £900k+ gets paid by the rest of the customers?
Funny that Channel 3 consulting who stitched up several NHS trusts with 2e2's services is still bragging about the fact on the front page of their website.
Perhaps Sussex Partnership are wishing they had spent more than three weeks on the due diligence of a £37m contract.
Every time something like this happens, as with the Megaupload fiasco last year, it demonstrates in huge 500 point Arial Black exactly why relying on The Cloud is a really, really bad idea. And the sooner everyone realises this and decides to keep their own files on their own servers the better it will be for the whole industry and the public as well.
This business of giving control of my data to others has irked me from day one, and anything that hammers this fact home to businesses and consumers everywhere is always welcome in my book. One day, people will get the message: you give control of your data to other companies only at your extreme peril.
So is it like outsourcing making lasagne?
Once again I have to say that this really just isn't the case. Why should a data centre be any less trust worthy than a company employee. There are countless examples of pissed off sysadmins destroying data and backups held on internal company machines.
If you want stuff in house recruit your sysadmins very very carefully. If you're going to outsource, pick you outsourcing suppliers very very carefully. It's as simple as that. There is nothing inherently more risk about outsourcing.
That is why you have a reliable DR set of procedures that take periodic copies of the data out-of-easy reach, and (importantly) that you control the process and the media.
E2E may have the best DR procedures around, but if they 'own' the media that the backups sit on, they are just as unavailable to the end user as the servers that they back up. This is why it is more dangerous.
I wonder how many outsourcing contracts contain clauses that immediatly revert the ownership of the backup media and documentation for the backup process to the end customer in case of insolvency of the outsourcer. Sounds like a good clause to me.
In the case of trusting your administrators, you should spread the responsibility to more that one administrator to make sure that it is performing properly, but this will allow for the situation that if someone trashes all the data on your live systems, you just invoke DR.
Of course, if someone is really malicious, and is allowed to screw with the systems for more than your longest off-site backup is viable, then it is still possible to destroy all the data, but you hope that there is no conspiricy and that such behaviour would be spotted. But you could say the same about the financial director, the pruchasing officer, or even the caretaker. It's not just IT people who could be holding a grudge.
I think some are missing the point, if you have outsourced your entire IT support operation then you don't have either the hardware or staff to keep backups.
The NHS and other public sector organisations using 2E2 need to wake up and ensure in future they carry out full due diligence, don't assume because they are on a government framework that they are sound financially because the GPS did a credit check on 2E2 in December and there were no issues reported.
They also ensure suppliers have insurance in place to mitigate against this.
"If you're going to outsource, pick you outsourcing suppliers very very carefully. It's as simple as that. There is nothing inherently more risk about outsourcing."
Well, there is. SImply put, putting your data in the hands of a company that you have no control over is a lot more risky than if it was in your own hands.
I quote from my original comment.
"and (importantly) that you control the process and the media."
I think you just made my point over again.
My response to "Dr Who" was actually about rogue administrators in an un-outsourced IT department being as big a risk as an outsourced operation going bad. But I tried to make it a little relevent to the original story as well.
I upvoted you, because you made sense.
How in the world could someone who is supposed to be educated (possessing MBA degree) could not possibly foresee the havoc the crash of an outsourcing provider, especially one providing mission critical functions, can be anything but a disaster for a company.
I do now understand what motivates such deranged individuals - executive bonuses for incompetent
I disagree, and see this as possibly an over-simplified reaction (for the record, no I'm not with a hosting provider). Businesses are progressively realising that what they are strong at is running their business, whether it's providing products or services, not necessarily running IT organisations.
That's not to say that this use case is suitable for everyone, but the ability to scale elastically on-demand, pay as you use, and leverage broad infrastructure together with a number of diverse and specific skillsets without the millstone of regular and heavy capital investment/maintenence, can provide businesses with improved agility, or simply allow them to focus on what they do best.
If you outsource you lose control over the process and the media. If, somehow, you don't you lose control you might as well DIY.
Businesses are progressively realising that what they are strong at is running their business,
"Running a business" includes everything from paying the electricity bill to keeping the works canteen going. IT comes in there somewhere along the way. What managers focus on is management and if they are not broad-minded and intelligent enough to properly delegate things they might not understand themselves to people who can do the job, they should not be managers, let alone directors and CEOs.
Now write something without quoting wholesale from your MBA course notes.
So 2e2 dont even own the assets they are hosting their clients services on!
IT Directors of the world - you have been warned. Mind you thats assuming that you genuinely have the strategic interests of your business at heart...and not just the desire to make your bonus at any cost and sod off elsewhere while watching the flames grow.
No, the article, as many other on El Reg has been, is massively over-simplified.
The HP kit is only "some" of what is in the DCs.
Outsourcing isn't necessarily a bad thing for all businesses. Cloud infrastructure (hate that phrase, personally - should be much more akin to utility based computing or some such phrase in my personal opinion) is also not necessarily a bad thing.
But as others have pointed out, failing to perform due diligence (surely the amount of turnover, profit posted and debt were readily available to these customers? And surely it would be clear that it was a house of cards built on a bed of shifting sand?) AND failing to put contingency in place is business-suicidal.
I remember doing some troubleshooting in one of the datacentres of one of the companies that 2e2 bought some years ago. The staff at the datacentre were very pleased with all the alternatives they had provided - alternative electricity supplies from multiple energy suppliers, a standby generator and UPS's, alternative points of access to the internet via multiple exit points within the datacentre but still only a single data centre for an awful lot of the customers.
I've lost count of how many times I've had the conversation about simple things such as cloud backup. Great. But, for example, only if getting that data back down a fairly narrow bandwidth internet pipe isn't a concern. Oh and what happens if that backup company goes under - how do you / can you get your data back?
Like many projects, this comes down to customers offloading their responsibilities for what they perceive as benefits - usually in cash terms - without sitting down and talking about it, and more importantly, thinking about it - 'it' primarily being worst-case scenarios.
I'm curious about one thing - it's been a while since I had any kind of involvement in offsiting to datacentres but close to a million pounds to run for one week? Seriously?
What a farce and hopefully a wakeup call to those people that think outsourcing everything to a single supplier without any contingency is an awesome idea because it saves cash. At least our hosting provider has insurance in place to protect us against this kind of failure happening to their business, its something we require from all our major suppliers and I am amazed its not for others!
or blackmail, whatever the term.
As an employee of a large company who has a significant amount of kit in their DC, I was privy to a conversation yesterday where our head of IT finance came in and said in no uncertain terms that part of the business was ****ed.
I am absolutely certain the legal wheels have already started turning - although I'm not sure I see the point.
Nevertheless it's not as bad as was made out - thanks to the fact that we've just employed our DR strategy for that site.
Would have been nice to have a little more notice though!
that is all
Want somewhere to move to? www.acscloud.com
We can handle the Cisco Hosted Unified Comms, as well as IaaS and other requirements.
Meanwhile the stupid banksters keep on outsourcing to India! ROFL Hahahaha!