back to article Stricken 2e2 threatens data centres: Your money or your lights

The collapse of UK IT contractor 2e2 descended into farce tonight as its largest data centre customers were told to each pay £40,000 just to keep the lights on. Clients of debt-crippled 2e2 were told to cough up the cash to keep systems running until they transition to another provider, sources close to the situation have told …

COMMENTS

This topic is closed for new posts.
  1. JimmyPage Silver badge
    Facepalm

    Sounds fair to me ...

    although I can forsee a slew of downvotes ...

    1. Destroy All Monsters Silver badge
      Mushroom

      Re: Sounds fair to me ...

      Yeah well, that STRONGLY depends on the SLA.

      Does it stipulate that in case of wind-down operations, customers may have to pay to defray any running costs plus VAT, may have to wait weeks to exfil their data from servers that belong to HP and to which they have no physical access and generally be up shit creek without a paddle unless they have a second cloud on retainer? (An even then they will probably want to nuke their first cloud by wiping disks etc. but that is another matter still)

      Can anyone verify?

    2. Mike007

      Re: Sounds fair to me ...

      Call it a "stupid tax" and it sounds fine to me. If keeping your servers up is that critical that you'll throw £40,000 for a possible extra week or so (nothing stopping them taking the cash and shutting it off anyway), why don't you have a backup you can switch to?

      1. Captain Scarlet Silver badge

        Re: Sounds fair to me ...

        "why don't you have a backup you can switch to"

        Maybe thats with 2e2 as well

  2. Anonymous Coward
    Anonymous Coward

    well that sounds like

    Hey, you have some nice data there, it would be a pity if anything happened to it.

    I guess companies that outsource such operations will have to think hard about where the data is and how to get it back in the future.

    1. Fatman
      FAIL

      Re: well that sounds like

      This is a 'poster child' for one of the many reasons why you do not outsource mission critical functions. It is also another horror story in my book of horror stories that gets used anytime a wet-behind-the-ears MBA type starts spouting "cloud" at WROK PALCE.

      I keep telling those 20-something who head into the CEO's office trying top sell her on the "cloud" to do their homework. Most of the time, they haven't.

  3. Adze
    Alert

    Heavy clouds...

    ...usually precede rainfall. Anyone else tempted to move their data to someone elses servers and services today?

  4. Anonymous Coward
    Anonymous Coward

    Seems to indicate that the ~400 remaining staff now officially have no hope.

    :-(

  5. Anonymous Coward
    Anonymous Coward

    The point is ...

    if you build your business on a single point of failure, with no contingency or business continuity compensating controls, then you are the author of your own misfortune.

    1. Anonymous Coward
      Anonymous Coward

      Re: The point is ...

      Exactly right. I was telling a customer of 2e2 they were at considerable risk some months ago. Not because of 2e2 specifically, or anything I knew about 2e2's situation, but because they had no, zero, zilch continuity/contigency plans that took account of a 2e2 technical, financial or corporate failure. They must be in complete panic-mode right now, this could conceivably bring their business down as well.

    2. Fatman

      Re: The point is ...then you are the author of your own misfortune.

      Try telling that to the wet-behind-the-ears MBA types that can not grok what could possibly go wrong when your cloud connection dies?

      They like to think in big-vision terms, and often they are not able to see the underlying problems because they lack the ability to appreciate nuts-and-bolts issues.

      1. asdf
        Trollface

        Re: The point is ...then you are the author of your own misfortune.

        >Try telling that to the wet-behind-the-ears MBA types

        And that is why other than useless crappy social network companies (that flop post IPO) the upper level people making the decisions are not in their 20s.

  6. Anonymous Coward
    Anonymous Coward

    Hey! We're Having a Bankruptcy Party

    Join Us.

    Bring Your Own Money.

    1. Destroy All Monsters Silver badge
      Trollface

      Re: Hey! We're Having a Bankruptcy Party

      Is this Enron?

      1. Anonymous Coward
        Anonymous Coward

        Re: Hey! We're Having a Bankruptcy Party

        It must be Blackberry!

  7. nsld
    Facepalm

    So thats

    £10K to run the data centre for a week and £950K to keep the administrators steering the ship!

    Seems odd they cannot get someone to take over that aspect of the business as it clearly has customers.

  8. Anonymous Coward
    Anonymous Coward

    What did I say...

    Well I did say yesterday that goodwill alone wont keep the data centres running.....

    £1m to keep it running for just over a week, no wonder 2e2 made no money

    1. Anonymous Coward
      Anonymous Coward

      Re: What did I say...

      Probably find a large proportion of this is for the fees of the administrators...

      1. Fatman

        Re: What did I say...fees of the administrators...

        administrator fees > 90% (maybe??)

  9. The Alpha Klutz
    Devil

    That's the problem with clouds

    They have a tendency to just disappear in a shower of rain. I bet their customers are all really happy with the "savings" they've made....

  10. Anonymous Coward
    Anonymous Coward

    You want what?

    Oh Mr Customer, you want what? The admin password to access your virtual servers? That will be £500K thanks

  11. Duffaboy

    the money is for

    the Overpaid contractors.

  12. The Godfather
    FAIL

    you're havin a laff.......

    From an insolvency perspective, this will have to go into a file of its own; not only in terms of how does one handle an MSP/hosting or services business Administration but also ineptitude in the lead up to Administration and beyond.

    Rule is, if you outsource, you'd better be comfortable with your chosen partner and have solid terms and conditions backed by back-up alternative actions to by-pass disruption and avoid being blackmailed.

    Don't ever recall this outfit handling any other IT insolvency.

  13. Anonymous Coward
    Anonymous Coward

    Err... isnt it getting to the point where its time someone gives the public a list of who these affected customers are? considering it appears they're either about to have their budget blown to bits to the amounts of whatever the 2e2 administrators demand, or lose their data centre.

    Seems the fallout from this is about to get a whole lot bigger.

    1. Destroy All Monsters Silver badge
      Devil

      Whirr..... WHIRRR!!!

      That will probably drive the spinwheel in Brussels another couple of rounds.

      Ready for the Cloud Directive!

  14. Ex-BenhamBoy
    WTF?

    What a state

    Seems to me FTI Insulting are in a worse state that 2e2 were!

    With no money left and saying they'll not be liable for any costs in any way why not take the money, turn the lights off and draw a line under it all? Probably the biggest Administration cock up ever!

  15. kain preacher

    Savings

    I'm sure they can afford with all the savings they made from outsourcing.

  16. Anonymous Coward
    Anonymous Coward

    What happens if

    What happens if half the big customers refuse to pay, I am guessing the £900k+ gets paid by the rest of the customers?

    Funny that Channel 3 consulting who stitched up several NHS trusts with 2e2's services is still bragging about the fact on the front page of their website.

    Perhaps Sussex Partnership are wishing they had spent more than three weeks on the due diligence of a £37m contract.

  17. Steven Roper

    I love it!

    Every time something like this happens, as with the Megaupload fiasco last year, it demonstrates in huge 500 point Arial Black exactly why relying on The Cloud is a really, really bad idea. And the sooner everyone realises this and decides to keep their own files on their own servers the better it will be for the whole industry and the public as well.

    This business of giving control of my data to others has irked me from day one, and anything that hammers this fact home to businesses and consumers everywhere is always welcome in my book. One day, people will get the message: you give control of your data to other companies only at your extreme peril.

    1. Destroy All Monsters Silver badge
      Trollface

      Re: I love it!

      So is it like outsourcing making lasagne?

    2. Dr Who

      Re: I love it!

      Once again I have to say that this really just isn't the case. Why should a data centre be any less trust worthy than a company employee. There are countless examples of pissed off sysadmins destroying data and backups held on internal company machines.

      If you want stuff in house recruit your sysadmins very very carefully. If you're going to outsource, pick you outsourcing suppliers very very carefully. It's as simple as that. There is nothing inherently more risk about outsourcing.

      1. Peter Gathercole Silver badge

        Re: I love it! @Dr Who

        That is why you have a reliable DR set of procedures that take periodic copies of the data out-of-easy reach, and (importantly) that you control the process and the media.

        E2E may have the best DR procedures around, but if they 'own' the media that the backups sit on, they are just as unavailable to the end user as the servers that they back up. This is why it is more dangerous.

        I wonder how many outsourcing contracts contain clauses that immediatly revert the ownership of the backup media and documentation for the backup process to the end customer in case of insolvency of the outsourcer. Sounds like a good clause to me.

        In the case of trusting your administrators, you should spread the responsibility to more that one administrator to make sure that it is performing properly, but this will allow for the situation that if someone trashes all the data on your live systems, you just invoke DR.

        Of course, if someone is really malicious, and is allowed to screw with the systems for more than your longest off-site backup is viable, then it is still possible to destroy all the data, but you hope that there is no conspiricy and that such behaviour would be spotted. But you could say the same about the financial director, the pruchasing officer, or even the caretaker. It's not just IT people who could be holding a grudge.

        1. Anonymous Coward
          Anonymous Coward

          Re: I love it! @Dr Who

          I think some are missing the point, if you have outsourced your entire IT support operation then you don't have either the hardware or staff to keep backups.

          The NHS and other public sector organisations using 2E2 need to wake up and ensure in future they carry out full due diligence, don't assume because they are on a government framework that they are sound financially because the GPS did a credit check on 2E2 in December and there were no issues reported.

          They also ensure suppliers have insurance in place to mitigate against this.

          1. Peter Gathercole Silver badge

            Re: I love it! @AC 10:57

            I quote from my original comment.

            "and (importantly) that you control the process and the media."

            I think you just made my point over again.

            My response to "Dr Who" was actually about rogue administrators in an un-outsourced IT department being as big a risk as an outsourced operation going bad. But I tried to make it a little relevent to the original story as well.

            1. Anonymous Coward
              Anonymous Coward

              Re: I love it! @AC 10:57

              If you outsource you lose control over the process and the media. If, somehow, you don't you lose control you might as well DIY.

      2. Test Man
        Stop

        Re: I love it!

        "If you're going to outsource, pick you outsourcing suppliers very very carefully. It's as simple as that. There is nothing inherently more risk about outsourcing."

        Well, there is. SImply put, putting your data in the hands of a company that you have no control over is a lot more risky than if it was in your own hands.

    3. Fatman

      Re: I love it!

      I upvoted you, because you made sense.

      How in the world could someone who is supposed to be educated (possessing MBA degree) could not possibly foresee the havoc the crash of an outsourcing provider, especially one providing mission critical functions, can be anything but a disaster for a company.

      I do now understand what motivates such deranged individuals - executive bonuses for incompetent management manglement.

    4. Joll3295
      Meh

      Re: I love it!

      I disagree, and see this as possibly an over-simplified reaction (for the record, no I'm not with a hosting provider). Businesses are progressively realising that what they are strong at is running their business, whether it's providing products or services, not necessarily running IT organisations.

      That's not to say that this use case is suitable for everyone, but the ability to scale elastically on-demand, pay as you use, and leverage broad infrastructure together with a number of diverse and specific skillsets without the millstone of regular and heavy capital investment/maintenence, can provide businesses with improved agility, or simply allow them to focus on what they do best.

      1. Anonymous Coward
        Facepalm

        Just Consultant Speak

        Businesses are progressively realising that what they are strong at is running their business,

        "Running a business" includes everything from paying the electricity bill to keeping the works canteen going. IT comes in there somewhere along the way. What managers focus on is management and if they are not broad-minded and intelligent enough to properly delegate things they might not understand themselves to people who can do the job, they should not be managers, let alone directors and CEOs.

      2. PhilBuk
        Headmaster

        Re: I love it!

        @Joll3295

        Now write something without quoting wholesale from your MBA course notes.

        Phil.

  18. Anonymous Coward
    Anonymous Coward

    Ransom bu outsourcing

    So 2e2 dont even own the assets they are hosting their clients services on!

    IT Directors of the world - you have been warned. Mind you thats assuming that you genuinely have the strategic interests of your business at heart...and not just the desire to make your bonus at any cost and sod off elsewhere while watching the flames grow.

    1. Anonymous Coward
      Anonymous Coward

      Re: Ransom bu outsourcing

      No, the article, as many other on El Reg has been, is massively over-simplified.

      The HP kit is only "some" of what is in the DCs.

  19. Anonymous Coward
    Anonymous Coward

    The perils of business continuity...

    Outsourcing isn't necessarily a bad thing for all businesses. Cloud infrastructure (hate that phrase, personally - should be much more akin to utility based computing or some such phrase in my personal opinion) is also not necessarily a bad thing.

    But as others have pointed out, failing to perform due diligence (surely the amount of turnover, profit posted and debt were readily available to these customers? And surely it would be clear that it was a house of cards built on a bed of shifting sand?) AND failing to put contingency in place is business-suicidal.

    I remember doing some troubleshooting in one of the datacentres of one of the companies that 2e2 bought some years ago. The staff at the datacentre were very pleased with all the alternatives they had provided - alternative electricity supplies from multiple energy suppliers, a standby generator and UPS's, alternative points of access to the internet via multiple exit points within the datacentre but still only a single data centre for an awful lot of the customers.

    I've lost count of how many times I've had the conversation about simple things such as cloud backup. Great. But, for example, only if getting that data back down a fairly narrow bandwidth internet pipe isn't a concern. Oh and what happens if that backup company goes under - how do you / can you get your data back?

    Like many projects, this comes down to customers offloading their responsibilities for what they perceive as benefits - usually in cash terms - without sitting down and talking about it, and more importantly, thinking about it - 'it' primarily being worst-case scenarios.

    I'm curious about one thing - it's been a while since I had any kind of involvement in offsiting to datacentres but close to a million pounds to run for one week? Seriously?

  20. Anonymous Coward
    Anonymous Coward

    What a farce and hopefully a wakeup call to those people that think outsourcing everything to a single supplier without any contingency is an awesome idea because it saves cash. At least our hosting provider has insurance in place to protect us against this kind of failure happening to their business, its something we require from all our major suppliers and I am amazed its not for others!

  21. Anonymous Coward
    Anonymous Coward

    this is extorsion

    or blackmail, whatever the term.

    1. Test Man
      FAIL

      Re: this is extorsion

      Extortion.

  22. Anonymous Coward
    Anonymous Coward

    ****ed

    As an employee of a large company who has a significant amount of kit in their DC, I was privy to a conversation yesterday where our head of IT finance came in and said in no uncertain terms that part of the business was ****ed.

    I am absolutely certain the legal wheels have already started turning - although I'm not sure I see the point.

    Nevertheless it's not as bad as was made out - thanks to the fact that we've just employed our DR strategy for that site.

    Would have been nice to have a little more notice though!

  23. The Jase
    Trollface

    LOL

    that is all

  24. Anonymous Coward
    Anonymous Coward

    We have capacity

    Want somewhere to move to? www.acscloud.com

    We can handle the Cisco Hosted Unified Comms, as well as IaaS and other requirements.

  25. Anonymous Coward
    FAIL

    I hope the DataCentre staff walk before they are pushed!

    Meanwhile the stupid banksters keep on outsourcing to India! ROFL Hahahaha!

  26. Velv
    Boffin

    Head >>> Bury >>> Sand

    Keep beating the drum - Business Continuity Planning

    It's more likely a data centre owner will go bankrupt than have a plane fall on the building, therefore the plans should already be in place (if not in motion) to get out of 2e2 (or any other data centre, building, POD, tent).

    So you're sitting back smug because you don't use 2e2? Suggest you get out your BCP and your list of suppliers and you see what your contingency is if you lose any one of them.

  27. Arachnoid
    Holmes

    So where does copyright of intellectual material come in to this scenario after all do the uploading companys not still own the data even if they refuse to pay?

    1. RealBigAl

      Yes but if there devices are powered off as there is no money and 2e2 don't own them how do the uploading companies get access to them?

    2. gh4662

      Its like me having a Jessie J CD and then putting through our CD shredder (possibly a good idea) is not relevant, doesn't affect copyright or IP

  28. Anonymous Coward
    Anonymous Coward

    Oh the sheer folly....

    ....of outsourcing.

  29. ducatis'r us
    Mushroom

    the vanishing clouds

    As well as the £960k demanded to keep the data centre running until the 15th the admionistrators' letter states: ( http://www.2e2.com/docs/jacqui/letter-for-non-data-centre-customer.pdf?sfvrsn=2 )

    'As you will understand, we have received a number of requests from customers seeking to gain access to their data

    immediately. Unfortunately, the levels of data held in the Companies’ Data Centres are such that this process could

    take up to 16 weeks and we will need to ensure that the integrity of third party data and security is maintained.'

    That is a lot more money yet to be demanded. I assume that as the numbers of customers with data declines the demands on the remainder will go up.

    This is a haemorrhoidal clusterf**k

    1. Anonymous Coward
      Anonymous Coward

      Re: the vanishing clouds

      Indeed. I wonder how many more weeks before the mainstream media finally catch onto how big this is?

  30. Anonymous Coward
    Anonymous Coward

    Client list on the left of this link

    http://wisdom.co.uk/case/ba.php

    British Airways

    DuPont Teijin Films

    Isle of Man Government

    ITV

    Legal Firm

    MoD

    Nuffield Health

    Philips

    Pret A Manger

    RCM

    Retailer

    Tie Rack

  31. Duffaboy
    Joke

    the money is for

    Its to keep 2e2.com online.

  32. Anonymous Coward
    Anonymous Coward

    Ex-Employee

    2e2 have treated their staff like an expensive overhead since they tried outsourcing our jobs to India 3 years ago.

    2e2 have not understood that their biggest asset is their staff instead of their wallets.

    Thats why you are all drowning at sea and 2e2 are rowing away, laughing, in the last of the lifeboats weeks ago whilst deciding it would be best to tell you two days before payday.

    They didn't even have the bottle to tell you face to face, like last time

    2e2 don't give a toss about the staff. They are asking you to work for nothing so they can wring out a bit more profit.

    Having worked there and left, along with many other good technical guys I know how cold and un-caring 2e2 can be.

    Thats because the original concept, thought up by engineers was taken over by accountants who do not understand anything technical except how to put everything on the slate and run away from their responsibilites when it goes tits up.

    Good luck to you all and I hope you all find something quickly. O2 seems to have their head screwed on the right way at least.

    Good luck - and *rseoles to 2e2

    ..Look on the bright side, whatever you get it can't be any worse :O)

  33. Anonymous Coward
    Anonymous Coward

    Costs

    Total Staff Related Costs 253,336

    Operational Costs Cedar licencing costs 26,087

    Microsoft SPLA licencing 22,382

    Other Software Support 28,011

    Connectivity 41,258

    Newcastle 3rd pty data centre 21,028 <--- Why not just find out who this is, and pay them to keep your data?

    Data Centre R&M 11,714

    Customer cheque printing 2,759 <--- So I'll carry on paying you to bill me :)

    Data storage (tape) 1,488

    Offshore apps support 1,484

    Other Costs 5,904

    Service Credits 1,526

    Total Operational Costs 163,643

    Sub Total Costs 416,979

    Datacentre Costs Gateshead (Rent, rates, utilities) 5,471 <---keep this one

    Reading (Rent, rates, utilities) 27,784 <--- shut this one down

    33,255

    Central expenses (including a provision for the Administrators' costs and legal expenses) 150,000 < SHIT THE BED!

    Grand Total Costs 600,234

    1. Anonymous Coward
      Anonymous Coward

      Re: Costs

      Central expenses (including a provision for the Administrators' costs and legal expenses) 150,000 <!!!!!!!

      So, adminstrators can actually charge for extorting money from the end customers? Parasites like that dont deserve to be paid. Silly customers who outsource all their data and don't have contingency plans deserve a fair few things, but not to fall victim to extortion from people who make their living out of the misery of others.

      1. Anonymous Coward
        Anonymous Coward

        Re: Costs

        Administrators are bound be certain legal rules. They get paid for following those rules. Without administrators there would be no access to this data. The company would have went insolvent rather than into administration and no one would have a chance to get access to anything bar the tin (which would almost certainly contain comprimised data) which would no doubt end up in court with law firms getting paid even more to sort it out. I does seem like extortion but the alternative would be worse. If I'm reading correctly the alternative may still happen. That is switch off, walk away and have a firesale of the equipment at an auction.

        1. Anonymous Coward
          Anonymous Coward

          Re: Costs

          "They get paid for following those rules. "

          Rule one - if no-one else is getting paid, it is immoral for you to try to extort money from customers in order to pay yourself first, then try to work up the energy to look around to see if there's any money left to pay anything else.

  34. Anonymous Coward
    Anonymous Coward

    The PDFs still on their site are great

    http://www.2e2.com/docs/jacqui/the-always-on-network_ms-1-.pdf?Status=Master

    http://www.2e2.com/docs/brochure-pdfs/2e2_Journey_to_the_Private_Cloud_v7.pdf?sfvrsn=0

    http://www.2e2.com/docs/technology-newsletter/Storage_in_the_cloud_Final.pdf?sfvrsn=4

  35. WaveSynthBeep
    Mushroom

    Cloudburst

    Clouds are a bit like banks. When they burst, everyone wants to withdraw at the same time. And then you realise it isn't actually possible to withdraw all your money/data at the same time as everyone else. In the case of clouds it actually costs them more than normal for their links to be hammered 24/7, so even if you're prepared to wait, getting your data out is more costly than just keeping going. Maybe there should be a clause in the contracts that triggers extra keepalive payments when insolvency is declared.

    On second thoughts, there's always the bulldozer method to retrieving your data...

  36. asdf

    hmm

    This is one of the rare times when the US corporate bankruptcy laws seem superior. Far too often the courts bail out companies that should be dissolved but this case shows the perils of having an administrator with an obvious vested interest and the real threat of immediate dissolution.

  37. Anonymous Coward
    Anonymous Coward

    Migration fun (!)

    I've had to migrate one of 2e2's customers today who received one of those pre - legitation invoicing (akin to ACS:Law) demands for £4k ... there were quite a few 'choice' phone calls ....

    Plenty of IT Managers fretting about their own jobs because of silly decisions....

    So a quick thought if you may ... for the developers who had a lot of stress over the past 24--48 hours, and a big thought out for the 2e2 employees.....

    Ironic though that 2e2 service desk were hiring new staff some 4 weeks ago!!!

This topic is closed for new posts.

Other stories you might like