back to article Telecity fix nixed: Borked UK internet hub 'had no UPS protection'

Telecity cancelled its plans for a second shot at fixing the stricken power supply in its Docklands internet hub at Sovereign House. Engineers for the data centre operator were due to try fixing the broken system again on Wednesday night/Thursday morning, sources told The Register. However, The Reg has learned, these plans …

  1. Your alien overlord - fear me

    Companies not returning your phone calls? Has Apple's influence spread or are you ticking off too many people nowadays?

  2. Stuart 22

    RackShack - the people's provider

    Anybody remember them? The pioneers of the $99 dedicated server with no contract? Bringing DataCentres to a whole new value/budget market.

    They had a power grid outage. The substation serving their datacentre actually blew up leaving them off the grid for about a week. And none of the operators of the 17,000 servers ever noticed. But they did get the full de-brief a little later. How the UPS silently took the load, how the two diesel generators powered up to provide the power. How one overheated (because they had only tested one at a time and the ventilation of one affected the other) - but they got carpenters to build a new whole ventilation system before it was lost. How they were having to get two tankers a day into refuel.

    All for $99 about ten years ago. Somehow I had expected at least something as professional today in London as opposed to the Texan outback. Oh happy days!

    * There was one other problem - their telephone system wasn't on the UPS so headsurfer and crew had to drive into the hills to a place their mobiles would work to call in help. But then few disaster recoveries are truly perfect ;-)

  3. Anonymous Coward
    Anonymous Coward

    UPS or no UPS - which is it?

    The headline says the borked uk hub had no UPS protection, the article appears to say they are awaiting part for the UPS so which is it? Looks more like they had some sort of serious power issue which fried the UPS and it is now running on mains bypass until they can get some parts.

    1. Captain Scarlet

      Re: UPS or no UPS - which is it?

      Yeah this also confused me, I assume they have a typical DC UPS area which I always assumed had multiple power feeds to areas of the data centre (So issues with one row of UPS's can be isolated).

  4. Anonymous Coward
    Trollface

    It's the old story...

    Whenever you make something fullproof, someone comes along and invents a better fool....

  5. Vince

    Not able to reach Telecity by phone you say?

    And the news article is where?

  6. Anonymous Coward
    Anonymous Coward

    A really poor place to have such infrastructure anyway

    Its vulnerable to potential Thames flooding and Docklands is an obvious terrorist target so if something did happen there and staff couldn't get to work ...

    1. Captain Scarlet

      Re: A really poor place to have such infrastructure anyway

      Just thought, that would mean the staff there would also not be able to go home :(

      1. Commswonk

        Re: A really poor place to have such infrastructure anyway

        And (I would have thought) an extremely expensive one at that.

        Am I the only old f*rt around who can remember that the (then) Post Office used to have Continuity Sets to maintain service at critical network points in the event of a supply failure?

        It may or may not be my age but I go all misty - eyed at the thought of a nice big diesel generator running somewhere.

        <sigh>

        1. Alan Brown Silver badge

          Re: A really poor place to have such infrastructure anyway

          "(then) Post Office used to have Continuity Sets to maintain service at critical network points in the event of a supply failure"

          That was on top of everything running off float-charged 50V batteries (where battery can bean anything up to "multiple arrays of cells with total volume more than matching most small houses")

          In general you only needed to start the genny if the power was off for more than 15 mins.

          My personal pets were ancient dual 300kW mirlees diesel sets in a TX station (bottom of http://maritimeradio.org/himatangi-makara/himatangi-radio/ - in their better days)

          1. Corp-Rat

            Re: A really poor place to have such infrastructure anyway

            All the BT Exchanges still have the generators (with automatic start these days) and the battery backup.

      2. Anonymous Coward
        Anonymous Coward

        Re: A really poor place to have such infrastructure anyway

        Just thought, that would mean the staff there would also not be able to go home :(

        Originally I thought it's about the only place in London where storing a couple of inflatable boats could be a viable part of emergency plans, but then I realised I don't know the flow rate of the Thames. Would be rather annoying if staff had to be rescued from the Thames Barrier :)

        (I realise it may show that I know zip about boats, ships and other floaty things on account of getting seasick if I just *look* at one of them - curiously, not a problem I have with flying things).

    2. TeeCee Gold badge
      Facepalm

      Re: A really poor place to have such infrastructure anyway

      Well most tech companies would expect their staff to connect remotely if they couldn't physically get to the office. Looks like this might not be a viable option for Telecity though as a prerequisite would be a reliable internet connection.

    3. streaky

      Re: A really poor place to have such infrastructure anyway

      "Its vulnerable to potential Thames flooding"

      No it isn't.

      "obvious terrorist target so if something did happen there and staff couldn't get to work"

      As somebody who used to live right next to it there's *serious* security around it. The only way you could protect it any more is by parking tanks outside and burying it 40 meters underground which isn't exactly viable anyways.

      I hope you're not thinking you're the only one who's ever considered terrorism on the peninsula before - hell it's like 100 meters from the site of one of the largest bombs ever set off on the mainland by the IRA - the place is a fortress; and not for nothing all the stuff that's there is precisely where it needs to be which is precisely why it's there in the first place.

      1. Anonymous Coward
        Anonymous Coward

        Re: A really poor place to have such infrastructure anyway

        "No it isn't."

        Yes it is. The Thames Barrier has already come close to be overtopped a few times and unless they

        hurry up and build a new one the risk just keeps getting greater.

        "As somebody who used to live right next to it there's *serious* security around it."

        Well I work near it and I can't see this "serious" security you're talking about. Unless they're snipers

        hidden in the trees. Anyway, I'm not saying the terrorists would try and take the place over, all it

        needs is staff not to be able to get in for a few days (any staff there would be evacuated) and an issues

        start to go unresolved and then it all unravels.

        "s precisely where it needs to be "

        Really? That would be why most of the big banks have emergency back-up sites outside london

        I suppose.

        1. Anonymous Coward
          Anonymous Coward

          Re: A really poor place to have such infrastructure anyway

          That would be why most of the big banks have emergency back-up sites outside london

          There is also one at Vauxhall, more or less diagonal from the MI6 building (which looks in better shape than that latest Bond movie suggests :) ).

        2. streaky

          Re: A really poor place to have such infrastructure anyway

          Yes it is. The Thames Barrier has already come close to be overtopped a few times and unless they

          I lived right next to it when it was at the highest level ever recorded and I can assure you it was never close to overtopping - and even if that happened it would be limited volume. You're confusing total breach with some water goes over it and you're just wrong anyway; it did not happen there was over a meter to spare. It probably needs replacing in the future but that date isn't now and there's no immediate threat. It's being closed more frequently but that doesn't equate to risk of actual flood merely risk that London would flood if it wasn't closed - and even then it's precautionary and have you seen how raised up above the water line the peninsula is at high tide? There's no risk here and the flood risk maps confirm this.

          The barrier is closed at low tide so there's enormous capacity for extra water even if it did overtop - but again that isn't going to happen within a significant period. You know the barrier isn't watertight right?

          Well I work near it and I can't see this "serious" security you're talking about.

          Well then you're not paying attention - I'm not going to sit here and explain them for fairly obvious reasons. Telecity could probably do more to their own premises in the area and the government should probably directly help them with that but the peninsula itself has various passive and active protections that you'll see if you're paying attention. Plus frankly who only uses a single site anyways.

          which looks in better shape than that latest Bond movie suggests :)

          Spoilers. :p

          1. allthecoolshortnamesweretaken

            Re: A really poor place to have such infrastructure anyway

            "Telecity could probably do more to their own premises in the area and the government should probably directly help them with that but the peninsula itself has various passive and active protections that you'll see if you're paying attention."

            That somehow has the ring of 'private army' - possibly with enough armed personnel to occupy Paris? Okay, bad example...

            1. streaky

              Re: A really poor place to have such infrastructure anyway

              I meant more passive physical security, I won't state the obvious about their premises down there but yeah..

          2. Anonymous Coward
            Anonymous Coward

            Re: A really poor place to have such infrastructure anyway

            which looks in better shape than that latest Bond movie suggests :)

            Spoilers. :p

            Nah - the authors took a dislike to it during Skyfall (I guess they were not allowed to film there :) ) so it's simple continuity. I'm in total awe of the realism, though - I had to go and check afterwards just in case something happened while I was abroad :)

          3. Anonymous Coward
            Anonymous Coward

            Re: A really poor place to have such infrastructure anyway

            "Well then you're not paying attention - I'm not going to sit here and explain them for fairly obvious reasons"

            A raised ground floor to prevent vehicle incursion, CCTV, internal barriers and heavy doors, right?

            A group of terrorists with guns & high explosives could be inside in 10 mins.

      2. Vic

        Re: A really poor place to have such infrastructure anyway

        there's *serious* security around it.

        It's not that serious.

        Years ago, while "in-between" proper jobs, I had a spell driving vans full of rack kit to Canary Wharf. It was a big van - a Merc Sprinter with a Luton body - and was generally pretty much full. I would drive in at least once per day

        There's a big police line I had to cross. Occasionally, they would inspect my paperwork. Not once did they actually look in the van to see if I was carrying what I claimed I was...

        Vic.

  7. Wensleydale Cheese

    "...hadn't actually got what they'd paid for."

    "A source told The Reg customers had paid Telecity for the use of dual, fully independent power suppliers to avoid outages but the fact their service was down indicated they hadn't actually got what they'd paid for."

    Who'd have thunk that from an internet company?

    Well I'm shocked I tell you, totally shocked!

    1. Dan 55 Silver badge

      Re: "...hadn't actually got what they'd paid for."

      I'm sure they religiously pay their taxes though.

      1. Anonymous Coward
        Anonymous Coward

        Re: "...hadn't actually got what they'd paid for."

        I'm sure they religiously pay their taxes though.

        They do, and you can check that because they're a UK listed company with published group accounts. £21.2m of corporation tax on profits of £81m.

  8. Adam 52 Silver badge

    All those sysadmins who wholeheartedly support colo hosting and decry the cloud have gone a bit quiet.

    AWS had intermittent packet loss (enough to disrupt interactive sessions but not fault-tolerant protocols) but their scale lets them work around within a few hours and their customers could even have relocated to Frankfurt, it's the SMEs with their own kit that are stuck in Sovereign House.

    1. Anonymous Coward
      Anonymous Coward

      As long as that data remains in EU hands I'd be inclined to agree with you.

      If, however, you are obligated to keep data close to your chest you'd be up the creek without the proverbial - I think it's not unreasonable for people to expect that a company has actually done its job, so unless their contract specified that outages are just tough luck I suspect this could get costly in compensation claims.

      1. Oli 1

        *THUD

        -Whats that?

        -Its the sound of a beancounter being ejected from a window head first

    2. Nate Amsden

      what is this bullshit you are spewing again? amazon suffers from power outages too

      http://www.theregister.co.uk/2012/07/03/amazon_outage_post_mortem/

      "But despite all the precautions – and there were many – the US East-1 region of Amazon's cloud was brought down after an electrical surge.

      And not because of the power outage, but because of faulty generators and some bugs in some of the control software that Amazon has created to deal with failover and recovery of various cloudy services."

      ---

      I've had equipment hosted at one data center that suffered a power outage(I mean equipment failure which resulted in an outage of both power feeds at the same time), one data center in the past roughly 15 years. Moved out of that data center shortly after(I was a new employee at the time), 2-3 years later that facility suffered a large 30-40 hour outage due to a fire ( http://projects.registerguard.com/csp/cms/sites/web/news/cityregion/16561418-57/story.csp )

  9. Anonymous Coward
    Anonymous Coward

    And we are being told to put everything in the cloud. Why?????

    1. Robert Moore

      >> And we are being told to put everything in the cloud. Why?????

      So that a decade from now, we can spend again to bring those services back in house. Of course.

      1. Anonymous Coward
        Devil

        Hey! IT migration consultants have families to feed to, you know!!

    2. Anonymous Coward
      Anonymous Coward

      This is about data centres. Even with this case, it is better to have stuff in there than an office.

      Unless your office is a data centre!

    3. Roj Blake Silver badge

      Because when done properly cloud is better than doing it in house.

      The moral of this tale though isn't which model is better, it's that no matter which model you use if you want round the clock access you need both redundant UPSs and redundant backup generators.

      1. Terry Barnes

        "The moral of this tale though... ...you need both redundant UPSs and redundant backup generators."

        I'd say not really. If you can't afford downtime the moral is that you need site resilience.

        Buildings can be affected by more than power outages or other infrastructure failure - they can flood, catch fire, there can be rioting, disease, hurricanes - if you want to stay up your n+1 needs to be in a different building, a different city is better, or if you can afford it a different country is excellent.

  10. Commswonk

    That reminds me...

    Its the sound of a beancounter being ejected from a window head first

    To misquote Oscar Wilde An accountant is a man who knows the price of everything, and the value of nothing.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like