back to article Level 3's UPS burnout sends websites down in flames

UK sysadmins woke up to a headache this morning after a major power cut at Level 3's data centre in Braham Street, East London, lasted for approximately five hours. The electrical fault developed at 3.35am, according to an error report seen by The Register, and knocked dozens of companies offline. The facility provides space …

COMMENTS

This topic is closed for new posts.
  1. Anonymous Coward
    Anonymous Coward

    Maybe the bean counters should of listened to Simon and the PFY

    1. Anonymous Coward
      Anonymous Coward

      s/ of/\'ve/

  2. Anonymous Coward
    Anonymous Coward

    Should _have_, dear boy.

  3. K Silver badge
    Megaphone

    OMG it affected LCHost...

    That must have caused a nightma.... hold on, who the f*ck are LCHost?!

  4. Charles Smith

    Buy power meters?

    I can visualize the bean counter or spreadsheet manager.: "Install power monitoring equipment and software to keep track of the energy budget? That sounds expensive... why on earth would we need that? Surely what we have now works just fine."

  5. Stuart 22

    Just Guessing

    A tenner says the engineers' fix was to bypass the UPS ...

    1. RICHTO Silver badge

      Re: Just Guessing

      Yes, 1 DC hall is on RAW mains until the UPS is fixed.

  6. Anonymous Coward
    Anonymous Coward

    redundancy never works at DCs ?

    I could well be wrong. But it seems that all the redundancies that are put into place at datacenters never seem to work. They always seem to have multiple power systems, battery backups, generators etc. But yet we continue to see story after story of large datacenters going offline for hours at a time.

    How come these redundancy efforts never seem to pay off?

    It's rarely the case that every backup and redundancy method failed but rather that those methods never kicked in to start with :S

    1. GavinC

      Re: redundancy never works at DCs ?

      That may be the case, but how many stories are we not seeing where the backups work perfectly and isolate the single failure without interruption?

      1. Anonymous Coward
        Anonymous Coward

        Re: redundancy never works at DCs ?

        > how many stories are we not seeing where the backups work perfectly and isolate the single failure without interruption?

        Well, how many ?

    2. An0n C0w4rd

      Re: redundancy never works at DCs ?

      I would suggest that the system wasn't fully redundant, or the A and B circuits were not sufficiently isolated from each other.

      Its actually surprisingly difficult to make a fully redundant power setup in a DC without introducing single points of failure. Partly for technical reasons, largely for cost. You'd have to have two independent UPS and genset systems, each of which can handle 100% of the load in case the other fails. That's a lot of capacity sitting around for a failure that is relatively infrequent compared to disturbances in the primary supply from the grid.

      GavinC is also correct - most of the time end users, and even customers in the DC, never know when there is a problem in the power supply as the redundancies or backup systems work as designed. At my last job we moved to a new DC with a new UPS and genset, and there was a component in the UPS that failed and brought the DC down twice before it could be replaced. The next 5 years the system worked as designed and the power was up without interruption from lightening, brownouts, surges, poles being knocked down, etc.

      1. Charles 9 Silver badge

        Re: redundancy never works at DCs ?

        It suppose it all goes down to cost/tradeoff. Are the sites you serve so mission-critical that any lengthy downtime would be costly enough that they insist on an uptime guarantee? If they pony up enough for the level of service, then you'd be financially (not to mention contractually) motivated to install a fully-redundant system.

        In this case, it's just a classic Murphy strike. You have a backup plan, but then the backup plan fails on you. Happened to me in a smaller instance when the UPS on the central computer broke down suddenly on my watch. And in this case, it probably wasn't worth it to have a fully-redundant system. In any event, they informed everyone of the situation and instituted plans the got things back up relatively quickly (a few hours for such a problem I say is pretty decent--we've had worse).

        And I wouldn't be surprised if they bypassed the UPS temporarily until a more permanent fix came in. I had to do that for my scenario. Again, there's a risk, but getting the thing back up was considered worth the risk as long as it only ran like this temporarily.

    3. RICHTO Silver badge
      Mushroom

      Re: redundancy never works at DCs ?

      Its a bit more complex than that. Large melting pieces of copper created large amounts of smoke - which triggered the fire systems and shut down most of the cooling and power for safety / security reasons.

  7. massdosage
    FAIL

    again?!?!?

    I work for a company that hosts some of its machines in this data center and has been affected by the outage. I wish I could say this is the first time that something like this has happened but this is becoming an all too regular occurrence. We use 3 different data centers and this one in Braham street is by far the most unreliable with overheating, coolers breaking and not failing over properly, breakers tripping and even people breaking in over the past few years. They may be cheap but you most certainly get what you pay for. I wouldn't recommend hosting anything mission critical here.

  8. Anonymous Coward
    Anonymous Coward

    Is that place cursed?

    Intrusions, HVAC failures and now a UPS burn out, that place sure seems prone to problems.

    Glad my ISP or mail host dont have kit there

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019