back to article Level 3's UPS burnout sends websites down in flames

UK sysadmins woke up to a headache this morning after a major power cut at Level 3's data centre in Braham Street, East London, lasted for approximately five hours. The electrical fault developed at 3.35am, according to an error report seen by The Register, and knocked dozens of companies offline. The facility provides space for …

COMMENTS

This topic is closed for new posts.
Anonymous Coward

Maybe the bean counters should of listened to Simon and the PFY

0
4
Anonymous Coward

s/ of/\'ve/

0
0
Anonymous Coward

Should _have_, dear boy.

5
0
K
Bronze badge
Megaphone

OMG it affected LCHost...

That must have caused a nightma.... hold on, who the f*ck are LCHost?!

2
1

Buy power meters?

I can visualize the bean counter or spreadsheet manager.: "Install power monitoring equipment and software to keep track of the energy budget? That sounds expensive... why on earth would we need that? Surely what we have now works just fine."

1
0
Bronze badge

Just Guessing

A tenner says the engineers' fix was to bypass the UPS ...

1
0
Bronze badge

Re: Just Guessing

Yes, 1 DC hall is on RAW mains until the UPS is fixed.

0
1
Anonymous Coward

redundancy never works at DCs ?

I could well be wrong. But it seems that all the redundancies that are put into place at datacenters never seem to work. They always seem to have multiple power systems, battery backups, generators etc. But yet we continue to see story after story of large datacenters going offline for hours at a time.

How come these redundancy efforts never seem to pay off?

It's rarely the case that every backup and redundancy method failed but rather that those methods never kicked in to start with :S

3
0

Re: redundancy never works at DCs ?

That may be the case, but how many stories are we not seeing where the backups work perfectly and isolate the single failure without interruption?

6
0

Re: redundancy never works at DCs ?

I would suggest that the system wasn't fully redundant, or the A and B circuits were not sufficiently isolated from each other.

Its actually surprisingly difficult to make a fully redundant power setup in a DC without introducing single points of failure. Partly for technical reasons, largely for cost. You'd have to have two independent UPS and genset systems, each of which can handle 100% of the load in case the other fails. That's a lot of capacity sitting around for a failure that is relatively infrequent compared to disturbances in the primary supply from the grid.

GavinC is also correct - most of the time end users, and even customers in the DC, never know when there is a problem in the power supply as the redundancies or backup systems work as designed. At my last job we moved to a new DC with a new UPS and genset, and there was a component in the UPS that failed and brought the DC down twice before it could be replaced. The next 5 years the system worked as designed and the power was up without interruption from lightening, brownouts, surges, poles being knocked down, etc.

1
0
Silver badge

Re: redundancy never works at DCs ?

It suppose it all goes down to cost/tradeoff. Are the sites you serve so mission-critical that any lengthy downtime would be costly enough that they insist on an uptime guarantee? If they pony up enough for the level of service, then you'd be financially (not to mention contractually) motivated to install a fully-redundant system.

In this case, it's just a classic Murphy strike. You have a backup plan, but then the backup plan fails on you. Happened to me in a smaller instance when the UPS on the central computer broke down suddenly on my watch. And in this case, it probably wasn't worth it to have a fully-redundant system. In any event, they informed everyone of the situation and instituted plans the got things back up relatively quickly (a few hours for such a problem I say is pretty decent--we've had worse).

And I wouldn't be surprised if they bypassed the UPS temporarily until a more permanent fix came in. I had to do that for my scenario. Again, there's a risk, but getting the thing back up was considered worth the risk as long as it only ran like this temporarily.

0
0
Anonymous Coward

Re: redundancy never works at DCs ?

> how many stories are we not seeing where the backups work perfectly and isolate the single failure without interruption?

Well, how many ?

0
0
Bronze badge
Mushroom

Re: redundancy never works at DCs ?

Its a bit more complex than that. Large melting pieces of copper created large amounts of smoke - which triggered the fire systems and shut down most of the cooling and power for safety / security reasons.

0
1
FAIL

again?!?!?

I work for a company that hosts some of its machines in this data center and has been affected by the outage. I wish I could say this is the first time that something like this has happened but this is becoming an all too regular occurrence. We use 3 different data centers and this one in Braham street is by far the most unreliable with overheating, coolers breaking and not failing over properly, breakers tripping and even people breaking in over the past few years. They may be cheap but you most certainly get what you pay for. I wouldn't recommend hosting anything mission critical here.

0
0
Anonymous Coward

Is that place cursed?

Intrusions, HVAC failures and now a UPS burn out, that place sure seems prone to problems.

Glad my ISP or mail host dont have kit there

0
0
This topic is closed for new posts.

Forums