Maybe the bean counters should of listened to Simon and the PFY
Level 3's UPS burnout sends websites down in flames
UK sysadmins woke up to a headache this morning after a major power cut at Level 3's data centre in Braham Street, East London, lasted for approximately five hours. The electrical fault developed at 3.35am, according to an error report seen by The Register, and knocked dozens of companies offline. The facility provides space …
-
-
-
-
Tuesday 10th July 2012 13:49 GMT Anonymous Coward
redundancy never works at DCs ?
I could well be wrong. But it seems that all the redundancies that are put into place at datacenters never seem to work. They always seem to have multiple power systems, battery backups, generators etc. But yet we continue to see story after story of large datacenters going offline for hours at a time.
How come these redundancy efforts never seem to pay off?
It's rarely the case that every backup and redundancy method failed but rather that those methods never kicked in to start with :S
-
-
Tuesday 10th July 2012 18:23 GMT An0n C0w4rd
Re: redundancy never works at DCs ?
I would suggest that the system wasn't fully redundant, or the A and B circuits were not sufficiently isolated from each other.
Its actually surprisingly difficult to make a fully redundant power setup in a DC without introducing single points of failure. Partly for technical reasons, largely for cost. You'd have to have two independent UPS and genset systems, each of which can handle 100% of the load in case the other fails. That's a lot of capacity sitting around for a failure that is relatively infrequent compared to disturbances in the primary supply from the grid.
GavinC is also correct - most of the time end users, and even customers in the DC, never know when there is a problem in the power supply as the redundancies or backup systems work as designed. At my last job we moved to a new DC with a new UPS and genset, and there was a component in the UPS that failed and brought the DC down twice before it could be replaced. The next 5 years the system worked as designed and the power was up without interruption from lightening, brownouts, surges, poles being knocked down, etc.
-
Tuesday 10th July 2012 19:18 GMT Charles 9
Re: redundancy never works at DCs ?
It suppose it all goes down to cost/tradeoff. Are the sites you serve so mission-critical that any lengthy downtime would be costly enough that they insist on an uptime guarantee? If they pony up enough for the level of service, then you'd be financially (not to mention contractually) motivated to install a fully-redundant system.
In this case, it's just a classic Murphy strike. You have a backup plan, but then the backup plan fails on you. Happened to me in a smaller instance when the UPS on the central computer broke down suddenly on my watch. And in this case, it probably wasn't worth it to have a fully-redundant system. In any event, they informed everyone of the situation and instituted plans the got things back up relatively quickly (a few hours for such a problem I say is pretty decent--we've had worse).
And I wouldn't be surprised if they bypassed the UPS temporarily until a more permanent fix came in. I had to do that for my scenario. Again, there's a risk, but getting the thing back up was considered worth the risk as long as it only ran like this temporarily.
-
-
-
Tuesday 10th July 2012 20:17 GMT massdosage
again?!?!?
I work for a company that hosts some of its machines in this data center and has been affected by the outage. I wish I could say this is the first time that something like this has happened but this is becoming an all too regular occurrence. We use 3 different data centers and this one in Braham street is by far the most unreliable with overheating, coolers breaking and not failing over properly, breakers tripping and even people breaking in over the past few years. They may be cheap but you most certainly get what you pay for. I wouldn't recommend hosting anything mission critical here.