Lights go out at Telecity in London data outage Anyone relying on Telecity for hosting, backup or email services will know this already: a power failure at one of its London centres has left dozens of businesses offline. The company's PRs are currently in a meeting so we have no official response. But several Reg readers emailed us to say power had gone for about an hour …
In fact, not much really ... 2 floors at 6/7 Harbour Exchange apparently.
Meridian Gate, Sovereign House, 8/9 Harbour Exchange, Bonnington House, Prospect House, Olivers Yard, PowerGate ... now if that whole lot went down :-)
BTW - I am no apologists for the Telecity Group - I think they are money grabbing so and so's!
"Dear Customer,
We are currently experiencing issues with the mains power supplying the 7th floor infrastructure at Harbour Exchange 6/7. Power has not been transferred to the UPS and generator systems due to a fault that is currently being investigated and this has resulted in a loss of power to 7th floor customer equipment at the site."
OMFG you sir, are a moron. They didn't go to GenSet or UPS power becasue there's a fault.
"Power has not been transferred to the UPS and generator systems due to a fault...."
Why would you put GenSet or UPS power on to a known faulty circuit and risk fire/damage. Do us all a favour and STFU n00b !
Back to Edison's original plan (selling heat and light).
More seriously, the gas infrastructure is straining too - and apart from most CC setups being relatively noisy there's still the emissions issue. London's air quality is bad enough as it is.
My suggestion of setting up the HPC centres where the infrastructure is readily available instead of down the hall from the users (empire builders) didn't go down well.
There is a wider problem here isn't there.
I've coloed and rented from a lot of data centers and hosting companies over the years, and without exception all of them have had an "unplanned" outage for an extended period at some point or another. Without exception all of them have also boasted a robust and redundant UPS system and hot fueled diesel generators. Of those whose outage was a power issue where the UPS and generators where supposed to ensure an uninterrupted power supply, without exception they all failed.
In one instance it was a issue with power blowing out the UPS circuit hence cutting UPS power and disabling the path of power from the generators. In another it was the transformer room having it's floor and walls blown in which severed the physical connection to the generators. A mobile genie was trailered in that was wired into a hastily reconstructed power panel and was then found not to be man enough for the job.
It seems that whenever a scenario occurs that UPS and generators are called for, other factors impede and these intangibles are never envisaged. There is a mind set here that says, "We've got UPS and generators, so power redundancy is sorted."
Well it's not is it, and there seems to be a whole area here of power systems design that is in need of far greater scrutiny.
Now, what makes you think that you know about every single power event at the data centres you've used? I suspect they've transferred to emergency power many times successfully without you knowing!
Admittedly I've had similar experience with data centres, but I appreciate that unexpected problems do occur. It does seem to have improved over the years though.
I probably shouldn't say it, but our current data centre has been 100% since we started with them over 2 years ago.
I said, "Of those whose outage was a power issue.." I'm sure that the UPS has kicked in successfully on many occasions. My point is that when there is a power related outage, it's never because the UPS itself has failed or the generator(s) failed to start it's always some other part of the power system that prevents the UPS / genie from doing it's job.
UPS and genies don't guarantee power redundancy, data centres always seem to be caught out by some other part of the power supply system. I've never had a situation where the hosting company said, "We're off line because the power went and the generator didn't start." It's always, "We're off line because the power went and the generator started but the transformer that regulates the power from the generator blew."
By the law of averages it seems to be that other areas of power supply are not given the same priority and are therefore not as robust as the UPS and generators.
It's a bit like the data centre I visited in Leicester, where the bod showing me around boasted about the multiple and diverse fibre providers to the building and multiple fully duplexed routers, who then admitted that they only had one switch. There's your weakest point right there and it makes everything else pointless.
My God! What ceiling heights are people specifying nowadays? I heard of extra for the overhead cabling, and I've seen a data centre where they specified the false floor to be 18 inches rather than 18cm high, but 1km from one floor to the next floor (or two) around the walls. Sir, I congratulate you!
Anon because of history in this area!
ok, I've had an update from our provider, which can be summarised as " A Telecity engineer switched off the feed to the UPS, but didn't notice until the battieries ran flat."
Even the UPS in my garage beeps loudly when it loses the mains! Are Telecity really telling us they don't even have a beeper or flashing light to say 'I'm on backup power' ?
Oh well. Stuff happens.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
