Protection at last: Operation Emergency UPS succeeds for Telecity UK internet hub Telecity has successfully completed emergency maintenance on power equipment at its Sovereign House facility. It was Telecity’s third shot at repair work in a week. Telecity has now, The Reg understands from sources, installed systems allowing for the safe switch-over of customers’ servers and storage to …
Dunno if it's true but should be...
In a Data Centre far far away in space and time (Watford many years ago). The data centre was having some work done in the car park. A JCB driver was digging a trench and hooked up the mains incoming supply. The Data Centre seamlessly switched to the UPS batteries and the techies sighed a huge sigh of relief as the well tested Diesel Gennies would soon kick in. Silence; No clouds of black smoke from the block house, and the batteries getting flatter by the minute.
The reason for the silence??
The trigger circuit for the generators and their supply to the building were in the same trench\duct as the mains power feed...
There but for the grace go we all..
The trigger circuit for the generators and their supply to the building were in the same trench\duct as the mains power feed...
There but for the grace go we all..
Not sure that I would have used the term "trigger circuit", but that's a small detail. You have nicely illustrated the concept of a Single Point of Failure and it is arguable that someone should have spotted what could happen. Did anyone advise the contractors that there were supply cables somewhere in the vicinity of where they were going to dig, and if not why not? And did the contractors have a formal Permit to Work? With the U/G cables in the car park being external to the premises then the cable may have been the property and responsibility of the DNO and they tend not to look favourably on people digging up their cables without checking with them beforehand to find out where the cable runs were. At the same time I'm not sure that they would have allowed the output from the generators to use the same trench at the time of installation.
Always the remember the Rule of Ps: PPPPPP. There simply must be other El Reg subscribers who know what this means.
(For the avoidance of doubt I am not having a go at kmac499 as there is no reason to suppose that he was in any way responsible; the points are rhetorical.)
I believe you can be have both in your job title (my father claims to be an electrical engineer, it even says it on his works office workshop door.)
Though I know his place has 3 hv lines but had the disaster recovery tested that by having the basement of the factory 2 under water including its substation.
Many (MANY) years ago I was involved in transferring mainframe workloads between data centres.
This was before any major network bandwidth was available so we carried a van full of magnetic tapes. [Yes, in those days sneakernet was the only option!]
Oh, and one of the TLAs [IRA] was active.
Just as we drove into the data centre car park all the lights went out and there was an enormous explosion and a cloud of black smoke from behing the data centre. We thought the place had been bombed and were about to leg it (well, van it) when the lights came back on and we could hear loud diesel noises.
The brief moment of sheer terror sticks in my memory, as does the cloud of black smoke.
Uninterruptable Power Supplies. BY Definition Uninterruptable means no disruption (in electrical power), so there must be a battery system and inverter system somewhere. Both of those items should have taken over long before the generators kicked in. Since they did not, then there was NO UPS. Alarm bells should have been ringing at that discovery. Gensets kick in when the main power cuts out or when the UPS is faulty. Both happen often enough that there is no excuse for being so lazy.
There seem to have been multiple points of failure here, not the least of which was selecting Telecity as a provider. The next failure was believing Telecities hype.
Sovereign House uses DRUPS (Diesel Rotary Uninterruptible Power Supply), which is a combined diesel generator and flywheel UPS with a clutch arrangement.
There are two independent DRUPS systems in Sovereign House (System A and System B), but I believe you have to pay extra to have feeds from both. System B is the one that has been affected by the recent power issues.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
