Ref the horror stories - Even when it all works...
One of our DCs was built to withstand everything, including no Grid forever (assuming diesel supplies). Six massive gennies in the basement (2 in, 2 spare, 2 servicing), a stack of UPS batteries that would have given Musk a chubby, and so on. Bloody impressive stuff, tested religiously each year. In isolation. Because as long as the components work, the whole does, right?
Turns out - eventually - we weren't testing all the components, because some of them had been hidden behind partition walls over the years, and disappeared off the documentation. Old building, been in use for a long time.
Then, one night, Local Substation went pop. I wasn't there, being sales, but I did get hauled over many coals by clients for weeks afterwards.
The sequence of events was, roughly, this;
1) Power goes
2) UPS - lifed at hour, tested a month previously, goes 100% to 21% almost instantly. Panic
3) Genny control panel does nothing (turns out it had it's own little genny we didn't know about, hidden away behind a stud wall). Main Gennies do nothing at all. More panic.
4) Someone finds the manual start button. Less panic.
5) Except Gennies won't sync onto the ring because the control panel is down. UPS down to 3%. Major flap now.
6) Read technical manual. Gennies can be manually synced onto the ring. "Sync them manually chaps, charge!" Except they can't. Every single breaker in 300,000sq-ft building tripped.
7) Brief pause of disbelief
8) One other, minor, technical thing then rears its head - the dead Genny control panel also controls the exhaust flaps. Massive generators still merrily chugging away with closed hall exhausts. Generator hall is now mostly not oxygen. FM200 system triggered. Everybody out.
9) For reasons I was never told, the water suppression on floors 4 and 6 also triggered... making floors 3 and 5 a bit damp too.
It was almost too farcical to behold - indeed, one of my banks actually said it couldn't have been that much of a cluster****. No Mr Bank, it was.
The supreme irony?
We were a DR provider...