Tales from the Crypt of 15 years in the IT world.
1. A certain global company had 3 operations centres. I worked in the EMEA one. The server room had lovely UPS rated for a couple of hours and was hooked up to an external generator.
When the local utility company sawed through the mains for a laugh, everything went A-OK. When it became obvious the generator would be needed, it was spooled up, but it required someone to be physically in the server room to let the juice from the generator in (a security feature apparently). Except, the only way into the server room was via a badge reader. Yup, the lock was electric and not hooked to the UPS. The only key holder for the physical lock was 10 hours flight away..with the key. Queue panic and IT managers running for the fire axes.
2. Same global company later outsourced half the ops work to a famous Indian outsource brigade. The lovely IT campus on the East coast of the subcontinent had survived the tsunami a few months earlier, but the whole area was regularly flooded. I was there during one such flood. The servers were safely tucked away at least 5m above the highest theoretical tsunami/flood mark. Except the emergency generators were....in the basement! Queue a small army of locals shifting sandbags and running bilge pumps/chucking buckets by hand.
3.Same company had operations in just about every landmass with more than a couple of thousand inhabitants. The SAP rollout required as much data as could be sucked in - meaning IT infra in places served by meagre comms - and lacking in anyone vaguely qualified in IT. The best they could be expected to do was wire a plug. One local expert did well to physically build the server and attach the storage. It had been shipped imaged - already in the domain etc - then stripped down for the journey. The build process required a testing of the RAID array, which is where it all went horribly wrong. Queue 3 day journey for the 3rd level support guru by train/plane/boat to discover the poor local had thought that testing the hot-swap meant taking out all the disks sequentially (at the same time) and putting them back.
4. A certain behemoth of an IT company I worked for. We needed to move an NT server that had become mission critical from an office space to the nuke-proof data centre at the end of the corridor, it genuinely did have real 1960s blast doors, sprung floors, faraday cage etc). Months of planning, contingencies coming out our @rses. One thing we didn't think of was that the machine hadn't been cold booted in about 4 years. Click, click, click. Oh dear....processor dead. In those days you couldn't just whack the disks in another box, it had to be exactly the same hardware.
I could go on and on....