Lots of UPS, but no backup AC
I remember at a job about 15 years ago now, when we built out the "server room" as we called it (about 11 racks of gear, much of it low density stuff sitting on rack shelves in 2 post racks), I made sure to equip it with tons of battery backup, quite literally there was probably 1,200+ pounds of APC Smart UPS rack mount systems in various racks. Battery run time was probably at least 90 minutes.
One Sunday morning my phone wakes me up with a message saying the system is going on battery power. I was tired, and said, oh neat, the system works. Took me what I guess was about 30 seconds to realize oh shit, I forgot the AC is now down and the room is going to turn into an oven. So I rushed over to the office(2 miles away) and started manually shutting shit down(it would of shut down automatically when the UPS battery got low but I didn't want to wait that long) .
Fortunately there was nothing truly mission critical there (90%+ was development systems that weren't in use on weekends, lots of Ultra Sparcs, PA-RISC, a couple Alphas, buncha x86 too). Room was pretty warm when I got there but at the end of the day nothing damaged, nothing lost. Power came back on probably an hour or so later.
All my jobs after that our stuff was always in a co-location facility, short of one facility in Seattle that was prone to issues(I moved my then-company out years before their big fire which caused 30+ hours of downtime), never had to worry after that.
Biting the hand that feeds IT
