2 posts • joined Saturday 7th July 2007 16:55 GMT
Deluge at "Stargate Command"
Curiously we had a similar situation arise last November when a colleague and I returned from a morning Coffee break to "Stargate Command" (as we dubbed our Mission Control) to find what looked like heavy rainfall pounding down fron the ceiling straight onto ALL the Servers / UPS's / TFTs.
In true heroic fashion that would have put the real SG-1 team to shame, my horrified colleague and myself threw ourselves into the room and without much ado, (or indeed concern for our own Health and Safety) commenced emergency shutdown protocols of all the live Servers. Water cascading off our backs we hunched over each Server in turn, barking commands and instructions back and forth as we hurriedly took all the aggregates offline. Within a minute we were totally soaked. By the time we got all the Servers wheeled out of Stargate Command into the adjacent corridor, we looked like we'd just been fished out of the Sea! Bedraggled Turkeys. Everything, albeit sloshed with water, somehow survived!!
It was only after we had wheeled everything out and we stood in the Server room in almost an inch of water, it suddenly dawned on us that if any of that cascading water had got into the UPS's or the Server PSU's and hit a live circuit, we would have received embarrassing "Darwin Awards" for our troubles.
The final ignominy was when a member of Senior Management turned up after the rescue and ordered us to have it all up and running in 2 hours. (We did NOT comply of course - it all needed drying out and checking).
The next day, special thanks for averting total disaster went to the guy from an unrelated dept who went to the local DIY store to buy a valve for the leaking water pipe. We got "Foxtrot Alpha".
To this day, 95% of the staff have NO idea what really happened.
My colleague and I later theorised that in a parallel universe somewhere, we actually "bought it" that day, Toasted to a turn when water got in a UPS which fell off a trolley onto the floor.
On our office wall, a little poster commemorates the day:
"We risked our lives to save xxxxx's Network Infrastructure and didn't even get a lousy T-shirt".
Next time, we definitely walk away.