Reply to post:

Picture the scene major £5M 18month long building refurb so not too shabby. The whole building need to be powered down for a weekend, but before this happened a brief outage (20 mins TOPS) was required to give the switch gear a jolly good looking at. This involved removing a panel on the switch gear and looking to check what needed to be done.

The guy PM’ing the build paid us a visit in IT. “A bit of time has become available to do the power down we’d like to do it Friday 1700h”,

“ermmmmmm hold your horses there fella, best not do that kind of job on a Friday just in case it goes tits”,

“hmmmmmmm ok how about Monday”,

“Ok so we have generators available?”

“Nope its only going to be for 20mins TOPS, no work is being done its just a jolly good looking at the switch gear, and the generators were going to be a few grand so a bit expensive”

“Ermmmmmmm so no generators!? We really need the gen’s they’re for just in case it goes tits not to run the DC, our UPS will last about 50mins but kit will need to be powered down way before that as the CRAC’s don’t run off the UPS.

“sorry to expensive and it’ll only be 20mins TOPS”

Monday comes the users have all buggered off its 1700, and the power is still on, 1705, 1710, 1715, 1720 and OFF the power goes so 20 mins late already. Go in to the DC hang on lights are still on what’s up, well they had a little bit of bother throwing the main breaker so decided to pull a breaker for the rest of the building instead so the DC is still on supply. Okayyyyyy so why was it late, well the bloke doing the work couldn’t undo the screws holding the hatch on the switch gear he need access to! FFS the switch gear is vintage mid 70’s vintage and never touched the building is within spitting distance of the sea so bugger me its corroded together who would have thunk it! Maybe someone should have gone own and given them a turn before hand, but never mind.

Right well as we’re on supply we’ll hang around until the rest of the building is back on supply “just in case” I’ll pop to the plant room for a chat and see how things are going. Off I pop to the plant room everything is going fine. Speak to our sparky, yeah everything is fine had a spot of bother with the main breaker pulled the air brake and it didn’t trip, pushed the stop button and it still didn’t go. Well it is old electro\mechanical probably ceased up a bit, give it squirt with some WD40!

Power comes back 1740 everything is fine so off we go home.

Next day 0800……………….

Just abut to sit down at my desk coat coming off, user comes in, can’t login, hmmmmmm odd. Colleague and I go to the DC next door to the office. Open door, quiet, pretty quiet, out of the 20 racks about 80 were OFF!!!!!!!!!! WTF!!!!!!!!!! Well there’s your problem right there! Frantic get things powered on then find out WTF happened. Managed to get most stuff powered up over the course of the day, lost a few disks and a couple of old Linux boxes, the main hit was the hyper-v cluster who’s storage was in a pickle and none of the VM’s would come up! Manage to get them up but that would come back to haunt us later.

Track down the sparky and the sheep’ish PM. Turns out that the main airbreak breaker they tried to trip at 1700 but didn’t trip DID actually trip at about 1800!!!!! Being electro\mechanical it was gunged up with old grease and the sear got stuck, after an hour gravity won and it tripped!! Sparky, who’s not a switch gear expert then couldn’t get it to reengaged after frantic calls to a mate of his he suggested, giving it a squirt of WD40!!!!!! Which got it working only trouble was this took over 1.5 hours way past the 50 mins runtime of the UPS! Result being lots of kit didn’t shut down properly. The hyper-v cluster one of them,although it should have shutdown it was held open by a buggy backup agent so didn’t shutdown when the UPS software told it too.

The knock on lasted weeks the hyper-v storage got corrupted and that took ages to fix, backups which backed up to a remote site want to replicate 15Tb of data across our 1Gb line so we landed up having to ship the replica from the offsite and replicate over our LAN. All for the sake of a generator hire for 1 day! Which was a few hundred £

Anyway after this total cluster feck the main power outage a month later had 4 generators, 1 for the DC 1 for the rest of the building plus backups for both!