BOFH: Computer room deluge

Brilliant

The BOFH just keeps getting better and better.

Been there...

The best one was when the Fire Brigade insisted on pulling the plug on our UPS before they went in the building to deal with the flood. Bloody facilities manager said OK!

What??

Underneath a pond??

What genius came up with that design...

you gotta get a drill mate!! :D

Underground

There's an underground stream running under our building... it has its own nice little crawl space, which happens to be the only under-floor means of cabling from one end of the building to the other. We've sent many cussing engineers down there in waders with one end of a cable... it would be so tempting to route the power through to the particularly whiney ones. I'm sure we'd get away scot-free too - I think the stream only comes out once it arrives in Southampton docks...

BWHAHAHAHA

'I think the stream only comes out once it arrives in Southampton docks...'

I'm sure I know where you are. Give us some warning first so I can be not on duty that day :)

Dr Who vs BOFH

Any chance of the next season of Dr Who matching wits with BOFH?

I've had the waders out

In the computer room at a certain Portsmouth Hospital, the engineers had contrived to put the overflow pipe from the air handling units at an angle - upwards.

I may be known a miracle worker, but even I can't get water to flow up-hill.

Water flowing uphill

You can do it with a big enough fan.

I recall during the U.S. East-coast blackout, one hospital (near a river) had assigned the sump-pump for the below-ground generator room to the "non-essential" circuits. Worked for an hour or so...

This is one of the episodes...

that looks completly realistic and an every day event. At least i've seen problems like this one. Our last case was a frozen air conditioner pipe which burst, so the ac shut it down and as soon as the ice melted the whole fluid content of the ac system left the pipe through the crack and the pipe was ducted above the main server room. The cause of the problem turned to be an aged thermal resitor in one of the internal units.

Bright sparks

Have a look at this Google Earth link:

http://bbs.keyhole.com/ubb/showthreaded.php/Number/737492

The original design for the grassy bit in the middle was for an attractive water feature. Then someone pointed out that the server room was directly underneath, and that having several tons of water sitting on top of some very large supercomputers was probably not such a good idea...

Brilliant Broadcasting Corporation

The main control centre in one of their offices - the one which is responsible for all the local tv and radio broadcasts - is on the first floor.

The canteen - with the sinks, dishwashers etc - is on the second.

Guess what has happened the week I am there on a contract?

Deluge at "Stargate Command"

Most entertaining!

Curiously we had a similar situation arise last November when a colleague and I returned from a morning Coffee break to "Stargate Command" (as we dubbed our Mission Control) to find what looked like heavy rainfall pounding down fron the ceiling straight onto ALL the Servers / UPS's / TFTs.

In true heroic fashion that would have put the real SG-1 team to shame, my horrified colleague and myself threw ourselves into the room and without much ado, (or indeed concern for our own Health and Safety) commenced emergency shutdown protocols of all the live Servers. Water cascading off our backs we hunched over each Server in turn, barking commands and instructions back and forth as we hurriedly took all the aggregates offline. Within a minute we were totally soaked. By the time we got all the Servers wheeled out of Stargate Command into the adjacent corridor, we looked like we'd just been fished out of the Sea! Bedraggled Turkeys. Everything, albeit sloshed with water, somehow survived!!

It was only after we had wheeled everything out and we stood in the Server room in almost an inch of water, it suddenly dawned on us that if any of that cascading water had got into the UPS's or the Server PSU's and hit a live circuit, we would have received embarrassing "Darwin Awards" for our troubles.

The final ignominy was when a member of Senior Management turned up after the rescue and ordered us to have it all up and running in 2 hours. (We did NOT comply of course - it all needed drying out and checking).

The next day, special thanks for averting total disaster went to the guy from an unrelated dept who went to the local DIY store to buy a valve for the leaking water pipe. We got "Foxtrot Alpha".

To this day, 95% of the staff have NO idea what really happened.

My colleague and I later theorised that in a parallel universe somewhere, we actually "bought it" that day, Toasted to a turn when water got in a UPS which fell off a trolley onto the floor.

On our office wall, a little poster commemorates the day:

"We risked our lives to save xxxxx's Network Infrastructure and didn't even get a lousy T-shirt".

Next time, we definitely walk away.

Evil for the Computer Technician, Volume 2

This is at least chapter one. Have to start with a zinger straight off the bat.

Now, all I need is a building, basement sub-station and a bean counter to drill at the critical point... ;)

Reminds me of....

A certain defence contractor in Basildon, that had a UPS on a computer in a first floor computer room catch fire one weekend. After it had burnt its way through the wooden floor, it landed in the paint store below!

That cunning bit of layout foresight resulted in pretty much a whole building going up in smoke!

The Life Aquatic with Me.

Last year we were getting a new roof put in our Jr. High Library. And the roofers forgot to tighten down some of the bolts. Well it rained and rained. Some where around 4 inches and there was about a foot of water in the library because of seals on the doors to prevent flooding. Anyways 26 computers got fired and alot of books. But most of these books were 20 years old so no biggy. I know none of it is essential but they had to redo the card catalog system.

Nothing so disastrous here, but...

Before we retired our old ND5700 machine, and could switch off half the cooling in our computer room, we had almost yearly floods in the summer.

What happened was that the humidity in the air condensed on the cooling system and dripped onto the floor. (And for some weird reason, when the 'modern' system was installed, the contractor never put in any 'sump pan' or drain)

The servers were in no danger, of course, as they were on the raised roof, and the UPS was in the basement(the computer room is on the third floor)

We knew of the problem, but really didn't care.

We just mopped it up every time it seeped out under the office walls.

(Typical thin office walls as used in office buildings.)

Dust, though...

I do know that at the local firestation a PC used to monitor offsite systems(a tunnel or something) once caught fire because of the buildup of dust inside it.

(They told me about it when I was there, installing another monitoring PC)

Sump pumps

Where I used to work (an industrial facility), there was a bit of remodeling in the basement, due to an expansion. So, in this basement, was installed all sorts of controlling and power cable for the facility above, now much more, and everything was made water-tight (IP 68 insulation means you could go scuba diving in that basement, should it fill up.)

Some contractor didn't notice that tiny detail when he installed another sump pump in there, its control panel was meant to be installed upstairs (together with the first one). The genius installed the panel down there, the manual override was in that panel... Well, level sensors fail, too... do the math.

The day it failed, the whole basement flooded, shorting out only the pump panel. The short in that panel showed up in the mains supervisory system, so we could shutdown everything safely, if we deemed necessary, at least. When the first pump (correctly installed and working) finally emptied the basement, the panel was deep roasted (440V shorting out, hmm), and finally people understood why they shouldn't have saved 100 feet worth of cabling for the upstairs installation of the panel in the first place.

It would be nice if someone attempted to dive in that water to start the pump on manual...

Just what is a risk?

At one of our offices a risk assessment showed up a slight risk of flooding of the ground floor. It hadn't flooded in that area in recorded history, but I suppose the assessing types have to put something down to justify their tedious little lives.

Management got the vapours and ordered the computer suite moved to the first floor. A new bit of first floor was duly fitted out and all the kit moved. On the roof of the carefully-planned new location was the water tank for the office. Time and corrosion did their bit for poetic justice shortly afterwards.

We still (years later) have the location. The ground floor still hasn't flooded.

TeeCee

Risk reduction

Or the chip designer that had multiple redundant server rooms on 4 floors - it's just a shame that they were one above the other and when the roof leaked, the water just ran down the cable ducts from floor to floor and took out all four.

Maybe next time, one in each corner of the building?

Out of the frying pan....

My employer had a computer room on the top floor of a building. Directly underneath a huge water tank on the roof. When it leaked, engineers were hastily despatched to the computer room to cover all the cabinets with plastic sheeting.

This computer room was then replaced with two data centres. One is located in the basement of building where flooding is prevented by a pump that is in constant operation (except when it breaks down). Thankfully, the two data centres are mirrored so when the pump breaks down, the data centre can be shut down and we can operate via the other data centre. Well, sort of.

The other data centre is built on the flood plain of a major river. Not only is it on the flood plain, it is also very close to the river.

Maybe we should all become data centre consultants and earn stupid money for stupid designs.

£20k a pop.

I used to work on a nuclear power plant (won't say which one) shortly after construction, so it was decided that the pre-designed living quarters next to the Control Room should be expanded, taking some space from the Process Computer Room, a.k.a "the real power plant operator". A small part of that computer room, with raised floor, AC and everything, was then converted in an kitchen, and the computers layout changed to fit the tighter space. Remember, this is no ordinary kit, or low reliability operation here, it is a "nuclear kit" after all, so it took a while to move everything without shutting the Plant down... only 9 months...

No problems with the moving, but somebody forgot that the Fire Prevention system should also be changed, now it would use FM200, a gas that replaces Halon because it is breathable, but it costs some dosh. Of course, a lightly burnt toast isn't reason to discharge $20k worth of gas in a place where it used to be a computer responsible for averting a possible nuclear meltdown.

So, as per new approved design, we blocked the optical smoke detectors on that room, until they were changed to a temperature-delta type. Up to this day everybody is scared of even boiling water for coffee on that kitchen...

But during those 9 months, it rained like Hell, and the basement of the Control Building flooded 3 times... thank the heavens everything down there was water-tight.

Basement Store Blowout

I work in a Optical store that cuts its own lenses on-site so we use alot of water for cooling and cleaning. When the lab was first built the drain (very large 6inches for possible equipement breakage and leaks) was located next to the equipment. However, being the true genius of construction, the contractor gave the floor a slight slope to help drain any excess water - only problem was it sloped AWAY from the drain (didnt find out right away).

A few months later we learned this when one of our small water pipes blew (same contractor) and flooded the lab. We then heard from the store downstairs that there was a constant rain in their utility room (computers, phone system,etc). When the contractor was brought back by the building management an independent inspector found so many problems he ran out of paper to list them. That contractor is no longer in business...

What a headache and mess. Took us weeks to dry and air out the lab to prevent mold.

Been there (sort of)

Last flood we had was from a science lab pipe bursting during the night the floor above the computer rooms. Fortunately the only room not to get wet housed the servers. But it made a mess of 3 IT suites, still have computers not directly soaked at the time failing from water damage a year later!!