back to article Secret weekend office bonk came within inch of killing sysadmin

Welcome once more to On-Call, in which The Register celebrates users' many, many failures by sharing your stories of being asked to clean up behind them. This week, meet "Matt" who sent us a story from 2004 when he "worked for a company who specialised in building high speed printers." Matt was the new kid on the block, so …

  1. Mycho Silver badge

    Keep stuff off the floor

    I worked on the second floor of a building that I labelled prone to flooding. People mocked me at first but every time something flooded I reminded them of that fact and they started agreeing with me. Pipes burst, aircon leaked, you name it.

    The lowest electrical equipment I was responsible for was stood on 2cm wooden blocks. The lowest UPS I was responsible for was stood on a glorified metal coffee table.

    1. This post has been deleted by its author

  2. Richard 12 Silver badge

    And sometimes the flood is concrete

    Relay room flood

    Hats off to the people who fixed that one.

    1. Olivier2553

      Re: And sometimes the flood is concrete

      We had water two meters high in the data center during the great flood of Thailand in 2011. Since then, the data center has been relocated to the upper floor and classrooms are at the lower floor: loosing chairs and tables is not a too big loss.

      1. Anonymous South African Coward Silver badge

        Re: And sometimes the flood is concrete

        How did they clean that mess up? (Concrete flood) Should be interesting.

        1. Kirk Northrop

          Re: And sometimes the flood is concrete

          They sent someone out to a nearby shop to buy sugar, and tipped it in. Sugar stops concrete from setting, apparently.

        2. BinkyTheMagicPaperclip Silver badge

          Re: And sometimes the flood is concrete

          They did exactly what I guessed they'd do - added something to slow down the concrete hardening, and shovelled it out.

          That thing was : sugar! Apparently more than 1% sugar in a concrete mix can stop it hardening almost indefinitely.

          I definitely wouldn't want to be the person that had to clean all the equipment, though..

          1. EvilDrSmith
            Headmaster

            Re: And sometimes the flood is concrete

            "...a small quantity of sugar (about 0.05 per cent of the weight of cement) will act as an acceptable retarder: the delay in setting of concrete is about 4 hours....A large quantity of sugar, say 0.2 to 1 per cent of the weight of the cement, will virtually prevent the setting of cement."

            Properties of Concrete (A M Neville). 3rd edition (which probably shows my age, somewhat).

            I have a dim and distant memory that the effects of sugar retarding the setting of concrete were first discovered when sacks previously used to transport molasses were then used to transport cement. But that might be me imagining things.

          2. Stevie Silver badge

            Re: They did exactly what I guessed they'd do

            "I definitely wouldn't want to be the person that had to clean all the equipment, though"

            *I* wouldn't want to be the one told to "fold all that sugar into the concrete with a shovel tootsweet!"

            1. Weiss_von_Nichts

              Re: They did exactly what I guessed they'd do

              Yes. My first thought was "how the hell are they going to disperse all that sugar within the wet concrete before hydraulization kicks in?". My second thought was about a lot of shovels, water buckets, sugar and the luck of not having to do it.

        3. pffut

          Re: And sometimes the flood is concrete

          Hopefully not the way Mythbusters 'cleared' their concrete truck...

      2. Anonymous Coward
        Anonymous Coward

        Re: And sometimes the flood is concrete

        data center has been relocated to the upper floor and classrooms are at the lower floor: loosing chairs and tables is not a too big loss.

        Or students, presumably

    2. Solviva

      Re: And sometimes the flood is concrete

      Can someone please decode the article's headline to English?

      "Victoria Tube line part shut hit by wet concrete flood"

      1. Phil W

        Re: And sometimes the flood is concrete

        It's only really missing a comma to make it acceptable

        "Victoria Tube line part shut, hit by wet concrete flood"

        1. This post has been deleted by its author

          1. Bernard M. Orwell Silver badge
            Coat

            Re: And sometimes the flood is concrete

            Commas are important, they can mean the difference between "At the weekend I helped my uncle, Jack, off his horse."

            And....

            Well....

            1. W4YBO

              Re: Commas are important...

              "Commas are important, they can mean the difference between "At the weekend I helped my uncle, Jack, off his horse."

              Let's eat Grandma!

              Let's eat, Grandma!

              1. Anonymous Coward
                Anonymous Coward

                Re: Commas are important...

                the difference between knowing your shit and knowing you're shit

                1. David Nash Silver badge
                  Headmaster

                  Re: Commas are important...

                  "the difference between knowing your shit and knowing you're shit"

                  Knowing the difference between commas and apostrophes is also important!

                  1. Anonymous South African Coward Silver badge

                    Re: Commas are important...

                    Obligatory angryflower strip

                    1. Jeffrey Nonken Silver badge

                      Re: Commas are important...

                      Though Bob forgot to mention possessive pronouns.

              2. TheRealRoland
                Coat

                Re: Commas are important...

                >Let's eat Grandma!

                >Let's eat, Grandma!

                "Calling Donner party of five?"

                "Over here! Now four!"

            2. Sgt_Oddball Silver badge
              Headmaster

              Re: And sometimes the flood is concrete

              It also saves lives - "Lets eat, Grandma".....

              The horror...

              1. Anonymous Coward
                Anonymous Coward

                Re: And sometimes the flood is concrete

                How big is a Let? How many did it take to finish Grandma off? Could they be encouraged to eat FCC Chairpersons? Oh, wait, apostrophes again. Darn...

            3. This post has been deleted by its author

            4. Someone Else Silver badge
              Coffee/keyboard

              @Bernard M. Orwell -- Re: And sometimes the flood is concrete

              Says it all --->

    3. Doctor Syntax Silver badge

      Re: And sometimes the flood is concrete

      It's just as well it only leaked into the relay room. Just think how much concrete they'd have poured in if they'd tried to fill the entire Victoria line.

    4. albegadeep

      Re: And sometimes the flood is concrete

      Thinking of the concrete-encased equipment:

      C3PO: "Oh. They've encased him in Carbonite. He should be quite well protected. If he survived the freezing process, that is."

    5. Anonymous Coward
      Anonymous Coward

      Re: And sometimes the flood is concrete

      "Hats off to the people who fixed that one."

      AC because I was (very loosely) involved in the recovery operation.

      When we saw the photos, the joke was that they wouldn't get it working again for a week. In fact it was up again in time for start of business the next morning. Two reasons:

      1) The sugar trick already mentioned; and

      2) Within an hour there was a van packed full of spares heading up the motorway.

      As it happened, the raised gap designed to protect against (water) flooding limited the damage, and the electronics came back up first time. The bloke in charge of supporting the recovery was later given a well-deserved award from LU.

  3. Chris King Silver badge
    Holmes

    Aircon Leaks

    One of our LAN rooms suffered an aircon leak, with water pouring out of the unit.

    Some bright spark decided that regularly emptying a bucket was tedious, and put a large plastic recycling bin under the aircon so they didn't have to visit the room so often.

    A bin with no wheels, which was rather heavy and very difficult to move when full of water.

    1. Allan George Dyer Silver badge
      Facepalm

      Re: Aircon Leaks

      Surely you have enough time to buy a garden hose and remind yourself how a siphon works while the bin fills?

      1. m0rt Silver badge

        Re: Aircon Leaks

        Would you want to start the siphon action going on water extracted from an aircon?

        "Hey, kids, fancy some legionnaires' disease?"

        1. BinkyTheMagicPaperclip Silver badge

          Re: Aircon Leaks

          Alternatively use, a bucket to extract water from the recycling bin until it's light enough to move.. It's not rocket science.

          1. Tigra 07 Silver badge

            Re: Aircon Leaks

            Alternatively replace the bin with one that has wheels...Or...Fix the leak?

        2. OldCrow
          FAIL

          Re: Aircon Leaks

          I would. By these easy steps:

          1. Submerge whole hose in water, so that no air remains in hose (bucket must be large enough).

          2. Squeeze or otherwise block one end of the hose.

          3. Pull hose into position (out the nearest window is usually best, as sinks are typically installed too high, relative to bucket).

          4. Release blockage from end of hose.

          5. Enjoy, while the bucket empties itself.

          Who primes any kind of hose by sucking on it, these days? Almost no liquid is anywhere near safe for your lungs or stomach lining. In some places, not even tapwater.

          1. Anonymous Coward
            Anonymous Coward

            Re: Aircon Leaks

            Who primes any kind of hose by sucking on it, these days? Almost no liquid is anywhere near safe for your lungs or stomach lining. In some places, not even tapwater.

            I do.

            Admittedly it's with homebrew.

            1. Anonymous Coward
              Anonymous Coward

              Re: Re: Aircon Leaks

              "Who primes any kind of hose by sucking on it, these days? Almost no liquid is anywhere near safe for your lungs or stomach lining. In some places, not even tapwater.

              I do.

              Admittedly it's with homebrew."

              If my Dad's 'Homebrew' of old is anything to judge by then :

              I refer you to the "... Stomach lining. ..." above. :)

              Self inflicted does not change anything :)

              On Topic, I don't understand why anyone would have a AirCon Unit that drains into a Container when there are quite expensive racks of kit around !!!

              Surely, it is worth the cost to 'plumb in' a proper drainage pipe to avoid the potential for flood.

              1. Mark 85 Silver badge

                Re: Aircon Leaks

                On Topic, I don't understand why anyone would have a AirCon Unit that drains into a Container when there are quite expensive racks of kit around !!!

                Surely, it is worth the cost to 'plumb in' a proper drainage pipe to avoid the potential for flood.

                I've seen the "drain" get plugged and building maintenance in all their glory can't find the plunger, a long length of wire, etc. to clean it with. So they call a "professional" who can't be there for week or two...

          2. Anonymous Coward
            Anonymous Coward

            Re: Aircon Leaks

            That also works with radiator fluid, oil... and everything else you'd find in a gas station storage room.

          3. caffeine addict Silver badge

            Re: Aircon Leaks

            Who primes any kind of hose by sucking on it, these days? Almost no liquid is anywhere near safe for your lungs or stomach lining. In some places, not even tapwater.

            I do, most weeks.

            If water filtered once by rocks is pure then water filtered thousands of times by Fluval and fish kidneys must be really pure, right? Right?

            1. CrazyOldCatMan Silver badge

              Re: Aircon Leaks

              times by Fluval and fish kidneys must be really pure, right? Right?

              Well - it hasn't killed my ye

              <Carrier lost>

            2. ibmalone Silver badge

              Re: Aircon Leaks

              Kidneys are good at filtering! Unfortunately you're on the wrong side of the filter.

          4. Prosthetic Conscience
            Coat

            Re: Aircon Leaks

            Pwah, are we still talking about aircon?

          5. CrazyOldCatMan Silver badge

            Re: Aircon Leaks

            Who primes any kind of hose by sucking on it, these days?

            [Waves hand].

            I really must replace the hand-pump on the end of the fish tank drain tube..

            (But I don't inhale)

          6. Cuddles Silver badge
            Paris Hilton

            Who primes any kind of hose by sucking on it, these days?

            See icon -->

        3. Anonymous Coward
          Anonymous Coward

          Re: Aircon Leaks

          1. Insert the entire hose in the air-con water. It is preferable the usage of a hose where both sides can be sealed by your thumb or working watertight valves.

          2. Fill the entire length of the hose with liquid. Keep both sides submerged in the tank. Avoid bubbles.

          3. Seal both sides.

          4. Displace one side of hose above suitable drain. Open the other (submerged) side. Nothing should happen, yet.

          5. Remove thumb/open valve on the drain side, but keep hose in position aimed at drain while water flows at high speed.

          Bonus: No need to contact your mucous membranes with air-con water.

          1. David Nash Silver badge
            Headmaster

            Re: Aircon Leaks

            My hoses have ends. The sides are pretty well sealed already.

          2. Karl Vegar

            Re: Aircon Leaks

            1,2 as stated *

            3 seal drain/output side

            4 Displace one side of hose above suitable drain.

            5 as stated.

            * or get fancy:

            - Stick tube down into the bin, one end first, make sure to let air escape through the other end. When there is an inch or two left above water, block the dry end. Dry end is drain end. move on from 3. If done right, all that gets wet is your thumb.

        4. This post has been deleted by its author

        5. Anonymous Coward
          Anonymous Coward

          Re: Aircon Leaks

          "Hey, kids, fancy some legionnaires' disease?"

          IIRC that infection is spread by atomised water being inhaled into the lungs. Hence why showers are often the source of the problem.

          1. Alistair Silver badge
            Windows

            Re: Aircon Leaks

            And why it rips through cruise ships in a massive hurry. Something about greywater cycling .....

            (First question you should ask when boarding)

        6. dan_in _ohio

          Re: Aircon Leaks

          Hey, I've already been there. August 1978......was in the hospital for 20 days and another week off from work. Don't EVEN kid about that desease.....just about put this guy (who was 19 at the time) in the ground.

        7. PNGuinn
          WTF?

          "Hey, kids, fancy some legionnaires' disease?"

          In actual fact, that'd be perfectly safe. Don't know about anything else living in the water though. Anyways, wer'e talking about condensate from a compressor aircon here, aren't we - Out of the cooled air, into the bucket, not a cooling tower for an evaporative cooler?

          Leigonella ingested is harmless. It's only dangerous when it's breathed in as an aerosol. - like the fine spray from a shower rose. It multiplies rapidly in warm water.

          I gather that before we knew exactly what it was and exactly how to test for it the standard test was to sniff the culture in the Petri disk. Apparently it has a characteristic odour.

          The beast can survive in water up to near 60 degrees C, which is why the greenie advice to turn your water cylinder stat down to 50 or 55 degrees to save energy was quietly dropped. (AFAIK no one from the greenie camp publicised that they'd dropped another Bo**ock, so there's probably lots of water cylinders running dangerously cool.)

          My worry is that many condensing boiler flues can run way below 60 degrees, and if the system is not used to generate stored hot water, or the boiler stat’s wrong, or the system is a combi with hot water on demand set to well below 60 degrees (which should be safe, since any legionella in the cold main doesn’t get stored at nice cosy breeding temperatures) that flue could never get to sterilising temperatures.

          The water produced when methane or propane burns, which you see as a plume of condensed water droplets from the flue of a condensing boiler, "has a ph similar to tomato juice" and has strict disposal rules - you aren't supposed to run it untreated to a fresh water drain, feed it into a cast iron system that does not have other water regularly running through it or let it contact concrete or brickwork. (The condensate can dissolve more than half an inch of mortar / pointing in a year.)

          I wonder what ph is necessary to kill leigonella, and whether the bug can evolve to cope.

          I've asked the question at training and assessment courses but never had a satisfactory answer. Sand, meet head.

      2. ricardian

        Re: Aircon Leaks

        Another hazard of siphoning

        http://www.nydailynews.com/news/world/australian-thieves-siphon-sewage-tank-petrol-article-1.2504298

        1. jake Silver badge

          Re: Aircon Leaks

          I hear a variation on this story every couple years. I always question it. Why? Well, I don't know about you, but I've pulled the cap on both fuel tanks and the vent/cleanouts of blackwater tanks. They look absolutely nothing like each other (first clue), they obviously have completely different purposes (second clue) ... and of course their smell is a dead giveaway.

    2. Chris King Silver badge

      Re: Aircon Leaks

      For those of you offering solutions, the facilities people came up with the bright idea of using the bin because it happened over a weekend, and soon after the "LAN room flooded and on fire simultaneously" incident I've mentioned in a previous On Call.

      Problem is, they forgot to tell us until the bin overflowed and water started seeping under the door.

      1. Anonymous Coward
        Anonymous Coward

        Re: Aircon Leaks

        How about this one... leaky drain pans in the ceiling, part of the HVAC system. Maybe the drain hoses were clogged. So install large plastic totes (bins) to catch the drips. Installed on the ceiling tiles. Then forget about them. Ticking 'time bombs'.

        Eventually, weeks later, it was raining totes in the office area, each brimming with 50+L of icy cold filthy water. Oh, another. And another...

    3. Mark 85 Silver badge

      @Chris King -- Re: Aircon Leaks

      For some reason, the old Rocky & Bullwinkle comes to mind: "Bail! Bail! Bail!".

      1. Herby Silver badge

        Re: @Chris King -- Aircon Leaks

        Of course, Boris said to Natasha "Stroke, stroke, stroke".

  4. malle-herbert Silver badge
    FAIL

    Near death experience ?

    How about installing a bunch of servers in a building that's still (partly) under construction, walking out the fire-escape door for a quick smoke while waiting for the install to finish, only to find out the fire-escape ladder wasn't installed yet...

    I almost dropped 18 feet down onto the concrete pavement below...

    1. Anonymous Coward
      Anonymous Coward

      Re: Near death experience ?

      Smoking kills, didn't they tell you?

    2. Tempest 3K
      Coat

      Re: Near death experience ?

      Well, they do say smoking will kill you.....

    3. 0laf Silver badge

      Re: Near death experience ?

      Wasn't there some poor chap that walked through door that had no walkway on the other side and fell into a vat of boiling something or other?

      1. BinkyTheMagicPaperclip Silver badge

        Re: Near death experience ?

        It's ok, Batman has him under control.

        1. Chris King Silver badge

          Re: Near death experience ?

          When you need Batman to enforce your Health & Safety regs, maybe you need to look at your recruitment procedures...

      2. Richard 12 Silver badge

        Re: Near death experience ?

        The "Juliet" doors over a theatre stage have caused many deaths.

        They're supposed to be locked properly when there's nothing the other side, yet somehow people still forget to do it.

        Laziness kills.

    4. Chris King Silver badge

      Re: Near death experience ?

      "Watch that first step, it's a doozy"

      1. Anonymous Coward
        Anonymous Coward

        Re: Near death experience ?

        "Watch that first step, it's a doozy"

        Is that a Disc World quote - possibly "Going Postal"?

        1. lglethal Silver badge
          Go

          Re: Near death experience ?

          @AC I'm assuming you're not trying to be funny - it's actually a quote from Groundhog Day...

          1. Anonymous Coward
            Anonymous Coward

            Re: Near death experience ?

            Ah - thanks - it had seemed to ring a bell about that door in Lord Vetinari's office.

            1. Anonymous Coward
              Anonymous Coward

              Re: Near death experience ?

              A lot of lift accidents are caused this way. Lobby doors open but there's no lift car.

              1. TRT Silver badge

                Re: Near death experience ?

                BOFH

              2. DropBear Silver badge

                Re: Near death experience ?

                I was always curious how exactly those can possibly happen; our elevator definitely isn't any sort of fancy, but the doors are literally, physically interlocked to the cabin with a small latch you can see turn and unlock the door each time the cabin arrives. So okay, these are "open by hand" doors not the modern self-opening kind, but still... (why) aren't all doors unable to open without a cabin present?

  5. Evil Auditor Silver badge
    WTF?

    Maybe it's a too early a morning for my brain to fully process this On-Call. Why "near-death experience" - because of water and electricity? (No RCDs in the data centre?) And how's the bonking relevant and why does Matt know about it?

    1. imanidiot Silver badge

      As I read it they're saying the air-con was left switched off by the installing contractor, but it was turned on to make said bonking more enjoyable/less sweaty. It was then left running when done with said activity. Thus causing the flooding. I presume the bonking was found out because there was a thorough investigation into WHO turned on the air-con unit. Thus access logs were inspected and security camera footage reviewed. One or both involved parties then fessed up to what they were doing when asked.

      1. Evil Auditor Silver badge

        Thanks! That makes sense. I probably suffer from EMIS - early morning illiteracy syndrome.

        1. WonkoTheSane
          Thumb Up

          "Thanks! That makes sense. I probably suffer from EMIS - early morning illiteracy syndrome."

          Nah. You're just undercaffeinated.

          1. Evil Auditor Silver badge
            Happy

            Nah. You're just undercaffeinated.

            Same cause. Same effect.

        2. Rob Daglish

          Any number of us who have worked for the NHS have suffered from EMIS - nearly as bad as Crapita SIMS...

          1. David Roberts Silver badge

            EMIS

            One thing it did have from a patient perspective was a very good user interface.

            Very good for searching through test results and stuff.

    2. ssharwood

      Yes. Water plus electricity. And the bonking news emerged over time

    3. John Brown (no body) Silver badge

      ...although thanking Dell for having a 1" clearance on the bottom of the racks seems a bit over the top. Every rack I've ever seen has at least a couple of inches of "dead space" before you get up to the kit.

    4. TWB

      RCDs

      Where I have worked for the last 25+ years we never had RCDs (or RCBOs) on power circuits for apparatus rooms as we called them - except for test benches. IIRC they were deemed too risky from the point of view of losing service - and also a few mA leakage is a pass, but once you have a handful of kit all running on the same RCD (or RCBO) the combined leakage could cause a trip and not all our kit was dual PSU.

      I presume we passed all the regs or else the 600+ bays should not have been live.

  6. DECUS4ever

    Live Steam

    My former boss and mentor told the story of fun in the old days.

    He was a BOFH for a paper mill which, if you’ve never been in one, involves very large machines, toxic chemicals and energy in the form of steam. He was in charge of the computer systems and had a computer room in the middle of the mill.

    During a construction project, they decided the best place to run a high pressure steam line was directly through the computer room. There was no need for that extra five feet of pipe to go around the room, “we can save money by going right through.” So they did.

    Everything was fine until one evening when the steam yearned to be free and the pipe burst INSIDE the computer room. Since it was evening and everyone had gone home for the day, no one noticed. That was until the computers went down. The pager went off at 2:00am and after a quick trip to the mill, he walked up to the computer room where the computers were on display. The window was opaque and white. Hot and white. Hot, white and there was a screaming sound coming from inside the computer room.

    He was smart enough to NOT open the door but called the plant maintenance team who reluctantly turned off the steam line. After everything cooled down and they could enter the computer room, it was obviously a total loss. They scrambled to replace all the kit while the plant maintenance guys spent a larger chunk of money for enough pipe to route around the room.

    1. jake Silver badge

      Re: Live Steam

      And in those days, it came to pass that incompetent staff were summarily fired. And low, there was much rejoicing throughout the land, for they were difficult to work with and made everybody's life a living hell whenever possible.

      Well, that's the way it should work, anyway. Gut feeling is they got a raise for figuring out how to re-route the pipe after the fact ...

      1. Chris King Silver badge

        Re: Live Steam

        "And in those days, it came to pass that incompetent staff were summarily fired. And low, there was much rejoicing throughout the land, for they were difficult to work with and made everybody's life a living hell whenever possible".

        These days, they'd be more likely to sack the IT guy for insisting that the machine room be located where it was. THEN give the other guy a raise for re-routing the pipe. THEN give themselves a raise for solving the problem of an inconveniently-placed machine room.

    2. Anonymous Coward
      Anonymous Coward

      Re: and there was a screaming sound coming from inside the computer room.

      For a minute there, I thought it was going to be a much nastier accident than one with just computers...

      1. DuchessofDukeStreet

        Re: and there was a screaming sound coming from inside the computer room.

        Where is Simon?

    3. Voyna i Mor Silver badge

      Re: Live Steam

      I am unsurprised. I worked somewhere where a large plastic hydrogen vent line was routed above a suspended ceiling. Of course, the hydrogen vent system was completely foolproof and could never ignite. Ever. And the company that installed the approved system was so good at it that they went bust.

      And the pipe was PVC.

      The hydrogen containing system that could never ignite duly ignited and the entire large PVC pipe exploded, sending large fragments of PVC through the insubstantial suspended ceiling. Some of them stuck quite deep in the floor below. Mercifully nobody was hurt.

      It also turned out that the person responsible for the installation had been made redundant some time before so there was nobody to blame.

      Oh, and did I mention the hydrogen was radioactive?

      1. phuzz Silver badge

        Re: Live Steam

        "Oh, and did I mention the hydrogen was radioactive?"

        Where the hell were you working where you had pipes full of radioactive hydrogen?

        1. kain preacher Silver badge

          Re: Live Steam

          Clearly he is one of homers co workers.

      2. Alien8n Silver badge

        Re: Live Steam

        One company I used to work for had an unwritten rule. In the event of an actual real fire the fire assembly points were to be completely ignored and everyone should just get as far away from the building as possible. Such as the next village. Something to do with the large tank of hydrogen below the factory...

      3. Alistair Silver badge
        Windows

        Re: Live Steam

        wait.

        Radioactive hydrogen.

        PVC?

        There were no engineers with any real training on that project.

        1. uccsoundman

          Re: Live Steam

          "Real engineers?" They cost money! It's a lot easier to get the local handyman to do it. He'll get it done cheap and won't be asking any troublesome questions, nor bring up those pesky rules and permits that other people seem to care about.

          1. Someone Else Silver badge
            Trollface

            Re: Live Steam

            "Real engineers?" They cost money! It's a lot easier to get the local handyman to do it. He'll get it done cheap and won't be asking any troublesome questions, nor bring up those pesky rules and permits that other people seem to care about.

            Or, you could get Scott Pruitt to do it...

        2. Doctor Syntax Silver badge

          Re: Live Steam

          "Radioactive hydrogen.

          PVC?"

          Well, to be fair, tritium's a very weak beta emitter. PVC would stop it easily.

          What's that? Leakage? Possibility of rupture?

          1. Voyna i Mor Silver badge

            Re: Live Steam

            "Well, to be fair, tritium's a very weak beta emitter. PVC would stop it easily."

            I'm glad someone gets it. I left that as an exercise for the reader. I'm not going to say much more because I don't want to doxx myself - except to observe that a lot of places in the developed world used to use tritium, and our own beloved government used to think the best thing to do with the vast amounts of it they created while making bombs was to put it in the North Sea, you can guess where.

            But eventually the Irish complained and it was realised that actually 5keV electrons released inside fish were probably quite bad for them, so in a sudden reversal of policy it was decided that the best thing to do with the stuff was to release it into the atmosphere.

            Before anybody got worried about it, I visited a plant once where tritium was being used in a manufacturing process. In a corner of a shopfloor there was a lump of vacuum equipment and a small gas bottle of tritium-protium mixture connected to it, along with a fairly basic standard welding type extract.

            1. Doctor Syntax Silver badge

              Re: Live Steam

              "a small gas bottle of tritium-protium mixture connected to it, along with a fairly basic standard welding type extract."

              And the result - tritiated water.

            2. Doctor Syntax Silver badge

              Re: Live Steam

              "I'm glad someone gets it."

              It helps to have worked in a carbon dating lab. Carbon 14 is also a weak beta emitter, a bit different in energy but it meant the same counting technology would see both if the material going into the counter was a hydrocarbon.

              Tritium in the hydrogen supply was a concern. On the methane counting system there was a supply of "old" hydrogen as the source of the hydrogen in the methane. The later benzene counting system was a different matter. The hydrogen came from water (carbon and lithium heated to produce lithium carbide, add large excess of water to produce acetylene which is then catalytically converted to benzene). The best that could be done was distil the water to reduce the tritium content. That just left the radon problem. When your water supply comes from a granite catchment you have radon in your tap water.

      4. CrazyOldCatMan Silver badge

        Re: Live Steam

        Oh, and did I mention the hydrogen was radioactive?

        My last place, we had a wonderful mix of low-flashpoint solvents and alpha-radiation sources on site, in fairly close proximity to each other.

        Which is why our fire alarms directly alerted the nearest fire station and we had to pre-clear our fire alarm tests with them. And why we got annualy refresher training in fire extinguisher usage and our Elf'n'safety person allowed us to play^W test with almost expired extinguishers..

        (Although they did sometimes use them as ready-alert tests by forgetting to tell the crew on duty..)

      5. kain preacher Silver badge

        Re: Live Steam

        Voyna i Mor

        How they hell was that install even legal or passed inspections? Were I lived at any wires that or pipes that were hidden by false tiles had to be fire rated .

        1. Voyna i Mor Silver badge

          Re: Live Steam

          "How they hell was that install even legal or passed inspections?"

          Don't ask me, I didn't install it.

          I will just mention in passing that in Berkshire, not at all far from inhabited areas, are tanks of highly radioactive liquids. At the time I was aware of them people did not know what they contained. Some of them had to be cooled by water passing through internal pipes. Some of them might be leaking.

          Such is the insanity of nuclear deterrence.

          Military secrecy and the needs of national defence cover a multitude of practices that would probably never be allowed outside.

    4. werdsmith Silver badge

      Re: Live Steam

      They put a steam pipe through a computer room ?

      Even if it doesn't leak, it seems an odd idea to put a hot pipe through a room that is being kept artificially cool using air con.

      1. Anonymous Coward
        Anonymous Coward

        Re: Live Steam

        Toilets blocked - and caustic solutions didn't clear it. Pushing a drain rod hard eventually gave a satisfying feeling of the blockage released. Flush worked - job done.

        The comms engineer was called out later on. All the modem racks were covered with a foul mess dripping from the false ceiling. In the ceiling void was a 100mm toilet drainpipe neatly separated at a rubber seal join.

    5. Anonymous Coward
      Anonymous Coward

      Re: Live Steam - *seriously* dangerous

      I worked at a large chemical plant where steam is often used for heating - it allows heat to be provided to places that would object rather explosively to open flame.

      To transport such steam, it is sent overheated (read: well above 100ºC, it's basically a gas) through pipes at high pressure. They started a re-insolation project of all those pipes along walkways when one of the pipes developed a tiny leak and someone was pretty much cut in half by walking through that jet of invisible steam - the plume of that leak was only visible some 10m further.

      I don't envy the emergency staff at these sort of plants - they get to see some rather nasty stuff.

      1. el_oscuro

        Re: Live Steam - *seriously* dangerous

        I would agree. My dad used to work at a place that had steam pipes like that, and when inspecting them, they had a simple test.

        They would walk along side the pipe with a broom stick. Leaks were identified when the broomstick was sawed in half.

        1. The Oncoming Scorn Silver badge
          Flame

          Re: Live Steam - *seriously* dangerous

          A revist from one of mine last year.

          Just came back from a training weekend down south in Orange County (same airport as Harrison Ford "Schmucked up his aircraft landing") & tale of a high pressure steam leakage on a submarine was told to me by one of the attendees (quite a few ex-navy attendees actually).

          When a pin prick leak occurs in a high pressure steam pipe, it's visible as a "cloud" in the general vicinity, when that's spotted all in the immediate area freeze & slowly look about for something within arms reach to use as a guide like a spanner or wrench as the North Americans have it, then move it in the general area of the cloud to find where it intersects with the leak, the results are pretty obvious once it does so.

          A newbie submariner, forgot all his training on his trip out, spotted the leak & immediately went to point it out to his shipmates (he was also in an area of the boat he wasn't really supposed to be in) inadvertently using his arm as the detection device.

          His forearm connected with the high pressure steam jet & a Anakin Skywalker type injury occurred, as the jet sliced through the flesh & bone severing without hope of re-attachment (Both sides of the wound cauterized instantly).

  7. jake Silver badge

    WTF?

    One of the first rules of AC work is "Always have proper drainage for the evaporator." ... It's not an afterthought, or something you have to go back to the shop to get. It is designed into each individual system, just like the ducting and power. It's just something you do, like breathing. It's in the estimate, and/or the bid. The parts for any particular install are on the truck with everything else. As a contractor, if my AC guy didn't have this prominently listed in his/her paperwork, I'd find a new AC guy pronto.

    Also, I've got to ask ... How many cubic feet was that one system conditioning? It'd take an awful lot of air to wring out enough water to deposit an inch on a typical data center floor ... more if it's a raised floor, of course. This is especially true seeing as modern AC systems recycle interior air[0], meaning once the water is condensed out, it's out.

    [0] Yes, I know, "fresh" air is introduced in larger systems, but that happens at a heat exchanger, typically on the roof where the noise of the air handling system won't annoy anybody inside. That condensation is typically handled by the roof, same as rain water ...

    1. lglethal Silver badge
      Go

      Re: WTF?

      You've forgotten that when the beancounters are in control - cheapest bid wins. It doesnt matter if it's incomplete, dangerous, or whatever. It costs less!

      I can only imagine that something along those lines was involved in this incident.

      1. Dr Dan Holdsworth Silver badge

        Re: WTF?

        Ahhh beancounters, where would we be without a bunch of cost-obsessed maniacs to get in the way?

        Probably in the place of a certain northern English university that I'd better not name (not that I work there, but still).

        This university had a sports complex with a flat roof. Not the smartest option ever designed for use in England, and like every flat roof ever, this one developed a leak. The University Estates people were called, and failed to fix the leak to nobody's surprise since University Estates tend to have the collective IQ of yeast.

        So, the job was put out to tender, and the Powers That Be unfortunately included the beancounters in the decision. The problem was that there were only a couple of contractors locally big enough and reliable enough to do the job, and they were well aware of this and tended to cartel-like pricing. A third group also tendered, at a much lower cost; let us call them "Del-boy Trotter & co". The beancounters saw only the cost, and insisted on this bunch being hired to do the job.

        Thus it was a week or two later that what looked like the contents of several scrap yards, plus exhibits from the Museum of Dodgy Roofers turned up. Shortly after Security tried to turn them away fearing a gypsy invasion, it was determined that these were the new roofers. Quite quickly, they got to work and started messing about with boiling tar, roofing felt and endless cups of tea, and a fortnight or so later pronounced the job done and asked for payment. In cash.

        This was refused until after a couple of good rainstorms showed that yes, the leaks were gone, so Del-boy and hist motley crew were paid and duly disappeared, never to return.

        Several thunderstorms later the leak re-appeared. The loft space of the sports complex was investigated, and it was found that the large plastic bin placed under the largest leak by the roofers was now full of water and overflowing.

        One of the usual suspects for roofing was duly engaged, after it was discovered to nobody's surprise save for the beancounters' that Del-boy & co had vanished.

        To this day the beancounters of this university have their every decision questioned with the words "But remember the sports hall!"

        1. CrazyOldCatMan Silver badge

          Re: WTF?

          Ahhh beancounters, where would we be without a bunch of cost-obsessed maniacs to get in the way?

          In one of my previous places, we'd have a server room with a UPS that didn't go in to bypass by the time we'd added about 60% of the kit (we were combining two server rooms and, on the technical side, we'd specced up the UPS to be able to carry twice the combined load because we knew there would be expansion in the new, larger room). Despite telling us that "sufficient budget" had been allocated for our design, it was discovered that the UPS *actually* bought by the purchasing team was only about 40% of the rating required "because of budget" and didn't have sufficient capacity for the existing combined load, let alone any future additions. And the UPS stack was surrounded by fixed racks and so we would have had to delay the project (and cost more money) to fix it. We ran on bypass for quite a long time.

          Same with the aircon. Original plan was to have two separate units (so one could be shut down for service), each capable of handling total load+50%. What we actually ended up with was two units, each of which could only carry 60% of total load. Which meant that, if we wanted to do service on one of the units, we had to shut down 40-50% of the servers..

          Beancounters - can't live with them and, for some reason, you go to jail for introducing them to the pit of quicklime..

        2. John Brown (no body) Silver badge

          Re: WTF?

          "But remember the sports hall!"

          The problem with those sorts of "lessons learned" is they are never documented. As people move on, the "lessons learned" go with them.

    2. Voyna i Mor Silver badge

      Re: WTF?

      Jake, it may be a rule but who enforces them?

      Some years ago my company moved into a managed office suite in a new block. The heating didn't work terribly well, and after a year a consulting HVAC engineer came round to investigate. I got talking to him and he told me this:

      The building had been designed with a condensing boiler system. It was designed for 55C, but the pipes were running at 82C.

      He had talked to the facilities manager about this and had been told that you couldn't run the system at the design temperature because if you did, the boiler room floor flooded.

      Sure enough, the water vents from the boilers had been left to drip on the floor by installers who had presumably not understood them, and not understood what the nearby drains were for.

      The system already needed repair due to deposits blocking thermostatic valves, all for the lack of a few £ worth of plastic pipe and a facilities manager who was totally devoid of clue.

  8. Ross Luker

    I've had an almost identical experience - spent a week walking into a server room to find a soaking wet floor, with no explanation why. Eventually discover, hidden behind a partition, the drain for the aircon in the adjacent post room - the aircon that the new cleaner not just turns off, but cuts the power to every night (no, I don't know why the aircon unit had a massive cutoff switch next to it) and dumps the water onto the server room floor...

    1. Anonymous Coward
      Anonymous Coward

      The mainframe operators came in one Monday morning. They noticed that there was a thin film of moisture on the console table. They then pressed the master "power on" button to sequence up the mainframe and all its peripherals.....

      It took the engineers three days to dry all the circuit boards with hair dryers.

      1. Anonymous Coward
        Anonymous Coward

        I'm just glad the Halon system where I used to work was never triggered. There were only two breathing masks available, so if it went off it would have been a case of who can remember where the masks were, find them through the clouds of Halon and put them on before you suffocated. Oh, and if anyone else was trying to do the same, well, let's just say survival is not a team sport under such circumstances.

        I always made sure I was in there with either

        a) people who would have idea where the masks were or

        b) smaller than myself so I could get the mask

        So yes, very glad it was never triggered. Anon for obvious reasons :)

        1. tony trolle

          Halon

          As a student on work placement many many moons ago, a Halon system was being tested.

          The "dry" run out tho the emergency door/hatch proved the 10 second delay in the Halcon flood was still going to be fine after some new kit was installed.

          But I asked why does it smell if its a dry run.

          Arr.

          Was told its just the bit of gas left in the pipes venting.

          But I smelt it while the 10 second alarm was going off.

          Arrrrrrr.

          Yes wired wrong. no delay been like that for 6 years.

        2. CrazyOldCatMan Silver badge

          b) smaller than myself so I could get the mask

          When running away from the dragon, you don't have to outrun the dragon, just the party halfling..

          (My wife, who tends to play halflings in Pathfinder, has a t-shirt with that message)..

        3. Doctor Syntax Silver badge

          "I'm just glad the Halon system where I used to work was never triggered."

          Working late in the lab one evening. I'm not sure just what gas was being used but I heard a distinct bang and hiss from downstairs, probably the vehicle inspection bay.

  9. Dodgy Geezer Silver badge

    You get much the same issue on boats - where the pipework is designed on the Olympic system (the bloke who gets there first runs his 1/4" pipe straight, and the big 18" low pressure steam conduit has to bend round it).

    For my sins I started work in a military command and control complex, where keeping the computers and signals equipment up under nuclear war conditions was a business requirement. So we had full replacements and engineering support on site 24hrs (or rather UNDER site, since we were well underground).

    The worst outage we had was due to a stationery delivery, when the trolley clipped one of the aircon pipes. You can't get rid of heat easily when you are underground, so that was critical for the mainframes. And of course we ended up with an airlock.....which lowered the defensive capability of the country for nearly 36 hours....

  10. Anonymous South African Coward Silver badge

    Also had a leak in the server room once.

    Now, the servers' located in the basement. Which is Not A Good Idea.

    However, the buidling is on a hillside, which allows the western part of the basement to be open. So flooding will not be a problem.

    Right?

    Wrong.

    Seems that the builders neglected to check on the horizontal level of the floor (which is now the server room). I've found out only after the fact that it is NOT truly horizontal, but tend to incline towards the western side, which has a wall. The door is to the east, where you enter and exit.

    It should be pretty waterproof, but we had to cut a hole in the wall for a fiber-optic (dang word, I never know if it's fibre or fiber, but anyway) conduit. It was sealed properly and everything.

    For the first year things went well. Second year, the conduit piping carrying the fiber inside started to crack due to vehicular traffic. Third year we had a nice water leak outside the premises... and yup, water entered the server room.

    Luckily the critical servers was high enough to keep their toesies dry. And I'm glad I did not get electrocuted - I waded into that mess barefoot. Water got warmer at a point, then I noticed that a kettle plug had immersed itself in the water - and I skedaddled out of there at a good pace, and refused to enter again until all the water had been pumped out.

    It woke manglement up, and the company had to spend some $$$ in getting proper drainage installed to keep water away from the server room.

    So far we've had a couple of rainstorms etc, but the server room remained properly dry.

    Only time will tell if we'll have a repeat performance or not. If it does happen again, the servers will be relocated somewhere else... which'll be a major PITA.

    And for the record, I will not try this experiment again, with a live electrical connection submerged...

    1. Aladdin Sane Silver badge

      Re: fibre or fiber

      Depends which side of the pond you prefer.

      1. okand

        Re: fibre or fiber

        It depends on if you're talking about networking or storage.

        1. CrazyOldCatMan Silver badge

          Re: fibre or fiber

          talking about networking or storage

          Or roughage in your diet..

      2. Voyna i Mor Silver badge

        Re: fibre or fiber

        "Depends which side of the pond you prefer."

        South Africa (see nick of GPP) is on the right, English-speaking side so it's fibre.

    2. Spanners Silver badge
      Terminator

      Re: fibre or fiber

      For speakers of English, the spelling is "fibre".

      "Fiber" is close to a childish word for someone who tells untruths.

      1. albegadeep

        Re: fibre or fiber

        For speakers of English west of the pond, it's "fiber" - for networking, storage, or diet. Though "fibber" is the same on both sides.

        1. Lilolefrostback

          Re: fibre or fiber

          "For speakers of English west of the pond, it's "fiber" - for networking, storage, or diet. Though "fibber" is the same on both sides."

          Not strictly correct. In Canada, we speak the Queen's English.

          1. psychonaut

            Re: fibre or fiber

            Apart from the quasi-french bits, shirley?

    3. Grant Fromage

      The lady must have been hot and dare I say ...moist, for the aircon to generate that much condensate ( apologies, I had to, no-one else has yet ).

      1. psychonaut

        i believe the term you are after is "frothing at the minge"

  11. peterm3
    Go

    one inch of water is nothing

    I once worked in a college with several computer rooms. Due to a blocked surface drain in a courtyard, a basement computer room started to fill up like a bathtup through the airbricks in the wall.

    The computers were on high benches and the students sat on bar stools. As long as their feet didn't get wet, they were happy with a good 40 cm of water on the floor!

    The computers were high and dry so to speak, so no problems :-)

  12. Christian Berger Silver badge

    It's actually not _that_ dangerous

    Electricity isn't like a swarm of animals, it doesn't "move" in water. If something like that happens, the bulk of the current flows very close to the points where the life conductors touch the water. In fact if you submerge a typical power supply, you're probably fine even close by. Even if you have some relevant current going somewhere the RCD will trip way before there's any danger.

    Safetly precautions are there to keep you away from danger as much as sanely possible. However even without them not everything is highly dangerous.

    BTW There's a great German short movie called "Staplerfahrer Klaus" (availiable on Youtube even with english subtitles) However that's perhaps not really SFW.

    1. Chris King Silver badge

      Re: It's actually not _that_ dangerous

      BTW There's a great German short movie called "Staplerfahrer Klaus" (availiable on Youtube even with english subtitles) However that's perhaps not really SFW.

      I've posted it here a couple of times, and nobody has whinged before - but if you're squeamish, there's a lot of fake blood and fake severed appendages towards the end.

      1. Christian Berger Silver badge

        Re: It's actually not _that_ dangerous

        The German Bluray release actually includes an interview with the producer in which he tells about safety specialists who, in all seriousness, see this as a safety instruction film. It is actually shown as a safety instruction film in many companies.

        1. imanidiot Silver badge

          Re: It's actually not _that_ dangerous

          Let's be honest though: "Staplerfahrer Klaus" probably contains all the rules typically applied in smaller companies and a clear demonstration of WHY those rules are there. Plus it's a heck of a lot more entertaining than the typical "occupational hazard" instruction material.

          1. DropBear Silver badge

            Re: It's actually not _that_ dangerous

            Hey - we clearly all still remember it: that means it's undeniably an efficient "safety rules" vehicle...

          2. Mark 85 Silver badge

            Re: It's actually not _that_ dangerous

            So the equivalent Stateside would be the "Blood Runs Red on the Highways" driver education films?

        2. Richard 12 Silver badge

          Re: It's actually not _that_ dangerous

          Yep. Used in the last four safety inductions I endured.

  13. Bob Wheeler

    Basement computer room

    I worked briefly at one customer site that had their computer room in the deep basement. At ground level, just in front of the building was a ornamental pond. A pond that needed to have a running water feed into it was it leaked badly.

    Guess where the water ended up....

    1. Dr Dan Holdsworth Silver badge

      Re: Basement computer room

      A very famous computer department in Manchester used to have a rooftop garden above their main machine room, complete with a pond. No prizes for guessing what used to happen on a regular basis; the pond leaked.

      The only highlight of this was that every year for a couple of decades, one lone duck would build a nest in that rooftop garden, the fledging chicks getting a free ride out into more suitable habitat when the time came; photo-opportunities with the Computer Science duck were a great honour.

      These days, the machine room is a big office. The pond is a skylight.

      It still leaks.

    2. collinsl

      Re: Basement computer room

      On similar lines, in BBC TV Centre in London the old VT (Video Tape) rooms were in the basement directly underneath the fountain in the courtyard.

      After a few months of occupation the fountain was turned off because the sound of it echoing off the walls of the circular courtyard made some of the female staff members keep visiting the loo every 10 minutes.

      This is not the point of the story, however. The point is that the fountain was also found to leak. When the Queen visited the site at some point in the 1970s the management decided to turn the fountain on to impress her, but in order to minimise the risk from the leak they decided to only turn it on whilst the Queen was outside. The Queen was seen to be approaching, so the fountain was turned on. She met the management, went inside and it was turned off. But then they had to turn it back on because the Duke of Edinburgh turned up in a land rover a few minutes later.

      The upshot of the story is a large part of the expensive VT equipment underneath the fountain was ruined!

  14. Jimboom

    I have also felt the pain of a wet server room.

    Went in to help one of the electritians wire in some new telephones. Noticed a pool of water underneath one of the server stacks. The worrying bit was when I mentioned it to said spark he just laughed and said it didn't surprise him with that building.

    Though in the end we discovered it was due to a 1st level support chap who couldn't be bothered to take the computers for cleaning away from he building when he air blasted the dust out of them. Which also was where the intake for said air conditioning was. Dust blocked pipe which in turn lead to the overflow water just dripping onto the floor.

    Fun times...

  15. Chris Jasper

    Turned up for work one day about, wow 20 years ago, to find no-one in the IT Office, one of the officey types said they had all sprinted down to the basement shortly before.

    Turns out there had been a major flood and the basement server room was 8 inches deep with black water that was only made blacker by the lack of lighting. Lots of power cables trailing in the water which I naively agreed to help lift out and place on higher platforms.

    In hindsight, I was pretty stupid and thoughtless back then.

  16. werdsmith Silver badge

    I imagine that we all have an air-con in server room water story.

    I have one, so bizarre that I can't tell it on here because my colleagues at the time will know it's about them and will not be thanking me.

    1. Aladdin Sane Silver badge

      That's what the Anonymous Coward option is for.

      1. DropBear Silver badge

        And how does that help when everyone involved knows exactly who the person telling the story is...?

  17. Anonymous Coward
    Anonymous Coward

    IIRC A new Data Centre was built next to a river that was subject to tidal surges. There were electric pumps in the surrounding yard in case a 50 year flood occurred.

    It was found that the mains electricity to the pumps failed in a severe flood.

    1. Bowlers

      Computer room leak(ers)

      Back in the seventies (sigh) the IBM mainframe computer room was 3/4 below street height. The windows were knee height on the street so easily accessible for viewing in or worse. Not 24hr operation then so when a couple of drunks decided to break in there was no water but a few "leaks" discovered next morning. Whole computer room moved at the next major HW upgrade.

  18. Anonymous Coward
    Anonymous Coward

    I worked...

    ...at an NHS trust many years ago where their cowboy fix for the aircon in one of the server rooms was to let it run down made up channels stuck to the wall into a bucket. It obviously relied on people checking the bucket wasn't full. I reported it once, got ignored.

    When you work in the NHS, you see why its such a mess.

  19. Anonymous Coward
    Anonymous Coward

    Similar Experience

    I once led a team updated the IT systems for a museum in London, somewhat sugar related. One of the sites on the south bank was converted from an old industrial site and the main computer room was in the basement. I got a call one morning from the server installation team saying there was a network issue so duly headed to site. Anyway long story short I wandered into the computer room, one with nice shiny metal floor tiles and agreed that yes for some reason the new network switches installed a few days before were all down. So wondering about the cabling I decided to lift a floor tile, so there I stood with a floor tile in my hand looking at water lapping up against the bottom of the tiles. I looked up and noticed a nice big 3 phase distribution board with all the cables exiting at the bottom and going under the floor. At that point I calmly suggested the team all leave the room (duty of care and all that) then got the hell out myself. It turned out there was a big water tank at ground level outside for the fire system for the buildings old usage and some bright spark decided to empty as it was no longer needed without first checking which way the ground sloped. Luckily it was Friday and we could head off early for the pub based weekly wash up meeting, my hands were only shaking a bit as I slurped my first pint!

  20. Anonymous Coward
    Anonymous Coward

    I worked at a site which had an incident where a series of problems resulted in a flood, failure of aircon and overheating. Such was the design of the building - open floor-to-roof in the large reception area - that the flood water evaporated then condensed up at roof level and came back down.

    Overheard from an engineer on the phone to head office about the problem: "I don't know. I can't see from here - I had to go outside to get out of the rain"

  21. Anonymous Coward
    Anonymous Coward

    Frozen

    Ah, the joys of an air conditioning system, low on coolant, in a humid environment. The evaporator tends to cause the humidity in the air to freeze into ice. Imagine my surprise that Saturday that I wandered into the machine room, and found a 300 pound block of ice hanging down from the ceiling. Oh, that's gonna hurt!

    1. Chris King Silver badge

      Re: Frozen

      I had a similar experience, but on a smaller scale. Nowhere near a 300lb block, but enough to seriously injure someone standing under it if it gave way.

      The aircon engineers were called in, took a look at the kit on the roof, and handed me a report which consisted of five words...

      "SEAGULLS HAVE EATEN THE PIPES !!!"

      Well, that explained the lack of coolant.

      The unit in the machine room was defrosted, emergency portable aircon brought in, and non-essential systems powered down to reduce the load on the portable kit until the pipework could be renewed.

  22. Doctor Syntax Silver badge

    Then there was the Great Melton Street Flood. A high pressure main in Melton St next to Euston Station ruptured, washing out a huge crater in the road and flooding the underground car park.

    It didn't affect our computer rooms. It found something much bigger and electrical: the Tube system. We had an operator on his way in to start a shift. He told us the train stopped. The lights went out. And then the water started coming through the floor....

  23. This post has been deleted by its author

  24. Anonymous Coward
    Anonymous Coward

    Bad AC design

    I once worked in an office which roasted in the summer, because there was only one small air vent for the 5-cube office. They eventually put in a "portable" AC unit (about 3' wide, 3' deep, and 5' high), and ran the exhaust hot air line up to the drop ceiling. Nowhere beyond that - it just pushed the hot air above the drop ceiling, causing the (still warm) air to work its way back down into the room. It essentially became a GIANT dehumidifier. (Which was still an improvement, though it was loud and we had to take turns emptying the condensate bucket.)

  25. sisk Silver badge

    Could be worse

    There are far worse things that can be found on the floor than water after that kind of "inspection". Granted such things tend not to be an inch deep, but still.

  26. JJKing Bronze badge
    WTF?

    I hope these were supposed to be humourous.

    which lowered the defensive capability of the country for nearly 36 hours....

    I'm sure that lowered defensive capability was more than covered by the 19,999 other "courier" deliverable buckets of instant sunshine.

    I was also astounded by the wheelie bin and garden hose solutions to the H20 disposal issue from the air conditioner problems. I am hoping they were ALL posted with humour intended behind them because why else would one suggest a couple of $$ (or ££) of plastic and a manual water control system to protect tens of thousands of $$ (or tens of thousands ££) of delicate and mission critical electronic systems that keep the company running and pay packets flowing to those employed below the level of HR, Bean Counters and other Management tosser.

    1. Yet Another Anonymous coward Silver badge

      Re: I hope these were supposed to be humourous.

      why ... suggest a couple of $$ (or ££) of plastic and a manual water control system to protect tens of thousands of $$ of delicate and mission critical electronic systems

      Because B&Q / HomeDepot don't offer the same sort of perks to customers as companies selling mission critical electronic systems

    2. Doctor Syntax Silver badge

      Re: I hope these were supposed to be humourous.

      "I am hoping they were ALL posted with humour intended behind them because why else would one suggest a couple of $$ (or ££) of plastic and a manual water control system to protect tens of thousands of $$ (or tens of thousands ££) of delicate and mission critical electronic systems"

      At the pay-grade of those who do things like that it's not delicate, mission critical electronic systems, it's just stuff. The water just needs catching and emptying it's the next shift's problem.

      I've seen a delivery driver want to drop a heavy, delicate and expensive piece of electromechanical kit off the back of his van because there wasn't a tail lift and they'd not arranged for a fork-lift. To him it was just another crate. There were enough of us to insist he didn't and to slide it down a plank.

    3. Baldrickk Silver badge

      Re: I hope these were supposed to be humourous.

      Sometimes low-tech is the best tech.

      The less complicated it is, the less there is to go wrong.

  27. Herby Silver badge

    On AC and condensate

    Many moons ago, I was traveling (for work actually) with a work companion in of all places Tyler Texas, where the motel rooms had occupancy sensors on the thermostats to "economize". Of course this was totally unacceptable as we wanted to return from our outings to a reasonably cool room rather than sweat it out while the AC attempted to cool things down in glacial time.

    Our solution was to open up the thermostat and "bypass" (I believe it involved a screwdriver adjustment) the motion sensor. This meant that we slept comfortably at night and came back to a nicely cool room after the days outing. When we did come back, we noticed that the condensate which drained from the AC onto the exterior walkway looked like a river for our room, but the other rooms had barely a trickle. We just smiled and carried on.

    Yes, Tyler Texas in the summer is both hot and humid.

    No need to be part of the gunpowder plot, this was back in the 80's and the company is long gone.

  28. Chemical Bob
    Facepalm

    The Story of Eustace

    Once upon a time, a long time ago, at a small company I used to work for, was a lad named Eustace. Eustace's parents were druggies so he grew up without a moral compass, good role models or a clue. One day his step-mother, taking pity on him, prevailed upon the owner to hire him so he could establish the beginning of a work history, see how normal people behave and maybe get a leg up in life.

    Eustace proved quite pedestrian at his work - light industrial and janitorial duties and kept asking me if I wanted him to clean out my office (my "office" was the only room on premises that had a door that could be locked for it was both the R&D lab and server room).I kept telling him not to bother for clutter is my particular idiom.

    One Monday the owner told me I might want a new desk chair and that Eustace was fired over the weekend. He then told me how these two things were related.

    On Saturday Eustace made himself useful and cleaned up what passed for my office. A bit later the owner showed up and immediately noticed 2 things:

    1. His boombox was missing

    2. Soft music was playing in the R&D/server room.

    Of course, he investigated.

    What he discovered in the de-cluttered room was not just soft music playing but cardboard had been taped over the window and candles were burning on the workbench, apparently to set a romantic(?) mood as Eustace about to engage in coitus with his girlfriend on my desk chair. Now I must point out that this was not the girlfriend he was living with and made a baby with and who was going to school so the three of them could have a decent life together. This was his *other* girlfriend, the hot one.

    I laughed my ass off upon hearing this. And then, figuring that Eustace lacked both the presence of mind and decency to put at least a sheet of newspaper on the chair, I dragged it around the back of the building to let mold and fungus clean out whatever might be soiling it.

  29. Griffo

    Worse Liquid Story

    One of my colleagues has a far far far worse story about liquids in a server room.

    This server room had been installed in a hospital, and had been in that location for several years. My old mate had just recently installed a couple of new blade centers and storage, and one Monday morning unlocked the door to the room looking forward to a day of playing with shiny new toys.

    Unfortunately, over the weekend, there had been a bit of a plumbing leak. What he at the time didn't know, was that in the ceiling above this server room, some bright spark had run a sewerage pipe. From the intensive care unit. Connected to "bed pan disposal unit". This pipe had for some reason popped off. Which meant that for all weekend, all the super disgusting sick person excreta had been dripping through the server room ceiling all over this brand new shiny kit.

    Needless to say, while specialists cleaners were called in to clean the room, all the kit was replaced under an insurance claim. Now that's a day you'd never forget.

    1. The Oncoming Scorn Silver badge
      Coat

      Re: Worse Liquid Story

      Another revist...(might have been on El Reg initially) about a shit\urine flooded comms\electrical room in a basement.

      Basically......

      The building designer\plumbing contractor had routed all the black water into a straight pipe for the entire building (not sure if a office or tower block) with a 90 degree bend at the very bottom.

      The inertial mass of free-falling turds with gallons of falling water from the top floors, had over time weakened the joint\pipe & the thing finally gave way. So the pipes contents instead of carrying on the journey to the sewers, the cellar comms room had taken on a new role as a septic tank.

      Something not quite similar happened when a "restroom" flooded at a slaughterhouse I used to work at.

  30. Anonymous Coward
    Anonymous Coward

    Anon as this is probably still there

    Previous life as Theatre Electrician - theatre in Scotland with a concrete encased basement electrics room containing three phase incomers and main lighting dimmer racks. Local Fire officer required the water sprinkler system remain functional even when it was pointed out water and serious amounts of electricity didn't mix. Even a hint of fire alarms and lx team was not seen for dust.

    Same theatre that due to the foibles of hydro power any matinee show in winter meant that at 5pm when the kettles and ovens went on for tea voltage could drop to around 190 and the lighting control boards would just give up - you could be left with every light in the rig live or complete blackout.

  31. Anonymous Coward
    Anonymous Coward

    RE. Re. Worse Liquid Story

    Not saying where, but had a very similar near-disaster caused by water somehow getting backed-up in long gutters on flat roof over several months.

    When someone finally got around to unblocking one of them the resulting torrent of filth ended up going through three sets (!) of ceiling tiles and 2.5" of water in the room like a tidal wave.

    Evidently the volume of water was underestimated, of course the servers nearly drowned but fortunately the only casualties were a few mains leads and other expendable items.

    Shared because it could happen anywhere and MRI scanners/X-ray machines *really really* hate water immersion, replacement cost would have been £4M+ and heads would have rolled for sure.

    AC

  32. Criggie
    Alert

    Me too

    Yep had that brown momemt,

    Due to various tectonic events, we had another company working out of our first-floor boardroom for a year. They had a couple desktops, a nice APC 1.7 kVA UPS on the floor (foreshadow) and all was well.

    An aftershock knocked over a water cooler, which was plumbed into the mains, not one of those bottle ones. It piddled away for a day and a half, releasing a lot of water whcih ran downhill. The boardroom was at the low end and was ~50mm deep in water.

    On Monday I walked in to see the damage, and became aware their stuff was still powered. So I turn off the wall socket.

    >CLONK< goes the UPS, as they do.

    The UPS was probably half under the filthy water and had just switched to batteries. I'm aware how a UPS is built inside, with the batteries below and the board along the top, so any waves could short it. Gingerly reach over and hold the power button on the UPS till it powers off and all the PCs go down.

    By how much did I avoid a shock? No idea, and I really don't want to know.

    TL;DR, earthquakes suck, AND they have aftermaths (afterbaths?) that go on for many years.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019