back to article Tech team trapped in data centre as hypoxic gas flooded in. Again

Welcome once more to “Who, me?”, in which we help Reg readers to unburden themselves by telling anonymised stories of big, bad, mistakes. This week meet “Luke” who told us that about 20 years ago he was in charge of disaster recovery “for an insurance company in San Francisco” and had “a series of ‘oh shit’ days.” Lue’s …

Page:

  1. Waseem Alkurdi

    Hasn't halon been banned or something in the '90s?

    And isn't the BOFH's site the only one still going with halon?

    1. Giovani Tapini

      Re: Hasn't halon been banned or something in the '90s?

      Banned for new installations, not to the point where it has to be ripped out of existing....

      So I imagine a fair bit lying around!

      1. Korev Silver badge

        Re: Hasn't halon been banned or something in the '90s?

        Banned for new installations, not to the point where it has to be ripped out of existing....

        So I imagine a fair bit lying around!

        Aren't they the BOFH's former colleagues "lying" around? You know, sales reps, beancounters, this week's boss?

        1. Doctor Syntax Silver badge

          Re: Hasn't halon been banned or something in the '90s?

          "Aren't they the BOFH's former colleagues "lying" around?"

          And is there still enough Halon about to replenish after the BOFH's accidental (every single one of them) discharges?

        2. el_oscuro

          Re: Hasn't halon been banned or something in the '90s?

          I think I have seen the BOFH. Is this her?

          http://dilbert.com/strip/2010-02-06

    2. Anonymous South African Coward Silver badge
      Pint

      Re: Hasn't halon been banned or something in the '90s?

      Upvoted for the BOFH reference.

      And have one (sans laxatives)

    3. I ain't Spartacus Gold badge

      Re: Hasn't halon been banned or something in the '90s?

      You would have thought that the BOfH would have used up all his halon by now, and have been forced to recharge his systems with something a bit more ozone friendly. Though he's probably got a source down the pub for getting hold of illegal halon. Everyone likes a drop of the hard stuff, now and then...

      1. Peter2 Silver badge

        Re: Hasn't halon been banned or something in the '90s?

        Like a CO2 flood, which is probably more lethal?

        1. Destroy All Monsters Silver badge

          Re: Hasn't halon been banned or something in the '90s?

          Like a CO2 flood, which is probably more lethal?

          You can't breath any (that's the point) so it's the difference between NATO 5.56 and Russian 5.45 x 39.

          1. Peter2 Silver badge

            Re: Hasn't halon been banned or something in the '90s?

            You can't breath any [Halon] (that's the point) so it's the difference between NATO 5.56 and Russian 5.45 x 39.

            Actually, no. Halon does not put fires out by displacing oxygen. It has some kind of funky chemical reaction which reduces the heat from the fire, which puts it out. IIRC, you can breathe the stuff for (IIRC) something like 15 minutes before you start getting a headache from it.

          2. Anonymous Coward
            Anonymous Coward

            Re: Hasn't halon been banned or something in the '90s?

            the difference between NATO 5.56 and Russian 5.45 x 39

            206.99? Of course if you consider the units things get a bit messy.

            1. Destroy All Monsters Silver badge
              Trollface

              Re: Hasn't halon been banned or something in the '90s?

              206.99? Of course if you consider the units things get a bit messy.

              You must be from a country with gun control.

              1. This post has been deleted by its author

        2. I ain't Spartacus Gold badge

          Re: Hasn't halon been banned or something in the '90s?

          CO2 is heavier than air, and is designed to displace the oxygen and starve the fire. So if you're a giraffe you should still find breathable air above the CO2 layer.

          Halon is designed to bond with all the free oxygen it can find, and so should mean that there's nothing to breath anywhere - either for you or the fire.

          So CO2 sort of might be safter, ish, maybe. Except that you can become unconsious when you're breathing air with more than 4% of it, if memory serves. It's definitely safer to be down the pub...

          1. I ain't Spartacus Gold badge

            Re: Hasn't halon been banned or something in the '90s?

            Ooopsie. Sorry about that. Well you learn something new every day - or unlearn something you'd remembered wrong. Just looked it up and I'd completely misremembered what halon does. It doesn't bind with all the oxygen around, it stops the fire's chemical reactions from working properly. So it's definitely safer than CO2, and can be breathed in low concentrations.

            It was apparently first used in extinguishers and fire grenades in the first decade of the 20th century. I'd read mention of those before and sort of vaguely wondered what was in them.

            1. Peter2 Silver badge

              Re: Hasn't halon been banned or something in the '90s?

              No probs. Kind of easy to assume the stuff is lethal given that the BOFH uses it as a mass murder weapon!

              Kudo's for checking and realising (and admitting) you were wrong as well. Increasingly rare these days, unfortunately :)

            2. SImon Hobson Silver badge

              Re: Hasn't halon been banned or something in the '90s?

              It doesn't bind with all the oxygen around, it stops the fire's chemical reactions from working properly.

              Correct. However, I once had a very interesting conversation with an expert in fire detection & suppression systems. He pointed out that while halon (or one of the modern replacements) does indeed stop the fire like that, if the source of heat is still there then you have big problems.

              So (for example) you have a cable that's overloaded and would ordinarily be burning. Most modern materials used in electrical and IT stuff is "self extinguishing" - which means that it won't carry on burning once the energy source (heat) is removed. Incidentally, you can easily test this for (say) a bit of cable by holding it in a flame till it starts burning and then remove it from the flame - it should go out on it's own.

              So now you have some PVC (where the C stands for Chloride, containing chlorine) still breaking down in the heat, but the free chlorine radicals cannot combine with the oxygen in the air because of the halon. Hmm, chlorine free radicals being stirred around (lots of fans remember) electrical equipment = lots of corrosion of exposed metalwork, especially connector contacts. So your kit isn't destroyed by the fire, but is rendered useless by the chlorine radicals attacking it.

              The "self extinguishing" bit is a clue - if you remove the power then the fire stops of it's own accord. Hence by far the most effective (and cheapest) method is to just "switch it off", aka an EPO switch (linked to a fire detection system where one's fitted).

              Of course, just "yanking the power" from a server room does have it's own downsides !

    4. dmacleo

      Re: Hasn't halon been banned or something in the '90s?

      when I worked in airline business (in USA) until december 2007 halon was still used in cargo compartment. and as far as I can tell was still (in 2017) used there.

      at same time handheld bottles and the like were NOT halon due to the regs.

  2. Phil Kingston Silver badge

    Seen it in plenty of places but never seen it used. I seem to remember someone telling me that if it goes off, all the dust that gets blown around is enough to kill a lot of kit. So you may as well have let it burn.

    One day they'll let me hit the button.

    1. Phil W

      " the dust that gets blown around is enough to kill a lot of kit"

      Not heard that one, but one known problem is that the sound pressure/vibrations generated by the gas exiting the nossles at high speed is sufficient to wreck mechanical hard drives.

      Ironically it's potentially possible to loose more data to the fire suppression going off, that if the fire had just been left to burn out.

      1. Anonymous Coward
        Anonymous Coward

        Ironically it's potentially possible to loose more data to the fire suppression going off, that if the fire had just been left to burn out.

        ========================================================================

        I doubt that.

        I was once visiting a company whose business was selling computer systems engineered to run as long as your redundant power and cooling systems were up, including during hardware maintenance and replacement. My company was considering buying some of those systems.

        Every part of the system was duplicated, at the least.

        When I was there, they were down, and their fully independent mirrored drive systems were down, with major data loss.

        One of the drives had caught fire... the smoke killed the rest of them.

      2. ~chrisw
        Coffee/keyboard

        Can confirm. Observed a recent event where an erroneous Argon release caused mass spinny disk crashes due to 200 PSI worth of acoustic shock. This, and the restoration work, sounded painful. If only someone had used the hold-off button...

    2. LDS Silver badge

      "if it goes off, all the dust that gets blown around is enough to kill a lot of kit."

      Depends. Some types of Halon (there are several) are clean - they leave no residues - that became a requirement for their replacements as well exactly because cleaning or replacing expensive kits, and slow down the recovery, is not nice...

      1. Alan Brown Silver badge

        Re: "if it goes off, all the dust that gets blown around is enough to kill a lot of kit."

        "Some types of Halon (there are several) are clean"

        Halon generally works by chemically combining with whatever's burning to extinguish the flames.

        It works well, anything near the fire has to be replaced. What you absolutely do not want in a data room is a dry powder extinguisher - something which got demonstrated to the H&S wonk who insisted on having one in there by letting one off in his office during a weekend when he wasn't around.

        1. Anonymous Coward
          Anonymous Coward

          "anything near the fire has to be replaced"

          Whatever you use to extinguish a fire, anything near enough to the flames and heat has a good chance to need replacement - it's the effect on whatever is away enough from the flames and heat that makes a difference, and the procedure needed to purge the room and restore it to working condition after the damaged items (and the fire cause, of course) are removed.

    3. Alan Brown Silver badge

      " all the dust that gets blown around is enough to kill a lot of kit."

      Yup. So it's worthwhile _insisting_ on a full deep clean of the entire room behind the ceiling tiles and under the raised floor before commissioning (and preferably before the cabling team moves in), then go over it again before actually trundling the racks in.

      In a small (8+5 metre) server room I pulled out several vacuum cleaner loads of nasty grit, destroying the machine in the process. Better that, than that the expensive hardware.

      The same room has a number of sticky floor pads and dust traps which are periodically cleaned out, as are the bottom of racks.

      When was the last time YOUR server room was cleaned?

      1. Chloe Cresswell

        Is this where people look up the time Cambridge Uni had a halon dump in their mainframe room?

        And also found out at the same time there was plumbers grease in the halon vent pipes.. the hard way.

      2. Captain Scarlet Silver badge

        "When was the last time YOUR server room was cleaned?"

        Erm 2009, the henry hoover found a dead rodent which meant lots of paperwork, never again

        1. Destroy All Monsters Silver badge

          Erm 2009, the henry hoover found a dead rodent which meant lots of paperwork, never again

          Shoot, Shovel and Shut Up, but without the Shoot part is the recommendation here.

          1. Anonymous Coward
            Anonymous Coward

            RE: Shoot, Shovel and Shut Up

            Reminds me of finding a dead bird in the ceiling void of a prep test kitchen at a food manufacturer.

            Was doing a contract running CCTV cable, found a small sparrow(?) up there dry as a husk. Told the supervisor on duty and let them deal with it.

            Anon because I don't want to ruffle any feathers...

        2. J.G.Harston Silver badge

          Hoover don't make Henries, they make Hoovers. Numatic make Henries.

          1. phuzz Silver badge

            Numatic make hoovers, but not Hoovers. One's a trade name, the other is a generic name, in the UK at least.

            I'm pretty sure the name 'Henry' was chosen to be alliterative with 'hoover' anyway.

      3. Tony Gathercole ...
        Happy

        re - When was the last time YOUR server room was cleaned?

        Reminds me of the occasion (circa 1982 or 83) when Trent Poly in Nottingham relocated its Computer Room into a converted college Library over the summer vaccation. Kit being moved included the single-decker bus sized DECSYSTEM-20 timesharing mainframe, with associated free standing disk and tape drives, all of which required a specialised County-level DEC LCG Field Service Engineer to supervise.

        Anyway, the relocation project was planned in detail and the lifting gear etc planned to turn up with the heavy gang with a limited amount of time and, of course, on a fairly tight budget. Time factors also quite critical as the old room had to be handed over to have asbestos cleared out.

        So, said LCG engineer turns up day before the big move and comes in to check out the new computer room. Looking around he's generally satisfied with the room - and then he tells us to lift up some floor tiles to check the under-floor void. Suddently he turns round to the Head of Department and sundry others with him and says: "No way is that machine moving in here with that floor!". He bends down and wipes his finger across the real floor and it comes up covered in dust to demonstrate what he means.

        Turns out that when the room was convered, the existing parquet flooring was sanded flat, cleaned and vaccumed and then left in that state.

        Rapid discussions came up with a plan that the floor could be sealed by varnishing it. Shortly after that four or five of us are piling into a car and haring off to a nearby DIY store and buying up almost all of its stock of large tins of floor varnish and brushes. In the mean time others in the department are busy starting to lift all the floor tiles.

        By now its mid-afternoon and M-day starts first thing the next morning.

        Teams of members of the department work through the night - with the bulk of it being done by the head of Application Services - and the whole job is just about finished by dawn the next morning. Mind you, anybody spending too long in the room would be high as a kite for the next few days with the vapours given off by the drying varnish.

        So, actually a happy ending to the story. All systems were moved during the planned window and we were able to get the service up in time to prepare for the new term. That was until several weeks into running in the new centre, when it was discovered that the "wrong type of air conditioning had been installed" and we had to have a kettle boiling in the room at times to increase the humidity to a level that worked for the kit ... but that's a story for another time!

    4. W4YBO

      "...all the dust that gets blown around..."

      I didn't quite see it, but I felt and heard the explosion of a flour mill (actually the grain elevator, but the mill disappeared when the elevator did) around three miles away. I hope no one allows that much dust build-up in their data center, but I've seen one that had to be close.

  3. Pen-y-gors Silver badge

    New version of Cluedo

    I think an IT-themed Cluedo could be fun

    "The PFY with the Halon in the ops-room"

    1. Symon Silver badge
      Coat

      Re: New version of Cluedo

      Maybe Col. Mustard Gas?

    2. Anonymous South African Coward Silver badge

      Re: New version of Cluedo

      And I think you're on to something.

      What about designing and selling an actual BOFH/PFY game a la Cluedo?

      The Boss in the Filing room with the Faulty CattleProd.

      Sharon in the Company Canteen with the Listeriosis Viennas. :)

      The Janitor in the Stairway with the PFY's patented slippery stuff.

      1. Martin

        Re: New version of Cluedo

        And presumably it is the Beancounter who is found dead in the cellar.

        1. Doctor Syntax Silver badge

          Re: New version of Cluedo

          "the Beancounter who is found dead in the cellar."

          Isn't the point of a demise in the cellar that the body's never found, it just disappears in a roll of carpet to a landfill north of Leeds?

          1. Anonymous Coward
            Anonymous Coward

            Re: New version of Cluedo

            > to a landfill north of Leeds?

            What, Scotland?

            :-)

          2. bpfh Bronze badge

            Re: New version of Cluedo

            Aha, the joker card - no body no murderer :)

      2. Allan George Dyer Silver badge

        Re: New version of Cluedo

        But it is always the BOFH/PFY that does it...

        OTOH, the game could be to discover who gets the blame...

        1. Anonymous South African Coward Silver badge

          Re: New version of Cluedo

          But it is always the BOFH/PFY that does it...

          OTOH, the game could be to discover who gets the blame...

          Something like that...

          1. Mycho Silver badge

            Re: New version of Cluedo

            I figured you were describing a game of identify the body.

      3. Alan Brown Silver badge

        Re: New version of Cluedo

        "The Janitor in the Stairway with the PFY's patented slippery stuff."

        And PFY shouting "boo"

    3. Qwelak

      Re: New version of Cluedo

      Already been done ....

      http://www.theregister.co.uk/2005/11/18/bofh_2005_episode_31/

    4. David 132 Silver badge
      Thumb Up

      Re: New version of Cluedo

      Now I have a vision of Tim Curry as the BOFH. Or maybe Madeline Kahn.

      Cracking film, by the way. Watch it if you haven't already. You're in for a treat.

      1. Swarthy Silver badge

        Re: New version of Cluedo

        Tim Curry as the BOFH, Madeline Kahn as Gina.

        Or a reasonable facsimile thereof, it has been over 30 years.

  4. Oengus Silver badge

    Oops

    At the computer centre I was working in during the 80's in the days of 1/2" reel tapes. We had 2 halon systems; a small system protecting the tape library and a much larger system protecting the remainder of the centre.

    The tape library was only "manned" during the day shift with the tape librarian preparing the trolleys of tapes for the overnight processing and filling the "Off-Site" storage boxes.

    One day (without notice or reason) the Halon in the tape library triggered. This was in the middle of the day when the Shift Supervisor was in the library working with the librarian checking the boxes to be sent "Off-Site". When the Halon went off the librarian and Shift Supervisor raced out of the room. To this day I swear that the librarian had the Shift Supervisor's foot prints up the middle of her back. I have never seen "Big Al" (the Shift Supervisor) move so fast.

    1. Alan Brown Silver badge

      Re: Oops

      "One day (without notice or reason) the Halon in the tape library triggered. "

      And THAT is my biggest worry about the bloody things.

      Honestly. I've been in mountaintop "temporary" installations where the halon system was a tank bolted to the ceiling with a mechanical thermal trip (like the heat detectors in fire aoarm systems) and still had to deal with the aftermath of the bloody thing having gone off for no apparent reason with noone around for twenty miles whilst the buillding was 4 foot deep in snow.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019