back to article BOFH: The Great Patch Mismatch

"It's just a minor ROM patch." the service engineer bleats "It'll only take five minutes." "Yeah... Nah," the PFY says. "It's minor - just addresses a couple of memory leaks and and cookie issues in the web interface." "Yeah. Nah," I repeat. "It's just the interface - the UPS will be completely unaffected!" "Nope," the …

COMMENTS

This topic is closed for new posts.
  1. Z-Eden
    Pint

    Needs more KZZZEEERRRTTT!

    Ah Friday. BOFHday...

    1. The First Dave
      Headmaster

      Don't think we really needed the footnote... *

      1. Quinch
        Holmes

        I concur. If you need the footnote, you won't get the joke.

        1. Steve Evans

          A slippery slope...

          If you're going to explain what a Halon system is, where are you planning to stop?

          Relays? UPS? Patch panel?

      2. An ominous cow herd

        RE: Don't think we really needed the footnote... *

        You're beeing way too polite with the author on this one... ;)

  2. petur
    Thumb Up

    Brilliant

    Thanks! Makes for a nice weekend :)

  3. Si 1

    Ah yes...

    ... the good old update that definitely, positively, absolutely won't need you to restart your device... except that it always does.

    If anything/anyone promises me that a reboot won't be needed I automatically assume there will be at least two forced reboots before everything is working again.

    Also, the more mission critical the device being updated the more likely the update is to go wrong.

  4. Anonymous Coward
    Anonymous Coward

    >My assistant and I subscribe to the belief that if it ain't broke it don't need fixing

    So in any attempt to find out where these two work we can exclude Ferrari

  5. Daniel O'Regan

    Real BOFH fans...

    Real BOFH fans don't need to be told what the halon system is...

    1. Anonymous Coward
      Anonymous Coward

      Re: Real BOFH fans...

      Judging by some of the comments on other articles I get a powerful impression that the only relationship some of the commentards have with computing is one to a distant relative, and even then only by marriage, who at one time browsed round a PC World store becuase it was raining out.

      1. mastodon't
        Gimp

        Re: Real BOFH fans...

        some of us know even less than that and use macs with virtual PCs...

        shudder

    2. Evil Auditor Silver badge

      Re: Real BOFH fans...

      Agreed, if you don't know what a halon system is, get a job not in IT. But, hasn't halon been banned for some years now? IIRC it was in the early 2000s when we were busy exchanging halon with other inert gas - the "oops, we just had a little fire in the data centre and the halon has been released automatically" kind of stuff.

      1. Citizen White

        Re: Real BOFH fans...

        It has been banned in new installations since then. Existing installations are grandfathered.

  6. Anonymous Coward
    Anonymous Coward

    Don't talk to me about...

    ... UPSs - my desktop APC one needed a new battery - which could be replaced without powering down. Great.

    But to make it realise the battery is now ok, and stop the bloody thing beeping at me every few hours I have to power cycle it.

    Fail.

    1. Anonymous Coward
      Anonymous Coward

      Re: Don't talk to me about...

      APC? If it is the entire unit will probably develop a mysterious fault where irt just decides to randomly switch itself off wihtin about 6 months of changing the battery.

    2. Captain Underpants

      Re: Don't talk to me about...

      Not as bad as having a racked APC UPS with what looks like a serial port on the back that's magically wired such that if you dont use the special APC cable and instead use, I dunno, any other bloody serial cable, you trigger an immediate shutdown of the UPS because they've done something bloody stupid with their NotASerialPort and helpfully decided not to put any warning stickers near said port...

      1. Andrew Moore

        Re: Don't talk to me about...

        I seem to remember that they short two of the pins in their un-serial cable.

  7. Lee Dowling Silver badge

    Just had to update an update of an update of an update that "would fix our issues".

    Turns out the expensive Management Information System that our school runs needed an update ages ago. We didn't apply it because we didn't need any of the changes it had. The next week, we were issued with an emergency update that fixed the corruption of the main databases caused by that update. We applied it weeks later when we had heard it was safe. Surprisingly, the new update backs up the old database before it touches it now (seemed kind of a big omission in such a large piece of software, but what the hell do I know?).

    So we do that, but that needed us to install IIS and ASP on the server that runs it, because the update-to-the-update also migrates you to their new "web-based" infrastructure even if you don't actually use it. Given that that server runs NOTHING else, we let it happen. Then we updated again to test the new HMRC PAYE tax notification system which was touted as a feature and we were on the trial for. Turns out that just gives SOAP errors because some silly sod forget to make the update-to-the-update-to-the-update actually activate ASP functionality (and given that the update came with install instructions that don't mention anything but installing IIS in its default config, that was kind of an oversight). We just had them log in to work out WHY it didn't do that and apparently there's going to be an update-to-the-update-to-the-update-to-the-update to fix that issue.

    I wouldn't mind, but we didn't have any issues to start with, only narrowly avoided complete database corruption by being stubborn and refusing to update, and we ended up losing the functionality to submit data to HMRC. Luckily nothing important until April but that's a pretty damn unusual testing procedure they have there. Oh, and in the process we annoyed the guy doing school reports because every update wiped out any school reports that had been taken offline and he was testing them for later in the year.

    Years of experience have taught me: Never update automatically. Never update immediately. And, if nothing's broke and it's not a "frontline" machine, never update. And certainly never update with a way to get back to EXACTLY where you were before the update.

    Applies to everything from modem / router / switch firmware up to and including network-wide application software.

    1. Cliff

      Updating ... and why not to

      @Lee Good post, frustrating when users then accuse IT of being slack/tardy for not rolling out updates without giving them a chance to be proven.

    2. DJV Silver badge

      Updates

      Yeah, I never update Ubuntu servers any more. Have been bitten once too many times (i.e. twice). Far safer to install a new version from scratch!

    3. Vic

      > the expensive Management Information System that our school runs

      I am *so* glad I don't have to deal with SIMS any more...

      Vic.

      1. Lee Dowling Silver badge

        Nearly, but SIMS isn't the only MIS that suffers these issues.

        Hint: Private school software a company in the West Country.

  8. Crisp
    Thumb Up

    Halon Gas

    When you absolutely have to kill every fucking problem in the room!

    1. pepper
      Flame

      Re: Halon Gas

      When all you have is Halon Gas, everything looks like a fire.

  9. Anonymous Coward
    Anonymous Coward

    This brings me happy memories

    In the good old days we had a detector break and set off the zone one alarm - the fire brigade arrived and we pointed out the problem, they duly removed the faulty pot and tested it in zone two - oh how we laughed when we they set off the Halon, it seems that if two zones go off then so does the gas. Even in all that equipment the fire brigade can run like bolt - who knew

    1. Lee Dowling Silver badge

      Re: This brings me happy memories

      I never got the point of halon.

      Let's prevent deaths from smoke inhalation by killing anybody who might be in the room wanting to breathe.

      Sure, halon leaves your equipment undamaged, but if you're important enough to worry about that, you have redundancy elsewhere anyway. Just install a separate sprinkler in those rooms and insure the servers if it comes to that. And then no H&S and legal nightmares, no extra equipment like masks and breathers, and no special installation and precautions necessary.

      I never got why computers, of all things, should take priority. *Especially* if those computers are supposed to save lives themselves (e.g. 999 datacenters) - then they should be highly redundant and distributed anyway.

      1. DJ 2
        Pint

        Re: This brings me happy memories

        I was told, as a young PFY, fleshy's can be found anywhere, computers are expensive and the data on them even more.

        Our halon system went off due to a fault, we opened a few windows and doors and evacuated the building until the fire brigade came by and checked it out to be safe. I was working on a big industrial chemical plant at the time, they were relived that the escaping gas was inert. The day the OTHER siren went off now that was a different story all together.

        1. Efros
          Mushroom

          Re: This brings me happy memories

          Worked at a major chemical plant on the NE coast of England, fire alarms everyone out, external sirens everyone in and close every exit tight. fire alarms and external sirens... find cupboard, stick head between legs, kiss ass goodbye.

          1. Anonymous Coward
            Anonymous Coward

            Re: This brings me happy memories

            I used to work there too, and worked on a project where I would call the on site ambulance service to say I am about start now, am wearing my breathing apparatus and have my gas alarms on, if the alarms go off its my body you need to carry out of the building when the all clear is sounded.

      2. J P

        The point of Halon

        They didn't just use it on computers. Libraries loved it; apparently priceless medieaval manuscripts and irreplaceable first editions respond slightly less well to sprinklers than do server racks (the concept of double redundancy isn't really compatible with *unique* archive material). The alternative gas systems are I understand considerably less effective, so more books will burn while it takes effect. IANAL*

        *I Am Not A Librarian

        1. Gerhard den Hollander

          Re: The point of Halon

          ook !

          1. Chika
            Coat

            Re: The point of Halon

            Exactly. Halon (and its delivery) aren't things to monkey around with!

      3. PatientOne

        Re: This brings me happy memories

        @Lee Dowling

        I've a bit of time, so I will explain it to you.

        Fire requires three things: Fuel, Oxygen and Heat. If you have a fire, removing one of those things will kill it stone cold dead. Water acts to reduce heat. All other fire prevention works to remove the Oxygen.

        In the event of an electrical fire, you do not want to use water: Water and electricity do not mix. So you use oxygen suppression.

        Halon is an oxygen suppression system just like CO2. Being a gas, it can get inside the servers to where the fire is (usually the PSU) and put it out BEFORE it gets out of control and sets the entire building alight, potentially creating clouds of toxic gas, putting neighboring buildings at risk, other lives at risk etc. Other systems such as water, foam and powder, would only coat the outside of the case and as such would be very ineffective in suppressing the fire. Indeed, in the case of water, it would only make things worse.

        Cutting the power is usual practice, too, but that is not always sufficient to stop a fire.

        Halon, CO2 and other gas suppression systems work best in a confined space. All will suffocate anyone caught inside that confined space as they are removing the Oxygen living beings, such as humans, require to live.

        As such, any area where Halon, CO2 or other oxygen replacing fire suppression systems are installed, will have things like oxygen masks, big red buttons to delay the release of the gas, release locked doors and so on - so if someone is caught inside the room they can act to protect their own lives. If they happen to be unconscious on the floor because they just electrocuted themselves and started an electrical fire... well, that's called Darwin's law of selection. Same as if they were rendered unconscious when they went to the aid of the person who got electrocuted and is still touching the live wire...

        Just as an side: I am a trained fire marshal and first aider. A few years back I was involved in a casualty simulation as described above. The simulation was used to show how we needed to think first, act second, and not add to the body count, and that in some situations, a body count is inevitable.

        I hope you are now the wiser.

        1. PatientOne

          Re: This brings me happy memories

          *Boot note: Oxygen suppression and Oxygen replacement is the same thing: Terms aren't exactly correct but I didn't reach for the manual before typing this. Bad me.

          Heavy inert gasses displace oxygen, hence 'suppressing' or replacing it.

          1. Marshalltown
            Trollface

            Re: This brings me happy memories

            Why not call the system an "oxygen displacement" system, since, as you point out, that's what they do. Ah, bueaucrats, can't live with 'em, can't suppress 'em.

        2. Nigel 11
          Boffin

          Re: This brings me happy memories

          Halon is NOT an oxygen suppressant. That's CO2 flood.

          Halon removes heat, not oxygen, in a rather clever way. When it is heated to its decomposition point, that is an endothermic chemical reaction. It absorbs heat, thereby cooling the flame. Further, the Bromine and other halogenated fragments that are released bond to the free radicals in the flame, so the flame is not only cooled but rendered incapable of catalyzing decomposition of its flammable substrate to generate more fuel for itself to feed on. It goes out and stays out.

          It's exactly the same property that makes it so dangerous in the upper atmosphere, where UV decomposes it and where the fragments then catalyze the destruction of Ozone. Which is why it's been banned except in the most critical of safety-critical fire-suppression roles. Computers gave to take their chances with sprinklers, or be put in a true lights-out room with a CO2-flood system that kills both fires and people.

          1. Captain Underpants
            Thumb Up

            Re: This brings me happy memories

            @Nigel

            You learn something new every day, and usually thanks to good folk like yourself willing to share knowledge. Cheers for an interesting read! :)

        3. Guus Leeuw
          Headmaster

          Re: This brings me happy memories

          Quote: Water and electricity do not mix

          I was taught that they mix quite well. In fact they mix so well, that you should never try to be in the water while mixing it with electricity...

          Just me olde 2 cents,

          Guus

      4. Bill Cumming
        Flame

        Re: This brings me happy memories

        Ahh... HALON, the IT's equivalent of a neutron bomb..

        gets rid of pesky people leaving data centre unscathed. ^_^

      5. Nigel 11
        Boffin

        Re: This brings me happy memories

        Let's prevent deaths from smoke inhalation by killing anybody who might be in the room wanting to breathe.

        Only in the parallel BOFH universe. Or you are confusing Halon with CO2 flood.

        In the real world Halon is almost inert and is not used in great enough concentrations to asphyxiate. If you were in the computer room when it dumped you would be off work with a monster headache the rest of the day and the next. It's not clear whether that's because of the huge noise it makes (only just short of eardrum-breaking), because of the flying ceiling tiles that the release usually causes, or because Halons have another use. The Anaesthetic gas that hospitals use is a halon.

        Maybe someone once got fully anaesthatized, and that started the urban legend about it killing fires and people and leaving equipment unscathed?

      6. Ross K Silver badge
        Joke

        Re: This brings me happy memories

        I never got the point of halon.

        Let's prevent deaths from smoke inhalation by killing anybody who might be in the room wanting to breathe.

        One quick call to your PFY-supplier and you're back in business once the halon's cleared and the body dragged out back behind the smoking nut.

        Dead PFYs and service engineers are less of a headache these days as you can always get a few "interns" to work for free...

  10. TechnoTechno
    Coffee/keyboard

    Excellent

    Eerily similar to an issue we had this week.. minus the Halon

    Good Laugh!

  11. philbo

    Damn, that brought back some memories

    In another life I was that PFY, in exactly that position. So much so I'm starting to wonder if my colleague's name back then was Simon Travaglia.

  12. cnorris517
    Facepalm

    Top work as usual, though I feel it's a sad indictment of us BOFH wannabies when you have to bootnote a Halon reference

  13. Richard Jones 1
    Holmes

    Halon Rocket Fuel

    One place I worked was being developed and the spare Haylon devices were in the builder's store.

    The builder's store took fire and one of the Haylon globes warmed up, and up, and up. In the end the outlet seal ruptured and, Werner Von Braun eat your heart out, the rocket was launched.

    It went through several walls on its way to stardom, though not to the stars. The police investigator was not amused when he was told that the flying cylinder was an extinguisher.

    A severe sense of humour failing followed

    1. I Am Spartacus
      Thumb Up

      Re: Halon Rocket Fuel

      That is priceless.

    2. Nigel 11
      Pint

      Re: Halon Rocket Fuel

      I've heard a similar story. Except it was a high-pressure air cylinder that someone had unstrapped from the wall and propped against another. It toppled, struck its neck on the bottom rung of the decorator's step-ladder, fractured, and went through three walls ending up embedded six feet into a hillside. No-one was killed. The decorator's deafness was temporary.

      Or so I was told, over a beer.

      (The REALLY scary stories involve overheated acetylene cylinders, which make unexploded WW2 bombs look friendly.)

      1. Alister

        Re: Halon Rocket Fuel

        I have a similar tale of an "F" size (about 4 foot tall and 8 inches wide) Medical Oxygen cylinder which was dropped in the gases store at an Ambulance station and snapped the valve off.

        It went through (in order) a brick wall, three ambulance vehicles, another brick wall, the stone retaining wall at the side of the driveway and disappeared into a wood. Later found half a mile away...

        You can still see the new brickwork where they patched up the building.

  14. I Am Spartacus
    Holmes

    Halon inspires striptease

    Richard's rocket fuel story reminded me of something that happened in about 83/84. The then new halon fire suppression system was in two electrical loops, some under the floor, some in the ceiling void. Fairly standard layout for a big data shop (mainframes filled the rooms then - it really was big iron).

    The system tests itself by having a trickle charge run through the electrical loop. This way it detects a break in the wiring. When you need to fire the halon, the deal is this: The siren goes and everyone heads to the exits; the halon is locked and can't go off until the shift supervisor has counted everyone out, locked the door, and activated the fire system. Simples.

    Unless the control box has a hissy fit and decides to blow a fuse. Seems that in this case it assumes there is a fire, that the door is locked, and blows the circuit.

    Pity the poor bod who was sat on a roll about chair in the bridge, under a large ceiling tile, when the globe above it fired. He got knocked off his chair and broke his arm.

    Double pity the young female tape operator who was walking over an air grating when the globe under that went off. She had her dress ripped off by the blast and got out wearing only bra and knickers with an attitude to kill at 50'.

    Cue another sense of humour failure!

    1. The Camel King
      Coat

      Re: Halon inspires striptease

      I think the most important question in response to that story is was she attractive?

      I was going to go with 'hot' or 'a bombshell' but I think they are open to misinterpretation...

      1. Anonymous Coward
        Anonymous Coward

        Re: Halon inspires striptease

        "I think the most important question in response to that story is was she attractive?"

        She worked in IT.

        Nuff sed.

  15. Jen 3

    UPS Clusters, who'd have 'em?

    Also helps if the engineer that comes in to service UPS1 of your very expensive UPS cluster is told by the UPS company that the recent firmware update now means that when he pushes the big red shutdown button it turns off BOTH UPS in the cluster, rather than just the UPS he is working on like he is used to.

    Ahh the sound of silence.....

  16. tim 4
    Mushroom

    HAAAALLLLOOOONN!!!!!!!!!!!!!!

    how's this for fun? we had a pcb plant server with super-clean specs to go with our super-clean area ; in which we made nasa and defense ind boards. the designer sold the company on a system that would protect the server from fire using halon, and a hermetically sealed room to keep everything unfiltered out. marvy! it was designed so that positive pressure from the aircon system kept the door sealed airtight , and in time of fire, airtight baffles would shoot into place to seal the ducts to prevent halon from venting into the production zones.

    the cdc would have been proud to use the setup as a lab. about a year into operation, lights and sirens erupted due to a short caused in an external upgrade of the server wiring system [ adding yet another strobe]. our current ITLord's habit of getting his coffee-cup [and wick] wet in hr saved his life. the halon fired, the resulting pressure being calculated for the ORIGINALLY DESIGNED server enclosure of nearly three times that size....

    someone could probably calculate the pressure/tons per squre inch for us, doesn't matter... the door could not be opened, the clever "in case of fire power-interruption circuits" inside the enclosure couldn't be reset, and the stupid allen-key lock-release for the door was useless because it couldn't be pushed in the quarter-inch to slide it over.

    they were forced to bust through one of the very costly walls with a sledge, putting concrete dust everywhere, and spreading it throughout production along with the halon. my roundly ignored , low-man advice of drilling a vent-hole and then patching the priceless door was roundly nay-sayed. so i didn't suggest drilling the wall [ half meter concrete with 1cm epoxy resin sealer-coat ]. costly clean-up , and a weeks production lost... because they jackhammered through the wall to get to the circuits..

    1. Wzrd1 Silver badge

      Re: HAAAALLLLOOOONN!!!!!!!!!!!!!!

      Some years ago, I was up to date on all work, everything was running smoothly and I had nothing to do. So, I decided to help out the desktop crew, as they were shorthanded and overwhelmed with work.

      So, I proceeded to the old mux room, which only had one rack of servers in it, the onsite backup tapes stored and the desktop service crew.

      It still had its halon system operational, which is part of one's briefing when one first starts work at the IS shop of the hospital, where all halon systems are and where the inhibit button is located.

      I began looking at a problematic model of portable nurse station cart. It had been a problem since it was first acquired and was losing ground quickly.

      I fired up one cart and within seconds, it appeared that I used real fire to fire it up. Smoke billowed merrily up from the motherboard, curling happily toward the halon system smoke detector.

      Three of us dove to the inhibit button, myself with one hand on the door opener. Our hands hovering over the button, we waited and were rewarded with silence.

      And so was born a new respect for Lowest Bidder Engineering Services.

  17. The New Turtle
    IT Angle

    Foot note should not be included in an otherwise excellent article. I've never worked in IT like that and yet it was still immediately obvious what the halon was for.

  18. Peter Gathercole Silver badge

    Where I work they use an argon/nitrogen mixture

    branded with the proprietary name argonite.

    Supposedly, if the oxygen concentration drops below 15%, almost all fires are extinguished.

    The system is calibrated to drop the oxygen content to about 12.5% by displacement, which will keep you alive if you are caught, but is probably quite uncomfortable.

    I don't intend to try it, not after going through an oxygen depletion experiment at school many years ago (probably not even allowed now), where I breathed the same air over and over through a CO2 filter. You don't start breathing heavily (it's the CO2 that makes your body believe that it's suffocating) and you just can't think or do anything. I was told that I looked very silly by the rest of the class not able to answer even the simplest question. Fortunately, the teacher liked me enough to take the mask off before I passed out.

    1. david 12 Silver badge

      Re: Where I work they use an argon/nitrogen mixture

      >I was told that I looked very silly by the rest of the class not able to answer even the simplest question

      Airplans are pressurised to 8,000 ft (2,400 m), giving about 75% osygen levels compared to sea level. Even at that level, writing reports and answering email is probably a mistake. More recent airplanes are pressurized to lower levels (the Airbus 380 runs at 5000ft equivilant).

      Pilots have an idependent oxygen supply they can use, but even so, it's probably lucky that they come down to a lower altidute before attempting to land.....

  19. SirDigalot

    There are two main problems in a datacentre

    Fire and ugly bags of mostly water.

    Halon or C02 takes care of both of these problems...

  20. Dadz

    underfloor

    Our halon discharge happened when a student stepped on a heat detector while running wires in the under-flooring. Heat detectors trigger when they notice an open circuit.

    We didn't obtain new halon afterwards. Although, the heat detector was originally a good idea, given the 50 cm thick layer of ~1600 mainframe cables under the three dozen port concentrators.

  21. tony trolle

    Halon or halonite

    The first place I worked at had a Halon 2402 system and at the far end of the room opposite the door was had a emergency exit trap door thing. Was told normally the red lights would flash and the fire alarm sounders would sound then 5 seconds later whoost . But they had just found out were was no 5 second delay after they had the system checked out by another vendor who serviced the rest of the building's fire systems.

    Nice.

    Was also told if it ever did trigger the replacement gas would be different as "this stuff decomposes when it gets hot and likely kill you".

    Funny they had a smokers area 50 feet from the main building in the back next to the on-site archive tape store.....

  22. Colin Miller
    Headmaster

    Pedant alert

    A ROM patch? Surely the engineer meant an E²PROM patch?

  23. Hairy Airey
    FAIL

    RTFM

    APC UPSs don't need a reboot when you change the battery - hold the power button in (the one marked "I") and it will commence a self-test (or start a self-test from the management interface). That clears the fault.

    I've never had to reboot one for a battery replacement.

    Young people today know nothing...

    1. ArmanX

      Re: RTFM

      The reason young people today know nothing (and I place myself squarely in that category) is actually pretty simple; it boils down to documentation.

      First, documentation today is generally terrible. It used to be that you'd get a giant manual with everything from a bag of case screws to the newest software; these days, you're lucky if it comes with a web address printed on the box. Or a box, for that matter. And good luck finding something that wasn't translated by someone using Google Translate.

      Second, in the 'old days', there were hobbyists that held on to documentation (or wrote it themselves); these days, there are so many versions of things, not to mention off-brands and knockoffs, finding complete documentation is almost impossible. Even the manufacturers don't know how their own products work. (1)

      And third, we have the Internet. What used to take hours searching stacks of books and dialing obscure BBS's now takes a two-second Google search - and that's what The Boss expects. Need eight hours to find the documentation to turn off an annoying beep? Ha! Live with it or reboot it!

      (1) In trying to debug some ancient, arcane hardware, I was given the private home number of a septuagenarian hobbyist - the only person in the world who knows how their hardware worked. Turns out he actually worked for their competition 20 years ago...

  24. Anonymous Coward
    Anonymous Coward

    different fire suppression angle

    once worked in a room with FM200 gas for fire suppression.

    Guess what all the Administrator passwords (complexity requirement: two or more letters, two or more numbers) were on the servers?

  25. Wzrd1 Silver badge

    Always remember

    All important patches on mission essential equipment should be applied one hour or less before COB on the last day of the work week.

    This will ensure your efforts will be awarded with the maximum amount of high rate overtime.

This topic is closed for new posts.

Other stories you might like