back to article 'Jet blast' noise KOs ING bank's spinning rust servers

Sound waves equal to a jet fighter taking off, generated by fire fighting equipment during testing, has been blamed for taking down ING's banking operations. Card and ATM transactions, internet banking and ING’s website were all felled in Romania for 10 hours on Saturday by the freak occurrence, according to Vice's Motherboard …

  1. Voland's right hand Silver badge

    130 db can knock you out and disorient you

    If you are suddenly hit by 130db siren without any ear protection you may end up in a state where you are not capable to escape the gas coming a few seconds after it. It is like being hit on the head with a mallet.

    They are lucky it was just a test, otherwise there could have been a couple of dead bodies in the server isles.

    1. Alister Silver badge

      Re: 130 db can knock you out and disorient you

      As I understand it, it wasn't a siren, the 130db noise was generated by the high-pressure gas escaping through the nozzles of the fire suppression system.

      1. Alan Brown Silver badge

        Re: 130 db can knock you out and disorient you

        0: Far too few organisations with these systems have done proper H&S and liability assessments.

        1: Tests aren't supposed to discharge the system. Inergen bottles are around £1000 a pop to refill. If you're substituting other gasses for the discharge you still shouldn't be testing this shit in a live server room.

        2: Whatever initiates discharge of the gas is supposed to shut the power off (ie, heads should park _after_ the discharge secondary alarm sounds and _before_ the discharge valve is opened), in order to remove any potential ignition sources. This is supposed to be interlocked.

        3: It's possible to quieten the gas discharge noise without impeding gas flow. A H&S review needs to be implemented if it really was 130dB. The mere possibility of someone being exposed to this would expose a company to near unlimited damages claims if it actually happened.

        4: You have to be extremely careful with discharge design. A data centre in Sydney had its floor tiles go several feet in the air during a discharge. Again, H&S and damage liability - especially if someone's been rendered incapable of dodging them by the sound - those tiles have sharpish edges and weigh 10-15kg apiece which means they're capable of breaking bones or more when gravity takes over.

        5: Inergen systems work by reducing oxygen levels in the room to under 10% (most things won't burn in that low a concentration), which equates to about the same partial pressure as being around 5-8000 metres higher in altitude - ie, 15-20,000 foot pressure altitude and can have major cognitive effects (up to and including death if someone's got impaired cardiopulmonary function)

        https://www.osha.gov/pls/oshaweb/owadisp.show_document?p_id=25743&p_table=INTERPRETATIONS

        Note the effects listed as oxygen percentage goes down.

        6: Inergen bottles sometimes leak (We've had this happen) and anyone who doesn't have oxygen level alarms in both the discharge(*) and storage areas could face unlimited liabilities if someone is affected as a result.

        (*) Our leak was into the gas bottle cupboard via a popped o-ring in the meter seal and that was also the electrical switchroom

        In other words, these things are EXTREMELY FUCKING DANGEROUS (so are older (mostly banned) halon systems as the gases get changed to toxic fumes when exposed to fire) and need to be treated as such.

        1. HausWolf

          Re: 130 db can knock you out and disorient you

          And did you see what country it occurred in? They probably don't have the same requirements to keep employees alive as does the UK and the US ( for now). That is one of the problems with globalization. The race to the bottom.

    2. Anonymous Coward
      Anonymous Coward

      Re: 130 db can knock you out and disorient you

      Intergen is not Halon. Nor is it like a CO2 system that smothers a fire (and any unfortunate occupants).

      ISTR that the older versions of the Intergen system generated byproducts when reacting to the fire. Said byproducts aren't supposed to be great for you, but then again, neither is smoke from burning electronics.

      Depending on the size of the room (and how well sealed it is) I'd also wonder about the air pressure changes...

      1. Alan Brown Silver badge

        Re: 130 db can knock you out and disorient you

        "Depending on the size of the room (and how well sealed it is) I'd also wonder about the air pressure changes..."

        Rooms with Inergen systems must have blow off external vents to take care of that. If you don't have that you risk damaging the building, let alone anyone in the room (sudden 3psi overpressure will knock you flying, 6psi will kill you)

        Usually these take the form of a set of 24*24inch holes in the wall capped externally by top-hinged steel doors with magnetic catches (enough pressure and they'll fly open). These create their own security issues. given the need to prevent wildlife or humans getting in, without impeding functionality.

    3. allthecoolshortnamesweretaken

      Re: 130 db can knock you out and disorient you

      No shit. Ever been close to a flash-bang-thingy (like the SAS et al use) going off?

    4. Anonymous Coward
      Anonymous Coward

      Re: 130 db can knock you out and disorient you

      Server Isles? Great beaches I hear

  2. wheelbearing

    Sounds nasty..

    Like crims and other ne'er do wells needed another attack vector - this could be a difficult one to protect against. The Romanian fire suppression hardware might be a different spec to that in use in other countries but alternatives like fire alarm speakers could theoretically cause similar damage if they have the juice and are designed to be cranked up to those levels (though very unlikely I would have thought).

    1. Crazy Operations Guy Silver badge

      Re: Sounds nasty..

      If miscreants have the ability to get deep enough into the building to re-wire the fire alarms, then you've already lost. They could just as easily sabotage the electrical system and destroy even more equipment, or just burn the whole place to the ground, destroying everything.

      Worrying about what someone could do with a loud noise like this would be like concerning yourself that a mugger with a heavy knife is going to bludgeon you with it, rather than the more obvious, and more effective method.

  3. Small Furry Animal

    But was it as loud as this ...

    http://3.bp.blogspot.com/-7yLgzEEHYcc/ThpBFpkJ-GI/AAAAAAAAByk/iMulydt2MnQ/s1600/Dead+Wall+1.jpg

    1. Anonymous Coward
      Anonymous Coward

      Re: But was it as loud as this ...

      But did they turn it up to 11?

  4. Anonymous Coward
    Anonymous Coward

    Fucking Romulans

    If they aren't trying to kill you with their warbirds, then they are attacking you with sonic gas discharge, the basta....

    Oh hang on - Romanians. Sorry, my bad. Carry on.....

    1. TRT Silver badge

      Re: Fucking Romulans

      I believe their data centres are populated with Quantum Singularity drives.

    2. DropBear Silver badge
      Devil

      Re: Fucking Romulans

      Well, being a vampire is a job requirement to get anywhere near the server room (it's a natural habitat for sunlight-averse creatures after all) which is why the safety of the systems involved was never a priority. Unfortunately, the designers seem to have neglected to consider its effects on the hardware itself...

  5. Anonymous Coward
    Anonymous Coward

    Funnily enough...

    On of the big UK utilities had a similar thing just a month back.

    Smoke in server room triggered smoke detector which triggered fire alarm which triggered gas discharge which gave the SAN cabinets a good shaking and took the SAN down.

    Not sure whether Intergen or some other system though.

  6. Anonymous Coward
    Anonymous Coward

    The amount of times a business continuity solution seems to cause more problems than not having it...

    Look at issues with the recent data centre outage for Vesk when a disk failed, that should have been handled by a RAID controller but took their operation offline. Or failover systems that end up corrupting data when they could have been sorted without automatically failing over. The report on the BBC also mentions that the noise from these fire suppression systems has caused failures in a number of data centres - and as it is very difficult to determine what disks have been damaged you could have problems for a long time afterwards.

    I have yet to hear of a fire in a normal comms room that would have been helped by a fire suppression system (i.e a spontaneous electrical fire).. In fact fires in comms rooms are extremely rare anyway and most, I would suspect are caused by human error and an electrical cutout, possibly with a well aimed fire extinguisher would have been a lot better than trashing your disks.

    1. Anonymous Coward
      Anonymous Coward

      ".... and an electrical cutout, possibly with a well aimed fire extinguisher would have been a lot better than trashing your disks."

      At 3am on a Sunday morning, it's a good chance a fire will take hold quite well before someone gets into there.

      If there is smoke and flames in a confined area, the LAST thing you should be doing is trying to fight it with a poxy fire extinguisher,

      1. Anonymous Coward
        Anonymous Coward

        "At 3am on a Sunday morning, it's a good chance a fire will take hold quite well before someone gets into there."

        My point is, you don't generally get servers bursting into flames at 3am on a Sunday morning. The risk of a server room fire in a decent, modern server room (one where they would have thought about creating a well running fire suppression system) is extremely small.

        A fire will often either start externally and engulf the server room anyway or it will be caused by someone in the server room who will be able to grab a fire extinguisher.

        Something that can kill the power will most likely be just as effective.

        1. Anonymous Coward
          Anonymous Coward

          My point is, you don't generally get servers bursting into flames at 3am on a Sunday morning.

          You clearly haven't heard of Murphy's law then :). Do you really think a server with a faulty component will nicely wait until office hours before blowing up?

          1. Anonymous Coward
            Anonymous Coward

            "faulty component" ... "blowing up"

            That is not a fire. A component can easily blow up without a fire starting. there is generally a lack of fuel source.

            The main area for fires is the UPS. If they turn Lithium there could be a bigger issue with server room fires, but until then servers don't blow up and catch fire. You'd be far better investing in backup and redundancy.

            1. Anonymous Coward
              Anonymous Coward

              That is not a fire. A component can easily blow up without a fire starting. there is generally a lack of fuel source.

              40+ years of working with electronics says otherwise.

              Anything that carries a current can burn. Anything with an electrolytic capacitor can pop, and quite enthusiastically so. Anything hot enough can carry a flame. Sometimes it doesn't even take a defective component. It could be a sparking connector, a power switch with a dodgy contact so it starts to arc, a solder joint might have been just a bit too dry, an elco could be off tolerance, a misplaced cable could rub off insulation and short/spark/arc - the list is endless.

              However, in all those years I have only ever seen a computer blow up once as it happens in Hollywood movies, and that was when a system was miswired during an oil rig revision. As soon as it booted up and the relais brought to A/D card online it emitted a loud bang and smoke, and its sturdy steel case went from rectangular to more ball shaped (which, in hindsight, must have saved us from injury because I suspect bits would have flown around otherwise, and we were 5 meters away from the thing when it happened, working on another server (our own, which had full optical insulators which rendered it immune to such mistakes). It turned out that electricians had miswired a junction box, so instead of getting the 30mA loop it was expecting, it was hooked up to raw 400V generator power, and the PC was *far* quicker than the fuse in expressing its dissatisfaction with that arrangement. There really was nothing usable left in the box, the surge had annihilated practically everything and cooked the motherboard and almost every PCB track on it.

              It was only at that point that my boss finally stopped grousing about me adding those (costly) insulators to our design because they reduced our profit margin a bit..

              1. Anonymous Coward
                Anonymous Coward

                And there you have it. No fires just components blowing up.

                None of which happened spontaneously at 3am, they happened because of miswires on specialist equipment, not well functioning server rooms.

                So my point still stands, spontaneous server room fires in professionally configured and installed rooms, using normal modern server room kit are almost (if not completely) non-existent - if you factor in ones where a fire-suppression system would have helped where an auto power cut wouldn't have then you be into the realms of - more likely to be hit by lightning.

  7. Anonymous Coward
    Anonymous Coward

    Well known

    I can confirm that the phenomenon of Inergen discharges killing HDDs is well known in the drive industry.

    The standing advice is "don't do it for shits and giggles"

    /works in the drive industry

    //but mostly on flash platforms

    ///Once saw the aftermath of a test with CO2 in a live data centre that blew down the entire false ceiling, together with decades of crud and cabling. Executive summary as overheard from next room: Click....whoooooooOOOOSSSSSHHHHHHH.....BANG...OH SHIIIIITT!

  8. myxiplx2

    Known problem, but not well documented

    I've had a customer affected by this in the past, if I recall it's down to the frequencies emitted by one specific design of nozzle, and there are advisories out there for the fire suppression systems advising against its use in data centres.

    The outage we saw affected 15k rpm drives, across several different SAN equipment manufacturers. 10k and 7k drives escaped largely unscathed.

    However the damage was severe enough to totally offline around 50% of those 15k rpm drives, to the point that they wouldnt even come back after a power cycle.

  9. Korev Silver badge

    Backup DC?

    Don't most banks have some form of backup DC for failover?

    1. DonL

      Re: Backup DC?

      "Don't most banks have some form of backup DC for failover?"

      A few years ago I heard about a company that had this but the data corruption caused by the blast of the fire supression (which went off because of a test or accident) replicated to the backup DC, making it mostly useless.

      Tricky stuff.

  10. Missing Semicolon Silver badge
    1. GrumpenKraut Silver badge
      Pint

      Re: You'd better believe it!

      Wow! If anyone told me that I'd think that would be a joke.

      Very instructive, thanks. ----------->

  11. Anonymous Coward
    Anonymous Coward

    Our original system was assessed several years back..

    and the floor tiles exploding all over the place was deemed a bigger risk than any fire.

    It was replaced shortly after.

  12. ma1010 Silver badge
    Trollface

    So the BOFH missed a bet?

    130 db can knock you out and disorient you

    No doubt Simon will be ordering a 130 db siren to add to the machine room for when the (black market) Halon system goes off. Why take chances that "unauthorized personnel" might find their way out in time to survive?

    1. kain preacher Silver badge

      Re: So the BOFH missed a bet?

      You mean 260db. Lets liquefy the poor chap.

  13. kain preacher Silver badge

    What's wrong with FM-200

  14. BernardL

    ...to the surprise of no-one who's ever seen (even on YouTube*) a Halon dump. Inergen is after my time, but it doesn't sound any better.

    Moral of the story: if you're in the room with the supression system on auto - and you shouldn't be - and you hear the discharge alarm RUN, don't wak to the nearest exit. And keep running until you're out of the building and well down the road.

    * other tubes are available.

    PS: Legend has it that the Amdahl factory in Swords had a halon dump (in the 80's?) because someone left the system in the wrong mode. Apparently it cost 40 grand to refill.

    PPS: 130dB seems way too loud for the alarm... I know they're supposed to be loud enough to drive you out of the room, but aren't there standards for loudness.

    1. Anonymous Coward
      Anonymous Coward

      +1 on the Run advice

      I remember being onsite when a customer had a Halon system installed to protect their shiny new AS400, the trainer basically said, if the alarm is triggered run for the exit, if the system triggers it will bring down the ceiling tiles at the same time that it blow random floor tiles into the air and you're disorientated by the 130dB gas discharge and lack of oxygen. They'd recently done a full system test (with the Fire Brigade onsite) at a different customer/location and no-one made it out of the server room.....

    2. Alister Silver badge

      PPS: 130dB seems way too loud for the alarm... I know they're supposed to be loud enough to drive you out of the room, but aren't there standards for loudness.

      It's not an alarm, the 130dB sound was created by the gas exiting the nozzles of the suppressant system.

  15. Ian Emery Silver badge
    Holmes

    Lucky me

    I spent a week working as a temporary relief at a combo office/warehouse site before finding a few days later, that - due to people forgetting to switch it back to "Auto", the Auto/Manual Halon switch had been bypassed, and it was in Auto all the time.

    Add this to everything else I found in the building,

    no effective earthing (unless the voltage was high enough to jump an 8 inch gap)

    bypassed auto fire door switches (and the doors themselves wedged open)

    crimp connectors joining power cable in the middle of trunking (and two different colour cables!!)

    Single phase isolators on two phase equipment supplies (someone got fried and also suffered thermal burns from the heating element their hand was wrapped around).

    And HUNDREDS of circuit breakers wired in bypass mode.

    Some of this stuff had been in place for 25 years without any of the regular staff picking up on it!!!

    When I brought it up I got fired.

    1. Anonymous Coward
      Anonymous Coward

      Re: Lucky me

      Some of this stuff had been in place for 25 years without any of the regular staff picking up on it!!!

      BT?

      1. Ian Emery Silver badge

        Re: Lucky me

        Kay&Co. AKA GUS, AKA Great Universal Stores,

        HOW BIG BUSINESS WORKS

        Feeling a bit disgruntled (as anyone actually BEEN "Gruntled"??), I handed my baby brother a full defect list, and he and a Fire Brigade team visited the place shortly afterwards.

        After finding everything on my list (it ran to eight pages), the Brigade presented GUS management with the list and said "No fix, no fire certificate"

        Management response was "Throw that list away, or we make everyone redundant, close the place, and move operations to Burnley"

        The Brigade folded and the issues were never fixed.

        10 years later GUS STILL closed the place and moved operations to Burnley.

        It makes you wonder what sweetheart deals they have got going on up there.

    2. Alan Brown Silver badge

      Re: Lucky me

      "I spent a week working as a temporary relief at a combo office/warehouse site before finding a few days later, that - due to people forgetting to switch it back to "Auto", the Auto/Manual Halon switch had been bypassed, and it was in Auto all the time"

      Even when it's in manual mode, the "discharge" button is normally active and only under a clipdown cover.

      We had (and have) a very real fear of some practical joker pushing the thing for shits and giggles (and it's not because of the £8k refill cost)

    3. Alan Brown Silver badge

      Re: Lucky me

      "When I brought it up I got fired."

      One phone call to the HSE would be in order, along with a lawsuit for unjustified dismissal.

      Presumably you documented it all.

      For added fun: Drop into the local fire brigade and let them know they need to do a surprise inspection.

      1. Ian Emery Silver badge

        Re: Lucky me

        @ Alan Brown

        HSE were part of the Fire Brigade Inspection team; I personally knew the guy leading the team and he was furious that the decision to drop it was made by higher ups.

        I appealed and my appeal was upheld, but GUS DIrector of Electrical Services, even after agreeing with me, said he wouldnt be giving me my job back, and paid me off.

        As I had been there less than 2 years (by a few weeks!!) I didnt get much.

        Before quitting the industry, I refused to KowTow over safety three times, with one other international and one small, local firm as the other culprits.

        I walked out on the small firm after being told to "hold my breath" after asking for a working gas meter, and the big firm sent me home, then, during the disciplinary meeting, I was bollocked for reading SciFi during my lunch break, the reason I got sent home was NEVER raised!!!!

        (Refusing to put a holesaw through a trunking full of live 240v and 440v cables).

        Yeah, thats the trouble with being multi-skilled and pretty good, they ask you to do EVERYTHING, from installing heavy machinery, to prototyping circuit boards, to moving server racks.

    4. John Brown (no body) Silver badge

      Re: Lucky me

      "Some of this stuff had been in place for 25 years without any of the regular staff picking up on it!!!

      When I brought it up I got fired."

      I suspect that your second sentence gives the reason why they were the regular staff and you didn't get to be there long enough to become one of them.

  16. Anonymous Coward
    Anonymous Coward

    You think your DC is "clean"................

    You should see the "crap" it stirs up after releasing it's load.........All the rubbish you thought had been cleaned up after the DC was built, well hidden more like............

    1. DropBear Silver badge
      Joke

      Well, I dare say it's usually still a lot cleaner than AC...

    2. Alan Brown Silver badge

      "All the rubbish you thought had been cleaned up after the DC was built, well hidden more like............"

      I made a point of lifting the entire tiled floor, rubbish picking (there was a lot) and vacuuming it along with every crevice before moving anything into ours. Along the way it was discovered that none of the earths were connected.

      No false ceiling thankfully but we did find a mountain of shit on various ledges.

      1. Linker3000

        Floor voids - more than just storage areas for rubbish

        I had the pleasure of visiting a building where the architects managed to design the server room with a FOUR FOOT flloor void rather than the desired 4 inches. Somehow this cock-up of NASA dimensioning proportions made it through all checks and that's how the building was constructed.

        By the time the building was completed and occupied, the data centre looked 'normal'... until you lifted a floor panel and stared into the void; this was one data centre where you **really** had to be careful if you left a floor panel out of place! There was so much space below the floor that the IT guys had installed servers down there - as my knees and back dscovered when I turned up to fit a new Ethernet card and was told which tile to lift to start my crawl.

        1. Anonymous Coward
          Anonymous Coward

          Re: Floor voids - more than just storage areas for rubbish

          That's not a first. I was at an MCI data center were if you popper the floor tile it was 4 feet deep with data and power cables underneath. this same data center refused to put a cover over the EPO.

  17. Manolo
    FAIL

    (No) business as usual

    ING's internet banking being off line is not a freak occurrence. In the Netherlands their systems seem to be more down than up.

    http://allestoringen.nl/storing/ing/overzicht

  18. Stevie Silver badge

    Bah!

    Well that worked.

  19. Fazal Majid

    Shouting alone has effects

    Brendan Gregg showed how shouting had a measurable effect on disk latency while at Sun:

    https://www.youtube.com/watch?v=tDacjrSCeq4

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019