'Jet blast' noise KOs ING bank's spinning rust servers Sound waves equal to a jet fighter taking off, generated by fire fighting equipment during testing, has been blamed for taking down ING's banking operations. Card and ATM transactions, internet banking and ING’s website were all felled in Romania for 10 hours on Saturday by the freak occurrence, according to Vice's Motherboard …
If you are suddenly hit by 130db siren without any ear protection you may end up in a state where you are not capable to escape the gas coming a few seconds after it. It is like being hit on the head with a mallet.
They are lucky it was just a test, otherwise there could have been a couple of dead bodies in the server isles.
0: Far too few organisations with these systems have done proper H&S and liability assessments.
1: Tests aren't supposed to discharge the system. Inergen bottles are around £1000 a pop to refill. If you're substituting other gasses for the discharge you still shouldn't be testing this shit in a live server room.
2: Whatever initiates discharge of the gas is supposed to shut the power off (ie, heads should park _after_ the discharge secondary alarm sounds and _before_ the discharge valve is opened), in order to remove any potential ignition sources. This is supposed to be interlocked.
3: It's possible to quieten the gas discharge noise without impeding gas flow. A H&S review needs to be implemented if it really was 130dB. The mere possibility of someone being exposed to this would expose a company to near unlimited damages claims if it actually happened.
4: You have to be extremely careful with discharge design. A data centre in Sydney had its floor tiles go several feet in the air during a discharge. Again, H&S and damage liability - especially if someone's been rendered incapable of dodging them by the sound - those tiles have sharpish edges and weigh 10-15kg apiece which means they're capable of breaking bones or more when gravity takes over.
5: Inergen systems work by reducing oxygen levels in the room to under 10% (most things won't burn in that low a concentration), which equates to about the same partial pressure as being around 5-8000 metres higher in altitude - ie, 15-20,000 foot pressure altitude and can have major cognitive effects (up to and including death if someone's got impaired cardiopulmonary function)
https://www.osha.gov/pls/oshaweb/owadisp.show_document?p_id=25743&p_table=INTERPRETATIONS
Note the effects listed as oxygen percentage goes down.
6: Inergen bottles sometimes leak (We've had this happen) and anyone who doesn't have oxygen level alarms in both the discharge(*) and storage areas could face unlimited liabilities if someone is affected as a result.
(*) Our leak was into the gas bottle cupboard via a popped o-ring in the meter seal and that was also the electrical switchroom
In other words, these things are EXTREMELY FUCKING DANGEROUS (so are older (mostly banned) halon systems as the gases get changed to toxic fumes when exposed to fire) and need to be treated as such.
Intergen is not Halon. Nor is it like a CO2 system that smothers a fire (and any unfortunate occupants).
ISTR that the older versions of the Intergen system generated byproducts when reacting to the fire. Said byproducts aren't supposed to be great for you, but then again, neither is smoke from burning electronics.
Depending on the size of the room (and how well sealed it is) I'd also wonder about the air pressure changes...
"Depending on the size of the room (and how well sealed it is) I'd also wonder about the air pressure changes..."
Rooms with Inergen systems must have blow off external vents to take care of that. If you don't have that you risk damaging the building, let alone anyone in the room (sudden 3psi overpressure will knock you flying, 6psi will kill you)
Usually these take the form of a set of 24*24inch holes in the wall capped externally by top-hinged steel doors with magnetic catches (enough pressure and they'll fly open). These create their own security issues. given the need to prevent wildlife or humans getting in, without impeding functionality.
Like crims and other ne'er do wells needed another attack vector - this could be a difficult one to protect against. The Romanian fire suppression hardware might be a different spec to that in use in other countries but alternatives like fire alarm speakers could theoretically cause similar damage if they have the juice and are designed to be cranked up to those levels (though very unlikely I would have thought).
If miscreants have the ability to get deep enough into the building to re-wire the fire alarms, then you've already lost. They could just as easily sabotage the electrical system and destroy even more equipment, or just burn the whole place to the ground, destroying everything.
Worrying about what someone could do with a loud noise like this would be like concerning yourself that a mugger with a heavy knife is going to bludgeon you with it, rather than the more obvious, and more effective method.
Well, being a vampire is a job requirement to get anywhere near the server room (it's a natural habitat for sunlight-averse creatures after all) which is why the safety of the systems involved was never a priority. Unfortunately, the designers seem to have neglected to consider its effects on the hardware itself...
On of the big UK utilities had a similar thing just a month back.
Smoke in server room triggered smoke detector which triggered fire alarm which triggered gas discharge which gave the SAN cabinets a good shaking and took the SAN down.
Not sure whether Intergen or some other system though.
The amount of times a business continuity solution seems to cause more problems than not having it...
Look at issues with the recent data centre outage for Vesk when a disk failed, that should have been handled by a RAID controller but took their operation offline. Or failover systems that end up corrupting data when they could have been sorted without automatically failing over. The report on the BBC also mentions that the noise from these fire suppression systems has caused failures in a number of data centres - and as it is very difficult to determine what disks have been damaged you could have problems for a long time afterwards.
I have yet to hear of a fire in a normal comms room that would have been helped by a fire suppression system (i.e a spontaneous electrical fire).. In fact fires in comms rooms are extremely rare anyway and most, I would suspect are caused by human error and an electrical cutout, possibly with a well aimed fire extinguisher would have been a lot better than trashing your disks.
".... and an electrical cutout, possibly with a well aimed fire extinguisher would have been a lot better than trashing your disks."
At 3am on a Sunday morning, it's a good chance a fire will take hold quite well before someone gets into there.
If there is smoke and flames in a confined area, the LAST thing you should be doing is trying to fight it with a poxy fire extinguisher,
"At 3am on a Sunday morning, it's a good chance a fire will take hold quite well before someone gets into there."
My point is, you don't generally get servers bursting into flames at 3am on a Sunday morning. The risk of a server room fire in a decent, modern server room (one where they would have thought about creating a well running fire suppression system) is extremely small.
A fire will often either start externally and engulf the server room anyway or it will be caused by someone in the server room who will be able to grab a fire extinguisher.
Something that can kill the power will most likely be just as effective.
"faulty component" ... "blowing up"
That is not a fire. A component can easily blow up without a fire starting. there is generally a lack of fuel source.
The main area for fires is the UPS. If they turn Lithium there could be a bigger issue with server room fires, but until then servers don't blow up and catch fire. You'd be far better investing in backup and redundancy.
That is not a fire. A component can easily blow up without a fire starting. there is generally a lack of fuel source.
40+ years of working with electronics says otherwise.
Anything that carries a current can burn. Anything with an electrolytic capacitor can pop, and quite enthusiastically so. Anything hot enough can carry a flame. Sometimes it doesn't even take a defective component. It could be a sparking connector, a power switch with a dodgy contact so it starts to arc, a solder joint might have been just a bit too dry, an elco could be off tolerance, a misplaced cable could rub off insulation and short/spark/arc - the list is endless.
However, in all those years I have only ever seen a computer blow up once as it happens in Hollywood movies, and that was when a system was miswired during an oil rig revision. As soon as it booted up and the relais brought to A/D card online it emitted a loud bang and smoke, and its sturdy steel case went from rectangular to more ball shaped (which, in hindsight, must have saved us from injury because I suspect bits would have flown around otherwise, and we were 5 meters away from the thing when it happened, working on another server (our own, which had full optical insulators which rendered it immune to such mistakes). It turned out that electricians had miswired a junction box, so instead of getting the 30mA loop it was expecting, it was hooked up to raw 400V generator power, and the PC was *far* quicker than the fuse in expressing its dissatisfaction with that arrangement. There really was nothing usable left in the box, the surge had annihilated practically everything and cooked the motherboard and almost every PCB track on it.
It was only at that point that my boss finally stopped grousing about me adding those (costly) insulators to our design because they reduced our profit margin a bit..
And there you have it. No fires just components blowing up.
None of which happened spontaneously at 3am, they happened because of miswires on specialist equipment, not well functioning server rooms.
So my point still stands, spontaneous server room fires in professionally configured and installed rooms, using normal modern server room kit are almost (if not completely) non-existent - if you factor in ones where a fire-suppression system would have helped where an auto power cut wouldn't have then you be into the realms of - more likely to be hit by lightning.
I can confirm that the phenomenon of Inergen discharges killing HDDs is well known in the drive industry.
The standing advice is "don't do it for shits and giggles"
/works in the drive industry
//but mostly on flash platforms
///Once saw the aftermath of a test with CO2 in a live data centre that blew down the entire false ceiling, together with decades of crud and cabling. Executive summary as overheard from next room: Click....whoooooooOOOOSSSSSHHHHHHH.....BANG...OH SHIIIIITT!
I've had a customer affected by this in the past, if I recall it's down to the frequencies emitted by one specific design of nozzle, and there are advisories out there for the fire suppression systems advising against its use in data centres.
The outage we saw affected 15k rpm drives, across several different SAN equipment manufacturers. 10k and 7k drives escaped largely unscathed.
However the damage was severe enough to totally offline around 50% of those 15k rpm drives, to the point that they wouldnt even come back after a power cycle.
"Don't most banks have some form of backup DC for failover?"
A few years ago I heard about a company that had this but the data corruption caused by the blast of the fire supression (which went off because of a test or accident) replicated to the backup DC, making it mostly useless.
Tricky stuff.
...to the surprise of no-one who's ever seen (even on YouTube*) a Halon dump. Inergen is after my time, but it doesn't sound any better.
Moral of the story: if you're in the room with the supression system on auto - and you shouldn't be - and you hear the discharge alarm RUN, don't wak to the nearest exit. And keep running until you're out of the building and well down the road.
* other tubes are available.
PS: Legend has it that the Amdahl factory in Swords had a halon dump (in the 80's?) because someone left the system in the wrong mode. Apparently it cost 40 grand to refill.
PPS: 130dB seems way too loud for the alarm... I know they're supposed to be loud enough to drive you out of the room, but aren't there standards for loudness.
I remember being onsite when a customer had a Halon system installed to protect their shiny new AS400, the trainer basically said, if the alarm is triggered run for the exit, if the system triggers it will bring down the ceiling tiles at the same time that it blow random floor tiles into the air and you're disorientated by the 130dB gas discharge and lack of oxygen. They'd recently done a full system test (with the Fire Brigade onsite) at a different customer/location and no-one made it out of the server room.....
I spent a week working as a temporary relief at a combo office/warehouse site before finding a few days later, that - due to people forgetting to switch it back to "Auto", the Auto/Manual Halon switch had been bypassed, and it was in Auto all the time.
Add this to everything else I found in the building,
no effective earthing (unless the voltage was high enough to jump an 8 inch gap)
bypassed auto fire door switches (and the doors themselves wedged open)
crimp connectors joining power cable in the middle of trunking (and two different colour cables!!)
Single phase isolators on two phase equipment supplies (someone got fried and also suffered thermal burns from the heating element their hand was wrapped around).
And HUNDREDS of circuit breakers wired in bypass mode.
Some of this stuff had been in place for 25 years without any of the regular staff picking up on it!!!
When I brought it up I got fired.
Kay&Co. AKA GUS, AKA Great Universal Stores,
HOW BIG BUSINESS WORKS
Feeling a bit disgruntled (as anyone actually BEEN "Gruntled"??), I handed my baby brother a full defect list, and he and a Fire Brigade team visited the place shortly afterwards.
After finding everything on my list (it ran to eight pages), the Brigade presented GUS management with the list and said "No fix, no fire certificate"
Management response was "Throw that list away, or we make everyone redundant, close the place, and move operations to Burnley"
The Brigade folded and the issues were never fixed.
10 years later GUS STILL closed the place and moved operations to Burnley.
It makes you wonder what sweetheart deals they have got going on up there.
"I spent a week working as a temporary relief at a combo office/warehouse site before finding a few days later, that - due to people forgetting to switch it back to "Auto", the Auto/Manual Halon switch had been bypassed, and it was in Auto all the time"
Even when it's in manual mode, the "discharge" button is normally active and only under a clipdown cover.
We had (and have) a very real fear of some practical joker pushing the thing for shits and giggles (and it's not because of the £8k refill cost)
@ Alan Brown
HSE were part of the Fire Brigade Inspection team; I personally knew the guy leading the team and he was furious that the decision to drop it was made by higher ups.
I appealed and my appeal was upheld, but GUS DIrector of Electrical Services, even after agreeing with me, said he wouldnt be giving me my job back, and paid me off.
As I had been there less than 2 years (by a few weeks!!) I didnt get much.
Before quitting the industry, I refused to KowTow over safety three times, with one other international and one small, local firm as the other culprits.
I walked out on the small firm after being told to "hold my breath" after asking for a working gas meter, and the big firm sent me home, then, during the disciplinary meeting, I was bollocked for reading SciFi during my lunch break, the reason I got sent home was NEVER raised!!!!
(Refusing to put a holesaw through a trunking full of live 240v and 440v cables).
Yeah, thats the trouble with being multi-skilled and pretty good, they ask you to do EVERYTHING, from installing heavy machinery, to prototyping circuit boards, to moving server racks.
"Some of this stuff had been in place for 25 years without any of the regular staff picking up on it!!!
When I brought it up I got fired."
I suspect that your second sentence gives the reason why they were the regular staff and you didn't get to be there long enough to become one of them.
"All the rubbish you thought had been cleaned up after the DC was built, well hidden more like............"
I made a point of lifting the entire tiled floor, rubbish picking (there was a lot) and vacuuming it along with every crevice before moving anything into ours. Along the way it was discovered that none of the earths were connected.
No false ceiling thankfully but we did find a mountain of shit on various ledges.
I had the pleasure of visiting a building where the architects managed to design the server room with a FOUR FOOT flloor void rather than the desired 4 inches. Somehow this cock-up of NASA dimensioning proportions made it through all checks and that's how the building was constructed.
By the time the building was completed and occupied, the data centre looked 'normal'... until you lifted a floor panel and stared into the void; this was one data centre where you **really** had to be careful if you left a floor panel out of place! There was so much space below the floor that the IT guys had installed servers down there - as my knees and back dscovered when I turned up to fit a new Ethernet card and was told which tile to lift to start my crawl.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
