back to article Thundering gas destroys disks during data centre incident

A few weeks ago in a Sydney data centre, something started making smoke. As its designers intended, the centre's reservoirs of inert gas started venting to stop any fire spreading. The smoke was contained, but not long afterwards tenants started to complain that some of their disks had not survived the incident, which was odd …


This topic is closed for new posts.
  1. myxiplx2

    Yup, kills SAS drives

    Saw this myself last year, we had a customer run a fire alarm test of some kind, fried a huge number of SAS drives. From our systems around 50% of the SAS drives were completely toast, unrecoverable. The SATA drives were fine, and I understand it was a similar story with other vendor systems too.

  2. LaeMing Silver badge

    Speak softly

    And carry a big disk.

  3. M Gale

    See also

    Don't get angry at your disk arrays. It upsets them.

    1. Terry 6 Silver badge

      Re: See also

      I've often told staff there was no point getting cross and shouting at the computer when it goes wrong, because it won't won't make any difference.

      Maybe I was wrong.....................

  4. Voland's right hand Silver badge

    Utter BS

    Most modern disks will get damaged if you significantly change the air composition and/or air density (they all have air vents). The head design relies on aerodynamics to maintain the correct distance from the platter. Air density affects that quite a bit.

    Just ask any Colorado resident - the stats for disk reliability around that parts are way off compared to stats in normal places around sea level. Some models which fare perfectly fine elsewhere regularly fail in 60 days or less.

    So the fact that someone releasing fire extinguishing gas (which is quite dense by the way) has changed the MTBF of drives does not surprise me in the slightest. SAS is likely to fare worse too as it spins at higher revs and is engineered to lower tolerances.

    Vibrations from "whistling", yeah some other time, b*ll*cks... The "storage engineers" that were asked need to get a clue what is a modern disk, how it is designed and operates.

    1. Mot524

      Re: Utter BS

      Lived in CO most of my life. Never experienced or heard of the air pressure negatively impacting hard drives, including friends who work at IT installations at CU Boulder and who live/work at ski resorts at 9000+ ft. (Also, airplanes are usually pressurized to around 8000 ft equivalent. Haven't heard of any hard drives crashing when used on an airplane.)

      1. Enzantl

        Re: Utter BS

        Short life! In the mid-80's (when dinosaurs still roamed the data center) DEC had hideous problems with their RA80 14" winchester drives. Apparently nobody had taken into account in the design cycle, that some highways in the US (Colorado in particular) reach altitudes of 11,000 feet / 3350m. As I recall, the disks were in sealed units with slightly pressurized gas; the seals failed at something like 9,000 feet / 2700m, the gas leaked out & or air leaked in, bad things ensued. Major sadness in the sales, services and support community. Took a bit to suss out that little mess.

  5. Anonymous Coward
    Anonymous Coward

    Rapidly released compressed air is freezing cold, I wonder whether there's a thermal shock factor to dumping enough inert gas into a room to cause an issue. This sound thing needs some more evidence to be compelling.

    1. Anonymous Coward
      Anonymous Coward

      "This sound thing needs some more evidence to be compelling."

      Watch the video ...

  6. Secman

    Not unprecedented

    This happened at Westhost too, May 2010.

  7. Steve Groom

    Oil paint

    I remember working for a large mainframe customer in the eighties where, over one weekend a large number of 3380 hard disks mysteriously crashed. The culprit was later found to be the use of oil based paints for renovating the data centre. Disk heads float on air, air + thinners = head crashes all round.

  8. Anonymous Coward
    Anonymous Coward


    Upon analysing our old Halon systems, prior to retrofit, we found a teeny weeny flaw. Before removed the old kit he guys looked at the system (along with some internal guys that know fire suppression systems way better than me), realised that the release would happen so fast (they estimated less than a second or two), that you wouldn't have time to die from asphyxiation, as the floating floor tiles being blown several feet into the air are far more likely to kill you.

    1. Anonymous Coward

      Re: Pah!

      Should have hired Burroughs. Their contractors would have perforated (drilled) the floor the data center...

    2. MrDamage

      Re: Pah!

      Email that anecdote to Simon Travaglia. I'm sure he would be glad to hear of another way he can remove a boss once he is locked in the server room.

      Speaking of Simon, where is he? I'm missing my monthly dose of psychosis. If I start suffering withdrawal symptoms, it will not bode well for the muppets I have to endure at work.

  9. Ed 13
    Thumb Up

    Nice demonstration

    It's an impressive video, but I am rather concerned he ought to have been wearing some ear protection in that sort of background noise level.

    He did seem to have to shout quite loud just to be heard!

    1. tfewster Silver badge

      Re: Nice demonstration

      Which begs the question - does the noise from the A/C and servers cause a problem too?

  10. John Smith 19 Gold badge

    Holy s**t.

    I would not have believed if if I had not seen it.

    Note it's not the level it's sudden (and steep) changes in noise level that does it.

    BTW. From nozzle theory almost any kind of pipe that has a gas tank with more than about 2 atm pressure difference between it and the exhaust will give an exhaust at the speed of sound.

    Low volume (70 psi) pneumatic industrial control systems leak like this.

    It's very loud.

    1. M Gale

      Re: Holy s**t.

      Hah, I distinctly remember cracking the valve open on a 5 foot tall helium bottle minus its balloon regulator.

      Never again.

  11. Aybee

    B*ll*cks? B*ll*cks!

    'Vibrations from "whistling", yeah some other time, b*ll*cks... The "storage engineers" that were asked need to get a clue what is a modern disk, how it is designed and operates.'

    What won me over was your carefully researched analysis and the trouble you took to substantiate your hypothesis.

    Have you watched the YouTube video linked above, BTW?

    That guy is a lot more slapdash in his scientific approach than you are. He just uses graphs and stuff and doesn't even bother putting his shouty words into this comment stream.

  12. Anonymous Coward
    Anonymous Coward

    Happened to us

    Direct lightning strike on the roof triggered the halon system. Even though the room was supposed to be suitably vented, the overpressure from the gas shifted some of the interior walls as well as destroying the false ceiling. Not at all surprising that some disks suffered, that sort of step pressure change could easily cause "flying" heads to bounce enough to polish the spinning rust.

    Remarkably, between a good UPS that absorbed the electrical surge, and proper backups, there was surprisingly little disruption to normal operations.

  13. Phil O'Sophical Silver badge

    Pressure changes can do all sorts of unusual things. In our Belfast office, 30-odd years ago, a car bomb in the street oustide shattered windows and broke internal walls. The offices had a good supply of VT100 and VT220 terminals. All the VT100s survived, all the VT220 screens were OK, but many of the VT220 keyboards didn't work afterwards. We never did identify which component of a keyboard was that pressure sensitive!

    1. Anonymous Coward
      Anonymous Coward

      " In our Belfast office, 30-odd years ago, a car bomb in the street oustide "

      --- Customer support here, good morning. Yessir, bit of an outage. No, nothing serious Sir, just another car bomb. We'll be back online in a few mins, just wiping off the glass shards now.

      1. Phil O'Sophical Silver badge

        --- Customer support here, good morning. Yessir, bit of an outage. No, nothing serious Sir, just another car bomb. We'll be back online in a few mins, just wiping off the glass shards now.

        You may laugh, but the explosion happened at 4pm on a Friday (we were in the pub, the building having been evacuated). The emergency plan went into action, some empty office space was acquired and cabled over the weekend. DEC delivered a lorryload of new equipment at 9am Monday, and by lunchtime everyone in the affected office had a desk with an operational phone & terminal. One customer who phoned expecting that their scheduled meeting was cancelled was more than surprised to be told that everything was still on track.

        DR planning works!

        Mind you, for weeks afterward we were opening files that had been inside steel filing cabinets, and shaking the broken glass out. Made us realise what that trite BBC phase "some people were cut by flying glass" could really translate to.

  14. Anonymous Coward
    Anonymous Coward

    mentioning no names...

    Happened in a London tier4 datacentre when fire system was test fired in 2012.

    Cue lots of swearing, lots of new disks, and ultimately, lots of lawyers.

    I can say no more...

  15. Anonymous Coward
    Anonymous Coward

    Posting anon for good reason, I work in enterprise storage.

    From an enterprise storage support prospective, I have seen this a number of times.

    In most cases the drives/data were recoverable, in a couple it was not.

    A lot of contractors seem to be putting together way too big of a system for the room (more profit for them). Also remember you have to "fill" the room quickly, you have sonic issues, you have a shock wave, you have a rapid change in room temp (it cools quickly), in addition to the pressure changes.

    It's gotten to the point that I would highly consider a well planned dry pipe water based system with proper drainage vs a gas based system. The only equipment to get soaked will be the equipment in the area of the fire (since water based systems only activate the head under the high intensity heat source), and you don't have to worry as much about the rest of your equipment.

  16. Anonymous Coward
    Anonymous Coward

    Gas release systems are there to stop the drives being damaged by fire.

    So, technically, they work, albeit someone may have misconstrued the requirements.

  17. Hairy Airey

    Never mind the disks...

    A sudden pressure change from these gas systems could perforate both your eardrums. Having worked in one of these centres I worked out how far I could run in the 30 seconds the system could be held off for. I reckoned I could just get safely clear before the gas went off because the stairs were a lot further away than the lift.

    If you work in one of these environments and don't run regularly - take up running now, because nothing can replace your hearing.

  18. pierce

    I doubt its SAS vs SATA. however, SAS disks in a data center are far more likely to be 2.5" 10k or 15k rpm, while SATA disks are much more likely to be 7200 rpm 3.5".

    just saying....

  19. RobHib

    Wonderful stuff

    Wonderful stuff, just proves how small those tracks on HDs really are.

    It's also illustrative of how vulnerable HDs really are, and thus how perilous our data is whilst it's held on mechanical drives.

  20. Rathkennamike

    This is a know problem with fire suppression systems in data centres. The gas nozzles must be fitted with diffusers to reduce the shock wave without reducing gas dispersion

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019