back to article Would YOU start a fire? TRAPPED in a new-build server farm

“What’s the smallest fire I could start to be noticed, but not so big that I risk burning down the building?” is one of the stranger thoughts to have entered my head, in many years of working in IT. No, I'm not a closet pyromaniac, so why was I entertaining such thoughts? I had found myself stuck in a data centre on a Sunday …

  1. Pascal Monett Silver badge
    Trollface

    Did I read that right ?

    You went around and randomly yanked on power cables to generate power alerts ?

    YOU MONSTER.

    1. SpeedEvil

      Re: Did I read that right ?

      No, you did not. Redundant power cables are where a power supply is fed from multiple sources so it can use any in the event of failure. If all input supplies are up - pulling the plug on one won't cause anything to happen, except a large 'POWER FAIL' warning.

  2. bpfh Bronze badge
    Flame

    Did you piss of Simon or the PFY?

    because even if the doors and cameras were off line and you were firmly locked inside, I can almost guarantee that the yet-to-be replaced Halon release would work very, very well...

    1. AustinTX

      Re: Did you piss of Simon or the PFY?

      You don't stick around when you pull the Halon lever.

  3. David Roberts Silver badge

    I trust that

    You reported them to Health and Safety for the blocked fire exit.

    You make it all sound humorous in retrospect, but...

    1. Justicesays

      Re: I trust that

      Yeah, that was my thought as well.

      1) Unattended work out of business hours in without regular check-ins

      2) Fire alarm and door access disabled

      3) Fire exit blocked

      4) No adequate supervision of contractors performing works

      5) Location full of open, unmarked voids.

      1. Denarius Silver badge
        Meh

        Re: I trust that

        sounded similar to my last data center experience in a brand new DC. One hall not fitted out was used as a storeage area with very unreliable exit card reader. Easy to get in to storage hall to grab new hardware for server farm under construction, damned scary getting out occasionally. Nothing like solid concrete all around to give a feeling of claustrophobia. Fortunately guards were aware of issue and checked often to ensure no profanities uttered.

    2. AndrueC Silver badge
      Stop

      Re: I trust that

      Some things that sound funny really aren't. Being trapped in a cupboard at home while doing some cleaning sounds funny. But it isn't.

      As I live alone I'm now very concious of the need to always have a telephone handy.

      1. Anonymous Coward
        Anonymous Coward

        Re: I trust that

        I was very happy I had my cell phone when the door to my camper shut and latched behind me. It is a HiLo camper and the top half drops down over the bottom half for travel and storage. It was in storage in my barn and I was checking on the battery to see if it was charged up. The battery monitor is on the inside and opposite the doors.

        The problem is the door is in two parts, one on the top half of the camper and one one the bottom. Both doors open out and the latch is on the bottom of the top and outside door. Works great normally when you are outside of the camper, but when the camper is down you cannot reach the latch from the inside. The battery was weak and I didn't have enough clearance to put the top up.

        I was also lucky one of my kids were home to rescue me quickly.

  4. Anonymous South African Coward Silver badge

    The BOFH can do a better job than this...

  5. phuzz Silver badge
    Unhappy

    I can imagine that being trapped in a building with no access to a loo would be pretty bad, because at some point you're going to have to pick which corner to wee in.

    However, at least in an office you might have the relative luxury of a bin or pot plant, in a data hall you're going to have to pick a spot that won't end up with you getting an electric shock in a very painful part of the body.

    1. fajensen Silver badge
      Joke

      Just lift one of the tiles and do it in there. Those of you who has ever had to go under a raised floor that has been in operation for some years have seen the abyss and knows.

      The only thing I haven't yet seen under a raised floor is a body and a naval torpedo!

      1. Allan George Dyer Silver badge
        Black Helicopters

        Clarification please...

        @fajensen - "The only thing", singular, so you've seen a body under a raised floor, and a naval torpedo under a raised floor, but not both together?

    2. DSGiTechGuy

      I must say in a similar situation (warehouse door release not working), with a dodgy curry inside of me waiting to get out. I did eye up the empty tote bins.... luckily saved at the last minute by someone who almost ended up joining me locked inside.

  6. Pinkerton
    Unhappy

    What if there had been a real fire...

    ...And the Halon had gone off?

    (Why do I always think of the worst possible outcome?)

  7. Daz555

    You could have had a wee into the floor space - hopefully their floor moisture sensors would have triggered a response.

  8. Anonymous Coward
    Anonymous Coward

    Been there. Done That

    Never had to resort to raising alarms though.... At one data centre, the access card would let you out at any hour, but not back in after 6PM. An alarm would be raised if the door was open for more than thirty seconds. That made going to the toilet somewhat problematic, if you were there by yourself.

    1. Anonymous Coward
      Anonymous Coward

      Re: Been there. Done That

      Presumably if you came across what looked like a forgotten bottle of apple juice you'd leave it well alone

  9. Anonymous Coward
    Anonymous Coward

    sounds like the site operator couldhave been fined really

    agree with posters above that the situation sounds more like some 1800's factory not a data centre - even a relatively old one.

    1. ChrisBedford

      Re: sounds like the site operator couldhave been fined really

      sounds more like some 1800's factory not a data centre

      Huhm or some 20-teens sweatshop in China or Bangladesh or the Philippines. Or South Africa.

  10. The Real Tony Smith

    Could have been worse

    You could have been locked in with some alien eggs.......

    https://www.youtube.com/watch?v=hU1YaowhYKM

  11. Disk0

    Sorry but...

    ...in this situation, power cables would stay out until I get out. You were locking in a death trap, with no way out, there is no amount of hazard pay that can justify that imo.

  12. imanidiot Silver badge

    stern words warranted

    That situation warrants at the very least a sternly worded letter to your boss and the highest boss of the DC involved. It all worked out this time, but that was a seriously risky situation to be in.

    Also, am I the only one who thinks it strange someone is allowed (let alone wants) to work in that situation without a buddy present? Accident happen. Even a minor one can turn deadly if there is noone to help you out. At the very least it'd be someone to talk to to while the time away.

    1. SImon Hobson Silver badge

      Re: stern words warranted

      Indeed, SOP for people who work alone and/or are particularly vulnerable should be to have some sort of check system. Either you have to call in at set times (and if you don't they start looking for you), or you have people checking in on you periodically (there were guards at this place right ?).

      In this case the site operator was criminally negligent and as above should have been reported for H&S breaches. At the very minimum, their security staff should have been checking on him at regular intervals.

      I have to admit that I've often had situations where I find myself thinking "that wasn't too clever, what if ..." when I realise that the wife won't be home to find me for another 8 hours. And I know for certain that 8 hours is well past my "personal endurance" ;-)

      1. imanidiot Silver badge

        Re: stern words warranted

        My current place of employment has a mandown system for those who have to work alone for some reason. Basically an oldtimey cellphone shaped thing you clip to your belt that starts beeping if you havent moved for over 60 seconds. You then get 30 seconds to cancel the alarm or it sends an automated alarm to the securitystation and the security callout line and also starts beeping like all hell.

        I personally hate the things but do see the use.

        1. quartzie

          Re: stern words warranted

          Handy as these things are, they probably rely on the missing cell signal...

          Wouldn't have been much help in this case.

          1. Stoneshop Silver badge

            Re: stern words warranted

            They usually work on a local radio system and I've also seen walkie-talkies and pagers with such a function. With everything inside the Faraday cage there's no reception problem. That said, for the two commercial DCs I've had to work in I had weak but sufficient phone and UMTS signal.

        2. ChrisBedford
          FAIL

          Re: stern words warranted

          starts beeping if you havent moved for over 60 seconds

          There would be a constant chorus of beeps if you implemented this at any given South African government office

  13. Phil W

    Design flaw

    "When I got to the exit however, my access card wouldn’t unlock the door."

    I find it very frustrating when i encounter doors to rooms like this that require a card for both direction through the same door, when the room in question has no other exits (besides fire exits which can't be opened from outside).

    Knowing who entered a room and when is very useful, but it is very rare that it is actually truly necessary to know who has left a room and when, a simple push to exit button would be more than sufficient in most cases.

    1. John Gamble

      Re: Design flaw

      But opening the door to exit and opening the door to let someone in are indistinguishable. If you're security-minded, you want to track comings and goings.

      But everything else... yeah, the lack of safety standards is appalling.

    2. cd

      Re: Design flaw

      How do these readers work? Could one just knock the reader off the wall and short some wires?

      1. Anonymous Coward
        Anonymous Coward

        Re: Design flaw

        No, the wires to the reader usually run to a control box somewhere that actually does the door control. The reader just reads.

        Of course, if you can get to the control box under the floor you might be able to turn off the mag-lock.

        1. Antron Argaiv Silver badge

          Re: Design flaw

          Mag locks, as I learned, come in two varieties: powerfail unlocked (required in the US by fire code for fire exits) and powerfail locked. Care to guess which one the data center in question would have used?

      2. Captain Boing

        Re: Design flaw

        no more star trek for you. :o)

    3. twsm

      Re: Design flaw

      You have to use the card to get out for security reasons, to prevent you passing your card to somebody outside so they can enter, i.e by pushing the card under the door or give or giving it to somebody who is leaving to allow somebody else to enter.

      One company that I visited would not allow you to use your card to mleave if you had not used it to enter, i.e. you cannot need to leave if you have not entered.

      1. Bob Dole (tm)

        Re: Design flaw

        This is too funny. Pass the card under the door? Why not just *open* the door and let them in? Far easier and without the fake Mission Impossible crap.

        What, exactly, do they think they are stopping here? If an authorized person is going to let an unauthorized person in then they will do it.

    4. ChrisBedford

      Re: Design flaw

      simple push to exit button would be more than sufficient

      Indeed, and when carrying notebook, toolbox, other toolbox, miscellaneous flyleads, and sundry hardware having to fumble around with a swipe card can be, umm, the last thing you need.

  14. kain preacher Silver badge

    This not even close to being funny. Some could of had a medical emergency and died cause fire/EMs could not get to them. I'm not a litigious yank but I do hope they get taken to the cleaners. better yet jail time.

  15. aurizon

    Well, stay there all weekend, think of the double and triple time you will get -- maybe

    1. John Brown (no body) Silver badge

      "Well, stay there all weekend, think of the double and triple time you will get -- maybe"

      Eating dust bunnies and drinking your own piss? I'd want a lot more than double or triple time for that!!!

  16. well meaning but ultimately self defeating

    Could you not

    Get a network connection and email someone. Surely you carry a bootable USB with you?

    1. imanidiot Silver badge

      Re: Could you not

      Did you not read the article? His laptop had already died, and there was no other equipment around...

      1. Stoneshop Silver badge
        FAIL

        Re: Could you not

        And to use that USB stick you want not only a system to be booting off it, but also a screen and keyboard. Which weren't there; if they were he could have logged into one of their systems and sent a message even without rebooting one.

  17. Stevie Silver badge

    Bah!

    And now for something completely different:

    Why oh why are we saddled with Unions who insist on all sorts of daft regulations and overmanning for every job and [fade out rant] ...

  18. ShadowDragon8685

    Good thing he went with the power cables and not the fire alarm, and yeah, someone should be paying for this in court.

  19. Me19713
    Mushroom

    I got locked into our NOC once, years ago. Despite my very explicit instructions that all electric door strikes must fail to the unlocked state, the contractor did whatever they please.

    I called facilities, but they couldn't get the door to release either. Fortunately, we had a big plate glass window into the lobby (so that people could see the pretty NOC without having to enter the NOC).

    I'm a pretty big guy, so I simply threw a swivel chair through the larger of the windows (the most expensive one) as a sign of my displeasure. Once I got access to my tools, I ripped the entire door frame out of the wall.

    I had no more trouble getting into and out of the NOC that weekend. :-)

    1. ChrisBedford

      threw a swivel chair through the larger of the windows

      Nowadays, of course, said plate glass would be bulletproof (or at least chairproof) so at best you are going to cause a few small cracks...

  20. Jim in Phoenix

    I would have raised hell if I had been in that situation. Now if I ever go to work inside one of these glass prisons, I'll take a hammer with me. Until now it never occurred to me that a 20 pound sledgehammer would be useful if one worked in IT.

    1. Antron Argaiv Silver badge

      Have an upvote, Jim, because a large hammer is *always* useful in IT...for very delicate adjustments to expensive equipment that refuses to see reason.

      "These look like hammar marks..."

      1. Triggerfish

        delicate adjustments to expensive equipment

        Really I tend to think of users as cheap throwaway items...

    2. imanidiot Silver badge

      Percussive maintenance is a tried and tested method proven to be 100% effective in showing why equipment is non-operational.

    3. kain preacher Silver badge

      a center punch works better and requires less force.

    4. Curtis

      United Cutlery makes a very nice "M48" hammer that I used to carry with me specifically for breaking glass or wood doors down. Management ONCE asked if it was necessary, and I just responded that I could bring the tomahawk (same line) instead. Never a question after that, but I was put on the Saftey committee responsible for fire drills, alarms, exit notices and such.

      Which then gave me the excuse to bring in 10 stories worth of climbing rope (7th floor) and 10 rappelling harnesses, with enough 550 cord to attach to them and haul back up. Management was grateful I never got to use it.

    5. Captain Boing

      or... like me, get yourself an automatic centre-punch. *click* and it shatters enough for you to just knock a way through. Great for car/coach/train windows too. About the size of a chunky pen so easy to pop in your tool wallet

  21. Herby Silver badge

    Just so you know...

    Most fire sensors are removable for maintenance purposes. When they are removed, the lack of a sensor usually raises an alarm. So, instead of "starting a fire" simply remove one of the sensors (commonly they "unscrew" a 1/2 turn or so). While it won't indicate a fire, it will indicate a fault which should alert someone.

    The other alternative I've seen is that some sensors have a magnetic reed switch in them for testing purposes. If you hold a magnet up to the sensor, it trips the alarm. The person testing the sensor wiring used a magnet on a stick to hold it next to the ceiling sensor. Truly a BOFH device when used properly (around noon on Friday).

  22. Anonymous Coward
    Anonymous Coward

    Done the reverse before

    Kick started a potenitally final build, nipped outside for a smoke while the 15mins of automated tests went though their paces. Went to go back in and relealised that my pass, building keys, phone and wallet were on my desk. No way back in and no way to contact anyone and the building alarm was behind a locked door. Luckily the toilet did not need a door pass.

    I had to sleep in the foyer, fortunately I woke before the first person to arrive the day after came in and pretended to have come out from the toilet, followed them through the door, went to my desk and committed my changes and it was assumed I worked all night to get the software ready for the testing before it was released the following day so got sent home for some sleep with a pat on the back and a full night of overtime.

    1. ChrisBedford
      Pint

      Re: Done the reverse before

      a pat on the back and a full night of overtime

      You sneaky rat. I'd have done the same ;)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019