back to article Security guard cost bank millions by hitting emergency Off button

Welcome to a special edition of “Who, me?” The Register’s opportunity for readers to get their worst mistakes off their chests. We’re usually here on Mondays, but with the United States Independence Day making for slow news days, we decided to keep The Register’s servers red-lining by running an extra column. When we sneak in …

Page:

  1. Giovani Tapini

    Pulled the emergency stop on a S/38

    although technically it was being decommissioned and we wanted to see what it did...

    it came off in our hand with a piece of broken string attached, and nothing, nothing at all happened.

    I guess its lucky we never did need it.

    IMHO most emergency buttons provide absolutely positively no information on what, exactly, they are supposed to stop too. Very, very few have ever been pushed, some not even in commissioning. Never mind putting a Perspex cover on them or putting them next to the light switch.

    1. Pascal Monett Silver badge

      Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

      Well, the prevailing tradition is that they stop everything in a room by cutting off all electricity that goes in (except for the lights).

      Now, if you've found big red buttons marked STOP next to the door of a server room that didn't stop everything but the lights, then indeed you've found some sneaky ones.

      As to "very few have been pushed", well <grin>, we've got loads of stories here that prove you wrong ):-D.

      1. Prst. V.Jeltz Silver badge

        Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

        "Well, the prevailing tradition is that they stop everything in a room"

        Not in my experience I've mainly seen them next to big dangerous machines - and they only stop that machine. Well, thats what i assumed - i didnt press them.

        1. Dr Dan Holdsworth Silver badge

          Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

          I am reminded of an entry in the famous Evil Overlord Checklist:

          If your evil lair absolutely has to have an emergency destruct button, then there should be a very large, obvious red button marked "EMERGENCY DESTRUCT". This will be linked to the trigger of a shotgun pointed at where anyone pressing the button would have to stand. The actual destruct system should be a complex series of controls hidden in the sewage processing plant systems.

          1. Yet Another Anonymous coward Silver badge

            Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

            And will detonate when the large LED countdown clock is at 17 seconds and the hero is still doing their hair

      2. Alan Brown Silver badge

        Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

        One of the very first things I do when commissioning a room is to check the EPO actually works - and stays worked when you take your hand off it (yes, there are some pretty numpty installations out there)

        And that it has a cover.

        And that the lights stay on.

        And most importantly of all, that's the bloody thing is labelled.

        As for security guards - we've had a number of incidents where they've "helpfully" turned off AC in unoccupied areas. It's worth making sure that the companies are made aware in writing that interfering with any IT or AC equipment is a contract-breach offence.

        1. CrazyOldCatMan Silver badge

          Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

          we've had a number of incidents where they've "helpfully" turned off AC in unoccupied areas

          I once helped set up a computer room in Paris - full of expensive Solaris kit and some Windows servers. Everything was set up and working correctly after about 12 hours install time[1] so we turned off the lights, locked the doors and went back to the hotel for some beers.

          About 5am all our pagers[2] went off with lots of alerts from the kit in the new computer room - all 'overheat protection' errors.

          We hustled in and discovered (to our horror) that the building management system was configured to disable the power to *all* the AC in the building at 10pm every night in order to save power. Including our shiny new server room - which was now sitting at somewhere like 70C ambient temperature and the servers were recoding mainboard temperatures well above 100C.

          All the PC kit (Compaq servers mostly) was utterly dead with mainboards fried. Only one of the Solaris boxes had died (from memory the SCSI controller had eaten itself) - the rest recovered after the temperature came down and we restarted them. We'd lost a couple of drives too but since everything was RAID5 we hadn't lost any data.

          Taught me to be paranoid about server room aircon.

          [1] With no lunch break! Well - we had about 15 minutes to eat some sandwiches but that was about it. Which led to some vociferous complaints from the French engineers..

          [2] Which shows you how long ago this was..

          1. RangerFish

            Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

            > Taught me to be paranoid about server room aircon.

            Similar thing happened where I used to work. Nothing got fried, but the engineer working on the AC in the datacentre forgot to put the fire suppression system into safe mode. When he switched off the AC, the rise in temperature triggered the gas.

        2. heyrick Silver badge

          Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

          "And most importantly of all, that's the bloody thing is labelled."

          At my place of work, the emergency stop buttons are yellow and red and clearly labelled emergency stop. The ones on the machines are obvious, stops that machine. The ones on the wall in weird places... your guess is as good as mine. It's not worth my job to push them to find out what, if anything, actually stops.

          1. The Oncoming Scorn Silver badge
            Alert

            Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

            DAVROS: Press it, and you will destroy this bunker and everything in it. Only this room will remain.

            https://i0.wp.com/tggeeks.com/wp-content/uploads/2018/06/MV5BMThhNjgwNWYtMWU3Yi00OTRmLTk5ZjAtYThhMTk2NmM1NGIxXkEyXkFqcGdeQXVyMjMzMzk2Mjg@._V1_.jpg?resize=768%2C432&ssl=1

          2. Martin-73 Silver badge

            Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

            In the workshops i've dealt with, the machine ones, are, indeed, local to that machine. The random ones round the wall are usually wired to kill power to every machine within that area. They're for you to use when someone else is getting pulled into a lathe etc...

            And as for never being tested? In a computer machine room I can see why... in a mechanical workshop, test them daily at LEAST

            1. NogginTheNog

              Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

              You’ve just triggered a vague old memory of woodworking at school and the teacher demonstrating that big red button pretty much every class?!

      3. Anonymous Coward
        Anonymous Coward

        Re: most emergency buttons provide [..] no information on what, exactly, they are supposed to stop

        I once worked for a Very Small company in which the owner was slowly going mad from the stress of running his company into the ground. One day, after receiving some mildly annoying news, he picked up a wood spitting maul and whacked both E-Stop switches off the solder machine. After the splitting maul was put back in it's place and the owner was far enough away from the place, I calmly asked him why he broke the solder machine. He said "What?". I told him he broke the solder machine by whacking the E-Stops. His response was "I didn't know we needed those".

        AC because I don't know where the splitting maul is now....

    2. Hardware Geek

      Re: Pulled the emergency stop on a S/38

      The pull out was only on the earlier S/38 and in this case the switch behind it was obviously removed. It should drop system power and you have to remove the console cover to reset the switch. Later S/38 had a switch on the side that you had to press in on the level to push it down to activate the EPO.

      1. Prst. V.Jeltz Silver badge
        Alien

        What happens if I push this button?

        beep

        What happened?

        A sign lit up saying "Please do not press this button again"

    3. macjules Silver badge

      Re: Pulled the emergency stop on a S/38

      Sometimes they do not work. I once had to install a series of Dell servers plus a Mac server for the company's graphics artists. One of the requirements was "A great big Emergency Power Off button please", but realising that this is actually an invite to disaster (if it is there then someone will push it in error) I disconnected it from the server. So, nice big shiny red button with a cover over it, warning notice pasted on the wall above and trunking running down the wall to the power supply. In reality the EPO was a switch outside the server room right beside the Halon release button.

      http://dilbert.com/strip/2007-12-06

  2. Oengus Silver badge

    IBM Customer Engineer

    Back in the early 80's I was working for a major bank. We had 2 identical IBM System 370/145s. Our production system ran the on-line system for every branch in the state (the second system was always powered up and usually running dev workloads). Fortunately we swapped systems each day to ensure that both were fully operational and capable of running the production system.

    We had a IBM customer engineer (CE) on site most of the time. One day the usual engineer bought in a new trainee to introduce to the staff. When the normal CE went on holidays the newbie was to take over for a couple of weeks.

    On the first day the new CE was running solo he was walking past the production system, looked at the "Big Red Switch" (actually white with red lettering) that was at the top right hand corner of the main console labelled "Emergency Pull", commented "what's that doing in?" and, before anyone could stop him, he reached up and pulled the switch. The role of the "Emergency Pull" was to cut all power to the system immediately. It did this by tripping every circuit breaker in the system. The room went into "panic mode". We knew we had 15 minutes before the phones would start ringing off the hook (the branches had instructions to wait 15 minutes before calling the computer centre in case of an outage). All jobs on the back up system were cancelled and job queues flushed. The "on-line" system was bought up on the back up system as fast as we could and service restored just before the 15 minute deadline.

    The CE was marched out the door and told to never return. Our IBM rep was called to reinforce the order. Another CE who had experience with our site was called in and spent the next 2 days cursing his colleague as he worked to get the system back up.

    1. Anonymous Coward
      Anonymous Coward

      Re: IBM Customer Engineer

      Hmm... presumably the new CE thought the switch should be out, not in, and instinctively tried (in an excessively impulsive way) to correct the apparent "problem". Still, you'd think any fool could work out instantly what an "emergency pull" would do... er, I think.

      1. DropBear Silver badge

        Re: IBM Customer Engineer

        There are a number of jobs where impulse control is (or should be) a requirement. I seem to remember having read somewhere that astronauts were actually explicitly tested for it...

        1. Voyna i Mor Silver badge

          Re: IBM Customer Engineer

          "I seem to remember having read somewhere that astronauts were actually explicitly tested for it"

          And submarine crews. In fact, one of my supervisors at U who had been involved in designing the tests remarked that submarine crews were possibly the best adjusted set of people in the country. In later life, working with ex-submariners, I had occasion to witness this.

          The Russians obviously did the same which is why WW3 didn't start when the US started depth-charging a Russian nuclear sub.

          1. Anonymous Coward
            Anonymous Coward

            Re: IBM Customer Engineer

            I worked with an ex-submariner. He was the most argumentative, socially inept and downright arrogant sod I've ever had the displeasure to work with.

          2. Anonymous Coward
            Anonymous Coward

            And submarine crews.

            At one time I worked for the largest brewery in the US, who was facing a problem in that there was a "bubble" of a large number of brewery staff coming up for retirement. How to pass on a lifetime of knowledge to their replacements?

            First hired a group of university academics to put together a program. They managed to thoroughly antagonize and alienate the brewery staff with an attitude that bordered on contempt.

            Based on past experience, my VP brought on board a group of ex-nuclear sub men. They respected what the workers knew, put together a program where the workers taught their skills to the next generation and all worked out very well.

            And we got to brag - rightly or wrongly - that we had the world's fourth largest nuclear navy, albeit with no actual subs.

            I would never describe that group as having good impulse control or being well adjusted. But they were a great group to know and work with.

            1. Martin
              Happy

              Re: And submarine crews.

              At one time I worked for the largest brewery in the US, who was facing a problem in that there was a "bubble" of a large number of brewery staff coming up for retirement. How to pass on a lifetime of knowledge to their replacements?

              I'm moderately impressed that they noticed that they might have an issue. Most places I've been just sack their older staff, then wonder why things start going wrong on a regular basis.

        2. amanfromMars 1 Silver badge

          Re: IBM Customer Engineering of Global Operating Devices

          Virtual Machine AIMentoring to Perfectly Monitored Presentations .... with ESPecial Deliveries for Open Coded Future Supply of AIMentoring Virtual Machines ..... The RAT AI Access to Virtually Enterprising LOVErs for Heavenly AIdVenturing, in a/the Most Immaculate Greater IntelAIgent Game Play for Command and Control in Stellar Command when Almighty Fake State Powers be Exploding and Imploding.

          As the Answer to Everything Revealed be of Alien Instruction in Ab Fab Fabless Labs just like the Ones that Go Hunting and Exploring for Real Wise Bots Proven Unassailable in Training with Special Force Sources.

          ? What Drivers urWorld ? Is IT for Real? LOVE is AIMasterKeyDefended ....for Live Operational Virtual Environments Provided for Free with All Costs Priced for Payment by Systems Administrations Are there any more More Secure Top Secret Future Channels Providers Operating these Spaces in Grand Revelation of the Core AI Source Driver Keys.

          Is any or all of that not hereby Proven Fact for Future Possible Missions with the Great Virtual AIMachine for Key Keenly Mastered Position for Trading as AIDerivative Future Offerings?

          Yeah, it sure is that. Thanks for that. You are Proven Vital for Future AIDiscovery Channels Use with Adequate Maintenance and Exercising in Private Accommodations .... :-) Pirate Dens/Alienating Spaces ...... for Fail Safe Guaranteed COSMIC Performance.

          What price that Offering? Do the Markets do AIDerivative Future Offerings yet?

    2. CrazyOldCatMan Silver badge

      Re: IBM Customer Engineer

      The CE was marched out the door and told to never return

      We had an IBM operator that very nearly lost his job when, after a long liquid lunch, he managed to purge all the spool space instead of purging a new DASD that had just been brough online.

      He got moved out of operations and told that, if he ever came into work smelling of drink again, he'd be out of the door so fast that he'd break the sound barrier..

  3. Mycho Silver badge

    I had to maintain a small server room with two UPSes and simple instructions in big letters about what to do to switch them off in an emergency.

    Problem came when another department got a machine in the same room and started treating my sign as a storage space.

    1. Alan Brown Silver badge

      "Problem came when another department got a machine in the same room and started treating my sign as a storage space."

      That's an easy one to fix. Unauthorised equipment ends up in the bin.

      1. Mycho Silver badge

        Only after someone else signs off that you can do that. I wasn't taking the blame.

        1. Charles 9 Silver badge

          "Only after someone else signs off that you can do that. I wasn't taking the blame."

          Also, the other department likely has a signature saying they were authorized, too. And if their signature is from someone over yours...

          1. Mycho Silver badge

            ... is it settled with greek wrestling?

            1. Mark 85 Silver badge

              ... is it settled with greek wrestling?

              Wouldn't Greek fire be a better way?

              1. Mycho Silver badge

                Greek Wrestling vs Greek Fire

                why not both?

                1. J. Cook Silver badge
                  Pint

                  Re: Greek Wrestling vs Greek Fire

                  http://knowyourmeme.com/photos/352470-the-road-to-el-dorado

                  Beer for me, I have the next 6 days off, and plan on spending half that time unconscious or stoned out of my gourd... on painkillers after a surgery.

      2. Anonymous South African Coward Silver badge

        That's an easy one to fix. Unauthorised equipment ends up in the bin.

        Had a rogue DHCP server on my LAN once.

        Investigated.

        Found it was some sort of D-Stink P-O-S which somebody's using as a switch.

        It got the wrong firmware uploaded somehow, and ceased to function.

        1. Anonymous Coward
          Anonymous Coward

          Had a rogue DHCP server on my LAN once.

          Just once ? Think yourself lucky.

          Then again, my internal IT customers are almost entirely techies...

        2. CrazyOldCatMan Silver badge

          Found it was some sort of D-Stink P-O-S which somebody's using as a switch.

          Yep. Been there, had that. Also had someone set up a port switch as a port mirror for the main LAN switch during some testing and then forget to remove it afterwards (or label it properly). Which made for great fun when we plugged a DSL modem into that port and brought down[1] our main LAN..

          [1] Well, theoretically it was still working but was trying to stuff 10GBs via a 10/100 connection of a 7Mbit DSL line which meant our main LAN spent most of its time waiting for the switch to clear its buffers..

          1. LeahroyNake Bronze badge

            D-Stink

            I had a similar issue with A Tp link 48 port Gb switch. For some reason it kept changing its management IP to the gateway IP.

            It got binned the minute a replacement arrived and after we had changed the gateway IP. Live and learn.

        3. TooheysN00b
          Devil

          I had a site recently that had a power outage.

          Afterwards their server "went weird".

          After hours of investigation turned out that there were TWO home grade routers acting as switches hidden on the site. Sure enough they had been configured by a predecessor to not act as DHCP servers and only dumb switches. After the power went out, they both booted up with a factory reset and started handing out IP addresses (192.168.x.x & 10.0.0.x) while the ACTUAL DHCP server was trying to hand out another range altogether and got quite sulky about the whole situation.

          I murdered the two routers and restart the DHCP service and everything calms down again...

  4. hammarbtyp Silver badge

    Its not unusual

    Power shutdown happens all the time, especially if you put the emergency power off button right next to the electronic door exit button...

    Of course moving it would be just tooooo expensive

    1. Tim Jenkins

      Re: Its not unusual

      "...emergency power off button right next to the electronic door exit button..."

      Also see server cases where the power button is ANYWHERE in the vicinity of the one for the optical drive...

    2. Alistair Silver badge
      Windows

      Re: Its not unusual

      @Hammar:

      I know where you've worked.

    3. Hardware Geek

      Re: Its not unusual

      I think the idea of having the EPO by the door is so that you can bang the EPO as you flee the burning room. Now when you couple that with electric door locks interesting things may happen.

      One instance I recall was a courier was delivering parts to a computer room that had powered sliding doors the operator let the driver into the room to drop off the parts, the driver then asks the distracted operator how to get back out and the operator utters the fatal words "Just press the button by the door"..... room goes quiet. Customer attempted to blame the hardware vendor that was having the parts delivered for the outage.

      Another one happened at one of by customers, following and narrowly averted disaster our branch manager decided to visit and was being shown around the room after being shown around and he was ready to exit the operator tells him to press the button by the door and then turns in time to see the branch manager reaching for the EPO and yells "Not that button!" The manager had been a large system service rep himself before becoming a manger and should have known what th big red button with "Emergency" in white letters was for.

      1. Anonymous Coward
        Anonymous Coward

        Re: Its not unusual

        Where I'm working now the fire alarm disables all the electronic door locks. If the first fire drill i experienced had been a real fire we'd have been in serious risk as we all headed for the front door then had to turn round and head out through the basement. At least now there's a little sign on the special button that opens the door when the alarms on. Fucking idiotic system.

        Thinking about it, par for the course for their software too....

  5. Nick Kew Silver badge

    Kim or Ken?

    Not sure which of them should be fired ...

    But surely not the security guard scapegoated in the first story. When you smell and see fire, you don't hesitate, you use the emergency button to shut down kit that could turn it into something much, much bigger and altogether more catastrophic.

    1. TonyJ Silver badge

      Re: Kim or Ken?

      @Nick - I came to say the exact same thing. What if he hadn't pushed the emergency stop and the entire place had burned down?

      And generally in what appears to be an emergency with smoke, fumes and flames, people don't tend to stop and have a quick chat first - they act.

      No way that poor chap should've been fired.

      1. Phil O'Sophical Silver badge

        Re: Kim or Ken?

        No way that poor chap should've been fired.

        Walk into an empty room and smell burning, you hit the emergency button. Walk into a room where people are working, and see that they haven't done it, it's just common sense to ask "why not" before taking extreme action.

        1. Nick Kew Silver badge

          Re: Kim or Ken?

          it's just common sense to ask ...

          That's the common sense that was applied at Chernobyl.

          1. James O'Shea Silver badge

            Re: Kim or Ken?

            "it's just common sense to ask ...

            That's the common sense that was applied at Chernobyl."

            Actually... no. It's a lot worse than that.

            1 there were some Big Red Buttons on various consoles

            2 no-one, I repeat, no-one in the building had any idea what the hell they did

            3 none of the available documentation said what they did, either

            4 every single staff engineer had been literally trained on the job by those who came before them

            5 management decided that it would be a good idea to find out what those buttons did and scheduled 'training' on one reactor which was not in use

            6 they found out what the buttons did. Ooops.

            See 'Ablaze' by Piers Paul Read for more detail.

            1. DropBear Silver badge
              WTF?

              Re: Kim or Ken?

              Actually... no. While what they were doing was not particularly advisable, it is established fact that the inherent and extremely dangerous instability of reactors of that design at very low power levels was not known at the time, and certainly wasn't known to anyone working at the plant. You might want to do some more reading...

              1. Alan Brown Silver badge

                Re: Kim or Ken?

                " inherent and extremely dangerous instability of reactors of that design at very low power levels "

                Virtually all light water reactors are unstable and have a nasty positive void coefficient at very low power levels, snapping into prompt-criticality at the drop of a hat, which is why this kind of operation is specifically avoided (It was the resulting steam explosion which blew the roof off at Chernobyl). The russians had a number of those in the 1950s and the americans deliberately experimented with them after the SL-1 accident to understand how they happen.

                It's a fundamental flaw of using water as the moderator. If the operators didn't know this, then soviet operating procedures were incredibly bad, as prompt criticality risks were well documented on both sides of the Iron Curtain

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019