back to article Crash, bang, wallop: What a power-down. But what hit the kill switch?

Well done, Reg readers, you’ve once again triumphed in the battle with the 9-5 and made it to Friday. Celebrate with a dose of On Call, our weekly instalment of tech conundrums and calamities caused by other people. This week, we meet “Clark”, who wrote in to tell us about a very particular kind of fault, which is probably …

  1. Andrew Commons

    Placement of kill switch and other quirks

    Probably similar timeframe - VAX 11/780s in late 1970s. Kill switch next to a phone on the wall... Engineer makes call on phone, leans against wall,.... lights out!

    Same installation. Telecomms 'electricians' removing a cabinet. Not sure if power is off. Large screwdriver between active and earth shows, momentarily, that power WAS on, lights out once again.

    Some air-conditioning fun from same location if On-Call is interested in that sort of thing.

    1. Joe W Silver badge

      Re: Placement of kill switch and other quirks

      Yeah, that's a classic. And might even have been featured here - I think I can recall that, but Friday, and a long history of caffeine abuse...

    2. A Non e-mouse Silver badge
      Unhappy

      Re: Placement of kill switch and other quirks

      In one of our machine rooms, the light switch is right next to the EPO button. We've asked and asked for them to be separated, but our requests are falling on deaf ears.

      (Yes, the EPO has been accidentally hit)

      1. Doctor Syntax Silver badge

        Re: Placement of kill switch and other quirks

        "our requests are falling on deaf ears.

        (Yes, the EPO has been accidentally hit)"

        Obviously not hit often enough.

    3. Anonymous Coward
      Anonymous Coward

      Re: Placement of kill switch and other quirks

      Sysadmin putting some old tapes back on a high shelf, which needed a stepladder to reach. He overstretched, the ladder wobbled, and he instinctively reached out to keep his balance. He said that even as he felt the main breaker handle start to move under his hand he realised that it wasn't the best thing to have grabbed. Too late...

    4. Graham Butler

      Re: Placement of kill switch and other quirks

      The server room in the old offices of a certain banana coloured directory in Reading had the kill switch next to the door exit switch.

      I think it was around incident 12 that management deigned to finally move it.

      1. Alister

        Re: Placement of kill switch and other quirks

        Yep, I've been in a datacentre hall recently where the door release and the kill switch are next to each other - one coloured Red, the other Green. Guess which one is Green hey?

        1. Anonymous Custard
          Trollface

          Re: Placement of kill switch and other quirks

          Especially for those who are colourblind...

          1. DJV Silver badge

            Re: Placement of kill switch and other quirks

            or switchblind!

        2. jbrownman

          Re: Placement of kill switch and other quirks

          similar thing happened here only with the fire alarm

          as it was activated in the basement it triggered a shutdown and a full evacuation of two buildings as they share the same basement

        3. the hatter
          Stop

          Re: Placement of kill switch and other quirks

          Problem is, EPOs are pretty much required to be somewhere quickly accessible near the space they protect, and obviously you want access to it protected to the same access level as the equipment. Then door releases obviously should be located near the door, and in places where people turn the lights off, that probably should be there. However, putting these switches all together at least shows the user there is a choice to make - the alternative it an exit, and an action to perform (release door/turn off lights) - if the user glances at one lonely switch, they're going to press it, so if their gaze falls on the EPO rather than the one placed not quite close enough, they're just going to hit it, with autopilot engaged.

          Molly guards, and putting EPOs out of direct sight line are options in some locations, but other times, local or national regulations prohibit them

          1. Andy Miller

            Re: Placement of kill switch and other quirks

            This placement makes a lot of sense when you are leaving and turning the light on. When you're coming in and groping for the switch to turn the light on, line of sight is not that relevant. Motion sensors for lights people... At least this story has made me feel better about taking out all the customer support gateways....

            1. paulll

              Re: Placement of kill switch and other quirks

              "Motion sensors for lights people..."

              As long as the timing's generous. Place I used to work there was barely time to read a Reg article or play a level of Plants vs Zombies before you'd have to do the penguin walk to turn the lights back on to ... conclude your business.

      2. Tomato Krill

        Re: Placement of kill switch and other quirks

        And look what happened to them - bankrupted them!

        You killed them you bastards!

  2. Anonymous Coward
    Anonymous Coward

    There's a New Mexico?

    1. John G Imrie
      Trollface

      Quick

      Build a new wall around it.

      1. joeW

        Re: Quick

        Some New Mexicans, I assume, are good people.

        1. Paul Cooper

          Re: Quick

          Los Alamos National Laboratory is in New Mexico; perhaps even Trump doesn't want to mess with people who build bombs!

        2. Francis Boyle Silver badge

          Re: Quick

          Even in the badlands? Better consult Marty Robbins.

        3. Ken Hagan Gold badge

          Re: Quick

          "Some New Mexicans, I assume, are good people."

          Yes, and they need to be protected.

    2. Phil O'Sophical Silver badge
      Coat

      And a new wall?

    3. This post has been deleted by its author

      1. Anonymous Coward
        Anonymous Coward

        Sadly, I suspect that reference is too subtle for most...

        1. defiler

          Only if they're too busy trying to change the world.

          I don't want to change the world.

          1. Anonymous Custard

            But sadly you will have to look for another girl...

            1. The Oncoming Scorn Silver badge
              Childcatcher

              Sadly all the girls I loved at school are already pushing (Grandkids) prams .

              1. ibmalone

                Sure it's not worth giving one of them accoll?

                1. Anonymous Coward
                  Anonymous Coward

                  If I did I wouldn't brag about it here

    4. bpfh

      North of Mexico which is south of New England with is west of England, which is east of South Wales which is north of New South Wales...

  3. StewartWhite

    Working with a v tall (almost 7 foot) colleague in the server room and he leant back from the rack as I took a look at the KMS screen only for his head to whack the kill switch and turn almost everything off. upside was that a couple of v old systems didn't go down as they were plugged into non protected sockets so we got those properly plugged in as soon as we bought everything else back up.

    1. phuzz Silver badge

      So if he was seven feet tall, and it was his head that hit the switch, how high up the wall was it? Could a shortarse even push the button if they needed to?

      1. EuKiwi

        Um... even if his HEAD hit it, not a shoulder or something, pretty much anyone above 5' 5" would be able to reach something 7 feet up, no?

        1. StewartWhite

          I guess it had originally been put up there so couldn't accidentally be pressed - before my time there started so couldn't say for sure. I presume they hadn't banked on somebody of Doug's height coming into the server room.

    2. Diogenes

      This happened where I once worked, 2 × ibm 3090 mainframes., supporting about 3k 3270 terminals and 2 3800-3 printers Switch was 6'6" high, operator 7' tall.

  4. GlenP Silver badge

    Not Unique...

    Big red kill button, box of paper on floor, operator bends down* to pick up paper and nuts kill switch.

    *If he'd had manual handling training of course this wouldn't have happened!

    I have also had a Unix server where the power button and CD Drive eject were very close together, that one got a card flap over the power button.

    1. MiguelC Silver badge
      1. UncleNick

        Re: Not Unique...

        I also remember that the old AT towers had a power switch on the front that was at ${mains}V across the terminals, as this was the good old days before ATX and soft-power on.

        The first time I ever had 240V was when I was digging under the IDE cables and the hand that wasn't holding the well-earthed case came upon the bare terminals of the switch. Certainly took me by surprise...

      2. PickledAardvark

        Re: Not Unique...

        No problem. You just hold the Reset button down for as long as the PC takes to complete its job...

        A colleague set the Turbo button in on his 486-66 for months without noticing. As I understand it, the Turbo button set the ISA bus clock speed into a compatibility mode for legacy ISA cards. My colleague's PC had a VL Bus graphics card and a 3Com ISA Ethernet adapter, neither of which seemed to be affected when running Windows 3.1. He noticed something after installing Linux and pressing the wrong button at some point.

        1. phuzz Silver badge

          Re: Not Unique...

          Ah yes, the reset button (or power button in the pre-ACPI days) which only engages when you take your finger off the button.

          I remember taping someone's finger to a switch after they pressed it at the wrong time, and as long as the finger was taped in place, the computer stayed up.

          1. Muscleguy

            Re: Not Unique...

            I got my work computer, G3 Power Mac, back after it went dead. Got called by the tech guy you had the side down (wonderfully accessible machines) and was looking at the board.

            'The only thing will be to hit the reset switch' he says (it was on the board) obviously not wishing to take responsibility for doing it. At which point my finger snaked out and pressed it, machine whirrs into life and I'm a happy scientist.

            It was still working well when I left that lab. I did personal surgery to that box, swapping a drive bay with another box to enable the installation of an internal Zip drive. Very easy to do. Compared to iMac surgery the G3/4 towers were a dream to work on.

        2. Long John Brass
          Headmaster

          Re: Not Unique...

          Did the Turbo-Button really change the clocks on the ISA bus? The VLBus I could believe, but the ISA one as well?

          1. Pascal

            Re: Not Unique...

            For a time, yeah. Early turbo switches changed the clock speed of the whole system, as isa bus was synched to cpu.

      3. Pirate Dave Silver badge
        Pirate

        Re: Not Unique...

        Remember the Minuteman rackmount UPSes that had their sexy round power button on the front of the faceplate, sticking out proud(ly) in the center? A button that only needed the merest brush to immediately and ungracefully shutdown the UPS. Only took twice before I decided to act. Went to the local hardware store, bought a cheap can of bright yellow spray paint. Threw the spray can away, duct-taped the yellow plastic cap over the power button. Problem solved.

      4. Roland6 Silver badge

        Re: Not Unique...

        The Sun 3 pedestal workstations which had the easily switched on/off switch at knee height and sufficiently proud of the front casing for a small knock to flick it to off... Funnily it took deliberate action and more force to switch the system on...

      5. Montreal Sean

        Re: Not Unique...

        Why is it that those AT case reset buttons always stuck out further than any other button?

        The reset button should have been slightly recessed, would have prevented so many accidental reboots back in the day.

    2. NorthIowan

      Re: Not Unique...

      "I have also had a Unix server where the power button and CD Drive eject were very close together, that one got a card flap over the power button."

      Yup, had to make a flap for the UPS on the home PC. It was a low flat box with the power switch on top and it was almost on the floor. A real problem with cats who walk anywhere they want to.

      But mice are a problem too. I had a mouse fall off the keyboard tray and it hit the power strip power button. For that one it worked to rout the power cords over the switch and velcro them together.

      1. SkippyBing

        Re: Not Unique...

        Surely the cats should deal with the mice?

        1. dvd

          Re: Not Unique...

          Have you had a cat? Cats only chase mice when they want to, not when you need them to.

          1. jake Silver badge

            Re: Not Unique...

            You're feeding your cats too much if they aren't hunting. Cut back their rations, they'll catch rodents that you didn't even know lived in your neighborhood.

            1. Anonymous Coward
              Anonymous Coward

              Re: Not Unique...

              I recommend that (reducing chow amount) only if they're indoor kitties on the portly side, otherwise the local ecosystem takes a hit.

              They're lovable little murder-balls, but I do enjoy seeing squirrels and songbirds too.

        2. Martin
          Happy

          Re: Not Unique...

          Bit off-topic, but what the heck, it's Sunday...and we were talking about cats dealing with mice.

          Many MANY moons ago when I was about six or seven, we had a toy cupboard with rather a lot of old paper in the bottom. Judging by the rustling, we were pretty sure that there was a mouse in there. So we got the cat up, who looked very interested at the rustling. We then poked around with a broomstick, and after a while a mouse shot out of the toy cupboard and across the room. On spotting the mouse, the cat howled piteously, jumped into the air and shot off down the stairs and out of the house. Didn't see him for hours.

          No idea what happened to the mouse.

          1. ICPurvis47
            Big Brother

            Re: Not Unique...

            When I was a kid, we moved house. The previous occupant used to keep parrots, in a purpose-built aviary at the bottom of the garden. Our cat, Mickey (long story), was extremely interested in the aviary, running back and forth across the front with his nose pressed against the wire mesh. We opened the door to let him in, and didn't see him again for three days. Every morning there was a row of dead mice lined up outside the back door, heads towards the door, eleven the first morning, seven the second morning, and five the third. Mickey then strutted back into the house with a smug air about him, but didn't bother with the food we put down, so how many of his victims had not made it to the line-ups we never found out.

    3. tekHedd

      power button next to eject

      Hah, my current desktop system (Thermaltake case), the power button is next to and shaped exactly like a USB port. I've stopped using the front panel USB ports.

      See also: laptops with a headphone jack directly next to the USB port, where if you're fumbling around the side and insert your headphone plug into the UB and short out the 5V supply, it kills the power supply and hard resets the system.

      1. Anonymous Coward
        Anonymous Coward

        Re: power button next to eject

        Brand new Dell Precision Xeon optioned notebook

        Headphone jack on right side of machine. USB A plugs 4mm from it.

        Honking huge vent above those.

        Plugs are in the way of external mouse for right handed people.

        Left side has a smartcard reader and an SD card slot, then dual USB-C.

        No interference.

        This was designed by a southpaw.

      2. imanidiot Silver badge

        Re: power button next to eject

        My solution would probably have been to disconnect the reset switch. But to each their own

    4. Ogi
      Facepalm

      Re: Not Unique...

      I have not yet triggered such an emergency switch. However at my new workplace, all the doors have magnetic locks. They have two buttons, the normal door release button, and the "emergency door release" button, which are always right next to each other. They are also both green (except the emergency button has "EMERGENCY" written in small green text above it).

      Both also do the same thing (release the door). but the emergency one will also sound the alarm (forcing a building evacuation) and call the fire and police. Multiple times already I found myself almost pressing the emergency button by accident, instead of the normal button when wanting to release a door.

      I really wonder who thought it would be a good idea to make both buttons the same colour and put them side by side like that. I have seen others in the office make the same mistake as well, but realise at the last minute before pressing. One of these days, while someone is in a big rush, I suspect the inevitable will happen.

      1. Pascal

        Re: Not Unique...

        Building codes changed a bit over the year, magnetic lock doors here now can't work like that anymore.

        You can get the usual ID mechanism (card reader or biometric or whatever on the side of the door) for normal access, but emergency access has to be the "normal door way" i.e. turn the handle / press the release bar in the middle of the door, except secure doors are allowed to have a timer of up to (15 seconds I think) with an audio warning before the emergency release happens / alarms are triggered.

  5. Chris Miller

    The scene is the late 70s - mainframes, fanfold paper and lengthy batch processing. Senior analyst called in to fix a problem with a run over the weekend. His wife is out so he has to bring the kids with him. It's raining hard, and the kids end up playing football in the computer room, until a perfect kick makes contact with the big red button ...

    1. jake Silver badge

      Yank version of the above.

      I was just putting the finishing touches on a small cluster of vaxen at SLAC one fine Friday afternoon. The annual Big Game between Stanford & Berkeley was to be the following day. A couple of grad students started passing a football (American version) between themselves. In the glass room (that wasn't glass). Just as I was threatening mayhem if they didn't knock it off, the ball hit the Big Red Button, Needless to say, a bunch of pissed off people couldn't attend the game the following day. The grad student's computer privileges were suspended for the rest of the academic year. Personally, I'd have hung them by the thumbs in the Quad as a warning ...

      (As an alum in good standing of both schools, all I can add to the above is "Go Bears!")

      1. baud

        Re: Yank version of the above.

        Do you keep that story on hand to reuse as needed?

        https://forums.theregister.co.uk/forum/all/2017/11/24/on-call/#c_3356387

        1. jake Silver badge

          Re: Yank version of the above.

          Yes, I do keep that story on hand. It's called a memory. I might post it again, eventually. Not everybody reading here today was here last time I posted it; It's new to them. And that would include yourself. Nice to see that you find my drivel important enough to read my backlog, though.

          Do you have anything on-topic to contribute, or are you just here to bitch?

          1. baud

            Re: Yank version of the above.

            Sorry if my comment appeared agressive, I just found it fun that you were retelling (via copy-paste) the same story, which I found nice.

            And I didn't go through your backlog, I was reading another comment thread the same day and found your other comment.

      2. tekHedd

        "an alum in good standing"

        How do you lose your standing as an alumnus? I'm trying to imagine a scenario, humorous or otherwise, that would result in this, and does not involve war crimes...

        1. Guevera

          Re: "an alum in good standing"

          Well among my university's graduates are Ken Lay and the woman the US military dubbed "Chemical Anna."

          1. jake Silver badge

            Re: "an alum in good standing"

            hint: Look up the meaning of "alum".

            (This comment brought to you by the "if you have to explain it" Group, LLC.)

    2. J. Cook Silver badge

      Two words: Molly Guard

      It's been around for... a while.

      1. waldo kitty
        Paris Hilton

        "Two words: Molly Guard"

        It has gotta make one wonder what Molly did to get a switch cover named after her.

        1. TSM

          Not really, it's right there in the linked definition: "Originally used of the plexiglass covers improvised for the [Big Red Switch] on an IBM 4341 after a programmer's toddler daughter (named Molly) frobbed it twice in one day."

  6. Steve Kerr

    And then there was silence

    Used to work for a large investment bank.

    Went from my floor to the dealer floor to help with an issue, when I got there, there were people standing up looking round like meerkats.

    Went back to my floor, people still standing up

    Turns out, the power was out in the computer room, 1000 cabinets worth of stuff.

    There was an electrician in the computer room, who was moving an electrical point, he had followed the cabling under the floor in direct line to a pair of power sockets on a pillar.

    Except......

    That cable, went round the pillar to the other side and another cable came round the pillar to the socket. On the other side was an emergency power off button.

    Back in history when the DC was built, all the EPO's were wired up with orange cable labelled EPO every 6 inches, the installers ran out of EPO cable so rather than getting more, they used what was there which was reels of power cable in grey. The poor electrician, couldn't get the final tile up as something was sitting on it so assumed that he had the correct cable.

    Outcome, a day of carnage for an investment bank :)

    The electrician was escorted from the premises never to return.

    1. Doctor Syntax Silver badge

      Re: And then there was silence

      "The electrician was escorted from the premises never to return."

      For something that wasn't his fault. I suppose it says something about investment banks.

      1. disgruntled yank

        Re: And then there was silence

        "For something that wasn't his fault. I suppose it says something about investment banks."

        After 2008, does it need be said?

      2. Anonymous Coward
        Anonymous Coward

        Re: And then there was silence

        "For something that wasn't his fault."

        Only it was his fault - his actions directly caused the power outage and I am assuming it was made very clear to him that powering down the building was bad.

        The correct course of action would have been to clearly identify what he was working on and if he was unable to do that, advise who he was working for that there was an issue and come up with a way that the risks involved could be reduced (i.e. carry out the work out of hours or determine how to investigate further or even discover the incorrect cabling and plan to replace it all). i,e, perform the task in a professional manner rather than bodge it.

        Instead, the electrician committed the same mistake as his predecessors who decided not to do things properly. The difference is they got away with it. And I suspect the investment bank likely had a history of working to JFDI with the unwritten rule being don't get caught.

        If you substitute the electrician for an IT contractor, does your view of the actions/consequences change?

      3. imanidiot Silver badge

        Re: And then there was silence

        He assumed. That's where he went wrong.

  7. Anonymous Coward
    Anonymous Coward

    Typical

    So... Installers mucked up - management failed to check the quality of their work - a lazy fat something was too arrogant to move their chair and the electrician was the one who got fired...

  8. Bob Wheeler

    It took me years...

    ... To overcome the urge to press the big red button every time I went inside a computer room.

    Bob (retd)

    1. MiguelC Silver badge

      Re: It took me years...

      We all know Big Red Buttons are irresistible!

      At CERN there's even a spoof one for visitors to press

      1. This post has been deleted by its author

      2. navidier

        Re: It took me years...

        > We all know Big Red Buttons are irresistible!

        > At CERN there's even a spoof one for visitors to press

        Ah, yes; I was manning a mock-up of a portion of the LHC at a Norwich science fair a couple of years ago. There was a faux Emergency Switch near one of the exits. Several times I had to encourage young kids to press it while they dithered; nearly as many didn't need urging.

        For my sins, I had two air-rifle pellets in my window on the way back to Cambridge; two other windows and the driver's side window were also hit. There must be SFA to do for young peeps in the Fens on a Sunday night...

    2. jake Silver badge

      Re: It took me years...

      I still have that urge. So I decided to build data centers for a living. Now I get to test that Big Red Button, pretty much whenever I like (really, it's in the contract) ... It's not really as much fun as you'd think.

      1. jmch Silver badge
        Thumb Up

        Re: It took me years...

        Yeah, Big Red Buttons that shut things down are boring.

        Now, BIG RED BUTTONS that make things go Whizz and Spark and BANG!!... That's where it's at :)

      2. steviebuk Silver badge

        Re: It took me years...

        It would be fun on the last day at a place you hate or hate and retiring.

    3. NBCanuck

      Re: It took me years...

      The only thing worse than a big red button is any button with a sign saying "Do not press". And if the button is also red and big? Isn't that just asking for trouble?

      1. The Oncoming Scorn Silver badge
        Stop

        Re: It took me years...

        ARTHUR:

        What happens if I press this button?

        FORD:

        I wouldn’t…Oh!

        FORD:

        What happened?

        ARTHUR:

        A sign lit up saying “please do not press this button again.”

      2. Anonymous Coward
        Anonymous Coward

        Re: It took me years...

        @NBCanuk: I have seen one that was effective. A little card with "Do not press this button. Severe environmental degradation will result" That card was hanging from cruise missile launch console. The console was fully secured and powered down. No-one in the visitors group was even tempted to disobey. Anon for safety of tour guide.

    4. Tomato Krill

      Re: It took me years...

      So for those years you gave in to that temptation?

      Very understanding employers!

  9. chivo243 Silver badge
    Facepalm

    Data Closets

    We have a couple of small data closets, and the light switches are just at bum level.... I'm a bean pole, and I can't even turn around with out knocking out the lights!

  10. wyatt

    When Cormorant came to be tested by the Army there was 2 emergency stop buttons. 1 by the PDUs where the operator would sit and 1 by the door. The one by the door was perfectly located so that when you reached the top of the steps and steadied yourself to go through the door you could quite easily hit the button. Most didn't, but some did. A modification was soon built from a Pringles tube and black nasty.

  11. Christoph

    Central computer room in open plan office. Since they might have to evacuate in a hurry if the halon was about to drop, the Big Red Button was placed on the wall outside - in the open plan office. With no warning label.

    Later, same solution - hinged box of clear plastic.

    1. The Oncoming Scorn Silver badge
      Facepalm

      In A Slightly Different Vein

      Not my story, but told to me years ago...

      RAF station had a power outage, while Top Top Top Brass was visiting & the UPS\Battery backup failed to kick in.

      TTTB demanded a full investigation & wanted the report on his desk ASAP, from someone who wasn't stationed at the base.

      That report highlighted two major things:

      It was full of crumbs & other flaky pastry detritus, where it had been used to heat then keep pies & the like hot & ready to eat, along with tea & coffee spills inside, on the sides & at the rear (with mug rings on the top of the unit).

      That the staff sergeant had signed off on, but not checked & topped up the batteries electrolyte levels, in fact It was almost empty and hence no charge, unlike the sergeant who promptly found himself on one & demoted to corporal.

      1. ricardian

        Re: In A Slightly Different Vein

        Same thing at a large RAF base in Cyprus in the 1960s. Major storm knocked out power for a couple of days. The telephone exchange had an enormous battery which should have provided power for at least 36 hours but died after about 30 minutes. It transpired that a sergeant in the Royal Corps of Signals (who looked after all RAF telephones overseas) was responsible for ensuring that the electrolyte was topped up and regular voltage tests were carried out; he hadn't checked for ages and there was hardly any electrolyte in any of the cells.

        After that same storm and flooding there was a signaller (private) in the Royal Corps of Signals who was using a canvas bucket to bail out a cable pit that was full of water. An officer in the Royal Corps of Signals drove up in his LandRover and asked the signaller how he was getting on. When the lad replied "Not very well, sir" he was told to put his back into it and bail more quickly. Shortly after that the SNCO from the Royal Corps of Signals arranged for an RAF fire engine to pump out the cable pit - a job that took about 4 hours because all the cable ducts were acting as drains and feeding into the cable pit.

  12. Stoneshop
    Facepalm

    No? Break.

    Data centre for a telco. Needs No-break installed. Generator shed gets built, stonking big diesel genny gets carted in and the guys from Perkins (or whoever manufactured that beast) gets the thing chugging. They listen, probe, do a few adjustments, listen some more, consider it Fine, let it run some more and figure it's time to push the Engine Stop button, sign off on the job and hand it over to the sparkies to let them do their bit. Alas, the Engine Stop button didn't., as it apparently wasn't yet hooked up. "OK, maybe it's that other button".

    That didnt't stop the engine either, but it did stop the computer room.

    1. ITS Retired

      Re: No? Break.

      The very first thing you need learn about big, noisy engines of all kinds, is how to stop them. How to start them is a few more steps doing the list.

      1. jake Silver badge

        Re: No? Break.

        The very first thing I teach people about my big, noisy engine is "Keep your hands off if you want to keep them!" ... She's a 1915 Case traction engine.

        1. The Oncoming Scorn Silver badge
          Holmes

          Re: No? Break.

          Nice I work for a Case dealership.

      2. Robert 22

        Re: No? Break.

        During WW2, when the first prototype Wright Tornado engine was started up, someone had arranged the throttle linkage incorrectly so it was unintentionally set to full throttle. The unloaded engine accelerated abruptly and, in a few seconds reached its ultimate limits and destroyed itself.

        1. whitepines
          Megaphone

          Re: No? Break.

          Got a reference for that? I'd like to use it for a cautionary tale at work :)

    2. whoseyourdaddy

      Re: No? Break.

      That's honestly the most hilarious button story. Well done, sir!

  13. Anonymous Custard
    Headmaster

    Hardware option

    On a more hardware side, some of our older kit (semiconductor manufacturing machines) shipped with their bright red EMO mushroom buttons entirely exposed, as it was mandatory for CE marking (being able to whack them from any angle).

    Of course this lead to the inevitable misoperation, from being smacked by various operator body parts (the front panel buttons were at roughly arse level for example, and the ones at the rear usually around chest height).

    Needless to say the first thing most of our customers did after the tools were installed (sometimes before they were even powered on) was glue perspex collars around all the buttons to try and prevent this.

    Forget lost program runs, try losing 150 wafers worth of chips-in-production. Not cheap...

    1. Anonymous IV
      Joke

      Re: Hardware option

      > EMO mushroom buttons

      You had EMOs all that time ago? I thought they were called Goths...

      1. Anonymous Custard

        Re: Hardware option

        We wouldn't be allowed in the cleanroom with all that make-up. And they (sadly) don't make black bunny suits..

        Of course the fact that the EMO buttons didn't actually kill the power to everything was only recently realised, and indeed on the odd occasion didn't actually do anything at all. But that's a whole other story.

        1. imanidiot Silver badge

          Re: Hardware option

          Ahhh yes, the great EMO button that doesn't actually do anything.

          Reminds me of the time I was working late in the cleanroom, far away from anyone else, getting sprayed in the face by 40 liters a second of water, desperately trying to keep the flow contained, just barely able to press the nearby EMO on the water conditioning cabinet, only to realise the flow doesn't stop and the pump is still running full blast...

          Then you're left with 2 options. Release the broken L-joint, duck left, sprint to the nearest console and enter the required commands for a remote shutdown or option 2, release the broken L-joint, duck right, sprint to the main shutoffs in the control cabinets at the end of the lane and just close the valves, letting the spike in water pressure force the pumps into an auto shutdown.

          I'm glad I'm not the one that had to report a 40 000 gallon water spill at a later time into the sub-fab basement at a customers fab though... That must have been a tough talk.

      2. Throatwarbler Mangrove Silver badge
        Trollface

        Re: Hardware option

        I wouldn't use EMOs; they'd keep cutting themselves off.

  14. lee harvey osmond

    Old story

    http://catb.org/jargon/html/M/molly-guard.html

    1. Anonymous Coward
      Anonymous Coward

      Re: Old story

      Also available in a software version

      1. Groaning Ninny

        Re: Old story

        Sadly not, I can't get hold of it for RHEL/CentOS, last time I looked....

  15. R J
    Facepalm

    This somehow reminds me of my school days

    A long, long time ago, I can still remember.. when the "cool" kids used to set off the fire alarm at school.

    This, of course, didn't only interrupt any teaching going on, but also triggered the fire department. Not really all fun anymore, and quite costly, or so they claimed.

    So the brilliant solution? Replace the glass cover for the alert button with a metal one.

    1. Groaning Ninny

      Re: This somehow reminds me of my school days

      I trust they chose a metal with a veeeery low melting point ;-)

      1. jmch Silver badge
        Devil

        Re: This somehow reminds me of my school days

        Or Magnesium

  16. Anonymous Coward
    Anonymous Coward

    Could be worse...

    A guy I occasionally chatted in ye olden days of FidoNet used to be a sysadmin at "an intelligence facility" where the mainframes had been fitted LITERAL kill switches - as in, thermite charges attached to various critical components. Don't know how those charges were supposed to be triggered but hopefully (from the point of view of people working there) not by an unprotected and unlabelled button on the wall.

    1. lglethal Silver badge
      Joke

      Re: Could be worse...

      But you have to make it unassuming and hidden, otherwise that bad guys will know what to protect to get the data. Hiding the kill switches amongst the light switches! Thats how you do it, no one will think to protect those!

    2. Anonymous Coward
      Anonymous Coward

      Re: Could be worse...

      A/C because it's an ex-colleague of an old friend - not my story to tell, really!

      Anyway, this chap was sent around the British Consulates and Embassies to perform upgrade work in their server rooms. On the back of the (very strong indeed) server room doors were metal hammers. If the building is invaded by unfriendly forces, the idea was to haul the hot-plug servers out and hammer the hell out of them, ideally before they stopped spinning.

      I understand more modern devices take a SCSI command and detonate a vial of solvent within the drive to destroy them. Maybe others know more.

    3. whoseyourdaddy

      Re: Could be worse...

      https://www.theregister.co.uk/2015/08/09/how_to_destroy_your_hard_drives_without_burning_down_the_data_centre/

  17. Admiral Grace Hopper

    Father Dougal

    I won't type it out again, but I am sure that everyone has worked with a Father Dougal at some point in their career. There's something so seductive about a Big Red Button.

  18. Andrew Moore

    Sounds familiar...

    Was in a client's comms room with my boss, sometime during the 80s. Suddenly the power goes dead and we all turn round and there's my boss at the kill switch. His response was along the lines of "What? I just wanted to know what it did." Needless to say, he was barred from the comms room and that ban still exists today.

  19. Gomez Adams

    There was the story of a shift operator showing some newbies some basic op console commands for a Unisys mainframe where I worked. All harmless until he decided to show them the $! command which did *not* need you to then hit Enter to action it to reboot the computer (think CRL+ALT+DEL)!

    Dollar, Bang by name and Dollar Bang by nature! :)

  20. Anonymous Coward
    Anonymous Coward

    Exit switch and shutdown switch - all next to the door

    I've seen this in a bank in London, in their computer basement.

    And yes, I was told the plastic cap over the kill switch was a later edition, as the result of a rather predictable incident..

  21. DontFeedTheTrolls
    Boffin

    Covers

    "The emergency button was enclosed in a new hinged box of clear plastic to avoid accidental shutdowns "

    Having experienced a similar accidental shutdown the IT "Problem Management" team were put on the case. Their recommendation was to shield the emergency power buttons with the metal shields or cages often seen covering fire call points, thus preventing accidental activation. Their smugness was short lived as I pointed out that the typical use of the Emergency shutdown button would be in the event of someone being electrocuted, and that perhaps a conductive material wasn't the best solution.

    1. Jim Mitchell

      Re: Covers

      If you're being electrocuted, you probably don't have the voluntary motor control to push the big red button anyways. This fear seems spurious.

      1. DontFeedTheTrolls
        Boffin

        Re: Covers

        True, but then you also don't want to have the person who finds you stuck with the same problem

  22. Anonymous Coward
    Anonymous Coward

    Fred gave head and the mainframes blew their loads.

    SCNR. Yeah, I will see myself out.

  23. Falmor

    Cray Red Button

    I used to work at a site that had a lot of mathematical modellers. One of the Cray Supercomputer models we used was just the right height for a raised elbow. The designers hadn't considered this but did consider the perfect place for an emergency stop button was on the top of the box. We put a plastic cover over ours so no one ever accidentally shut the thing down. Seeing this a Cray engineer tells he'd been to a site that had used half a coconut shell for the cover.

    1. Nick Kew
      Coat

      Re: Cray Red Button

      Must've been a cray for help?

      Sorry. Like other commentards, I'm sure I've read this story at on-call (or maybe who-me) too recently before. Mine's the one with ... oh, they all are.

      Edit to add: Here's a much more dramatic rendition.

    2. The Oncoming Scorn Silver badge
      Pint

      Re: Cray Red Button

      Thanks to that comment, I discovered Monty Python & The Holy Grail was on Netflix last night, so that covered part of my evenings entertainment.

      "What is the airspeed velocity of an unladen swallow?"

  24. Delicolor
    FAIL

    Buttons that you shouldn't press...

    I once had a role that evolved into looking after a 60 rack Data Centre. It was located in the shell of a former 1960s computer hall although most of the vintage services had been stripped out or abandoned.I spent a year or so reverse engineering all of the services as the electrician who had worked on it over the years died in unfortunate circumstances.

    Whilst we didn't have an ESD switch, the room did have an FM200 gas suppression system which was disarmed whenever staff lifted floor tiles.

    One weekend, I received an urgent phone call from a colleague with the immortal words "There's been a bit of a cock-up". The gas system had gone off, two thirds of the cooling had shut down and Servers were gradually shutting themselves down due to localised overheating. Not knowing how the interlocks worked, I eventually got the Air Con up again by pulling the battery leads off in the alarm panel. (There was a discharge detection sensor that required a manual reset, something that went into the documentation once we were told about it). Fortunately, the cooling interlock wiring had not been extended to unit 3 (and standby unit 4) so we were able to avoid compromising the really important stuff.

    It turns out the staff member got confused when re-enabling the fire panel and got it into his head somehow that it would reset if he pressed the button marked RELEASE under the hinged flap with the seal on.

    Getting the FM200 replaced was another saga, it took weeks...

    1. ibmalone

      Re: Buttons that you shouldn't press...

      I received an urgent phone call from a colleague with the immortal words "There's been a bit of a cock-up".

      If there is a silver lining to this kind of incident, it's the opportunity to come up with a truly memorable understatement. Make it good people!

  25. Antonius_Prime

    Needs pointing out...

    Sir PTerry knew ages back...

    “Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry.”

    ― Terry Pratchett, Thief of Time

    1. navidier

      Re: Needs pointing out...

      > Sir PTerry knew ages back...

      > “Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry.”

      > ― Terry Pratchett, Thief of Time

      When I were a lad, we used to sometimes go into the Big Smoke (for us, that was Byron Bay) to do our shopping. There was a big department store on the corner of Jonson and Lawson where you were served individually at the counter and your bill and cash whizzed up on a cable to the cashiers on the second floor who then let the receipt and change fall down on the same cable. Next to the main counter was a small wooden sign:

      IN CASE OF FIRE, PLEASE RAISE THIS FLAP

      Underneath it said:

      NOT NOW, STUPID -- IN CASE OF FIRE!

      1. J.G.Harston Silver badge

        Re: Needs pointing out...

        Well, it did say 'In CASE of fire'. Do you think it should have been labelled 'In *EVENT* of fire'.

        In case of rain, take your umberella.

        In *event* of rain, open umberella and hold over head.

    2. ibmalone

      Re: Needs pointing out...

      As a very small child in London for the first time, I obviously pushed the emergency stop button on one of the long zone 1 tube station escalators. Not sure everyone was amused as I was.

      Got to do again it many years later as an adult when someone fell over behind me while getting off one. Just lucky I'd been waiting for a second opportunity...

  26. Roger Kynaston
    Pint

    Kill switches

    Two incidents come to mind. Once, at the local authority where I worked, we had building work going on in our computer room. I was working upstairs when all went dead. I rushed down with others to find that the builders had been hammering so hard on the door that they had triggered the emergency kill switch near it. Half an hour of doing the headless chicken dance and all was back with the world. In those pre outsourced, pre SLA, pre critical indident days there was a bit of a meeting which involved reading the riot act to the builders and all went back to normal. I dread to think what would happen now.

    Second was at the same authority but a different computer room. We had to shut down for weekend electrical work. I was doing the Sun boxes while colleagues were doing the Windows stuff. The facilities guy was there to turn off the power when we were done. I had just got the payroll DB server to the OK prompt and typed the words power off when the facilities guy hit the off button at the exact same moment that I hit return. Cue silence and a lot of swearing from the Windows people who had only got half way through their shutdowns.

    Beer cos it is Friday.

  27. smudge
    Facepalm

    Deja vu all over again

    We've covered this before, however...

    The company I worked for had a large computer room, basically long and narrow. The doorway was partway along one of the long sides.

    One day, one of the senior bosses was showing some prospective clients around. Took them into the doorway of the computer room.

    Spread his arms out emphasise how big it was.

    But you're wrong - that was OK.

    Then, to let them get a better view, he stepped to one side of the door, and leaned back...

  28. Barking House
    Facepalm

    Beware the Chairman of the Bank

    It's the early-mid 1990's we are an outsource supplier to a very large UK bank stock brokering operation - They have invested heavily in a clustered VAX infrastructure that has eliminated most of the single points of failure, except one big one in that the cluster (And shadowed disks) were in the same physical data centre and in fact in the same large bank customer only data center hall/room.

    This clustered VAX infrastructure worked extremely well and the bank was very happy with the high availability, much improved speed and the service we were delivering, so much so, they wanted to bring a bunch of their senior executives to our facility to show them what the banks money had been spent on.

    Cue to day of the visit, everything was going really well, the final part was an escorted visit to the actual data room where all their kit was housed - Some questions asked by executives but suddenly the chairman announced that he understood that it was impossible to shutdown the computers (Great selling of high availability by the account manager) and suddenly the chairman reached out to the big red button on the wall. Literally mill-seconds from disaster I managed to intercept the chairman's had (with quite a bit of force) and stop him hitting the EPO.

    However the aftermath was me apologising for the "assault", the bank chairman thinking we had something to hide and overall 12 months to rebuild the customer confidence in us - Even though we did stop a complete powerdown which would have cost them hundreds of thousands.

    Lesson learnt, never ever take customers into the data center even if they own it - So next step was a couple of windows installed so customers could see but not touch.

    1. PickledAardvark

      Re: Beware the Chairman of the Bank

      Perhaps a communication failure by the marketing team, or those who fed them their spin?

      When talking about disaster recovery, getting back to normal performance within X hours, etc it is imperative to stress that recovery is VERY expensive. And a disaster is different from somebody pressing the Big Red Button.

    2. tfewster
      Flame

      Re: Beware the Chairman of the Bank

      He may think he owns them. But if I'm responsible for the, they're mine.

  29. Doctor Syntax Silver badge

    I've mentioned before the bomb alert being persistently retriggered by a cleaner vigorously polishing the big red bomb alert button. I'm sure some enthusiastic cleaner somewhere must have polished an emergency shutdown button.

  30. KittenHuffer Silver badge

    I can't resist saying it .....

    ...... Right Fred's Head!

    1. Robert Carnegie Silver badge

      But

      Doesn't the modification make a serious head injury more likely next time?

      I suppose that may be intentional.

  31. Rtbcomp

    Non Accidental Power Interruptions

    I worked on a local government ICL 1900 site and the system kept crashing. Over time we noticed a coincidence between the crashes and the presence of a particular operator in the line printer room.

    It transpired that this guy had a chip on his shoulder and found he could crash the system by switching the circuit breaker at the rear of the printer off and on.

  32. disgruntled yank

    Who needs colleagues?

    I did once flip the reset switch on a client's machine. I had been planning to do so later on, to give them some practice in running the equivalent of fsck, but not quite then. This was during the installation and training phase, not production, so nothing was lost but time (ten minutes?) and face.

    On the other hand, I was at a tech day when my phone informed me that certain of our servers were down. This proved to be so, and I hopped the Metro back to the office. It turned out that the SAN tech, while doing I know not what preventive maintenance, had flipped the power switches with his shoulder. Everything started up again nicely, to our and his relief.

  33. Jason Bloomberg Silver badge
    Pint

    The fickle finger of fate

    "And don't press this", he said, as he was showing us around the new computer room, with a little more overreach than anticipated.

  34. Anonymous Coward
    Anonymous Coward

    BBC1 and 2 control rooms...

    ..used to have the big red buttons at same height as the head rests of the chairs. Fortunately the control room kit was just effectively remote control panels - all the actual kit was a floor below and ran happily without control panels being active. After a few brown trouser moments big flip covers were fitted.

  35. Anonymous Coward
    Anonymous Coward

    what does that do, click, CLUNK

    Not a DC but a whole college campus. Back in the late 80's I did electrical engineering at college. As part of an assignment we had to go in to the main electrical switch room and draw and lable what we saw. Anyway there was a very large airbrake style breaker the ones that look like a one arm bandit fruit machine, now I knew what it was and NOT to touch the large handle. BUT something in my head said I wonder what that does and I flicked the small flapper swich on the side of the air breaks case. CLUNK air break tripped and the entire college lost power! It was the FECKING breakers test button that I pressed!

    My mates all helpped out and said my A4 ring binder folder had accidently knocked the switch. For the rest of the term the college clock and bell was out by about 10mins as thats how long it took to get the power back!

  36. Giles C Silver badge

    At my previous employer we had the same problem. This was a few years ago though

    EPO. Upton under the door exit one and someone pressed the wrong one, replaced with a bigger button that you had to hit hard to trigger it.

    The same person (I think) was repsonsbile for unplugging a generator for the building (somebody stole the cables to the substation) from the main supply to run his envelope inserting machine. It took an hour before the ups ran out and the site died....

    The better one was the day that the power overloaded, entire room down (nobody was in there), went in with torches and the breakers in the room were fine. Went outside and we found that the 100A board in the comms room was fed by a 40A breaker outside....

  37. ICPurvis47
    FAIL

    Unintentional Emergency Shutdown

    When I was working for a very large electrical engineering company, building motor control gear for a well known maritime organisation, I also caused mayhem with part of my anatomy. One of the units was in Test, and the testers were doing a heat run, running the equipment at full chat whilst the observers from the customer looked on. I had gone into Test to take some photographs for the Instruction Manual that I was preparing (I was in Technical Manuals Department at the time) and I had to scrunch myself up into one corner of the roped-off area in order to get all of the cabinets in shot. Suddenly everything went dark, and the high pitched whine of the invertors wound down the scale to inaudibility. Cue furious shouts from the Test Engineers, I had inadvertently backed onto one of the emergency shutdown buttons that were located at various points around the department, and that had shut off all power to the Test area and surrounding parts of the building. A complete morning's heat run ruined, and the customer's observers were distinctly unimpressed. The heat run had to be rescheduled for the next morning as it had to start from cold. Needless to say, I was NOT allowed into Test whist a heat run was being performed on that or any further equipments.

  38. Petrea Mitchell
    Alert

    Switching off the oxygen

    Some years ago, a past workplace of mine got one of those systems installed which could deoxygenate the server room in the event of a fire. The control panel was mounted right by the door, and unfortunately was very sensitive to jostling, like the vibrations from the door slamming because someone was carrying something bulky in and didn't have a hand free to close the door gently.

    Luckily, this system would wait a few seconds before actually activating, giving someone the chance to stop it first. So those of us non-admins who worked nearby remained blissfully unaware of the hazard until the inevitable day when someone didn't manage to deactivate it in time. A lazy summer afternoon was interrupted by the almighty FFFFWRRRMMM of the system activating, followed by a couple panicky minutes of realizing what the sound must have been, trying to locate someone with server room access who was not actually in the server room at the time, visualizing colleagues passed out from hypoxia... (It was not until after this incident that we were told it didn't lower the oxygen level so far as to become totally unbreatheable.)

  39. swm

    Legitimate use of EPO buttons

    In the computer center the chief operator was an ex marine gunnery sergeant. In New Hampshire we always complained about the low humidity. The HVAC folks said they could fix this. The next moment the machine room started filling up from the bottom with fog. The operator calmly went to the EPO button and pressed it. It worked and he probably saved a lot of spinning rust.

    When designing the operating system I had to debug the power off shutdown fault (meaning that there was 160 usec before CPU power dropped). There was just this time to save all of the essential state in core (yes, real core). So I sat powering down and up the main cpu - a huge monster - to test this. The field engineers were not happy as they said that the CPU wasn't designed for repeated power cycling but never had any problems.

    So there are some legitimate uses for these buttons.

  40. cam
    Joke

    Big red button?

    That just says "Push Me. Go on, you know you want to. Puuuush Meeee."

    I'd opt for a two small grey buttons at opposite ends of the room, like in Superman 3. That'll work!

  41. Charles Calthrop

    i heard an awesome story about a guy in the dell factory at limerick. He was working on the production line when he pressed the emergency stop button by accident, and caused the entire line to shut down. It caused a huge rumpus as it cost $$$, so much so that someone from Dell HQ flew over to see what had happened, to try to see if there were workarounds they could implement. So this big shot was talking to the operative

    'Walk me through exactly what you did' the big shot guy said

    'I accidentally pressed the emergency stop button, like this', the guy said. Pressing it again to show him

  42. Andy A
    Happy

    I was once helping out by clearing a backlog of fault calls at a railway depot. Suddenly the PC I was working on went dark and all went quiet.

    Enquiries revealed that someone up the road had chopped the BIG cable with an excavator, so it was not going to be a quick fix. I went to the server room and gave things a graceful shutdown before the UPS gave out. This was late afternoon after the office workers had left, so only the engineering staff were around. I headed over to them to inform them that I was heading off too, and commiserated with them that they would now be unable to brew any tea. "No problem" they said. "This buffet car gets its power from the overhead line".

  43. HWwiz

    Almost

    Almost as good as the Contractor DC engineer at Heathrow in a certain DC that hit the kill switch that nearly brought down British Airways a few yrs back.

    Bet he has had a hard time getting a new job.

  44. Montreal Sean

    Petrol stations

    I work for a large privately owned IT services company in Canada, and one of our clients is a large energy corp with petrol stations across the country.

    I have lost track of how many times I've been sent to a site to troubleshoot why the paypoints can't communicate with the petrol pumps, only to find someone had accidentally pressed the big red emergency shutdown button for the pumps.

  45. Unicornpiss
    Happy

    I have this above the EPO "red button" in our server room..

    Red Button

  46. Marshalltown

    The underground tree squirrel blackout

    Never saw a colleague pull a "Fred's Head," but once, with all the staffing happily doing staff things, the building suddenly went dark. The UPS started to scream and the server shut down, though by then all the other work stations were already dark. The lights were out and the phone system dead. The boss had us checking the breakers to the building, but they were fine. The city utility said, "it wasn't us!!" An electrician was called in and found that we had power to the pole and down the descender to where it entered a conduit under the drive way into the parking lot. At the riser though, where the line entered the building, there was no power. The enter undergroun line had to be exposed and the conduit dismantled before the "cultprit" - a tree squirrel - who had for some squirrelly reason mades it way into the conduit and half way to the far end before settling in to chew through the power line with fatal results.

  47. prouton

    But what if you designed it so it couldn't accidentally be pushed...

    Back in the day I was a computer operator on an IBM System 370 box, which required something called a motor-generator unit to supply 3-phase power to the system. The unit itself was a plain looking box about the size of a wide freezer, with doors on the long sides for access to the guts. Toward the top of one of the doors was a cut-out through which the EPO button could be accessed. Now some clever engineer just knew you didn't want to place a push button at waist level that would kill everything in the room, so they smartly made it a "pull" to activate switch.

    Unfortunately, one day the IBM CE was enjoying some free coffee, leaning against the unit at just the right spot, and when he moved forward a belt loop on his pants managed to snag the rim of the switch and Bam! down goes the system. Okay, that's embarrassing enough, but as luck would have it the wall panel relay also died in the incident, and it took a full day to find a replacement for the part.

    Shortly after that the versatility of the 80-column punch card was put to use -- one was taped over the hole in the motor-generator's door panel.

    1. W.S.Gosset

      Re: But what if you designed it so it couldn't accidentally be pushed...

      > the versatility of the 80-column punch card

      See, SD cards and USB sticks just AREN'T as good as the good-ole tech!

  48. Debil

    A typically British response..

    We had nice sticky out kill switches and after being activated several times, ok I did it once simply by stacking boxes on a table handily put under a switch, management came up with a missive to prevent accidental activation.

    This turned out to be a load of plastic cups and me trotting around the place covering the red bit with a plastic cup. Seemed to work as I can't recall them being set off again.

    Oh yes and we had the halon gas stuff and raised floors. One data centre had an electrical fire that caused the halon to release.

    The whole computer room was wrecked as the injectors were under the raised floor and several floor tiles went into action and became part of the PDP11's.

    On the plus side room was refurbished and had the injectors sticking out of the cieling tiles.

    1. Colintd

      Re: A typically British response..

      I remember that halon system going off in the taperoom for the Cambridge university mainframe during the late 80s. Minor fire went out, but the level of dust raised meant all the tapes had to go through a special wash/cleaning machine (actual liquid bath with sponge pads). Extremely disruptive.

  49. Compuserve User
    Facepalm

    Remote Support in the dark

    Worked with an outfit supporting small medical office in the UK. We were still using XP (haha) and I had to manually patch each machine with the latest updates. We had one particular office that was unable to be contacted, and we had to call in for the office manager to check the server. She was compliant and the server came back on but only for a few minutes then it was not responding.

    I picked the short straw and had to go down to the facility to see what was going on. Log story short, the office manager replaced the server power cable with the kettle cable as the other one was longer, and every time she went into the server room and plugged it in and left, the door closing would pull the plug out. Even better, the server was moved next to the door as the kettle plug didn't reach and damaged the patch panel.

  50. Scroticus Canis
    Childcatcher

    "... a new hinged box of clear plastic to avoid accidental shutdowns in future...

    We got one of these in the machine room following a school visit.

    We certainly thought of the children following the incident - and banned them!

  51. Will Godfrey Silver badge
    Coat

    Lightbulb Moment

    Reading these comments, (and noting the number of them) It dawned on me that El Reg Commentards must have an average age of 70.

    The one with the farthings and a silver thrupenny bit in the pocket.

    1. jake Silver badge

      Re: Lightbulb Moment

      I don't know about an average age of 70, but there are quite a few of us who have been making money with computers since the 1960s or early 1970s. Isn't that groovy?

      Peace, man!

  52. Homeboy

    Around 1980 I was the site engineer for a large 24x7 data centre in the midlands which was full of IBM and Amdahl CPUs and rows and rows of IBM and StorageTek disk and tape running all the usual mission critical work. It was a new, purpose build data centre, wide corridors everywhere, double doors into the dc itself one of which was usually kept locked and the other side was opened with a card key for staff to enter.

    Everything was running fine, all the kit was up and running, lots of sounds of spinning things, until some new kit was being delivered during one busy day and the other half of the double doors was opened to allow it in. Yup - the (very new, recently installed) shiny red EPO button was not quite far enough along the wall and the second door hit it...cue sound of lots of spinning things slowing down.

  53. Colintd

    Kill switches that go bang

    Rather than silence I can speak from personal experience about what happens if you hit the kill/field dump switch on an NMR/MRI machine. All the stored energy in the coil vaporises the liquid helium coolant, followed.by a very loud bang/woosh as several thousand pounds of now gaseous helium, with about x1000 volume goes up the vent pipe. Had to be done as some idiot had come in with a steel gas cylinder which was literally dragged off the trolley into the coil. Not an experience to be repeated. Try leaning near the kill switch if you go for an MRI and see the techs.reaction...

    1. tinman

      Re: Kill switches that go bang

      at least your MRI did shut down. In this report the kill switch had been disconnected so the two staff in question had to suffer entrapment for several hours until a GE engineer came to shut the system down

      https://medicaldialogues.in/ge-pays-rs-1-crore-settlement-to-technician-who-got-stuck-in-mri-at-tata-memorial/

  54. simonpearse

    power outage

    One of my customers installed 3 PC's to run a taxi booking/dispatch system. They decided to do things properly, so they got an electrician to install an independent ring with sockets for the computers, the screens, printer, router, radio TNC, and some spare sockets. The ring was run off a smallish UPS.

    So far so good.

    Since the ups would only provide a short duration in the event of a power failure they bought a small generator, so if mains went, the ups would take over for long enough for them to get the generator out of the cupboard and connect it to the input of the ups.

    Seemed very good.

    Some years later a digger in the street severed the mains. Remember those 'spare' sockets.... two electric heaters and a kettle had found their way onto the circuit so the UPS threw a tantrum. They couldn't find the key for the cupboard containing the generator. The fuel in the generator was several years old and no-one new where the starting instructions were. An arguing mob of irate drivers soon had sore arms from pulling on the starter handle. then some bright spark noticed that the shop two doors away had power. And they had a long extension lead. Soon they had a working computer, screen, router, and TNC.

    Now an even brighter spark reasoned that if they removed the socket from the extension lead and replaced it with a plug, and plugged that into one of their wall sockets, it would power their 'ups' ring and all their computers would work. I suspect that their degree of familiarity with H&S laws was about as advanced as their knowledge of three phase mains distribution (yes the shop down the road was on a different phase)...

  55. G.Y.

    vacuum

    I once had a system was no backup tape; they told me "back up 1 disk to the other"

    "what if there's a power cut?"

    "you're on a UPS"

    Then a technician plugged a vacuum cleaner into the UPS, 4 times running, and blew it.

    I politely inquired as to why she was not shot at dawn.

  56. Martin
    Stop

    On a smaller scale...

    A friend of mine was once working on a PDP-11 machine, typing in a lot of code using TECO.

    Someone else in the office came over and started playing with the front switches on the PDP-11. For no reason, you understand - he was just fiddling for the sake of it. It probably wasn't going to do any harm, but not unreasonably my mate asked him to stop playing with the switches in case something went wrong. "Oh, look, it won't do any harm", says the other guy, "but if it makes you happy, I'll turn the keyswitch to lock so that the keys CAN'T do anything."

    So he reached for the keyswitch, and turned it the wrong way - which switched off the PDP-11....

  57. tony trolle
    Stop

    why did he do that ?

    Service call goes out for a mainframe tape deck, engineer turns up later on & phones to be left in.

    Someone walks down to main door and lets him and another man waiting next to him in.

    A minute later second guy said "oh big red button" and presses it.

    Operator and engineer look at each other, engineer said "why did he do that? "

    Turns out it was some random nutter they let in.

  58. OzBob

    Probably said before but still funny

    I was powering on an HP EVA Array after maintenance, and as I flipped the main breakers on the main PDUs, my co-worker snuck up behind me and clapped really loudly. Laughs all around from him, until I start clutching my chest and reaching for my medication (I had a severe heart attack a few years before), at which point his expression changed to "uh oh, bags not doing CPR". Fortunately the chest pains subsided after a few mins and we had a good laugh about it later.

  59. SPiT

    In case anyone is still reading, the regional control centres that the UK government commissioned for the FireControl project (go look it up) had their equipment room EPO buttons next to the magnetic locked exit door AND they didn't have an exit button for the door so it was the only button present. The situation was in fact even worse than that in that their official process at the time I came across it was to allow third party staff into the room with no means of exit and expect them to phone security to get out, a scheme that didn't even work reliably as during the delivery phase the security office wasn't reliably manned. All in all a rather impressive breach of health and safety and lone working arrangements.

  60. TeeCee Gold badge
    Facepalm

    EPO by proxy.

    Back in the eighties, a friend worked at at '38 shop as lone operator. Handily placed behind the two band printers was a wide, highly polished and almost frictionless marble windowledge. His method of getting reports off the printers for distribution was to flick the perfs on the fanfold at the appropriate point, stack the reports on the ledge and once he had a decent stack, give the stack a shove along the smooth, polished surface where it would stop at the end. Repeat until all reports are sat in a series of stacks along the ledge. Easy and efficient.

    One day he took the day off. The developers were busy with a software release required that day so it fell to The Boss to sort the reports out. He'd observed the operator's process and decided that it was the best idea, so that's what he'd do.

    Turned out that knowing the maximum permissible height of the first stack, so it wouldn't smack into the EPO button at the end of the ledge when slid that way, was a crucial part of the process that he'd missed.

  61. Anonymous Coward
    Anonymous Coward

    EPO inside data centre :-(

    We were experienced IT staff, so the Emergency Power Off was a large twistable lever located in the middle of the data centre complex (so that people would hurry the least distance on average whilst their coworker was in trouble). Not such a great position when the computer room started to fill with water and a volunteer was needed to wade across the sparking computer room to turn the EPO switch. Nowdays there's a second EPO switch inside a cabinet on the outside wall of the building -- that's got some theoretical issues, but none of those issues put people's lives at risk.

  62. W.S.Gosset
    Thumb Up

    "More Magic"

    A related story, wherein the Big Red Button is A Secret and A Joke:

    A Story About 'Magic' : ESR/catb.org

    Some years ago, I (GLS) was snooping around in the cabinets that housed the MIT AI Lab's PDP-10, and noticed a little switch glued to the frame of one cabinet. It was obviously a homebrew job, added by one of the lab's hardware hackers (no one knows who).

    You don't touch an unknown switch on a computer without knowing what it does, because you might crash the computer. The switch was labeled in a most unhelpful way. It had two positions, and scrawled in pencil on the metal switch body were the words ‘magic' and ‘more magic'. The switch was in the ‘more magic' position.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like