back to article Sysadmin left finger on power button for an hour to avert SAP outage

Welcome to the seventh instalment of Who, me? The Register's new column in which readers share stories of the times they broke stuff without any help at all from users. This week, meet "Jeremy" who back in 1999 scored his first "real" IT job "as part of a team sent out to run the IT at a big publisher." Said team was working …

Page:

  1. highdiver_2000

    Why didn't you eject the CD player using Windows Explorer before touching the server?

    1. pstones578

      I will give you that one given the timeframe but if this was SAP then I would guess it was not running on Windows. These days servers never come with CD drives anymore

      1. Lee D Silver badge

        Nope, but they do come with ID lights.

        It's a really dumb thing to press the button on the wrong server. And... if we're talking about an era where holding in the power button doesn't kill the machine hard in 5 seconds, and where NT is running, and where it doesn't auto-power-off on the Turn Off Your Computer screen, then we're back in the age of floppy disks and maybe even pre-CD in your average server.

        But whatever era, there will have been a better way to indicate what server you mean rather than just guessing.

        1. Anonymous Coward
          Anonymous Coward

          a) PL1000 / 1500 did not come with CD ROM per default - they were optional and expensive!

          b) the Y2K Updates were done by Floppy - ROMPaq Updates

          c) Hostnames were put on brownish labels and consisted of 8 positions: 2 letters for the city and then 6 numbers. No clue what kind of server it actually was - the lists were all you had.

          After several hours of too many dB, temperatures between 15 and 40 °C (Depending on where you were in the DC) - one tends to get a bit "unfocussed" - as was the case with my pal J here.

          (Yes, I post this anonymously - but I am said "Jeremy" ;)

          1. Emmeran

            Jeremy spoke in class today

            This same sort of thing happened to a friend and co-worker and we did indeed make him stand there holding the button in until we could the users out and the apps shut down.

            I still recall that forlorn look on his face as he stood there alone in the data center, it brings a smile to my face.

          2. Marco van de Voort

            Moreover the default cdrom was not exactly standard. It was connected on the onboard SCSI (the IDE was connected to the floppy ?!?!) and the system firmware could only boot from devices that had a special (512byte sector emulation?) jumper on and used floppy emulation. In the mid 2000s the only distro that booted was Slackware 8.1

            1. Alan Brown Silver badge

              "Moreover the default cdrom was not exactly standard. "

              No, but ejecting it makes it clear WHICH server you want rebooted.

              1. Marco van de Voort

                ejecting

                I know, my point was more that it was not a given that would work. The first Proliants were mightly quirky beasts, and the CDROM BIOS support was minimal and ancient.

        2. Anonymous Coward
          Anonymous Coward

          "[...] there will have been a better way to indicate what server [...]"

          Head came round the door "All yours". So I headed off to the machine room to do my testing on the cold stand-by comms processor. The console was mounted on top of the unit. Hit the keys for debugger mode - machine stops. Sudden howl of anguish behind me.

          There were two comms processors. That day they had decided to use the stand-by one for the official acceptance time trials. Thankfully the presiding government official allowed that as a genuine mistake that did not affect the acceptance criteria - and the repeat run was ok.

          After that there was a large notice on whichever one was the live machine.

        3. Dave K Silver badge

          Well, I'm assuming that as these were Y2K updates, we're talking of a server from around the mid-90s. I still saw plenty of AT (ie, mechanical) off switches in those days.

          As it is, hindsight is a wonderful thing. However, mistakes do happen...

  2. Anonymous Coward
    Anonymous Coward

    The weather

    I did turn it back on, but we had to cut back to the sofa for a few seconds while I figured out what I'd done.

  3. Anonymous Coward
    Anonymous Coward

    I did that once, not on a production server but home computer back in the days when a power off could kill it forever or severely mess up your next reboot. It teaches you a lesson about computer placement that you never forget. These days I have two pieces of cardboard wrapped in black tape over the top of the buttons because they are on top of the case.

    1. TechnicalBen Silver badge

      New PC case.

      I have a wonderful mini-itx PC case. Only problem is the power button is on the top, just where I may rest something for a moment. Like a game controller or whatever.

      I only did it the one time. ;)

      1. Boothy

        Re: New PC case.

        I like my current Antec Tower case (had it years now, triggers broom ya knows).

        It has a full height door on the front, hiding things like 5.25 and 3.5 bays (all unused these days), but it also hides the Power and Reset buttons.

        I don't know if by design, or accident, but the edge of the door also has large (finger sized) air vent holes from top to halfway down, the bottom one of which lines up quite nicely with the buttons.

        So no way to hit them by accident, but you can still use them without having to open the door.

      2. d3vy Silver badge

        Re: New PC case.

        "I have a wonderful mini-itx PC case. Only problem is the power button is on the top, just where I may rest something for a moment. Like a game controller or whatever."

        I used to work in an office where there were two banks of desks fed from two sockets (with extension leads - but thats not the wtf here..) the sockets were at about the same height as the head rest on your average office swivel chair and positioned right behind someones desk.

        The number of times that the power got knocked off started to get daft so the managements solution... Not to move the socket... not to change the sockets so there was no switch... Shove a few old PSUs under the desks to feed the PCs if the switch gets hit.

        1. JimboSmith Silver badge

          Re: New PC case.

          At home I have spike protected power strips on all my equipment. I did once have something that was fried when the power came back on after a power cut and am now slightly paranoid. Most of them have either no power switch or a recessed one to prevent you switching things off accidentally. The one place it does have a switch is the living room entertainment area (TV, DVD, Blu ray, Satellite receivers, CD, Amp etc.) My housemate was (out but) recording something she wanted to see and I didn't, so I thought I would tidy up the cables around the back. I was busy doing this when I heard the TV power off and standby lights which were reflected by the coffee table go out. I then spotted the switch that my knee had just hit which turned the blasted strip off.

          The recording was now stopped and it would take a couple of minutes to get everything back up and running. So I switched off and then back on the power to her room at the circuit breaker and claimed it was a power cut. She only lost approximately 4 minutes of the tv prog but but I vowed then and there to replace that strip. I now have a 19 inch rack unit which has proper power distribution strips (with spike protection) screwed to the back. These have sunken switched to prevent accidental presses.

    2. ma1010 Silver badge
      Alert

      Cats can be a problem

      Years ago in college, we worked in groups, and one of my group mates had a cat. One Sunday he was at home compiling all his hard work and watching the cat play around the computer, "Aww, how cute..."

      Then the cat nosed the big, red RESET button and thrashed his work. "Get out of here, you damned thing!" Not so cute, then, apparently.

      1. DJSpuddyLizard

        Re: Cats can be a problem

        @Cats can be a problem.

        Toddlers too.

        On our PCs at home I had to bypass the power and reset buttons and install switches with a key.

        Good old cold-war missile bunker key switches on both PCs, just in case.

        1. elDog Silver badge

          Re: Cats can be a problem

          Yeah - animals and kids and red buttons. Sounds like the state of affairs in the US right now.

          Wonder if he'll press it, change his mind, and keep it pressed in until the B-52's can be recalled.

    3. Oh Homer Silver badge
      Facepalm

      Re: I did that once

      My only fatal "power off" incident involved a PC with a mechanically failing drive (bearings failure, I believe). It was borked anyway, but in my desperation to recover at least some data I made the mistake of turning it off to try some other method. It never spun up again.

      Then there was the classic where I formatted the wrong drive, back before I knew that drives could be "un-formatted". Lost everything that day. I also gained a greater appreciation for backups.

      There was one happy tale, though.

      I had an Amiga 4K with a CyberPPC SCSI controller, which I'd been using without issue for years, until one day I decided to meddle with settings I didn't understand at all, in CyberPrefs (the SCSI controller's firmware settings). I came in first place for the Darwin Awards that day, and ended up with an unrecognised drive.

      Just to absolutely prove how stupid I was, for some reason I didn't make the connection between my meddling and the fact that my drive had mysteriously disappeared. As my brain sunk deeper into hibernation mode, I gave up completely and bought a PC - my first ever in fact, from the sadly long defunct First Computer Centre in Leeds. So in a way, I have that SCSI controller (or my stupidity) to thank for many happy years playing Doom and Quake, then eventually getting fed up with Windows and switching to Linux.

      Years later I fired up that Amiga 4K, went straight into CyberPrefs, changed back the incompatible setting, and rediscovered my "missing" drive, along with my long-lost youth.

      1. ds6 Bronze badge

        Re: I did that once

        I wish I could have recovered my digital childhood, if even it was mostly poor attempts at MS-DOS scripting, those mail-order screensaver bundles, and directories of disgustingly quantized GIFs... Frankly, I was too dumb to know what a backup was.

  4. Anonymous South African Coward Silver badge

    Nowadays the "hold-the-button-in" trick won't work, because of ACPI, especially on a modern PC.

    1. Steve the Cynic Silver badge

      Nowadays the "hold-the-button-in" trick won't work, because of ACPI, especially on a modern PC.

      "A modern PC"? The first one I ever saw that had the necessary bits was in 1996. Twenty-two years ago.

      1. imanidiot Silver badge

        I guess that just goes to indicate his age. Now get of my lawn!

      2. Anonymous Coward
        Anonymous Coward

        1996? I doubt that. Probably 1997 or 1998 for you.

        ACPI only got released in December 1996, and the first PCs with ACPI sold in 1997.

        Widespread adoption was only in 1998/99.

        Windows 95 had no ACPI support, Win 98 came with disabled ACPI. Only Linux 2.6 and Windows 2000 and onwards supported ACPI. And those OS even disabled ACPI on pre-2000 hardware, as ACPI v1 was quite buggy.

        https://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface

        1. JBowler

          Huh?

          >1996? I doubt that. Probably 1997 or 1998 for you.

          You know Steve, the Cynic, then, Anonymous Coward?

          >ACPI only got released in December 1996

          Duh..... duh..... Like, someone developed it dude.

          Quoting from Wikipedia just proves you work in a troll farm for putin.

          I don't know Steve, the Cynic, but I do know what I was doing in December 1996 and it certainly wasn't released until some time in 1997.

        2. Steve the Cynic Silver badge

          Hmm. Well, it was around that sort of time, possibly 1997 and certainly NOT 1998, that I ended up with a computer at $JOB that switched itself off from Windows 95. Possibly not full ACPI, but not ordinary pre-ATX either.

          1. StargateSg7 Bronze badge

            My old Compaq 386-series did that and that was late 1987 or so. You could shut the computer itself down completely via software by some pushing some values into the x86 registers and calling an interrupt which was NOT part of the MS-DOS or IBM PC-AT BIOS standard INT calls. AND if your terminal display was SCART compatible like ours were (they were basically industrial-grade 20 inch Sony Trinitron TV's used as computer displays with 800x600 pixels of resolution), we could even shut down the monitor from software in 1987! ..SOOOO.....this isn't new technology.

      3. Shart Tank

        I protest! Anything past trumpet winsock is modern!

    2. Anonymous Coward
      Anonymous Coward

      "Nowadays the "hold-the-button-in" trick won't work, because of ACPI, especially on a modern PC"

      It'll work fine. You just need the users logged out, and everything shutdown within 4 seconds. A challenge, but perfectly achievable for a true boffin!

      1. Montreal Sean

        It'll work fine

        "It'll work fine. You just need the users logged out, and everything shutdown within 4 seconds. A challenge, but perfectly achievable for a true boffin!"

        Bah! Users should be saving their work every 10 minutes or less.

        If they didn't, too bad for them.

        Ok, I may be a bit of a bastard...

        1. Anonymous Coward
          Anonymous Coward

          Re: It'll work fine

          I worked in a college with a CTO who thought like that. On multiple occasions I saw him cheerfully reboot terminal servers, kicking all users out without warning, to 'fix' stuck print queues. On examination days.

          I'd then get lumbered with the job of helping anxious teachers fill out the 'extenuating circumstances' paperwork for the roughly 10% of kids who'd basically been given an instant exam resit. Bafflingly, the college never lost its exam centre status, and the CTO was never disciplined in any way, as all the other bosses seemed to view it as an act of God.

      2. Alan Brown Silver badge

        Nowadays you don't need to hold the button in, unless you REALLY mean to switch it off.

        An accidental press won't shut things down. Unless you set it up that way.

  5. alain williams Silver badge

    Typed 'Reboot' where ... ?

    Telnetted into various Unix machines, wanted to restart the one in the server room. Whoops - I forgot which machine I was logged into and typed 'reboot' to a machine on the other side of the planet. It did not come up, had to wait until teatime for the guys there to come in and push a button :-(

    1. wyatt
      Thumb Up

      Re: Types 'Halt' where ... ?

      I can hold my hands up to that one as well. SSH to a server then a workstation, 1 letter (c/s) difference between them and I wasn't on the client.. Fortunately it was a reboot rather than a shutdown and it also happened during another major outage so the impact was minimal.

      1. Chris King Silver badge

        Re: Types 'Halt' where ... ?

        I was on the receiving end of that once.

        An acdemic had moved to another uni, and my opposite number at the new uni was helping him transfer his files from our OpenVMS machine to theirs.

        In another telnet window, the IT bod was logged into his test OpenVMS machine, preparing to test a patch for a nasty little crash bug that anyone with telnet/SSH access could trigger - no extra privileges required.

        Yes, he got the two windows mixed up, and our box dropped dead.

        Boom, crash dump and P00>>> prompt at the system console.

        Fortunately, it happened in the middle of a change window, ironically to install and test the very same patch. Still, it's not really the sort of thing you want to see when logged into SYSTEM at the console and installing patches.

        The other guy phoned me a couple of minutes later and 'fessed up to his mistake.

    2. Aitor 1 Silver badge

      Re: Typed 'Reboot' where ... ?

      A work colleague (admin) lost his job that way many moons ago.

      Hethought he was putting my code (well, a version update of the project I lead) into the integration environment.. but put it into production, as he had both terminals open, and made the huge error of pulling from command line. I had told him before to put into the server, and execute from the server.. as a friendly suggestion.

      He was lucky in the sense that there were no bugs in the code, so in a sense the systems kept working, unlucky in the sense that this was in the client/server era, so the decision was to push the updated client. 45 minutes down time for 50/100 ppl (dont remember well).

      He lost his job for a single mistake in two years, I am still a bit angry about that.

      1. Evil Auditor Silver badge

        Re: Typed 'Reboot' where ... ?

        He lost his job for a single mistake in two years, I am still a bit angry about that

        Angry that he made this single mistake or angry that he lost his job?

        Depending on the type of business 45 minutes downtime may or may not be reason for dismissal. Apparently, more than 5 hours of downtime for approx. 90% of the staff of about 20k (a bank) was no reason for dismissal. Than again, it wasn't due to an operating error but a management decision to implement a half-baked release.

        1. dotdavid

          Re: Typed 'Reboot' where ... ?

          He lost his job for a single mistake in two years, I am still a bit angry about that

          Indeed, seems a bit of a stupid decision to me, especially as the fired guy is definitely going to be the one person you are certain would never make *that* mistake again.

          1. Cynic_999 Silver badge

            Re: Typed 'Reboot' where ... ?

            "

            Indeed, seems a bit of a stupid decision to me, especially as the fired guy is definitely going to be the one person you are certain would never make *that* mistake again.

            "

            The same might be said of a driver who accidentally hits the accelerator instead of the brake and ploughs into a bus queue. But I guarantee he would lose his licence at the very least, and be lucky if he escaped jail. Usually it is the severity of the act that is punished, but sometimes the consequences of a simple mistake are so severe that they are taken into account as well.

            1. Adrian 4 Silver badge

              Re: Typed 'Reboot' where ... ?

              The punishment isn't for the guy who caused the problem. It's a warning for the rest of you.

              And it's only applied by the sort of manglement that values numbers, not individuals.

            2. Wayland Bronze badge

              Re: Typed 'Reboot' where ... ?

              Cynic_999 "accidentally hits the accelerator instead of the brake"

              It's not the same at all because the driver is using all the controls constantly with no problem. To catastrophically make three errors all at one and persist with those errors until people are run over is nothing like being a bit late on the brake peddle.

              There is a video where a man is chased into a layby onto the pavement by a bus which smashes through the front of a shop. The man managed to escape but the driver claimed he hit the wrong peddle.

              From what I can remember about driving (have not driven since yesterday) you don't hit the peddles with your feet you gradually press them to cause the amount of acceleration or deceleration you need. You begin doing this in plenty of time and you can press the peddles harder if you need more effect.

              In a rack of servers it's an easy mistake to be looking at the wrong server, hence the little button that lights up so you can figure out which one you want to work on.

              1. Cynic_999 Silver badge

                Re: Typed 'Reboot' where ... ?

                "

                From what I can remember about driving (have not driven since yesterday) you don't hit the peddles with your feet you gradually press them to cause the amount of acceleration or deceleration you need

                "

                It is easier than you might think.

                Imagine that you are closing slowly with the car in front. So you press gently on the brake but you see that you are still closing with the car in front. So you press a bit harder - and see the gap is now closing *really* fast, so you panic and jam the brake pedal full to the floor. Only later do you realise that your foot had been on the accelerator rather than the brake.

                Or while stopped you start reading a text on the phone in your lap when out of the corner of your eye you suddenly see that your car has started slowly rolling forward because you forgot to set the handbrake. Sudden adrenaline rush and panic, you stamp hard on the brake to stop the car before it rolls into something - except it isn't the brake.

              2. Cynic_999 Silver badge

                Re: Typed 'Reboot' where ... ?

                "

                There is a video where a man is chased into a layby onto the pavement by a bus which smashes through the front of a shop. The man managed to escape but the driver claimed he hit the wrong peddle.

                "

                You really think the driver did it deliberately? You have obviously never reacted in a panic.

        2. I am the liquor

          Re: Typed 'Reboot' where ... ?

          @Evil Auditor

          Depending on the type of business 45 minutes downtime may or may not be reason for dismissal.

          If 45 minutes downtime is that much of a problem, then sacking the tech who caused it by a simple finger fumble is nothing more than scapegoating. More reasonable would be to sack the executive who failed to put in place systems ensuring a simple human error couldn't cause such a serious problem.

          1. Mark 85 Silver badge

            Re: Typed 'Reboot' where ... ?

            More reasonable would be to sack the executive who failed to put in place systems ensuring a simple human error couldn't cause such a serious problem.

            In a perfect world, yes that would be the right thing to do. In the real world, the execs protect each other and everyone else is cannon fodder and/or scapegoats.

          2. Shart Tank

            Re: Typed 'Reboot' where ... ?

            You must be new to this....

          3. Evil Auditor Silver badge

            Re: Typed 'Reboot' where ... ?

            @I am the liquor

            I fully agree with you.

      2. Dave K Silver badge

        Re: Typed 'Reboot' where ... ?

        It's unfortunate if he was otherwise a good admin. After all, if there's one thing you do know afterwards is that you have an admin who isn't going to make that mistake again in a hurry...

    3. Anonymous Coward Silver badge
      Facepalm

      Re: Typed 'Reboot' where ... ?

      Boss: "Client z has a problem, so I'm just rebooting server x"

      Me: "OK, so why have I just had a notification that server y has rebooted?"

      Boss: "Oh shit"

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019