back to article Techie was bigged up by boss… only to cause mass Microsoft Exchange outage

Welcome once more to Who, Me?, our weekly column in which readers confess to their worst IT cock-ups. This time, we meet “Freddie”, who must have a good supply of luck on his side, as he managed to not only get away with his error, but win praise for fixing it. At the time, Freddie was a busy man. He worked for a managed …

  1. Rich 11 Silver badge

    we’re on the hunt for spooky tales for a Halloween special of On Call

    To be titled 'Ghost in the Machine'?

    1. Anonymous Coward
      Anonymous Coward

      Or "Ghost in the Shell Server."

      1. Buzzword

        It'll be a crossover with the movies which starred Matt Damon as an undercover spy: "Ghost in the Bourne Shell". The sequel will be "Ghost in the Bourne Again Shell".

        1. Symon Silver badge
          Coat

          JSON Bourne.

          {

          "firstName": "David",

          "lastName": "Webb",

          "isAlive": true,

          "age": 48,

          "address": {

          "streetAddress": "415 East 71st Street",

          "city": "Nixa",

          "state": "Missouri ",

          "postalCode": "65714"

          },

          "Preferred shell": [

          {

          "type": "linux",

          "name": "Bourne"

          },

          {

          "type": "when out and about",

          "name": "bash"

          }

          ],

          "children": "2",

          "spouse": "Dao"

          }

          1. David Webb

            Re: JSON Bourne.

            Great, now the damn assassins are going to be after me again, thanks for that mate!

            1. Anonymous Coward
              Anonymous Coward

              Re: JSON Bourne.

              Your cover is safe - in Arkansas they call each other "Bubba", not mate, so either you are an import (not easy to do in this day and age), or you are living in a different country, possibly in the southern hemisphere .

          2. Stevie Silver badge
            Pint

            Re: JSON Bourne.

            Well done, Symon, well done. E-beer for you.

    2. Mark 85 Silver badge
      Pint

      we’re on the hunt for spooky tales for a Halloween special of On Call

      To be titled 'Ghost in the Machine'?

      And here I was thinking they were looking for more horror stories about Windows 10. I'll go to the naughty corner after I offer this virtual beer.

  2. Lee D Silver badge

    Which is why you don't work direct at the server but over RDP unless something really critical has happened, because then "shutdown" is something you can only really do with a command-line (or if some installed program offers to do it for you, but then you test that on a non-server first, and generate yourself a mental install script, right?)

    1. Anonymous Coward
      Anonymous Coward

      Ehm, no. you can do a shutdown from RDP too - even if it will warn you - if you have the permissions.

      Anyway, there was a time when MS server applications had great remote administration applications, which let you working from your workstation without any need to access the server directly, or via RDP or whatever.

      Then came the Linux crowd where there were no such applications, and everything must be done directly connected to a server via SSH to manipulate some text files (while maybe wiping your machine with rm -rf *, or shutting it down if you happen to work in the wrong terminal).

      Unluckily MS saw how cheaper it was (and easier to offshore), compared to developing full remote administration applications, and copied it with PowerShell. Now, even management applications are wrappers over PowerShell scripts, so everything is far slower, bloated, and error handling a joke.

      1. LeoP

        Re: while maybe wiping your machine with rm -rf *

        Well, who wants remote clicky-mousy for server administration instead of a text file that can be copied, edited, copied back just in case, etc. might actually be "wiping your machine with rm -rf *".

        From the point-of-view of the universe, he or she might also do less harm in a Windows environment.

      2. K Silver badge
        Trollface

        RE: Then came the Linux crowd where there were no such applications

        CLI works for linux because every application is decoupled... With Windows its a peace of dogs meat, as all they've done is take the same monolithic products and stripped away the GUI - For analogies,

        Linux is like a sleek F1 car, each part is self-contained and held together with screws. It can be hard to get the components working together at first, but they're easily swappable.

        Windows is like a big American RV, humongous and put together with rivets... it works and is very comfortable, but don't expect to swap anything without rebuilding the whole thing.

        Troll, cause its Monday morning and I'm stuck on a train going out of my mind --->

        1. Anonymous Coward
          Anonymous Coward

          Re: RE: Then came the Linux crowd where there were no such applications

          "Linux is like a sleek F1 car, each part is self-contained and held together with screws. "

          Nowadays, that's unfortunately less true than ever. To illustrate this, I suggest you try the Gentoo installation process. When you start with a stage 3 tarball and use OpenRC rather than systemd, the number of dependencies required to get a *lightweight* functioning desktop with a suite of useful applications which use your hardware properly is scary.

          1. asdfasdfasdf2015

            Re: RE: gentoo

            what are you on about?

            xorg-server mate slim firefox thunderbird conky dconf-editor libreoffice galculator corefonts dejavu roboto vlc audacity spotify ghex gimp conky

            there. you have a gentoo desktop.

            1. Norman Nescio Bronze badge
              Pint

              Re: RE: gentoo

              >> "Linux is like a sleek F1 car, each part is self-contained and held together with screws. "

              >> Nowadays, that's unfortunately less true than ever. To illustrate this, I suggest you try the Gentoo installation process. When you start with a stage 3 tarball and use OpenRC rather than systemd, the number of dependencies required to get a *lightweight* functioning desktop with a suite of useful applications which use your hardware properly is scary.

              > what are you on about?

              > xorg-server mate slim firefox thunderbird conky dconf-editor libreoffice galculator corefonts dejavu roboto vlc audacity spotify ghex gimp conky

              > there. you have a gentoo desktop.

              Um. You put conky in twice. I know it's good, but it is not that good. Good effort, though, so please accept the virtual pint --->

              And for most normals, that would be part of a truly scary command line, which would be accompanied by pity when you proudly point out you composed it from memory.

          2. molletts
            Linux

            Re: RE: Then came the Linux crowd where there were no such applications

            "...try the Gentoo installation process. ... the number of dependencies required to get a *lightweight* functioning desktop with a suite of useful applications which use your hardware properly is scary."

            At least Portage deals with all those dependencies for you automagically, even if you do need to type a few package.use incantations to keep it all clean & tidy and free of unnecessary cruft and avoid filling the OCD-pleasing 4GB partition. (I'm afraid I had to cave in and resort to allocating 8GB on my desktop systems some years ago.)

            Getting a fully-tricked-out Windows system requires downloading Mammon knows what from Baal knows where (they'll give you hints if you bribe them with shares in your soul), then trying to install it all in the right order without accidentally letting any of those packages install the bundled spyware, take over a bunch of file associations that you would rather stayed with some other package or replace your preferred media player with SuperVideoMediaDoober Plus Max (Cloud Edition). Then you find that one of the things you need only works properly if you install .Net Framework 1.1 _after_ you've installed .Net 4 but before you install .Net 2, unless you install .Net 3.5 first, in which case you have to install the Visual C++ 10,11,12,13,14,15,16,17 and 18 runtimes in an order determined by consulting a medium, after slaughtering a goat at New Moon.

            (Oh wait, I'm getting confused. The bit with the goat is from the NetBSD install procedure.)

            1. TRT Silver badge

              Re: "... headless MS servers do indeed exist..."

              ARRRGGGHHHHH! Are the halloween stories starting already?

              Or there's the JK Rowling one, "Nearly headless 'nix server".

              1. SVV Silver badge

                Re: "... headless MS servers do indeed exist..."

                Headless MS servers are rather like headless chickens. They can run for a little while, but soon conk out.

                They can also both be described as "dead fowl".

                1. TRT Silver badge

                  Re: "... headless MS servers do indeed exist..."

                  I suppose they might be a turkey solution.

            2. phuzz Silver badge
              Gimp

              Re: RE: Then came the Linux crowd where there were no such applications

              "Getting a fully-tricked-out Windows system requires downloading Mammon knows what from Baal knows where"

              Ninite. One installer, no spyware, done.

              To stick with the point about upgrading a CLI based OS to a full GUI, in Windows you just do: install-windowsfeature server-gui-mgmt-infra,server-gui-shell.

        2. Khaptain Silver badge

          Re: RE: Then came the Linux crowd where there were no such applications

          @K

          You do know that modern PowerShell is actually available and actually worlks... and that headless MS servers do indeed exist...

          1. Zippy´s Sausage Factory
            Windows

            Re: RE: Then came the Linux crowd where there were no such applications

            @khaptain

            You do know that modern PowerShell is actually available and actually worlks...

            Hahahahahaha

            and that headless MS servers do indeed exist...

            No dooont... stooooppp... my sides are hurting...

            1. David Given

              Re: RE: Then came the Linux crowd where there were no such applications

              *No dooont... stooooppp... my sides are hurting...*

              Psst! Don't tell anyone, but there's a Linux port of PowerShell! There's even a Debian repository:

              https://docs.microsoft.com/en-us/powershell/scripting/setup/installing-powershell-core-on-linux?view=powershell-6

        3. phuzz Silver badge

          Re: RE: Then came the Linux crowd where there were no such applications

          "Linux is like a sleek F1 car"

          You have to replace half the components after every few hours of use, and just chuck the old ones away?

          A rally car might have been a better choice (they're designed to have components swapped in and out), but even then it's still not a good analogy.

        4. Smoking Man

          Re: RE: Then came the Linux crowd where there were no such applications

          Just dreamin':

          [Quote]

          Linux is like a sleek F1 car, each part is self-contained and held together with screws. It can be hard to get the components working together at first, but they're easily swappable.

          [/Quote]

          Then there was systemd: https://youtu.be/hYOWIdPHXts

        5. vincent himpe

          Re: RE: Then came the Linux crowd where there were no such applications

          A sleek F1 car... where everyone brings one piece of the car along when they come to the racetrack. They all go out to the pit and put the car together piece by piece, arguing non-stop about what kind of car they are supposed to be building. Violent fights erupt between KDE / Gnome and VI / emacs, Linux/Debian user groups ... In the end the bodywork is not installed because nobody can agree on the color scheme to be used. There's not steering wheel ,shifter or pedals. Instead there's a panels with some cryptically labelled buttons. And a MAN page explaining in which order to press the buttons to steer the car, accelerate and brake...

          The car takes off without paint or bodywork but is declared a huge success because.. well look at the 'open'-ness of it... you can see every part ... and it's very lightweight too... so it's fast. But it still is only half baked. If it breaks the people who built it simply tell you : we gave you the list of parts (the sourcecode) you go fix it. We've moved on to the next car. If you dare raise the issues of missing bodywork and paint : prepare to be scoffed at. As for the steering steering wheel and pedals : this is not a kids bike. Only little children need steering wheels and pedals.

          coat..

        6. Anonymous Coward
          Anonymous Coward

          Re: RE: Then came the Linux crowd where there were no such applications

          Linux is like a sleek F1 car, each part is self-contained and held together with screws. It can be hard to get the components working together at first, but they're easily swappable.

          Ha Ha Ha Ha Ha Ha

          Linux is like a series 2 land-rover defender with bits and bobs screwed on from Later models, discoveries and Range rovers.

          Of course in reality some bits aren't screwed on and need welding or you may have to drill some holes in the chassis rails or even remove the engine.

          Some bits wont work together at all and may cause your defender to stop in the middle of a stream.

        7. Field Commander A9

          Linux is like a sleek F1 car ... Windows is like a big American RV

          Good analogy! Especially considering the fact that F1 cars usually are designed with less than ~100 hours of life span.

      3. mathew42
        Mushroom

        Simply seeing shutdown on the same menu as log off sends chills down my spine every time I see it.

        Sure I can understand how it makes perfect sense for a desktop system, but for a production server where I don't have access to restart the box. Not good.

        1. Anonymous Coward
          Anonymous Coward

          shutdown/reboot

          Agreed. Where's the Windows version of Molly-guard?

        2. EnviableOne Bronze badge

          especially when your in the office in slough, and the server is in telecity, and you dont have inteligent hands.....

        3. Anonymous Coward
          Anonymous Coward

          ..."where I don't have access to restart the box"

          That is quite rare nowadays though. Usually they'll be VMs which you just start up through the relevant console or for hardware boxes/vm hosts then you normally have a management interface that works with the box switched off (but obviously still powered). Failing that the intelligent power supply can be cycled and as long as you have it on power on on power restore that should work. I've never let a truly remote box get zombied by an errant shutdown.

        4. Field Commander A9

          shutdown on the same menu as log off

          Did you know that you're supposed to click on your avatar (instead of the power icon) on the start menu when you're trying to log off or lock the screen?

      4. Jim 59

        "“I had just finished with five minutes to spare, but instead of hitting 'sign-out', I accidentally hit 'shut-down' on the primary server,” Freddie said."

        Placing the logout button next to the button for "shutdown" is one of the worst pieces of design in the entirety of Windows, and that's saying something. Freddie must be one of thousands of people who have made and continue to make this mistake.

        With Windows 10, the buttons have been parted (but only by half an inch), and are no longer in the same submenu. And they are better marked, but it is still one of the most obtuse and needless mistakes ever to come out of Microsoft. Putting a massive risk at customers' feet for no reason.

  3. Anonymous Coward
    Anonymous Coward

    Been there got the t-shirt

    When using robocopy to move an empty (as in no users) EDB file to a new drive, make sure you specify the Database Name in the script.

    We were running out of space so added some more storage, then moved some of the databases to that new drive.

    We had 9 days worth of space left so it was a bit of a rush job once the storage was ready.

    Anyway, without the variable set, I robocopied the half empty drive over the live one...

    Script cancelled but damage done.

    Thankfully Exchange sorted it all out, failing over to another copy with about 10-15 mins downtime. It took it's time to think about it, but it sorted itself out! Phew.

    You learn by your mistakes!!!

    1. Nunyabiznes

      Re: Been there got the t-shirt

      When you are lucky you learn by others' mistakes!

      1. Alien8n Silver badge

        Re: Been there got the t-shirt

        Our previous contractor was asked to leave his employment as a result of an errant Robocopy statement.

        He left the MIR switch in his script when copying over some errant files that hadn't copied due to permission issues, 2 days after the file server had been replaced. Result? We lost 2 days data. His biggest mistake though was doing all this before checking to make sure we could actually access the backups of the new file server. Small mercy being only 2 days data lost, and it wasn't the database server.

  4. Giovani Tapini
    Windows

    Note to Microsoft

    the default on an option on server OS should be logoff not shutdown.

    1. Dave K Silver badge

      Re: Note to Microsoft

      At least on later versions of Windows Server (from 2008 onwards I believe), you're required to provide a reason for the shutdown/restart that goes into the event log. This is usually about the same time that you realise you hit the wrong option when trying to select "Log off", and does at least give you chance to cancel the impending doom you've initiated.

      1. joshimitsu

        Re: Note to Microsoft

        I think that feature was there in Server 2003 as well.

        And Windows desktop had an "are you sure?" screen.

      2. Peter2 Silver badge

        Re: Note to Microsoft

        Having heard all of the horror stories of people shutting down servers since the 2k days I have always put in a GPO on my sites to remove the "shut down" and "restart" options from the start menu of my servers. (plus other housekeeping bits like logging off people who leave their sessions connected after a decent period.)

        This forces shutting machines down to be done with the "shutdown" command (start->run "shutdown /i") which forces very deliberate action be taken in shutting machines down.

        1. Anonymous Coward Silver badge
          Facepalm

          Re: Note to Microsoft

          @Peter2

          "I have always put in a GPO on my sites to remove the "shut down" and "restart" options from the start menu of my servers"

          I tried that with one particular customer. They were in the habit of rebooting the server whenever there was any problem, so I thought that removing the menu items would stop them. Instead, they resorted to yanking the power cable every time.

      3. LeahroyNake Bronze badge

        Re: Note to Microsoft

        All well and good until it restarts at 0930 because 'updates' Server 2012. I hadn't logged on locally to that server for several weeks and the updates should restarts should have happend outside the backup window / between 7-8 am. Thanks MS.

    2. Prst. V.Jeltz Silver badge

      Re: Note to Microsoft

      "the default on an option on server OS should be logoff not shutdown."

      I think it is these days

      Also you get that "Let me record reason for shutdown" box - thats saved me a couple times. or is that only on reboot?

    3. vincent himpe

      Re: Note to Microsoft

      Real computers boot at power up and are shut down when they outlived their usefulness and decomissioned. They don't have reboot options and don't need them. Ask anyone running a Cray...

  5. Mk4

    Is there anybody out there?

    I did the same thing, accidentally shutting down the file and print server for the London HQ of AT&T (this was a very long time ago, so I feel safe enough naming the company) at 14:30ish on a Thursday afternoon. I was not sure I had really hit "shutdown" until I saw all the lights go off on the server. Starting up was a slow process - the disk arrays spun up their drives sequentially and the whole thing was offline for 5-10 minutes.

    We had all the big wigs in the building, the hundreds of staff etc. but there was not a single call logged. SMB mounts on clients would only have noticed if they tried to r/w while the server was down and obviously also no-one tried to print anything (or just tried it again 5 minutes later and it worked).

    Not a truly impressive story I will admit, but in those minutes before I could check that everything was back online I was convinced I was going to get the sack. :-)

  6. K Silver badge

    Reminds me of something I heard a long time ago...

    Take 10 mins to fix a business crippling outage, then get shouted at for a) causing it and b) taking so long to fix it... But, take 5 hours to fix it, then you're a rockstar, praised for being a saviour!

    1. frank ly Silver badge

      Re: Reminds me of something I heard a long time ago...

      If it took 5 hours it must have been a difficult job with lots of hard work involved, obviously, er ..... as I always told management.

    2. Andrew Moore Silver badge

      Re: Reminds me of something I heard a long time ago...

      Never finish anything too quickly- it gives management unrealistic expectations...

      1. TRT Silver badge

        Re: Reminds me of something I heard a long time ago...

        To be honest, Captain, I think you gave me too much time...

        So how long will it really take, Laddie? Eh? You told him exactly how much time it would take to fix? How are you going to keep your reputation as a miracle worker doing that?

        Montgomery Scott.

  7. Spazturtle Silver badge

    If your systems are too stable it's always a good idea to cause some 'unplanned maintenance', makes you look good for being able to quickly fix the issue and it reminds management that you do actually do a job.

    1. Rich 11 Silver badge

      Ideally you should time the alert to reach you just as everyone arrives for work. Ensure you're standing in the lobby juggling a hot coffee and a disposable but impressively heavy manual, drop the manual to grab your phone when it plays a klaxon at top volume, yell "Fuck!" when you see the screen, drop the coffee and race for the server room. When everyone arrives at their desks to find all systems up and running just as if nothing had ever happened, they'll remember you.

      1. el_oscuro

        I used to have the siren ring tone on my iPhone set for the ops center at work. I would get paged so often at 3am from that office, that hearing that ring tone on someone else's phone still sets me off. And I haven't worked there in years.

        1. David Given
          Unhappy

          I know from bitter experience that never, ever, ever use any sound you actually like as a pager alert. You *will* learn to associate it with terror, panic, and being woken up at four in the morning. It doesn't matter how pleasant or innocuous it is; using it on a pager will ruin it for you --- even Rich Evans' laughter will lose its charm.

          1. Anonymous Coward
            Anonymous Coward

            Yeah, know that feeling. I now use vibrate for overnight calls, but after a while I started getting shudders just putting my phone into vibrate mode - as it vibrates as it changes mode.

          2. ArrZarr Silver badge
            FAIL

            Count yourselves lucky.

            You changed the ringtone.

  8. steviebuk Silver badge

    A spooky tail...

    ...but I suspect bullshit. I'm a skeptic but still like listening to the stories anyway was this.

    At a certain NHS Trust, I can't remember which hospital (not one I worked at). A tech didn't like going down into the basement as he said it was creepy. To get to the IT cupboard down there you had to go through an old spare room with just chairs in it. He said they were scattered when he went in and when he'd come out of the IT room they were all arranged.

    He didn't hang around long.

    Again, I suspect a bullshit story to scare another engineer who REALLY didn't like going down there :) or other engineers sneaked in and arranged them silently (more plausible)

    I read that back and it's quite a boring story but the only spooky one I have.

  9. Martin an gof Silver badge

    Exchange seems to feature a lot

    Many of the "who me?" and "on call" stories seem to feature Exchange servers, possibly more than any other single type of story. Never (thankfully) having had to be responsible for Exchange, is this because Exchange really is a pig to administer, or is it simply that there's a heck of a lot of them out there?

    If Exchange is so bad, aren't there any alternatives?

    M.

    1. Korev Silver badge
      Joke

      Re: Exchange seems to feature a lot

      Lotus Notes

      1. Stevie Silver badge

        Re: Exchange seems to feature a lot

        Gah! I still have nightmares about the "choose your own toolbar buttons from eighty seven toolbars" feature. One false click and there went the feature you used every day but couldn't for the life of you remember where it started life so you could get it back again.

      2. niksgarage

        Re: Exchange seems to feature a lot

        When I briefly managed the Lotus Notes support team in IBM, quite a few of us played instruments, so we decided we would form a band to play in the evenings. Someone suggested we call ourselves "Replication Conflict".

      3. jcitron

        Re: Exchange seems to feature a lot

        Yup. Lotus notes runs on "stuff". I supported a few of those in the past and periodically they would fall over with smiley faces and stuff on the console. At first I thought it was the ancient hardware they ran on then I left that company and low and behold the same thing used to happen on their boxes too. I would get a call that "Notes can't be reached", and I'd take a walk down the hall to press the reset button.

        CC:Mail wasn't much better and that used to get the smiley-faces and stuff on its screen as well. A more than periodic fresh brains and we'd be up and running for another day or two.

        But on the other side when the company split from its parent that ran CC:Mail and notes, I ran an Exchange 5.5 server for 10 years and never had a problem with it. It's quirky, but once you understand the setup, it's not bad.

        I haven't used it in about 10 years, having retired in 2012, so I've forgotten a lot but it wasn't bad once it was up and running, and it's one of those pieces of software you don't do anything with unless you have to. You let it do its thing outside of periodic maintenance required and of course doing anything AFTER a successful back is a must as that guarantees there won't be a problem.

    2. Anonymous Coward
      Anonymous Coward

      Re: Exchange seems to feature a lot

      "is this because Exchange really is a pig to administer, or is it simply that there's a heck of a lot of them out there?"

      Obligatory UserFriendly:

      http://ars.userfriendly.org/cartoons/?id=19980316

      Dentist-turned-indoctrinator: "[The Microsoft drill bit] doesn't work well, but we all use it anyways."

    3. Trixr

      Re: Exchange seems to feature a lot

      I think Exchange is great, personally. It's people who screw around with it too much that run into trouble, or who are running legacy configurations that have been deprecated for years (the amount of orgs still running 2003 is scary). Related, those who have bosses unwilling to pay for upgrades or resources to keep it in good nick. Or admins who apply CUs that have been out for less than 3 months (before MS has a chance to fix the c*ckups they almost always introduce at release time these days due to their "agile" "testing").

      Some of the older issues were to do with MS trying to promote it as an application platform a la Lotus Notes back in the early 2000s. That was always a pile of garbage. Integration with voice services is still a pile of poo, but I've manged to avoid it.

      These days, if you're running it on adequately-resourced servers behind a real MTA to provide SMTP gateway services, you've got your HA with DAGs (database availability groups) set up nicely across at least 2 well-connected sites, and balanced databases that aren't too huge, then I really think of it as "set and forget" other than patching. As long as you set up your topology properly in the first place, and that's significantly easier post-Exchange 2010. 2016 is a doddle. It helps if you've come to grips with Powershell (since I started off with Perl back in ye olde days, Powershell is easy).

      There's certainly nothing on the market that's as good with scheduling, shared resources, and easy HA configuration. Outlook is shite, still, but that's nothing to do with the back end.

  10. Martin Summers Silver badge

    As I'm starting to realise, the largest horror story in IT, especially after the latest update, is Windows 10. There job done, Happy Halloween.

  11. timhowarduk

    Reboot anyone?

    Reminds me of turning up to a new site, arriving ahead of my boss, and having to spend considerable energy explaining who I was and why I was there to be allowed into the building, and then a good deal more energy to get permission to be led to the server room. We had been called in after the regular sysadmin had reportedly done a complete reinstall of everything, and then promptly gone on holiday for a fortnight, and the staff were saying almost nothing worked, specifically mentioning printing, internet access, Sage Line 50, their database etc. The parting shot was "OK, LOOK at the server room, but don't do anything that will take systems down without talking to us". So I set about analyzing what was there, reflecting that there was apparently little left running in terms of systems to take down even if I wanted to...

    The first odd thing was that server 1 was already logged in as Administrator. I had a good look around it, noting it seemed to be file and print, DHCP Server, DC, DNS and Microsft ISA Server. There was a netmask of 255.0.0.0 for the entire network (which was only about 40 workstations), a giant DHCP pool size and signs someone had been in a great hurry. The server didn't have a static IP address, in fact it was all really odd and nothing was configured 'normally'. I'd arrived at the conclusion that someone had basically just followed the install wizard and rapidly set up some shares. There were two other servers there and eventually I wondered what they were doing. The servers were all set up on a Belkin KVM switch. I was familiar with these switches and knew you change which server you are controlling by pressing scroll-lock twice then the server number you wanted.

    So I pressed scroll lock twice, and to my horror the server 1 promptly appeared to shut down in front of me. I looked down and the keyboard someone had installed was Swedish (I think) and it had a "Sleep" key in the place where scroll lock normally was.

    Within 10 seconds I had the guy running in asking why "everything is offline", and the chance of making a good first impression was lost forever. But after a very long day and a lot of configuration we left with them happy enough. I later learned the sysadmin was the son of one of the directors, and a lot of things suddenly made sense, like how he was still in a job after hosing everything then going on holiday....!

    1. iainr

      Re: Reboot anyone?

      Yes, I hated those belkin KVMs, we had a cluster of 40 odd linux desktops desktops with 5 kvms daisy chained together. THe choice of scroll lock as a hotkey for linux was bad, and not being able to change it to something else was worse. Pressing scroll lock whilst the kernel starts to boot pauses the whole process. so unwary colleagues would reboot a hung node, wait until is was starting to boot and then switch to work on another node, leaving the first one paused mid boot.

  12. DougS Silver badge

    Why would you have a maintenance window at 6:30am?

    I can't see why you wouldn't want it at 6:30pm since some people might be working late, but what's wrong with say 10:30pm or even 1am? That way if things go wrong (and inevitably they will go wrong at some point) you have plenty of time to resolve it before work starts. At 6:30am you have no time at all before they start complaining - as evidenced by this story.

    1. doublelayer

      Re: Why would you have a maintenance window at 6:30am?

      Good logic. My guess is that they wanted the sysadmin there during the work day, but didn't want to hire another one. They can't reasonably ask the person to be there at night and during the day, so they just schedule it for early in the day with the assumption that "We gave you time to run the maintenance, when we wouldn't complain. So if it isn't working at 7:00 when that ends, we feel we have all the excuse we need to yell at you".

      1. DougS Silver badge

        Re: Why would you have a maintenance window at 6:30am?

        They can't reasonably expect him to work from home at 10:30pm to take care of maintenance, but can expect him to show up in the office at 6:30am (and probably still want him to stay until close of business)

        I know which job I'd say "hell no" to if I had a choice between the two!

        1. doublelayer

          Re: Why would you have a maintenance window at 6:30am?

          I'd also rather the job where I can do some maintenance from home late at night. However, that option is dangerous. If something breaks that requires me to be on site, then I probably have to go there. At night, when access isn't always figured out, and transport systems might not run as often if at all. Worse, let's say that I say "Sure, I'll work at night for a bit running standard maintenance and also be there in the day. It still adds up to the same amount." Now, they have an expectation that I'll be there during the day. If the maintenance is usually thirty minutes or so, that's fine. When it becomes several hours for a major update or something similar, requiring you to stay up most of the night to work on it, you have less leeway to say that you won't be in. You have a perfectly good excuse, but that often doesn't convince the people who pay you.

          1. DougS Silver badge

            Re: Why would you have a maintenance window at 6:30am?

            If you break something at night then you'll have to show up at 6:30am to get it going, so it really isn't any worse than coming in at 6:30am for the maintenance in the first place.

            95% of the time you won't run into such problems and you can come in at a sane hour after having taken care of the maintenance the evening before from the comfort of your home.

    2. Ken Moorhouse Silver badge

      Re: but what's wrong with say 10:30pm or even 1am?

      I used to do a lot of work for a firm of solicitors in central London. I can't remember what the reason was now but I didn't want to do something that was going to disrupt everyone. So it was agreed I would go in, in the evening, do what I had to, and leave, and phoning for a minicab if needs be.

      The firm was on the 2nd floor of what was effectively a large house with another firm of solicitors on the 1st floor and a bank on the ground floor. I went in, as agreed, did what I had to, rang for a cab, shut the 2nd floor office door, put the keys through the letterbox and made my way down to the ground to wait for the cab.

      What I hadn't bargained for was someone working late on the 1st floor who left the building before me. He/she had dead-locked the front door to the building, so when I tried to get out, I couldn't. If only I hadn't been instructed to put the keys through the upstairs letterbox. The battery on my mobile had died by this time and I now had the minicab driver outside. I shouted through the letterbox to him, "I can't get out". He looked up and down, round about - where's that voice coming from? "Over here, I'm locked in." "Oh so you don't want a cab then." "Yes I do, but I'm locked in." Well miffed at being messed around, he went.

      What would you do at this point? Think before you answer, and then read on.

      Lots of things running through my head, my wife knew I was going to be late, but not as late as this could end up. Suppose the police think I'm a burglar, suppose the building caught fire, etc. Burglar? Fire? I know, I'll set off the alarm. The alarm company are sure to have details of the key-holder. So I did and it was your traditional very loud continuous bell.

      The police pitched up. They see me looking through the letterbox. "Can you open the door please sir?" "I can't, I'm locked in." "Are these your premises sir?" Etc. etc. They believed my story after a few runs through it, including the minicab driver. "No we don't have keyholder details, sir. Nothing we can do, apart from informing your wife."

      So I slept on the hallway floor with the alarm bell ringing. At about 7.30am the cleaners came in and nearly tripped over me.

      So, take my advice, 6.30am has some advantages...

  13. Hazmoid

    early morning shenanigans

    One of the joys of working for a Stockbroker in Perth Western Australia (The Australian stockmarket works on Eastern states time so is normally 2 hours ahead of local time) is that daylight savings in the Eastern states(ES) means that you have to start 3 hours earlier than normal. So for me that meant I had to find some way to get to work by 5 in the morning. (to ensure support for the early birds in the ES.) My solution was to ride a bike the 12 km into work, shower, change and have breakfast at my desk) . This worked well for a number of years (so well that I wore out 3 bikes in the years I was commuting).

    One morning whilst coming up the shuttle lift to the office from the basement, the lift jammed. No matter what I did I could not get it to move. So I let security know through the intercom, and decided to lie down in the lift and get some sleep, because they told me that the lift service company would not be there until 8. Great idea except that every 20 minutes they would page me "to see if I was OK" :( so I could have got 3 hours of sound sleep but instead got about 10 minutes at a time. When I got out of the lift I went to security and said thanks, but you could have let me sleep :)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019