back to article Sysadmins: Your best tale of woe wins a PRIZE

I'd like to share with you the plight of a good friend of mine. He's a systems administrator for a mid-sized American accounting firm. His story isn't particularly remarkable, but all the more important because of it. Bob has a bachelor's in computer science, and a lot more experience than I'd normally expect someone clutching …

COMMENTS

This topic is closed for new posts.
  1. Callam McMillan
    Thumb Up

    This sounds like a way of soliciting new stories for the Bastard Operator from Hell series!

    1. Captain Scarlet
      Windows

      Well not strictly true

      Like many of us I take the lessons of the BOFH and the PFY to basically be a git.

  2. Anonymous Coward
    Anonymous Coward

    *rubs hands*

    this should be good, El Reg readers always seem to have a never-ending stock of stories that amuse, entertain and horrify in equal measure

  3. The Axe
    Happy

    RBS?

    Any candidates from RBS? :-)

    1. SJRulez

      Re: RBS?

      This is their new recruiting drive, they are waiting to head hunt someone with a story about CM 7

      1. Ross K Silver badge

        Re: RBS?

        CA-7 even? Or were the BOFHs playing Championship Manager when they should have been working?

        1. SJRulez

          Re: RBS?

          Lol,

          They were probably to busy hacking it and pirating it for later.

          I must of been playing on something or slacking to not spot my mistake there....

  4. moonface

    One off the top of my head....

    A foreign system that I managed was down for a lot longer, than it should have been, just because I couldn't find the key in the remote session to produce the "£" symbol in my password. Always use "$" instead these days.

    1. Anonymous Coward
      Anonymous Coward

      Re: One off the top of my head....

      This did make me laugh, cheers!

      I occasionally have a similar problem when having to bounce through too many machines to get at the one I want (all Windows, combination of RDP and Dameware), working out which one has caps lock or sticky keys switched on can be a pain!

      1. Wensleydale Cheese

        Re: One off the top of my head....

        "I occasionally have a similar problem when having to bounce through too many machines to get at the one I want (all Windows, combination of RDP and Dameware)"...

        I've got a mixed language combination of RDP bounces which won't let me get at things like ~, [, ], \.

        And folks wonder why my command prompt has these weird characters in it :-)

    2. Wensleydale Cheese

      Re: One off the top of my head....

      @moonface

      There are times that I hate those who mandate "special characters" in passwords.

      I'll hazard not one of them has had to use the wide variety of foreign language keyboards that I do.

      And don't even mention "dead key" keyboards to me...

      And that's assuming that the login prompt is configured for the same keyboard layout as the user session.

      Mine's the bag with the US keyboard in it.

      1. phuzz Silver badge
        Facepalm

        Re: One off the top of my head....

        UK Apple keyboards don't seem to have a key for #, which makes logging in difficult if you have one in your password.

        And speaking of non English keyboards, Turkish keyboards have a selection of letters which look like, but aren't, English characters, which also makes typing in passwords difficult, should one be trying to check ones gmail from a hotel.

        1. teapot9999
          WTF?

          Re: One off the top of my head....

          It is Alt-3 which gives #

        2. KroSha

          Re: One off the top of my head....

          Option-3 gives #. Keyboard viewer is your friend!

          1. Wensleydale Cheese

            Re: "Keyboard Viewer is your friend"

            Yes indeed, but sometimes you cannot get the right characters at a Terminal prompt.

            The answer here is to add US Keyboard layout to your options in International. You can set up a shortcut such as Cmd-space to toggle input languages..

        3. Suricou Raven

          Re: One off the top of my head....

          You can get it through a key-combination (right-alt 3, I think? I can't remember without actually being sat at the keyboard), but this is rather non-obvious. No user would guess that, I had to look it up. It also doesn't work when I'm running windows and RDPed into another windows server - whenever I want to join a channel on IRC, I have to copy-paste a hash from somewhere.

    3. Reue

      Re: One off the top of my head....

      Sounds very familiar to the time I had remoted onto a virtual session hosted in India. Had the user on the phone and was desperatly trying to find the local keyboard setting for ". I was unable to and then spent a good 10 minutes trying to convince the user that all I wanted was for them to type " into a command prompt. In the end I email the " to them and copy/pasted it in.

    4. Psyx

      Re: One off the top of my head....

      Was your password for dial-in on-call fixes "£4nothingchicks4-3"?

      1. moonface
        Trollface

        Re: One off the top of my head....

        No, it was "Sh!tSyst£mFu(K£d"

        now it's "$h!t$ystemFu(Ked"

  5. Nigel 11

    Muppet engineer

    We once purchased a server with Windows Server pre-installed. By the time it was delivered plans had changed and it was reformatted to run Linux. Three months later it broke down. Looked like a simple failed PSU to me, but it was still on warranty, so we called for an engineer.

    Some hours later he told us it was OK again and left at a trot. We were surprised that he hadn't left it powered up, and dispatched someone to the machine room to boot it. But it was booted ...

    and once again running Windows! The muppet thought GRUB was a hardware error, so he reformatted the disk array and reinstalled Windows. Thank heaven for backups, and that it wasn't desperately mission-critical.

    You don't have to outsource to India to get muppets.

    1. Fatman

      Re: Muppet engineer..thought GRUB was a hardware error,

      Would have been shot for reformatting ANY of our servers with WindblowZE.

      We don't have these nice Tux stickers and posters around here for our amusement. The door to the server room doesn't say "Penguin Lounge" for nothing!

  6. Bodestone

    Slight issue

    If you go there and spend ages typing something in then you will lose all the text unless an already logged in member of their community. To register they seem to want a huge amount of your details before you can continue.

    1. Hieronymus Howerd

      Re: Slight issue

      Probably less of an issue than you might think, since most readers seem to have completely missed the point and are busy typing their stories into the Reg's comment form instead.

  7. Nigel 11

    Mouse problem

    I was called in to fix a workstation in an old Victorian basement bit of the site. Quad 140W Opteron thingy. Expected to have a certain amount of fun sourcing a beefy enough power supply. But the lights were on and fans were whirring.

    When I took the cover off, a very strange burned organic smell assulted my nostrils. A few seconds later I found a dead mouse with its head wedged between the fan blades and one of the heatsinks. I hope the poor wee thing's neck was broken in an instant. I fear otherwise.

    After the mouse was removed and the CPU allowed to cool down, it rebooted without a hitch. The mystyery remains, how did a mouse get into the case? There was no hole anywhere near big enough anywhere in the metalwork that I could see.

    1. David Webb

      Re: Mouse problem

      Mice can squeeze though gaps as small as 5mm so looking for a mouse sized hole means you're looking for a tiny little hole.

    2. Evil Auditor Silver badge

      Re: Mouse problem

      It slipped in as a mouse kid and fed on whatever came crawling along... Poor thingy. Presumably it committed suicide to end its meaningless life - I mean, who in his/her/its right mind would want to spend the life poking in computers?

    3. S4qFBxkFFg

      Re: Mouse problem

      The rule when dealing with mice is that anything you can stick a pencil through isn't mouse-proof.

      I never found out if that referred to the diameter of the hole, or whether a sharpened pencil could be shoved through.

      Probably either.

    4. tastytouring

      Re: Mouse problem

      What a great story. I work for SolarWinds and would love you to enter our contest, but unfortunately, we are not yet able to open our contests up internationally. We're trying to work that out with legal and hopefully will be able to offer our contests to all soon. For those of you who do live in the U.S. and would like to enter, here is that link: http://thwack.solarwinds.com/community/solarwinds-community/sysadmin-day-contest.

      1. Andy Fletcher

        Re: Mouse problem

        Dunno who downthumbed you. I did pop by and submit a story, and got some rather nice friendly PM's regarding it - seems like a nice community over there. Not being able to win the competition is kind of a bonus anyway - the price is something Apple made called an "iPad" or something. Pretty sure I wouldn't know how to use it.

    5. This post has been deleted by its author

      1. I ain't Spartacus Gold badge
        Happy

        Re: Mouse problem

        "If you get a pencil through a hole a mouse can !"

        How does the mouse hold a pencil?

  8. Supertommo

    I reckon the boys at RBS will have this competition in the bag

    1. Anomynous Coward

      "I reckon the boys at RBS will have this competition in the bag"

      They did but unfortunately they left the bag under a leaky sewage pipe and then threw it out when they noticed it smelled funny.

    2. John70
      Joke

      Didn't they out-source the bag to India?

      1. Aaron Em

        Not the bag

        Just the guy who was supposed to be carrying it. Turns out they didn't bother to check if he'd ever seen a shoulder strap before...

  9. Anonymous Coward
    Anonymous Coward

    Too many to choose from...

    Starting with the MD who thought he was still up to speed technically speaking and factory defaulted all of our coms gear to the helldesk tool who decided the best way to free up some space on a development server was to delete the contents of the system32 folder!

    1. Andy Brown
      FAIL

      Re: Too many to choose from...

      Years back, we had a 'Unix consultant' who decided that he'd empty the user trash can on the server - the directory called /usr/bin - then wondered why the server stopped working.

    2. Psyx

      Re: Too many to choose from...

      One of my ex's did that to her laptop.

      Luckily, she was hot and I forgave her.

      I may also have hammed up the difficulty of fixing it a bit, too.

  10. chuBb.
    Facepalm

    I'm allegedly a developer...

    I'm allegedly a developer... (admittedly one with a CCNA and an interest in system administration, Linux and other general geeky stuff)

    Except i am charged with Running exchange, the cloud platform, the virtulisation stack, all of the servers and desktops, the helpdesks (yes 2, as a so say manager was beguiled by a nice mug and stickers, didn't even hold out for a pen drive!) , documentation, router kicker (crappy dlink wireless router from PC world powers the office, and dont get me started on the ADSL line!!!!), 1st, 2nd and 3rd line support, explaining basic concepts to the "technical" director (yesterday talking to a potential client his response was "we send some HTTP", when asked a question about SSL) oh and taking the flack for when dev deadlines are missed for all of the above, well here are some WTF's from this job:

    1) We upgraded to SBS 2011 becasue the MD bought a mac book and couldn't connect to exchange, said mac now lives on a shelf gathering dust, Migration from SBS 2003 disnt work and we spent 3 weeks with a partial service.

    2) Exchange is hosted locally on an ADSL line which has a top down speed of 5Mbs, on a router which crashes 10 to 20 times a day, average mailbox size is 12Gb and the suggestion of archiving went down like a lead balloon and was forced to revery back to all you can eat inboxes.

    3) one of the helpdesk servers was a windows XP box bought in 2002 which died last week, that was

    fun to fix (now on a proper server)!

    4) we have a mixture of 1 to 10 Gig ethernet to every desk yet the MD insists on people using wireless to said crappy router because wires make the place look untidy...

    5) It took 6 months of nagging to take the UPS out of the empty comms rack and install it in the server rack

    6) The working helpdesk has been replaced by a cgi based web app which crashes when more than 5 people try and access it it also uses a VBScript based file upload and a scheduled task inplace of a service and requires administrative permissions to save to the file server, suffice to say new clients dont have the option to add attachments to support calls (we publicised the "upgrade" to our clients most are not impressed!)

    7) A mezzanine floor was removed, but a comms cupboard was left 20' up a wall, i needed to replace a switch in there, new switch wouldn't slide in shoved it harder heard a crunch then a loud buzz, there was a bloody wasps nest in the cupboard!!!! queue rapid decent down the ladder and no network to half the building until the exterminators had been!

    I could go on but the phone is ringing and someone needs their hand holding...

    I Love deadlines its the whoosing sound they make when they fly past....

    1. Anonymous Coward
      Anonymous Coward

      Re: I'm allegedly a developer...

      Are you me from the future?

      our sbs2003 -2011 migration hasn't happened yet, but the cupboard and wasps nest thing sounds familiar.

      Just got back from holiday and found:

      one of the HD in our RAID array has failed and the guy called in just took the faulty disk out without replacing it.

      The cooling to our server room was switched off because it was too noisy

      the "internet has been slow" so someone has been hard rebooting the servers every day by unplugging them.

      My work area has been "tidied" by putting all of my components and tools onto the warehouse racking at ceiling height.

    2. Aaron Em

      Oh God

      You tried to migrate straight from SBS2003 to SBS2011? You poor, poor bastard. I spent two weeks once on a 2003-2008 migration -- eight of those ten days with MS Product Support on the phone eight hours a day, for which service I ended up paying not one thin dime, and if you know how MS PS support works then that tells you all you need to know. (If you don't: Unless you have a support contract, which costs $stupid, there's a per-call incident fee of merely $silly, which is waived if the cause of the problem turns out to be Microsoft's fault. They waived it.) And I've got a 2008-2011 migration coming up this weekend which I am sure will also be a nightmare. Straight from 2003 to 2011? I'd rather have a root canal.

      I could go on too, but there's a server to bring back up and a couple things to set live before the day's work proper starts off -- back to the coal face...

      1. chuBb.
        Facepalm

        Re: Oh God

        It wasn't fun i can tell you that, and i have had a root canal before, was definitely more enjoyable, lots of painkillers straight on the nerve :D..., 2008 to 2011 is meant to be less painful but still a royal pain...

        Best bit is the one unfixable problem is that the one person in the company who needs access to every ones calendar is the one person who's account wont allow the permissions to be set oh the joy, tried explaining to "management" that exchange nevermind system administration is a full time job in its own right oh well u can but try...

  11. Nigel 11

    Simple things

    A PC wasn't working, looked like PSU fault. Took it back to my office to check and repair. Plugged in. Pressed power button. Very loud bang. Small amount of smoke. Lights went out. Oops.

    Outside office, also no lights. OOPS.

    100 yards down corridor, still no lights, lots of people peering out of their offices asking WTF. OOOOOPS.

    Somehow, the PSU fault hadn't taken out the 13A fuse in the plug (yes, it should have been a 5A fuse, but it did have the mandatory safety-tested passed-by sticker). Not the 30A breaker on the circuit either. Nor the 100A breaker above that. No, when the electricians finally located the fault, it was a 180A fuse in a box high up on a wall that had probably last been looked at when it was installed in the 1920s.

    It took a lot of phoning to source such a monster fuse and they paid for a motorcycle messenger to bring it down to London from Leeds. As far as I know it's still there. Too much effort to schedule replacing it with something modern before the next time (in the 2060s? )

    The PC was fine after a PSU transplant.

    You don't need complicated things to make a FUBAR. The simple things are also out to get you.

    I could also tell you about the exploding substation and the need do do a tap-dance to avoid getting burned by globules of molten copper pouring out under the door, but that's got no IT angle at all.

    1. Annihilator
      Coat

      Re: Simple things

      "it was a 180A fuse in a box high up on a wall"

      That's a 6 inch nail isn't it? Or would be in the BOFH stories...

      "It took a lot of phoning to source such a monster fuse and they paid for a motorcycle messenger to bring it down to London from Leeds"

      Take a note of your nearest B&Q for the next such outage :-)

      1. Callam McMillan

        Re: Simple things

        Having worked in B&Q many moons ago, they wouldn't know a 180 amp fuse if you rammed it where the sun don't shine. As for them stocking one, you have more chance of the devil calling a central heating repair service!

        1. Fred Flintstone Gold badge

          Re: Simple things

          "As for them stocking one, you have more chance of the devil calling a central heating repair service!"

          You've given the answer yourself: they DO stock copper pipe.. :)

        2. Annihilator
          Facepalm

          @Callam McMillan

          Hmm... B&Q do still stock 6" nails though, right?...

      2. Anonymous Coward
        Anonymous Coward

        Re: That's a 6 inch nail isn't it?

        Consumer unit fuses are bigger than the little cartridges in 13A plugs, and big industrial ones have lugs on the ends - a bit of copper pipe of the right size should do the job. :-)

        1. Cpt Blue Bear

          Re: That's a 6 inch nail isn't it?

          When I was a callow youth I worked as junior shit-shoveler at a medium sized engineering firm. One afternoon there was an almighty bang and the whole site went dark except for the emergency lighting. It seems both the main HT connector fuses had blown when one of the company sparkies grounded something he shouldn't have (and was very lucky to be alive). Anyway, need new HT link. Head sparky starts making calls but none to be found this side of the country. Nearest replacement is 1000km and two days away. No mobiles back then, so all this had to be done in full public view in the front office, with a growing and restless audience. Management by now is soiling it's collective self about downtime and missed deadlines.

          Cue junior sparky and a "hang on lads, I've got an idea" moment: one piece of one inch copper pipe, two blows with a hammer to flatten the ends, punch two holes to fit over the terminals and voila! We are back in business...

  12. Bodestone

    Bit of a WTF situation just now

    10 months down the track we are still trying to give us money to replace our primary (read only) flile server. If it powers down it will not come back until the following routine is followed:

    Unplug cable, pop lid, remove BIOS battery, bridge reset, insert power cord, remove cord, unbridge, replace battery, close lid, power up, reconfigure the BIOS again.

    Combined with the fact that we are still trying to also get money to replace the faulty tape drive. It goes for a couple of days then fails with unknown HW error. Only solution is to power down drive and server, power up drive, wait, power up server. Sometimes it works first time, sometimes takes a few tries. Our last successful backup was 9 days ago.

  13. Lee Dowling Silver badge

    I once was caught by a network outage whose ultimate cause was:

    An X509 certificate, in a Java package, had expired. The Java package was part of a UPS monitoring software (and nothing you'd ever normally play with). The UPS monitoring software went absolutely crazy and just hung the entire machine (as in, your mouse couldn't move across the screen in under a minute). The machines affected were pretty important and all of them had the problem. Because they still responded to some queries, pings, etc. (just about) nothing bothered to failover anyway but even the failovers had failed. And you couldn't log into the damn things or use any of them directly.

    And obviously, the entire network slowed to an absolute crawl affecting the entire place it was in. Sure, we could force failover to a clean machine but those servers had apps running and we needed those apps (data - pish, we had that, but if you can't submit the payroll without that heavily-locked-down smartcard tied only to those machines you were stuffed).

    Sounds like nothing but go through your standard procedures - everything responds, everything responds to ping, just some things are very slow. Can't narrow down to one server. Try to log in to them remotely and it timeouts. Go find the physical machines. Try to log into them. LITERALLY could not get the mouse to move. Suspect hung servers. Wait suitable length of time. Force a reboot. Server boots up at normal speed, does disk check, nothing wrong, gets to login screen, hangs like hell.

    Okay. Weird. Safe mode. Same thing. Okay. Disconnect from network. Same thing. Restore from latest backup (which takes HOURS). Same thing. Okay, restore from Known Good backup (more hours!). Same thing. Confirm the machine is fully operational with install from clean Windows CD (but too much software on there to just replace every server in the space of a day or so from nothing). Perfectly working, no problems at all. Okay, restore from ancient backup taken just after the Stone Age and which DEFINITELY, DEFINITELY, DEFINITELY works. Same thing. WTH? Try other machines. Same thing.

    Restore a backup to an identical machine acting as tepid-spare. Same thing. Bugger. Complete reinstall looking like only way to get it back and working. Clean Windows install on clean, isolated machine. Start reinstalling software from disks (known-good versions of everything that worked for YEARS). The reboot AFTER the UPS software is installed starts displaying the same symptoms. Damn! (But at least we can narrow down). After a couple of installation retries (and rebooting after literally every step), narrow it down to the UPS software installation. Update available? No. Reported problems? Yes, with same symptoms. No solutions.

    Fall back onto the last resort of IT: Phone random IT people and see if they've had the same thing. Couple of them experiencing it, one of them in the advanced stages of diagnosis. After much jiggery-pokery and literally just turning the software on under heavy debugging/monitoring (and LOTS of rebooting) and trying to identify cause, narrow it down to accessing a certain Java package and then, inside that, find an expired certificate. How do we know it's the cert? Can you fool it by putting the clocks back? Actually, yes. Before the expiry time, it works, after it doesn't. Can you replace the cert? Yes, but the software refuses to work even if it doesn't hang. Can you fake it? No. Can you remove it? No, the software won't work. Can you run your processes with an inaccurate clock? No. Can you run your server without UPS monitoring? Not really.

    So it takes this long to get to knowing what's happened and what you can do about it. The affected servers can have their filesystem accessed by something else (Linux boot disk) and have the UPS software disabled, because there was literally no other way to get the software turned off once Windows had started (and actually, I moved the UPS monitoring to a Linux machine that then would issue shutdown notices to the Windows servers instead!).

    Eventually, after about a week, the UPS manufacturer issue an update and pretend it never happened (and NOBODY installed that update without first installing clean or working out what the problem was). But you try diagnosing that bugger while having a whole company breathing down your neck and silence in the server rooms.

    What saved the day? Linux, contacts, careful investigation, no blind reinstalling, and not relying on the UPS manufacturer to actually DO anything about it.

    The last thing you expect when your whole network goes down (and only the servers, not the clients) is some UPS software that's been working FOREVER, on known-good versions, on known-good servers to suddenly stop working even on restoration of older, working backups (whose restoration had ALWAYS worked before that point when testing backups) and then hang the machine to the extent that you literally could not do ANYTHING on it at all. I was just about ready to kill Microsoft until we saw that no updates had taken place in the weeks leading up to it and the servers had been rebooted and backed up and test-restored since then.

    1. phuzz Silver badge
      Megaphone

      As a warning to the rest of us, which UPS manufacturer are we talking about?

      1. Fatman
        FAIL

        RE: which UPS manufacturer

        Yes, enquiring minds want to know!!!

        Especially considering that we have 6 UPS systems at WROK PALCE, one for each of 5 floors, and one for the server room.

        Most appropriate icon I can think of.

  14. Anonymous Coward
    Anonymous Coward

    Not a sys admin but...

    I've done something which might just about qualify - it's more hardware related than admin related to be honest. Actually, it's stupidity related. And as I'm not a sysadmin, I'll remain anon and discount myself from the prizelist.

    I was, err, playing kinda "bit of stick and a few rolls of paper cricket" in the server room with a colleague (yeah I know, I know...)

    My colleague scored at least a 4 as the "ball" sailed over my head and went behind a cabinet. In my hurry to ensure it didn't become a 6, I rushed round the back of the cabinet. And not just any cabinet, oh no. This was the cabinet where every terminal in the entire organisation had an RS232 patch lead going from the back of the racks into a patchboard on the wall.

    I tripped, and landed in amongst all the cables. Very few of them had covers, just three wires soldered onto the pins, and most of them were a pretty tight fit into both the router-thingy (I can't remember what it was called) and the patch-board. So it just ripped the wires straight from the pins. I must have destroyed about 80 or more cables.

    We had to sit soldering for at least 4 hours solid.

    To add to the fun, none of the cabling was labelled, so we had no clue which machine-connector went to which patch-connector, so we were plugging people into someone else's still-logged-on account right left and centre.

    You might imagine the first thing we did was panic. Wrong. The first thing we did was get rid of the bit of stick and the rolls of paper. Then we panicked.

    1. Anonymous Coward
      Anonymous Coward

      Re: Not a sys admin but...

      this is exactly why the helpdesk are not allowed in the server room, none of them can be trusted. Next time they ask why i shall forward them to this story...................

    2. Nigel 11
      Headmaster

      Re: Not a sys admin but...

      The greater FAIL was whoever connected and configured it in the first place. With the right jumper on the D25 (or D9) connectors, unplugging the connector would have been treated the same as a modem hangup, and that should have terminated the logged-in session if the software config had security in mind.

      1. Lee Dowling Silver badge

        Re: Not a sys admin but...

        Sod that - why do you have 80 or more unlabelled cables just plugging into an apparently vital device?

        Get the Dymo out, ffs!

        1. Anonymous Coward
          Anonymous Coward

          Re: Not a sys admin but...

          this is exactly why the helpdesk are not allowed in the server room, none of them can be trusted. Next time they ask why i shall forward them to this story.

          I'm not helpdesk, I'm a 26-and-counting-years-experienced dev. But, and far more importantly, I'm also a big kid, and I wouldn't have it any other way :)

          The greater FAIL was whoever connected and configured it in the first place. With the right jumper on the D25 (or D9) connectors, unplugging the connector would have been treated the same as a modem hangup, and that should have terminated the logged-in session if the software config had security in mind.

          This was pre-internet, so no IT threats from outside. As far as internal security goes, the payroll office got armed-robbed the morning of pay-day (twice, if memory serves), and the social club regularly got broken into and all the alcohol nicked. They plastic-injection moulded approximately 4 tons of a mixture of key-rings, cups, plates, and other trinkets for an open day, the entire lot of which got stolen from the factory floor the night before the open day. It was regularly seen for sale at market stalls for months afterwards. Security was something that was to be provided by big ugly burly men, the concept of information security would have sailed right over the MD's head like, well, a paper cricket ball. To be honest, the only threat from a keyboard would have been someome clocking you over the head with it to steal your lunch money.

          Sod that - why do you have 80 or more unlabelled cables just plugging into an apparently vital device?

          Get the Dymo out, ffs!

          No need. The entire factory and office block is now taking up valuable space in someone's landfill site.

          1. Anonymous Coward 15
            Devil

            Re: Not a sys admin but...

            Hmm. I bet you some of those thefts were internal.

  15. Jello

    During my first week as an ICT Techie at a school I encountered a problem with MS Office on one of the machines. Reading round a few of the knowledge base articles and support tickets the answer was to re-allocate the MS Office package to the workstation, basically un-installing and re-installing the whole of Office. Being as I'm such an efficient person I thought "Heck, why not re-allocate the package to the entire domain therefore fixing any other machines that may have the issue but no-one has noticed yet". Fantastic idea, especially 2 days later when people could finally use their computers again as it brought the whole network to a crawl.

    First week in a new job not knowing enough about what your doing is a bad combination.

    On the other hand, Office worked pretty well after that.

    ---

    I also recently had a colleague panic that one of the displays on a laptop had packed in, he went through the usual processed including contacting the manufacturer to check it's still in warranty before someone said "Have you tried the key combinations to increase the brightness" to which he replied "Of course I have, but strangely it's just started working fine" - hmm...

    Cheers

    1. Lee Dowling Silver badge

      I once crashed an entire school network (it was RM, so the classification of it being a computer, or a network, is dubious at best) by allocating an MSI package (I say that, but the RM ones are pretty butchered - embrace and extend again!) that happened to have a space in the filename. It was only a tiny MSI of something minor but I'd called it something like "packagename update" so I could distinguish it from the base package. I'd tested it on some machines and it was fine, all I did was rename it and deploy it for installation that night (you can't really deploy stuff during the day in a school).

      Every single client crashed at reboot (which obviously didn't happen until the next morning) and just hung there before the login screen because their crappy software didn't know how to handle a space (despite the fact that their network management tool on the server did to allocate you to allocate it in the first place). And there was little you could do about it except login and clean-install or (if you know the magic words) wipe out their package databases after removing it from the server.

      I made the RM guys fix it in the end, because I certainly wasn't going to be held responsible for their crappy software. And even though they told me the problem was "fixed" in the next hotfix, I never deployed another package with a space in the filename, just in case.

      1. Anonymous Coward
        Anonymous Coward

        It's a fairly safe assumption that spaces in a path will fuck something up at some stage, even with proper operating systems.

  16. Anonymous Coward
    Alert

    "The printer isn't working!"

    "Did you switch it on?"

    "Oh."

    1. Anonymous Coward
      Anonymous Coward

      Re: "The printer isn't working!"

      I once got woken up by a phone call at 5am - someone had gone into work early and the building was "dangerous" because it was too dark. "Have you tried turning on the lights?" "Oh."

  17. Zombieman
    Alert

    The day the M.D. acted as a doorman...

    A long time ago (this was before XP) I was running maybe five minutes late one day (co-incidences are so unfair), and as I approached the building, from inside the building the Managing Director opened the door eager and worryingly pleased to let me in...

    Expecting something about being late I approached. nervously, saying good morning, and he was strangely not sounding bothered about me being late. What he WAS bothered about was that no-one in the office was able to login to their desktops as he calmly explained the situation saying he hoped I would be able to help.

    The other local techie could login to our (then single) server, not his machine, and had found rebooting the server didn't help... I poked around for five minutes and saw a DHCP configuration change from about 2/3 days before, reverted it, and as if by magic everyone could login!

    An offsite techie had changed something which SHOULDN'T have caused an issue, but somehow did. He was confused because of the time delay, thinking any changes should've been immediate if there had been a problem. Every DHCP request was failing (never worked out why), so no machines were getting IPs for the network (having lost the rights to the IPs from the Monday), so couldn't communicate with the server to validate the login details.

    Took me WEEKS before I stopped nervously looking for the M.D. ready and waiting to open the door for me each morning.

    1. Lee Dowling Silver badge

      Re: The day the M.D. acted as a doorman...

      I've had techies come in and start up their own DHCP server (presumably accidentally by sheer incompetence) by leaving things like the DHCP options turned on on, say, an ADSL router or similar, and then leave it running. Things can work fine for a while until either a) machines leases expire, b) the server detects the other DHCP and shut down its own service (thus making you reliant on that little ADSL router to do EVERYTHING or sometimes c) machines "closer" (network-wise) to the DHCP server get one response, the rest a proper response from the server and sometimes the two overlapping and issuing leases for each other's IP's to each other's clients.

      Then I would get a phone call (I was doing the rounds for local schools and spent a day per week in each for many years) that nothing was working and had to work out what was happening when no documentation or even help from the original installers existed for the networks they were running (is the ADSL *supposed* to be giving out addresses? In what range? What ranges are the firewall etc. configured to use? etc.). But, of course, "nobody's touched the network in 28 days" and no problems are noticed until the DHCP leases die or the ADSL goes off or whatever.

      The nicer switches let you block DHCP from all but one port, which is a lovely option until you're not told that and try to replace a server that's falling over. And the places that have those nicer switches tend not to have the problem in the first place (though it is funny when someone replaces that switch and you find that there's dozens of DHCP servers all other the place in WAP's etc. and nobody was previously affected because of that safety barrier that they've just removed).

  18. Mayhem

    Lights, camera, silence

    First week running the bank branch on my own. Walked in as usual one morning to change the tapes, turned the lights on, and had the gut churning chunk sound of every machine in the server room losing power at once.

    Several hours of mainframe IPL later and we're finally back up with me on the phone to our electrical guys to work out how the hell the room lights were wired into our UPS power.

    Everyone assembled onsite the next day to prove that it wasn't possible. Took everything down just in case and brought them up in standby. Most satisfying junior admin moment ever when after 3-4 attempts by the sparky I walked up, hit all four switches at once and the room promptly crashed to dropped jaws from all inside.

    Turned out to be a bit of dodgy wiring in an unlabelled sealed over switchbox in that section of wall - it was an old power cutoff wired under the floor to the master cutoff switch on the far side of the room. Since the wall was just a cheap partition, the vibration of the door closing over time had jolted a wire loose, and turning the lights on at once was just enough vibration to make the wire jump free and take everything down from inside the UPS supply.

    Good lesson that even completely disconnected systems can be linked, if not in the fashion you would have expected.

    1. I ain't Spartacus Gold badge

      Sorry for the off-topic

      "Good lesson that even completely disconnected systems can be linked, if not in the fashion you would have expected."

      I remember reading in the 1980s about someone who'd bought a Ugo. Communist block car, manufactured in Yugoslavia. It was converted to right-hand drive for the UK. As it turns out, simply by sawing the old pedals off the left hand side, and covering them in carpet.

      Going down the motorway at 70. Passenger yawned, and stretched, and stretched, and BRAKES!!! Could have equal amounts of 'fun' with the accelerator at junctions...

      1. Anonymous Coward
        Anonymous Coward

        Re: I call bollocks

        They might have left the pedals in, but they wouldn't have been hooked up.

        1. Yossarian
          Stop

          Re: I call bollocks

          Wouldn't be too sure, I remember a Watchdog piece about a modern European car (I believe a Renault but not sure) where they had converted it to right hand drive by leaving the break linkage on the left hand side and bolting a bar behind the carpets to the right.

          The passenger could happily apply the brakes (the real brake pedal moved) by pushing hard on the carpet. I remember this only being a problem with the brake pedal, so helpful when the other half thinks braking is a silly idea when a tree will suffice.

        2. Anonymous Coward
          Anonymous Coward

          Re: I call bollocks

          Actually, being a total cheapskate I bought my (ex)wife a Yugo a few years ago. I discovered that you *could* brake and accelerate from the passenger seat. So, your call of bollocks on this occasion is wrong. Probably why she is an Ex :))

          1. Anonymous Coward 15

            Re: I call bollocks

            Good cars for driving instructors then?

  19. Captain Scarlet
    Meh

    The I Something Department

    Rescued a member of staff from a portakabin office after a door access system (Which was dumped on us after the person who brought it and someone asked how do we use this) was blaimed for keeping a door shut.

    After going to check and finding the physical lock was to blame we then had to help out the trapped member of staff through a window (and then raise lots of H&S forms). Still have no idea what this has to do with the IT (Sorry don't deal with technology anymore I S(omething) department) but makes some days interesting.

  20. the-it-slayer
    Devil

    Not really related an IT cockup...

    In my first IT job, I ended up being promoted to the network and sys admin above the trainee techs that I used to look after. There was only 3 of us in the office always bombarded by teaching staff and kids wanting their laptops fixed day-in-day-out.

    Anyway; one day, I called my boss about something that a member of staff had done to their laptop quite amazingly by the excuse they gave me. During this conversation where I was explaining things quite angryily, one of my junior techs and the stage technician were messing around with a can of compressed foam (that we used to clean up the old bits of kit and resend them out). Whilst throwing this can around, I will still on the phone and my boss agreed with my concerns. Suddenly, I heard this mini-explosion behind me. Only did the stage tech throw the compressed foam onto the corner of a metal file cabinet and I turn around whilst mid-conversation to find the room fully covered in foam (including my back!). I had to keep the conversation with my boss going which was luckily ending whilst the junior tech and stage tech saw me turn round to see what the hell went on. Let's just say that the evidence was still apparent on the posters we'd put up after we manically tried to clean up the office pronto before my boss walked in who only was based next door to us.

    We did tell him in the end, but that was the last time my junior tech ever got the near the compressed cleaning foam!

  21. Anonymous Coward
    Anonymous Coward

    Our VPN box...

    [user@server ~]$ date;date;date;date;date;

    Thu Oct 21 10:12:23 BST 2010

    Thu Oct 21 10:12:18 BST 2010

    Thu Oct 21 10:12:23 BST 2010

    Thu Oct 21 10:12:18 BST 2010

    Thu Oct 21 10:12:23 BST 2010

    Disabled anything remotely to do with ntp. Rebooted. Didn't make a difference.

    [user@server ~]$ ntpdate pool.ntp.org

    All back to normal. For five minutes. A day later it was a couple of minutes out again.

    Called the hosting company's support line. Three days of head scratching later, they say, "We'll swap the drives into another box." One day after that, they say, "It's all fixed". One more day and it's buggered again.

    VPN box became ftp box, ftp box became VPN box. Almost two years later:

    [user@server ~]$ date;date;date;date;date;

    Thu Jun 28 13:43:35 BST 2012

    Thu Jun 28 13:50:09 BST 2012

    Thu Jun 28 13:43:35 BST 2012

    Thu Jun 28 13:50:09 BST 2012

    Thu Jun 28 13:43:35 BST 2012

    1. Leigh Geary

      Re: Our VPN box...

      That smells like a badly configured Virtual Server mate. I've had VM's doing that in the past.

  22. KroSha
    Facepalm

    There's a big data centre in Wapping, with about 100 racks per floor. As it should be, this is access controlled to prevent the cleaning lady from wandering in with her polish. However, no one thought to tell the new security guard that you need to swipe your card to get out again. He just saw the big (red) button and thought that was the unlock button. He'd never seen an emergency power shutoff switch before.

    1000(ish) servers down in less than 2 seconds.

  23. Psyx

    Smoking boots and server outages

    I was once paged by a major client in order to deal with an 'outage' of nearly a hundred mid-to-high end UNIX servers at around 4am. A quick wander around revealed that every server had suffered a power outage that lasted nearly an hour, and nobody would tell me what the root cause was, and looked really shifty when queried about it. I started running file system checks and getting everything up and running again, and tried to find out what the story was.

    Eventually I discovered why everything had dropped and why the Ops Bridge crew were looking guilty and keeping quiet: A UPS engineer had been working on the system overnight. Due to the vendor being tight-wads and cutting contractor rates to the bone, said sparky did not have a mate with him and was working alone on the systems, and the vendor did not arrange a full outage because the systems were too critical to business. The guy fried himself and was killed and cooked over a period of time. If someone had been there with him, they could have killed the power in time and he'd have lived (he was still alive when the first medical guys arrived, but they wouldn't touch him as the power was still on). There was some delay in power-down due to concerns that HA systems would be affected. Which shows you the value placed on human life by some companies.

    So I fsck'ed a bunch of machines in the wake of this mess.

    The kicker: The department in question had a great safety record and all of the vendor's staff got free stuff while the record was maintained. In order not to jeopardise the clean health and safety sheet and cause awkward questions, the death officially went down as a suicide.

    Because obviously you don't work on a UPS alone unless you are suicidal...

    1. Anonymous Coward
      Anonymous Coward

      Re: Smoking boots and server outages

      and this is still a secret and no-one did/is doing time for this?

      You shouldn't be telling us, you should be telling the peelers. >:(

      1. Psyx

        Re: Smoking boots and server outages

        It wasn't in the UK.

        Life is cheap when it's outsourced and you're overseas.

  24. This post has been deleted by its author

    1. This post has been deleted by its author

  25. Psyx

    Another classic...

    Because I worked night-shift looking after a mainframe and a UNIX system, our glorious leader decided that our section would be ideal for offering overnight PC helldesk and tech support to every muppet on the airfield who had a PC. So my mid-shift naps were interrupted by a series of idiot-calls from the Air Force's finest on a routine basis.

    Picture a winter's eve. The runway is closed to to bad weather. The snow is coming down and settling heavily. It is now 4am.

    The phone rings.

    His monitor doesn't work. The light on the front isn't on.

    "Is it plugged in and switched on?"

    *suspicious lack of any pause to check this before replying*

    "Yes, of course."

    I offer to wander up to his office to take a look, and he now reveals that he is sat in a building over the other side of the runway. It's either a ten minute walk in the snow, or a nine-minute job of defrosting a vehicle and two minutes of driving over there.

    Great.

    "Are you really sure that it's plugged in and switched on"

    "YES!"

    "Ok then. I'll be over in fifteen minutes. I'm going to bring a screwdriver and a hammer. If it's broken, I will use the screwdriver to fix it. If it's not switched on at the wall, I will use the hammer to hit you. Now: Is... it... switched... on... and... plugged... in?"

    *actual sounds of rummaging*

    "Oh... it's working all of a sudden..."

    *hangs up*

  26. Psyx

    I used to hate handing out root passwords or any kind of admin rights to users, regardless of the number of letters from their boss telling me that the said user 'needed' it. I cannot count the number of times I've had to rebuild stuff that users screwed up.

    The worse example was when I gave someone rights to install software on his machine. The next day, I was called to his office. He'd tried to reinstall Windows for some reason that was never clear (I suspect he'd deleted the files labelled 'Windows' because they didn't appear to do anything useful). The problem was that he couldn't fit the floppies into the drive any more.

    The 'any more' should have been my warning.

    A visit to the office showed me that the machine was asking for the user to 'insert disk 4'. The user had successfully inserted disks one through three when requested. Concurrently. Without ejecting prior ones. They were all jammed in the drive.

    This was a guy who only 24 hours had received a written testimonial from his boss tersely telling me that he 'knew what he was doing with computers'. Clearly the guy DID know what he was doing on some kind of zen level of pure stupidity which I can never understand... because he'd actually got as far as disk 4 in the procedure successfully!

  27. jlf175

    A few

    Story 1: I work at a small regional bank in the US.

    As is often the case in a smaller IT shop like mine things seem to either be completely dead quiet or insanely busy, often with no rhyme or reason as to why since the problems that come in are often not related. A PC goes down in one office and not a few minutes later someone else has a problem in another part of the building. After several hours of people calling in with frantic requests for help - since their problem is obviously the most important thing - it can get to even the most seasoned of sysadmins resulting in a increasingly short temper.

    So near the end of a particularly rough day, I get a call from a teller who insists that they can not print out a transaction receipt for a customer to their little serial printer. I run through what I think is a pretty good checklist. Is the cable connected to the device? Is the print session to the AS400 open properly? Does it have power? I check the print device on the AS400 host and its online too. This of course all takes time to do but the employee is having none of that and is becoming increasingly agitated because of course if we don't fix it in 20 seconds then we're incompetent.

    "Fine I'll be out in a minute to replace it," thinking I'll just swap the stupid thing and worry about it in the morning.

    Arrive out there, and the teller and the teller's boss are fuming about this because they had to hand write the receipt, oh-heaven-forbid. I open the top of the little Epson receipt printer to swap out the paper roll and see the problem. "There's no paper on the roll." I tilt the printer so they can see, shut it, and walk out without saying another word.

    Story 2:

    "My keyboard won't work anymore" the person in the mail room tells me, "So I can't punch in to the time clock." There's few things that upset a user more than not being able to punch in immediately when they get to work. The idea of staying an extra minute or two at the end of the day to make up for it is often met with the shrieks and cries of the damned. Fortunately he called early in the morning so I'm in a relatively good mood and his pained cries for help are easily brushed aside.

    "Okay, probably just broke down, we have plenty of spares. Maybe you could punch in from someone else's desk and I'll be right down." Go to the storage closet and grab a spare, hoping the rest of the day will be this easy.

    Get there, spare keyboard in hand and disconnect his old one. As I go to throw it in the trash can next to his desk a stream of water comes out of the keys, resulting in a puddle of water on the carpet. "Um, did you spill something on this keyboard?"

    "Well, yeah, I spilled some water this morning. Does that break them?"

    "Well, computers run on electricity and these are electronics, so yeah."

    "oh.."

    Users..

    Story 3:

    Problem with the AS400 host system of some nature and users are in a panic. They can't process loans, can't open accounts, can't do anything. After the first few calls from users we're all getting a little bit nervous I send out an email to all users saying the AS400 is having a problem and we're working on it. The stress starts to increase when a few more issues crop up on other systems since those systems can't talk to the AS400 now either. So the stress factor starts to multiply.

    So we're making calls left and right to various vendors and support personnel for assistance. While trying to do these productive things people in another department start bombarding my email with messages about how the AS400 is down. I email them back tell them we're working on it. But the emails keep coming.

    After the problem subsides I go up to tell them personally and directly that everything's working now. They are all laughing and said they know they were just pulling a little joke on me.

    So the next morning before they came in I went up there with some tools and pulled all but one wheel off their rolling chairs. This resulted in security being involved in trying to figure out what happened and multiple emails going around asking for information. I let those emails circulate for a while and then told security what occurred and where to find the wheels. I left them behind the desks trash can of each user. I wasn't written up and that group didn't call the help desk for about two weeks after that.

  28. Rufus McDufus

    In my first job technically before I became a sysadmin, I worked at a top UK university. One day a female computer science professor came in holding a rather smelly and damp-looking Apple keyboard. 'It's broken' she said. 'How' I asked. 'Well, I threw up on it this morning and I figured the acid might damage it so I hastily tried to clean it off' she replied. Okay... 'What did you use to clean it with?' I asked thinking she'd dunked it in water or something marginally less stupid. 'Cooking oil', she replied. 'Hmmm, maybe we need to get you a new one then...'.

    Don't you love academics? I had a few people wash laptop keyboards under a tap as well, with the laptop still attached of course.

    1. Aaron Em

      Don't you love academics?

      No.

  29. Anonymous Coward
    Anonymous Coward

    No RAID

    This one almost sounds like an urban legend. I wouldn’t believe it myself, had I not run into the handy work of the ‘IT Guy’ in question myself. It happened before I started working where I am now, so isn’t my own story.

    When taking over a large amount of outsourced IT work from another company, it was discovered that some of the servers our company were now looking after happened to be running on single disc drives. When the previous ‘IT guy’ was questioned about this lack of redundancy he absolutely insisted he’d done his homework and fitted mirrored discs. He even went to the trouble of opening up one of the systems in question and pointing to the polished (as opposed to brushed or painted) finish of one of these ‘mirrored’ drives......

    1. Aaron Em

      "It happened before I started working where I am now"

      is another way of saying that you swallowed somebody's no-shit story all the way up to the sinker. Hey, we were all young once.

  30. nathharper

    competition is only for US citizens

    which is a shame, as I can sympathise with your mate Bob

  31. JiminyCricket

    Many years ago, I worked in 3rd Line Support for a reseller. We had internal IT for our internal systems, but as usual, when there's a problem, "you're a techie - fix it" is usually the way forward.

    Arrived at work before 9 o'clock one Monday morning to be greeted by the Sales Director to inform me that no-one could access their email. Popped into the server oom, and found that an entire rack was down. I quickly disvovered a faulty power strip, and as there was already a spare the other side, began moving all plus across to it.

    I had to connect the new power strip under the floor of the server room, so called one of the junior members to help with the lifting of tiles, and to make sure no-one fell down the hole when the floor was up. It all went well, and a short time later the rack was booting up nicely, so set off to leave the server room with the junior member to inform everyone that all would be well in the next few minutes.

    Now, to leave the server room you had to hit the door release button next to the door. Unbeknown to the junior guy, next to it - under a cover that had to be lifted - was also an emergency shutdown panic button that would kill power to the whole room. Yep, you guessed it - he lifted the cover and hit what he thought was the door release. 20 server racks powering off left the room feeling ever so silent!

    Cue spending the next three hours getting all of the systems up from scratch. :(

  32. Cannorn
    Pint

    Sales Guys

    So I've always had a mish mash of roles but in one role where I was doing Server and desktop support we had the sales guys running the show, selling projects we couldn't possibly deliver in the time frames they promised without days of unpaid over time and the blame on US if the sales failed despite our constant protests.

    Karma however, paid a visit one day.

    One of the most troublesome Sales managers pops over one day, the same guy who pocketed a 6 digit bonus for selling something (read, spent a fortune on the company card at titty bars and the pub) which us IT guys worked 3 months of stress filled overtime to deliver without so much as a cheers guys....came round at 10am.

    He DEMANDED a brand new laptop, built to X,Y,Z Spec, with PowerPoint installed and thoroughly tested with a VITAL sales presentation he needs to present at 2PM today, leaving the office at noon.

    We got it all setup, tested, spare batteries, spare power supplies etc etc.

    Handed it over but he insisted on powering up to test, fine, he does so and is happy.

    2PM I get a call, Answer to instantly get a 100 decibel stream of abuse down the line from Sales Nubbin that the laptop isn't working.....

    Apparently he's arrived on site, has me on speaker phone in front of the clients and wants me to explain to them why he can't show them the demo...

    OK whats the laptop doing.

    Nothing, it just beeps.

    Beeps? ok hold the phone to the laptop so I can hear it...

    That sounds like power...I'm guessing you didn't power it down when you put it in the car, is it plugged in?

    OF COURSE IT'S **** plugged in!! do you think i'm a complete idiot!!

    And where it's plugged in, is that switched on..

    OF COUR.........*hangs up*

    Well that made my day.

    1. perlcat
      Facepalm

      Re: Sales Guys

      I've always enjoyed laying a little karma down on the salesdroids.

      Major PC maker I worked for owned a crap video card maker -- they couldn't write video software to save their lives, and it was clearly unsuitable for corporate use (or any other for that matter). Naturally, businesses stayed away from it in droves. Naturally, since the margins on crap video cards is so good, they'd offer the corporate salesdroids spiffs to move them. (I remember one spiff was a speedboat -- the margins were good indeed) In support, we'd get hourly ass-chewings over these cards, and in desperate cases, the other techs would ship out video cards made by competent manufacturers to replace them. Then they'd endure the ass-chewings from management about containing costs.

      Not me. I'd carefully explain to the user about spiff's, crap video, what makes good video, track down their salesdroid, and drop the hot customer on the 'droid's line. It didn't take long for the salesdroids to decide that no spiff was worth that kind of epic ass-chewing.

      It was good for another teaching moment, too. One tech I talked to was going to order his own PC, with one of these cards. I asked him: "WHY, for the love of G0d, WHY?!?!" He said "Because I know every way possible to *fix* these cards. I asked him why he didn't use the other brand of card. He said "Because I never get calls on them." I just sat there smirking at him until he figured it out.

      1. Aaron Em

        Thought for a second you said 'spliffs'

        But that's not really a salesman's drug, is it? More cocaine or methamphetamine, really.

  33. Psyx
    Facepalm

    You made me jump!

    I once worked in a freezing cold server room, manning a few terminals with a co-worker. It was all UPS protected of course, but the lighting wasn't, and there were fire control systems. One such file control system was some form of fire-door in the air conditioning system. It was clearly large, heavy, and metal. It was also clearly located right above our desks somewhere.

    As a result, every time there was any form of power outage and UPS kicked in, there was this enormous *BANG* above our heads, and we were plunged into darkness for a few seconds.

    My oppo and I were doing some fairly critical work. I can't remember what exactly. But I remember the darkness... I remember the crash of the fire-door... but the most terrible sound of all was - in the wake of this cacophony - the quiet "oops" of my co-worker, as he'd jumped at the noise and spasmed on the keyboard.

    He'd only broken the whole thing with an inadvertent combination of key-presses. It took us the rest of the shift to fix.

    1. Anonymous Coward
      Anonymous Coward

      Re: You made me jump!

      A very quiet 'oops' is when you know things have gone properly tits up.

      1. Aaron Em

        Re: You made me jump!

        Either that or a loud OH SHIT followed by frantic keyboard noises.

        I like the idea of a steel fire door as a "file control system" -- apply it to the problem fileshare users often enough, and those gigabytes of MP3s just vanish like magic...

  34. Rufus McDufus

    General hospital

    I once had to spend a weekend installing servers at a hospital. The on-site tech told me the story of their nice new shiny server room.

    They got all the servers in racks, rolled them in, and went to turn them on. Hmm, something not quite right here - no-one's installed the power yet!

    OK, roll them all out, install power. Roll them in.

    Hmm, no network! Bugger. Roll them out, install network, back in.

    Hmm, it's getting rather warm in here. Ah, maybe we need air-conditioning?

    A few weeks later, air conditioning installed. Now we're getting somewhere.

    Except the air-con wasn't installed properly and flooded the server room several feet deep in water.

    Roll the servers out, dry them off, install a drain in the floor just in case, roll them back in. Yes, the dried-off servers still worked. They're braver than I am re-installing flood-damaged servers but that's maybe that's the way they do it in the NHS?

    Oh the joys of good server room design!

    1. vgrig_us

      Re: General hospital

      Well, i hope someone had this story on hand when my boss insisted on a drip pan under the pipe joint running over our ERP rack in just built server room. Everyone looked at him like he's crazy.

  35. paul clarke
    Facepalm

    Oh well....

    I work in a school as the "Network Support Officer" That means I am the SOLE IT bod in the secondary school with 16 servers and 300+ workstations, I maintain AD, Hyper-V, Exchange, IIs, VLE etc etc etc.

    I am also the first responder....

    I was once called to a classroom by a pupil who said "Miss can't use her mouse!!" I asked for more specifics but none were given. I gave the child a brand new USB mouse and off they ran. 10 minutes later while I was cleaning out a laptop that had clearly been dumped in the sandpit, the child came running back. "That mouse does not work either" so I plugged the mouse into my test rig, and it worked!! I then sent the child back to the classroom with instruction as to WHERE the mouse needs to be inserted.

    10 Mins later, the teacher calls me

    Teacher: "that mouse is broken",

    Techie: "Are you using the laptop?",

    Teacher: "Yes"

    Techie: "have you tried the touch pad?"

    Teacher: "yes"

    Techie: "what happens?"

    Teacher: "Well the mouse kinda moves then snaps back to the centre of the screen".

    Techie: "What happens with the mouse plugged in?"

    Teacher: "Exactly the same!"

    Techie: "Ok I am on my way"

    So I am now proceeding to the classroom to determine why the cursor would snap back to the centre of the screen. I arrive at the classroom with the teacher and whole class watching me with anticipation as I enter the room.

    I take one look at the classroom and immediately identify the problem, but as I have been dragged away from my desk i feel I do not need to share my findings just yet.

    I proceed to the desk, moving past the SMART board, and the object that was leaning against the SMART board. I had to move the object, while not making it obvious that is what I was doing!!

    I then proceeded to demonstrate that not only did the mouse function correctly, but the touch pad worked fine as well. Having done so, I then proceeded to allow the teacher to try it, and they were indeed happy. I therefore left the room as I had found it, complete with object leaning against the SMART board.

    5 minutes later, a slightly exhausted child returns to my office declaring that the mouse was broken again!!

    So I repeated my actions, moving the object that was leaning against the SMART board, and showing that the mouse was functioning correctly. The class now seemed to think I had magical powers and that I was some kind of messiah towards computers. Surely I could not be fixing things by simply laying my hands upon them? I kept my smile to myself as I replaced the object that was leaning against the SMART board.

    5 Minutes later the child appeared at my office once more. Feeling sorry for the child, I decided to pass on my mystical powers. I passed on the ancient secret of the SMART board.

    The child returned to the classroom and moved the object that was leaning against the SMART board. He moved it all the way to the back of the room, held it above his head and said to the teacher "try now miss" and it worked. The whole class clapped the young boy.

    Later that night, 4 bottles of Peroni arrived at my office with a note from the embarrassed teacher, "Sorry, and thanks for not making me look an idiot"

    I drank the Peroni without any guilty feeling. That teacher will now always check to see if anything is leaning against her "touch sensitive SMART board" before she blames her mouse or lack of ICT support.

    This is tale 1 of too many!!

    Paul

    1. Lee Dowling Silver badge

      Re: Oh well....

      Similar story.

      But stemming from the cleaner accidentally binning the SmartBoard eraser.

      To those who don't know - if you don't have the eraser or a pen in the right pocket, the light-sensor on them thinks you've picked up THAT pen / eraser, and determines what colour / whether to erase or not by what's stored in the holes and what's not. With no eraser detected, you're obviously holding the eraser, yes? All the time. No matter what you do with the pens / boards / cables / laptop / replacement laptop / rebooting / etc.

      They'd cleared 14 pages of the day's work by the time I walked in and applied a bit of blu-tak I happened to have at the time to the sensor in the eraser pocket..

    2. Anonymous Coward 15

      Re: Oh well....

      And that's why perhaps it should have been a Promethean. OTOH then you get the same situation as with "analog" whiteboards where working pens are like gold dust.

  36. Anonymous Coward
    Anonymous Coward

    time change horror

    First week as a new contractor in June 1998, and eager to impress.

    I noticed that one of the servers time had drifted and despite restarting the ntpd daemon, it was too far out to auto correct. However, as the time was 15 mins slow, I thought "Great, I'll just push the time forward 14 mins and let the ntpd drift catch up the remaining time". No danger to timestamps on anything as the clock was going forward.

    #> date 0628134500

    Notice the error???

    Yep, I forgot the "." before the last 2 zeros.

    Effect, date moved forward 18 months to June 28th 2000

    Unfortunately, this server ran the Maestro Batch control system.

    So, I now had 450ish daily schedules, 18 month end schedules, 6 quarterly schedules and two year end schedules all competing for resources.

    Realising what I had done, I immediately forced the database offline to prevent updates (the first part of the daily schedule had a delay in as it worked out how long the lock on the database would be), and began mass killing jobs.

    Got the server back (to the correct time) without further incident, then had to own up to my (new) boss.

    Their attutude was "You fixed it? OK, don't do it again".

    As a result, I have never tried to cover up any of my mistakes.

    As a side issue, while I was dealing with the server, a colleague later told me that the phones had lit up like christmas trees.

    Apparently the client software used to interrogate the server to get an estimate finish date/time, then display a "Service will resume in xxx minutes". But, the server date was 18 months in the future, and the client compared the estimated finish time from the server with the date/time on the client PC. People were complaining that the service wouldn't be back for nearly two years!

    I subsequently stayed with the company for several years, and even did their Y2K testing (I didn't mention that one of their servers had already been tested).

    1. Anonymous Coward 15

      Re: time change horror

      This wouldn't happen to be a certain royal financial institution north of the border, would it?

  37. yellowlawn
    Unhappy

    I used to work for this major high street bank, but last week they sacked me.

    Regards

    Johnny Fat Fingers

  38. mike2R

    EBKAC

    I've always preferred PICNIC (Problem In Chair, Not In Computer), simply because it is easier to work into conversation:

    "Any joy fixing that issue yet?"

    "Yeah, it was a picnic."

    1. AOD
      Thumb Up

      Re: EBKAC

      The other variant I've seen on that is the "error code",as in:

      "It was a code ID10T"

      1. Anonymous Coward 15

        ID10T

        Yes, but that's obvious if a luser sees it written down.

  39. Paul Woodhouse

    one word...

    Exchange...

    had a particularly crap spec 2K3 SBS server running exchange, average size of mailbox prob. around 10GB, naturally it would frequently die, the HDD died in a slow strange fashion at one point, still working to the point where it corrupted my backups as well... spent a full weekend getting that lot back, (mainly by hacking peoples OST files into PST ones) one of the directors woke me up at the desk I had finally passed out at on Monday morning and the first thing he said was do you know email isn't working?

  40. CraigW
    FAIL

    US-only contest

    Wtf??

    El Reg does remember which side of the Atlantic it's on, right?

    1. Trevor_Pott Gold badge

      Re: US-only contest

      Blighty, San Fran, Oz...seems like El Reg is everywhere, eh?

      Oh, and I'm from Canada.

      It's a good point though, I'll get the sub-ed to add a note about that...

    2. Psyx
      FAIL

      Re: US-only contest

      Really?

      Balls.

      I guess I need to add a RTFM bon mot, about the time I spent ages typing out a few funny tales for a competition which I wasn't even eligible for...

      Come top think of it, I did have to learn how to use a UNIX system the hard way... and indeed UNIX itself:

      One day, some guys came in from the States, took out our old kit away, and installed some Sun servers which ran this thing called 'SunOS'. They taught two people how to use them over the course of the weekend. One of those two transferred out of the section a week later, and the other one got caught sleeping with someone he shouldn't have and disappeared mysteriously within a month.

      So, we all had to teach ourselves UNIX, with no training budget, no courses... zilch. The one place on a US-based training course was taken by the section head (who never touched the machines and just went on it for a jolly) and his mate from a different section (they spent the week drunk and didn't transfer any knowledge).

      About a year later, I'm being talked through a problem with a guy from the install team in the States, and he was fairly shocked that our knowledge was pretty basic.

      "You should *know* all this!"

      "How? Nobody has taught us dick about it."

      "RTFM?"

      "We don't have any 'M'. We have zero documentation. nooothhhiiiinnnng."

      "What about the thirty instruction manuals, with the blue spines... all the system documentation and checklists?"

      It transpired that we did get given documentation. A grilling of the section head revealed that they'd sent everything to him. He'd locked it in a cupboard and never breathed a word about it to us.

      Yes: We got his ass kicked out of our department for that one.

  41. Anonymous Coward
    Anonymous Coward

    @phuzz

    APC. It happened to me too. Thanks be to the Elders of the Internet for being there.

  42. Blitheringeejit
    Boffin

    My favourite Helpdesk opening gambit...

    "'Ello, is that technical support? Well, I've put the thingy in the whatsit and it won't 'ave it, what should I do?"

    1. Aaron Em

      Re: My favourite Helpdesk opening gambit...

      "Either put on a rubber or buy her another drink, mate. Cheers!"

  43. NellyD

    MD: I want a netbook.

    Me: They're awfully slow.

    MD: I want a netbook!

    Me: It'll drive you nuts waiting for things to load.

    MD: I need a netbook, order me one!!

    *orders netbook*

    MD: I can't use this. It's too slow!

    Me: *awaits demands for an ipad that doesn't run what the MD needs to do their job*

  44. Martyn 1

    Remember Thin wire Ethernet?

    Years ago in one of the I was working at offices every morning a whole section of the office would have a network failure, causing some of the clustered boxes to reboot and untold woe and a gnashing of teeth. Investigations on the network continued for several days but still each morning the network would go down at about the same time of day. Eventually they tracked the problem down to a segment of thinwire Ethernet (coax cable with a BNC connector on the end), they then discovered that someone in that area had recently got a new monitor, and when connecting it up had decided that every socket on the back should have SOMETHING plugged in so he'd plugged a thin wire Ethernet segment into the Video out connector on his monitor, every time he turned the monitor on it put s spike on the network and brought the segment down.

  45. Herr Ferret
    Mushroom

    Tidying the Sysadmins Office

    A long time ago in the mists of time, when I was a lot fresher faced and could stand windows server support, my job was to support the IT manager, and cover for him when he was on meetings. This was in a quite well known high street name, so I thought they would be 'Super professional'. Oh was I wrong!

    Other than that, my job was to perform backups while having a nap, and to repair suspiciously broken laptops (They broke whenever another staff member got a better laptop than another, so you had to replace them all at the same time, or a week later we would get lots of 'Whoops, look what I did' with laptops). At one point I was shouted at for replacing a 'whoops' laptop, not with a new one, but with a different one that I had repaired. It lasted about a week until it too was returned, utterly destroyed.

    My manager however was a horder, and his massive office was floor to ceiling with junk. It was epic, with small paths through it and if you touched anything it wobbled. The IT manager left for a weeks holiday, and I came into work to see a skip, parked under his window. It was wonderfully placed so that items could be dropped/bombed out of his 3rd floor office. I got quite excited with the prospect of some legitimate destruction. All the directors had ordered it on the sly, with the intent that I clear out his room, and then suffer the wrath upon his return.

    I had an awesome afternoon making some serious noise, other staff helped. Whooping occured. His whole room went in that skip apart from his desktop, his cup and 3 files marked alphabetically. Upon his return he went mentalcakes.

    I dodged the bullet by showing him the small pile of shrink wrapped 286 PCs, that he had lost 10 years previously. Probably about 4 - 5 thousand pounds worth of kit he had 'Forgotten' in his homemade saarlac pit of an office. This wasn't to be mentioned ever again. I got a good reference when I left to a better, less windowsy job!

  46. Simon Dax

    Where do you begin :)

    The Dutchman sent over from the European headquarters in Rijswijk to supervise a machine room refit because HIS boss felt we "lacked sufficient experience". Who - for reasons known only to himself - decided that a wall mounted coax termination unit was the perfect place to park his fat arse while he ran patch cables from it to the back of the new VS300.

    Or the two salesmen from the second floor who - after reading a fairy-story about computer viruses in a home computing magazine - took the coax connections off the back of their terminals, carefully wrapped a condom round each of the bnc connectors and forced them back on. They left the tnc connectors off because "the threads were cutting through the condom".

    Or the secretary from one of the regional offices who got a paper jam in a old style, floor standing laser printer and decided to just leave it when I refused to make the six hour round trip from London to clear it for her. The resulting fire caused nearly £20,000 worth of damage.

  47. Yossarian
    Pint

    Three of my best

    I am a sysadmin now, but these come from before my time, so all but the last were someone elses problem:

    1) Bad company using two sockets for around 10 big servers in a tiny cupboard of a room with fans on the same sockets and a little over used, surprisingly had a fire in one of the sockets. They decided that instead of getting it fixed properly they would run an extension cable to the next room and run half the servers from that one instead. A guest CEO was wating in this room one day for a meeting and his phone complained that it was empty, the only socket he could see was the one by the door which was already in use, but he couldn't see what for so decided it wasn't as important as his phone. Bye bye stack of servers. The sysadmin put a postit note on the plug for the next year until we moved offices and built a proper server room with more than two sockets.

    2) At a customer the sysadmin had to shutdown a big honking Sun server for routine maintenance. After many hours of waiting he finally phoned to say we could start the applications whilst giggling away to himself. The story helped get over the massive wait late at night: He had tried to boot the server and found that the boot disk was corrupted, tried to repair it and massively failed. After hours of trying to get hold of a replacement he looked in a desk drawer and found a Sun disk with a massive boot print on the case and the words boot written on it. Plugged it in and it booted the server perfectly. So the big production server ran with a boot disk that had a boot print on. Makes it easy to identify.

    3) Standard story of the kid left to play with a production server (the one with the boot boot disk), everyone had to leave early (to go to the pub...) and left me to finish up an installation, I decided to clean up the tmp folder so ran "rm -rf *". Shame it was the wrong folder.

    Four hours later I finished rebuilding the server, fortunately it was an active-active system so I could just copy everything from the mirror and change all the config for server name / IP etc. I got free beers and a large amount of respect for not bottling it and single handedly bringing it back (I was about 1 year into the job and had no idea what I was doing). Unfortunately I had messed up one file which was only found a month later after someone who didn't know the story had been ripping his hair out trying to find the problem, so I only got a little shouting at.

    Final boot note, one sysadmin I worked with sent an email to the entire company informing them that the email server was broken and he was working on it... He didn't last long.

  48. Anonymous Coward
    Anonymous Coward

    Helpdesk calls

    I once convinced our IT trainer that leaving electricity sockets switched on with nothing plugged in caused electricity to leak out, and cost the company money! also told her that you could speed up access to 3.5" floppies by spinning the hub with your finger before you put it in the drive. Found out later she was cascading this "advice" to all the people she trained in the company!

    We are the technical "experts" in the IT team, and we replaced one team members desktop on his laptop with a screenshot of his desktop - he rebooted thinking it had crashed - We repeated this 3 times before he twigged!

    Once flew (well crashed) an R/C helicopter into an aircon unit in the server room, casing it to fail- rescued it before the facilities guys turned up.

    1. Psyx
      Facepalm

      Re: Helpdesk calls

      Ah... diskettes.

      I once fielded a call from an office where they had lost some data and had to revert to a back-up. On trying it, they called me up because "none of our back-ups are working". They backed up on 5.25" every week night, and had a month's work of weekly back-ups, but none of the data appeared readable.

      Now these guys weren't great, but I had them pegged as fairly tech savvy. Given the choice between it being every back-up corrupt, a drive issue, or them being stupid; I was actually thinking it was the drive.

      I was wrong. It was the other two.

      See, they had decided to keep their back-ups somewhere safe, obvious and easily accessible.

      They had labelled them very clearly and then put them up on their office whiteboard... attached with 'fridge magnets.

  49. Anonymous Coward
    Anonymous Coward

    At a now defunct Lancashire truck manufacturer.

    OS/370 during IPL - and as is pretty normal for this type of machine, a couple of thousand users depended on it.

    Operator starts the IPL - Machine throws up console message 'Today is my birthday'

    Ops reply r00, u

    Machine repeats console message 'Today is my birthday'.

    Rinse and repeat for at least 30 minutes.

    05:30, I receive a call detailing the above events.

    Through the fog of sleep and the beginnings of a particularly nasty hangover, I somehow managed to vaguely remember one of those conversations - funnily enough approximately 12 months earlier - that I had when I was working with another sysprog installing the new Amdahl machine. It went something like... "wouldn't it be a laugh if we ...?"

    Try r00,happy birthday I replied.

    I didn't need to wait for the confirmation that it had worked as the half screamed 'Oh FFS, which C*** did that?! came echoing down the phone.

    I rolled over in bed and drifted back off to sleep comforted by the very good call out fee that I had just earned.

  50. Anonymous Coward
    WTF?

    Surreal Day

    This one isn't really a tale of woe as such but I think it deserves to be told as one of the strangest hardware support incidents I have even been involved with.

    About 12 years ago I used to work for a large publishing firm as a Sys Admin. They had an Ericsson DNA server that handed the internal phone system. It was a standalone box running Windows 2000. Anyhow there was a failure of the phone system and I phoned Ericsson for support. An Ericsson engineer arrived and tried to fix it, but it appeared to be a hardware issue with one of the cards and so he had to phone for hardware support. This was not done by Ericsson but by a 3rd party company. A few hours later this large man appears at the company to fix it. He hasn't a clue what he's doing and is so camp it's unbelievable - I mean John Inman, this is a wind up, where's the hidden camera - camp. I swear he was saying every single dodgy innuendo possible. The Ericsson engineer and myself are cringing in embarrassment.

    Anyway needless to say Mr overtly camp can't fix it. Doesn't know what he's doing and so calls for his boss to come down. Meanwhile the Ericsson engineer goes to get a coffee and I go back to doing 2nd line support calls. We leave the HW engineer to his own devices in reception. A little while later I get a call from the girl at reception saying there's someone here to see you. I can hear her quietly trying to stifle a giggle and she over emphasizes 'someone' here to see you. So I go out to reception and there's a man there, wearing a skirt and blouse, looking exactly like Les Dawson would in a skirt and blouse... I'm speechless. He walks over to shake my hand and introduces himself as camp man's boss. The receptionist is pissing herself laughing and so are both security guards and the guys from facilities who just 'happened' to be there at that time.

    I'm now thinking

    1. Camp man now seems very normal

    2. Is Jeremy Beadle behind a pot plant somep lace?

    3. Les Dawson's dead right?

    So I lead Les and Camp man to the comms room (which is on the other side of the building from the server room) and find the Ericsson engineer in there. Ericsson man's facial expression is still etched on my mind as his brain rushes over itself to comprehend what is going on! I am no help as I'm not sure either. Camp man keeps calling Les 'ducky' (he really wasn't helping the stereotype here) and as the server is on the floor, Les hunkers down to take a look at it. Yes that's legs apart wearing tights, frilly underwear, pre-op. Myself and the Ericsson guy just had to leave the room pronto otherwise both of us would have died laughing. Just as I left them, I asked them to get reception to page me once they had gotten somewhere - couldn't ring see. Meanwhile I recount this to the others in the IT dept. It's greeted by serious cries of BS. One of them goes to the comms room and comes back saying F*&k. Les and Camp man got a lot of people opening the door then saying "oh sorry Bob's not here then" over the next 20 mins.

    To cut a long story short they had to source another motherboard which would arrive the next day. The next day I'm there, the Ericsson guy is there and another person from the HW support company is there. He has the job done in like 20 mins - doesn’t say a word and we don't ask anything. Ericsson man then reconfigures the new card and leaves.

    5 years later I'm at another place with another Ericsson PABX. It has a problem and the same Ericsson engineer appears to fix it. I recognise him and ask if he remembers that surreal day. He laughs and said that day was f&*king weird alright. Luckily for both of us that PABX issue didn't require HW support..... I'm not sure I could cope with another surreal experience again....

  51. Anonymous Coward
    Anonymous Coward

    Bob is now a janitor?

    Bob was always a janitor, he just didn't realize it ....

  52. Tom Maddox Silver badge
    Mushroom

    My CSB

    Years ago, I received a report that we had a faulted UPS at our data center, which, as it failed, would take out half the power supplies in one of our production racks. I duly headed down to investigate, saw that the fault light was on and the battery drained. Assuming that it might just be a temporary glitch, I hit the reset button on the UPS. The next thing I heard was a great, terrible silence as all of our production systems, including our core switches and SAN, went offline due to every single UPS in the data center shutting down. After I changed into my emergency pair of brown trousers, I called the most senior manager I could get hold of and explained that we were dead in the water, resulting in an "all hands on deck" call.

    The scenario which emerged was as follows:

    Due to a lack of confidence in the data center's UPS/generator system, we had installed our own UPS units. Fire safety laws mandated that all power in the facility be able to be shut down, so the UPS units were wired into an emergency power off circuit: one, single emergency power off circuit. When I reset the faulted UPS, it shorted back into the EPO circuit, which caused every other UPS in our cage to receive an EPO signal and shut down. The led to a bunch of us standing around the back of the culprit UPS with an electrician, trying to safely remove the EPO wire, which would not come out due to being physically fused to the plug. Eventually, we got to a Mission Impossible/James Bond-style scene where the wire cutter came out and we had to make the call to just cut it and hope that nothing worse happened.

    One crispy-fried electrician later . . . (I jest, of course)

    With the EPO circuit cut off, all the other UPSes came back on line, powering everything back up. We took out the offending UPS and began the cleanup process. Fortunately, it only took us about four hours from initial failure to final confirmation that our production systems were back up.

    The UPS manufacturer's response to this behavior by their faulty hardware amounted to "bummer, dudes." A few weeks later, we pulled them all out, having decided it was better to rely on the datacenter's power backup instead.

  53. Fred Flintstone Gold badge

    Coming to an oil drilling platform near you

    This was a while back, but it was fun.

    I once worked as sysadmin-and-anything-else for a London consultancy, and got the job to design some hardware for an oil platform (4 PCs in a single box on a mini network to process data and model stresses). I got flack for putting optical input isolators in the design (those things cost money), but I'm the kind of guy who plays safe - and boy was I proved right.

    I get to install the thing with a colleague, so we fly out. The rig is undergoing a refit, wires and people everywhere, but we find our space and start installing it. Initial tests all OK, and we give the go-ahead for input signals (wind, direction, anchor tension, that sort of stuff). Wind data was sourced from a system of another company, and they'd just finished rigging too, so they boot the box.

    Hard disk spins and rattles, and finally, a DOS bootup sequence appears (yes, it really was a while ago). Eventually, I hear the relays click in of their data acquisition card - and the thing literally blew up as in the movies. A massive bang, lots of sparks and instant smoke, the system steel case (think IBM AT style) is suddenly bulging - the works. The saving grace was the steel case because I think the DA card and boards would have acted as schrapnel otherwise - the bang was electrolytes getting something they were not designed for.

    We helped the poor chap collect what was left, and went back to shore with him for beer - we couldn't do anything more until that unit was replaced and we wanted to thank him for the show (and he needed the alcohol - he'd nearly had a heart attack).

    Eventually, we find out what had happened. That system was supposed to keep an eye on the power plant too, but a wiring fault meant it wasn't getting a 20mA loop signal, but a direct, enthusiastic 500V feed directly from the main generator - and they did NOT use optical separation.. So, relays kick in, and the poor DA card received 500V where it hurt most, immediately sharing that joy with the motherboard and PSU. Basically, every electrolytic capacitor in the box turned into a miniature bomb before the PCB tracks could burn out. I have never seen anything like it, not before or after - it was VERY impressive.

    It had a side effect - every time I now see a movie PC blow up I must smile :).

  54. Anonymous Coward
    Anonymous Coward

    When I say you need a new UPS, I do mean it.

    After discovering that the fortnightly UPS self-tests were bringing down a subset of our servers (probably the ones with smaller capacitors in their PSUs), I told management that they needed a new UPS for that rack. Nothing happened. Raised a PO myself to get a UPS. It was cancelled. Installed a workaround of rebooting servers late Sunday night, just after the self-test, to flush any memory corruption after the brownouts and micro-blackouts. All good for several months, until an electrical storm passed over the city one Sunday night. Got into work on Monday morning to find a dead server. When I finally got it back up, I could see in the logs: Power supply failure. Restored. Power supply failure. Restored. Dead. And it was time-stamped exactly to the storm. Management then went and bought that UPS- pity they hadn't done it 6 months earlier. I did enjoy the I TOLD YOU SO however!

  55. Nazgill
    Mushroom

    Customer never gets told No.

    So i a work for a company who's global reach includes the recent NHS Fiasco....yup that would be them. Having shifted and changed directions and paths over the years to further develop my skill set I currently am at the position where the thought of doing day to day sysadmin work fills my head with thoughts of suicide and murder. Not me of course, just assisting the customer in their final options.

    So over the years i have experienced much and seen much, and I'm currently overseeing a group of Sysadmins for Wintel and Vmware. These are people who still think that the customer is always right and that the customer gets what he asks for, regardless of the limitations of the human body to go without sleep. (124 hour week anyone?) So i'm taking a moment to reflect, when i hear (for a better term) one of the PFY's agreeing to a raft of demands from a customer about patching the server environments. Patch the entire environment, tonight, in 6 hours. That would be over 500 varying servers in 6 hours....a bit of a push to say the least, yet wait, we are to catch up with the missing patches for the last 6 months.

    Now any good sysadmin will decry ' you have not been patching the servers? how crap are you!!' but hold on a minute... these are customer servers, we can only patch when we get direct permission to do so.... which we had been chasing, for 6 months oddly enough. So its agreed (minus my vocal dissatisfaction and protest about time frames) the servers will be patched.... in 6 hours.... with 8 different OS's as minimum, after doing a normal day (14 hours) the poor PFY has been targeted as the victim who will patch the environment that same day till 6am... Of course it fails, the patching server cant cope, the environment is too messy and some of the time frames for some of the servers (shortest being 30mins for 10 servers) has left the pfy a broken man (with RSA and other conditions) and sends an email stating the successes and failures.

    Being the kind man i am, i let him crawl into a bed and proceed to die. Yet management decide that 8am is just about a good time to give the PFY a direct call to discuss the email he has just sent. And then again an hour later.... and then again 2 hours later.... and then again at 1pm. This must of been the time he gave up on getting any sleep and came into the office. His eyes told me of his pain.....The phone call i got a mere few moments after that even gave me a moment of pity for him....but that soon passed...he was, again.... to be patching till 6am. THIS time every server had to be patched, so, giving him no work for the rest of the day.

    I allowed him to prepare his assault on the server lists and the patches required. He did well and manged to get all the servers patched... emailed out his victory success and stated, that yet again he would crawl into bed and to please do not call him..... but the management team, who i can only assume had lost the ability to read an email... or anything for that matter, decided to call once more!... this time it was because the customer hadn't received a text stating the success of the patching. Supposedly this had been agreed by management ( and completely failed to be filtered down).. but that wasn't the best part, the customer was complaining about the fact the server was missing patches.

    To stick up for my team and the PFY's honour I decided to take a look myself. Nope the customer was wrong from date x to date y the servers had been patched and was considered fully compliant. Oh how foolish i was to think it was that simple, no, the customer was complaining about patches that were not on the server, from 2003!!! that's right, the tool the customer was using was the Microsoft security tool to check which patches were needed. The customer was outraged at the lie the PFY had emailed out.

    I went to bat on his behalf, asking the management team to kindly not call the engineer again (disturbed from 7:30am onwards) or i would view it as employee abuse. After a significant amount of wrangling and the discussion of the use of correct language ( i.e not foul) when asking for something to be done, management once more collapsed to the indomitable might of the customer and agreed to their demands... which now expanded to all the applications to be patched too......so that meant the patching requirements had just increased about 10fold.

    The PFY comes back to the office... eyes like black holes and a face so gaunt i could shine a torch through both cheeks and see the other side, and he was beginning to smell a little ripe too, but only just. Again he is tasked by management with the job to patch, he pleaded with them, i pleaded with them, our co-workers pleaded with them. But to no avail, management were determined everyone else was far too busy. The look on the PFY's face was that of a man on the verge of emotional collapse. But soldier on he did... a longer patch window had been agreed.....24hours constant patching with servers blocked into 2-6 hour windows in which we were ordained with permission to patch and reboot.... His 4th day of little or no sleep was at hand and reaching into a 5th, already he had clocked up a fantastical 82 hours of work and more was to come.

    The end came, he released emails, reports, screenshots, text messages everything......and STILL management called him to find out the status. That was it, he snapped (i don't blame him) i heard the phone call... i heard the language...i smiled...Management were shocked at his 'attitude' and came to me to complain. Oh for the 30 minutes of ear bashing i gave them, made what my PFY had said on the phone seem pale in comparison. No proper procedures, no proper practice, the customer had been told yes yes yes yes, and why? why was this all necessary. Company audit and the customer forgot it was coming and didn't plan to have the servers patched.

    The PFY isnt allowed to speak to the patch management and incident management team anymore. I have been given a polite warning by my manager.... the patch manager (term used loosely for someone that emails the customer asking permission to patch and doesnt actually patch or plan or do any work) is not allowed to accept any patching requests from the customer that are not 3 weeks in advance. So its a minor bonus i guess... :D

    1. Anonymous Coward
      Anonymous Coward

      Re: Customer never gets told No.

      Ahh, 'no'. That wonderful, short, and accurate word that is so underused by customer-facing types. Indeed, I sometimes marvel at the circumstance that no customer has ever come across it, given it is ne'er uttered to them.

  56. Anonymous Coward
    Anonymous Coward

    Many moons ago...

    OK, not a SysAdmin story strictly, but all of this reminded me of an event that occured some 12 years ago.

    I was contracting. This was in the days when literally anyone could be an IT contractor if they'd spent long enough fiddling with their own PC.

    I started off doing cabling, then installs and spent some very well-paid years installing and configuring Cisco 800 and 1600 routers. The next contract was a bit bigger, and we were tasked with installing and cabling a hospital. The whole infrastructure was being updated and we were doing night-shifts and many out-of-hours tasks to get the job done.

    Me and a mate were just finishing the last piece of the puzzle before the big "switch on" - installation of a colossal Cisco Catalyst. It was fairly high up a wall and we both lifted this hugely expensive bit of kit up to the cabinet. We slid it in, all seemed to be going well. We then pushed it back fully so that it would connect to the main chasis.

    Then we heard a crunch.

    Thinking we'd trapped something, we slid it back out. Around the back the connections, which were a sea of tiny pins (much like a brush or 20 combs on top of each other), were all bent back. Somehow, even though it was perfectly on the rail, we'd not connected squarely with the socket at the back and every single pin was now a tangled mess.

    We had a think for a few seconds. If this had been a few bent pins on a VGA connector we perhaps could've salvaged the situation. But this many? No way.

    There was only one way out of this situation.

    Push it back in, screw it up tight, leg it.

    We knew that this was the last day on the contract so, we started to pack up our things and head off.

    It was at this point that the Project Manager turned up, complete with many of the other contractors. They all came into the data centre and the Project Manager got us to all stand in a circle. He gave us a glowing appraisal, thanked us for our hard work and achievements. He wanted us all to witness "the final piece of this installation and the completion of the new hospital network".

    He turned to throw the switch for the final cab - the cab we'd installed. For a few brief moments there was a flicker of LED's followed followed by a small popping noise. There, in front of us, we witnessed the death of a hugely expensive Cisco Catalyst worth several thousand pounds.

    The Project Manager turned, saw no lights on and started to fiddle. He was joined by another engineer and they randomly poked around. Me and my mate started looking for an exit when they turned and said, "Hmm.. we'll have to raise a support call with Cisco on that, it's probably DOA. Anyway, thanks for your efforts guys, you can all clear off."

    We practically ran out and shoved our tools in the car with the tarmac flapping up behind us as we roared off.

    On the flip side, I think I've paid enough taxes to pay off the damage by now.

    1. Psyx
      Go

      More bent pins...

      I once took over a client's site from a guy who was moving on to greener pastures. He was walking me around the server room, informing me of every server's little foibles. We reach a Sun server... an E2k... might have been a 6k, come to think of it. I notice that it is about half-populated with boards, and that rather than being all nicely next to each other, they are spammed all over the rack/back-plane.

      "Right, these are all over the shop like this because of a thermal issue. It overheats if the boards are in any other configuration."

      "Really?"

      "No. Not at all. But that's the official story. They're really all over the place because I was paged in with a memory fault when I was massively pissed. And as I slid boards back in on the rails, I fucked it up a few times and broke the connections. The empty slots are where the connections on the backplane are buggered. Don't use them. I didn't replace the backplane because they're expensive, and I didn't fancy getting fired."

      And indeed, about two years later I was asked why the server was configured as it was in a server meeting. I told them the convenient fiction, they didn't really believe it, and wanted to 'tidy the boards up' [If it ain't broke, don't fix it, guys!!].

      So I spent an evening re-arranging the boards and having to suck my breath in and shake my head at the right time in order to fake thermal issues and pull the wool over the eyes of the junior sysadmin tasked to 'help' me [ie spy on me and make sure I wasn't pulling a fast one. You could see that the client could smell a rat, but they clearly didn't know what flavour rat it was...].

      He swallowed it and with pure innocence shining brightly in his eyes reported at the next meeting that there was indeed thermal problems and no other configuration worked.

      I was eyed somewhat suspiciously by clients in the room. But they had tasked their little spy to keep an eye on proceedings, and here he was; extolling my virtues.

      PFYs are far too easy to fool. Old age and treachery always wins over youth and enthusiasm.

      1. Anonymous Coward 15

        What flavour of rat?

        No, is filigree Siberian hamster.

        1. Psyx

          Re: What flavour of rat?

          They wouldn't have known what that was. Their English wasn't always that great (but my Arabic was hell of a lot better than what I let on. Handy, that).

          These are the guys who pulled me aside after a crash meeting to ask if I was in some kind of cult or religious sect that they'd never heard of. They figured that there had to be a sinister reason why I always wore black.

      2. Anonymous Coward
        Anonymous Coward

        Re: More bent pins...

        Straw poll (as a result of this post); a) how often do you get drunk when on call and b) have you ever been called, and c) what did you mess up?

        There is a video on the security system the BOFHs here have saved, of me drunk at 2am having been called out, throwing a LCD monitor against an empty cabinet because I could not connect to the back of a Server to press Ctr-Alt-Del like the application demanded. I stuck it in the "recycle" area and used a KVM appliance instead.

        1. Psyx

          Re: More bent pins...

          Why; none, of course!! /innocent.

          The guy I took over from on that contract seemed to be pretty permanently hammered. His pager had been in the swimming pool and dropped in beers (when he didn't feel like responding) at least a half-dozen times and still worked somehow. When I worked my first Saturday morning there, my secretary showed me where my predecessor stashed a duvet and pillow so that I could sleep under my desk like he did!

          I've been called in by on-call guys while *not* on-call to help, and had to get a lift to the site from my misses, because I was too bladdered to drive (but somehow sober enough to fix Sun E10k servers... go figure).

  57. Zombieman
    Facepalm

    AppleTalk / Thin Ethernet

    In the days before the joys of having a structured cabling network installation we had two, parallel, networks. A Thin Ethernet system with various Sun and Burroughs machines along with a couple of PCs, and an AppleTalk network for the Macintosh systems. A few very very special machines were on both networks...

    BOTH of these employed long segments of cable which plug into "something" and that something plugs into the computer. If the computer was not at the end of the chain, you had two long segments of cables connecting to either the Ethernet "T" connector, or the AppleTalk "dongle"...

    Barely a week passed without one of the IT staff needing to walk around and follow the cables around the rather "organically" laid out cable runs to find out where one, either or sometimes both networks had been "segmented" by someone having, for example, taken a machine home to work, or moved to another office and taken said "T" connector or "dongle" with them because they "thought it was a vital part of the computer"...

    1. Fred Flintstone Gold badge

      Re: AppleTalk / Thin Ethernet

      Yup - been there. I started off at a company knowing zip about networking, but something needed to get rigged quickly so it was ARCnet over coax with hubs and passive splitters, and PowerLAN on a box to act as file and print server. And a network problem usually had the same cause: someone done something with a T piece or left an end unterminated.

      It stopped when I started the rumour that it affected the telephone system too. As one of the few people qualified outside official suppliers to program the Panasonic phone exchange, I had the power to seriously mess with someone's phone (and head) - it's amazing how quickly the problems disappeared..

      BTW, that PowerLAN thing had nothing in the way of quota management so users got WAY too free with what they left on the server. Applied psychology: I wrote a Turbo Pascal program that created 10MB files in a directory only I had access to, and so it wasn't visible just how much space we really had left. The advantage of a backup with compression was that my 10MB files took no extra space as it was all zeros, and freeing space was a matter of zapping one or more of my files..

      Ah, the things we do for an easy life :)..

  58. pacmantoo

    Have you paid the bill...?

    My first, very humble job as a grunt working part time doing everything for a small charity. After a couple of weeks got a call from one office that the 'internet wasn't working'. Toddled over - checked the obvious - power on, network switch on, can I ping the router (cheapy BT thing), have any cables fallen out the back of your PC, can I ping something on the net...er, no...Looked at said cheapy ADSL plastic box - local LAN activity but no ADSL light. Ho -hum. Spend an hour getting the work experience girl to find the account details. Phone them up. "Yes sir, you've been disconnected as you haven't paid the bill..."

  59. vgrig_us

    Ok - not one of mine, but...

    ...from trusted source.

    Unix dev server with Sybase DB running. Suddenly Sybase won't start - developers start asking around.

    Here is what happened:

    - whole bunch of technical (supposedly) users were using this server.

    - these technical (supposedly) users discovered that if you save files to /tmp, permission don't, you know, "get in the way".

    - same technical (supposedly) users discover later that their files are gone after reboot.

    - sysadmin (with Unix knowledge supposedly) got tired of complains and decided to do something about it.

    Many of you guessed what he did, right? I can hear "Oh, nos!" and banging heads on keyboards.

    YES - HE CHANGED PERMISSIONS ON /TMP, restricting writes to it! And Sybase servers won't start.

    So, sometimes, oh, who i'am kidding - most of the times - BOFH is a moron BOFH. Major motivation for me - not to be one.

  60. vgrig_us

    Ok - now one from my experience...

    I used to work for s mall company in downtown New York.

    Owner of that company had a friend (frenemy?) on the same floor (owner of accounting firm). After I (and another very good SysAdmin) left the left the firm, said owner got some $90/h Ms Cert Partner consulting to take care of the network and systems.

    The internet setup we left behind was public IP network (remember - it's 2000) on a cisco router with partial T3.

    So, owner decides to get back some T3 cost and sell internet connectivity to accountant for $50/month. Consulting firm handles the detail. For a while everything is great! Things work. But all of the sudden random PCs in both offices can't access network (LAN or WAN). I just happened to stop by - they beg me to look at it. So, here is the multiple FAIL:

    - consulting firm just ran the cable from available Cisco interface to accountant's office: no NAT, no routing, basically just combined two layer 2 networks (a major no-no, but seemed to work for a while)

    - accountant decides at some point it'll be good to sublet a room to an attorney.

    - said attorney's "tech savvy" partner sets up a network with server (but it's own private IP subnet) on the same switch as accountant's network.

    - things would still work and noone would know how f%$#d up the setup is if said "tech savvy" partner didn't setup DHCP server.

    So, 2 DHCP servers on the same LAN (same vlan). PCs were just getting private IPs from attorney's server.

    I told them thir options:

    1. Cut the freaking L2 uplink.

    2. Pay me same ($90/h that idiot from consulting firm charged) to setup NAT and such.

    They opted for cut uplink, cheap bastards. :-P

  61. tastytouring

    Where to Vote + Post Your Story for Chance to Win.

    Thank you all for sharing these amazing, hilarious, and PAINFUL war stories. We are working on being able to open future contests to everyone, but for now, U.S.-based IT Pros have a chance to win an iPad, a $50 gift card to ThinkGeek, or a piece of original art based on your story.

    Whether or not you are eligible to enter, check out the current submissions and vote them up or down. The top winner will be chosen by the judges (which Trevor is one), but the other four will be based on community voting.

    Check those stories out here: http://thwack.solarwinds.com/community/solarwinds-community/sysadmin-day-contest

This topic is closed for new posts.

Other stories you might like