back to article Outsourcer didn't press ON switch, so Reg reader flew 15 hours to do the job

Welcome again to On-Call, our weekly wander through readers' recollections of their ramblings to customer sites after being called out to fix stuff. This week, reader Tim shares a tale from “About 7 years ago when I was working for a software-as-a-service company based in the UK.” Things were going well and the company was …

  1. Anonymous Coward
    Anonymous Coward

    Not sure I can top the story

    but I have been on a 30 hour round trip in Europe to configure a printer as the hawdware outsourcer was not permitted access to PC's that could execute its commands...

  2. chivo243 Silver badge
    Trollface

    Let me guess

    'I have power' lights and the 'I am running' lights.”

    Amber and Green?

    I would have loved to see the looks on the "other" guy's faces

    1. Voland's right hand Silver badge

      Re: Let me guess

      I had that one as well on a couple of occasions.

      It is not just Dell. Nearly all servers which have WOL and/or a management controller of some sort on board are like that - they will indicate that the power is connected, but not power on unless you tell them to.

    2. Anonymous Coward
      Anonymous Coward

      Re: Let me guess

      Easy to solve - ask the guy to use his phone to take a short video of the server front and back (or even a photo), and email it to you. All the information you need to say the words "Press that button there".

      Plus - if the servers aren't built or working at all, you don't even need this evidence to say "Press the button...." without any cost or risk at all

      1. I. Aproveofitspendingonspecificprojects

        How come Tim wasn't asked to stay another day to find someone less intellectual than a barnacle to take care of future issues.

      2. jonathanb Silver badge

        Re: Let me guess

        Isn't "have you tried turning it off and on again" usually the first item on the level 1 tech support script?

        1. Cynical Observer
          Facepalm

          Re: Let me guess

          @jonathonb

          ... my guess they would have been stumped. As the machines were not on, turning them off would have been impossible and having failed to complete step one, they could not countenance proceeding to step two under any circumstances.

    3. Chris Parsons
      Headmaster

      Grammar Nazi

      Guys', if plural.

    4. Eddy Ito
      Mushroom

      Re: Let me guess

      To be fair the inventor of "off" leds should be drawn and quartered followed by serious torture. Many years ago my mum bought a Sony receiver that had a red "off" led which turned green when it was powered on. The problem was up to then nearly everything had red "on" leds which went dark when powered off. Imagine my surprise when I pressed the 1/0 button to turn it "off" only to be met with Aida with the volume set at 11. Funny thing, not many devices still have that arrangement but unfortunately the evil bastards have discovered blue leds for damn near everything and they seemingly pump them at 20 watts.

      1. illiad

        Re: Let me guess ( off leds..)

        only answer is thick tape to cover them... likewise 'disk activity' leds next to power leds(mine are tape covered!)

  3. mix
    FAIL

    Surely some kind of telephone support would have worked it out? Are the servers on? Yes they have lights on, what colour are the lights? etc.

    1. Peter 26

      Exactly. This is incompetence from both sides.

      The people trying to login are so incompetent they can't do basic troubleshooting, so they pass it on to the techie. But unfortunately the techie although knowledgeable enough to fix the issue, is socially incompetent, so doesn't call the company to "clarify" the situation beforehand and save a trip.

    2. Anonymous Coward
      Anonymous Coward

      Re: what colour are the lights?

      ... of course, if you are red/green colourblind, working out whether that more-or-less greeny coloured light is a "green" or an "amber" one can be tricky.

      1. Anonymous Coward
        Anonymous Coward

        Re: what colour are the lights?

        In my experience with Dell servers, the only way to be sure they're on is if the fans are doing an impression of screaming banshees.

        1. TRT Silver badge

          Re: what colour are the lights?

          But what if you're deaf AND deuteranopic?

          1. illiad

            Re: what colour are the lights?

            to repeat....

            if color blindness is involved, ** get a Janitor *** or **guy from the street** to tell you what color it is!!!!!!

          2. Yugguy

            Re: what colour are the lights?

            Deuteranopic? Unable to see the fifth book in the bible?

      2. chivo243 Silver badge
        Pint

        Re: what colour are the lights?

        "Eventually management gave up"

        The whole management team and all the rack monkeys plus the on site rep are colorblind at the same colocation? What are the odds? I guess anything is possible ;-/

        We won't calculate the odds of them all being deaf too...

        1. Doctor Syntax Silver badge

          Re: what colour are the lights?

          "Eventually management gave up"

          This is usually when things start to improve.

      3. Anonymous Bullard

        Re: what colour are the lights?

        if you are red/green colourblind, working out whether that more-or-less greeny coloured light is a "green" or an "amber" one can be tricky.

        In that case, the answer to "What colour are they?" should be "Hold on, let me get my friend..."

        1. Anonymous Coward
          Anonymous Coward

          Re: what colour are the lights?

          I made this point on a different thread, but if you are red/green colourblind, a little piece off a broken brake light lens solves the problem. If it's green, it is dark (LED), or possibly dark blue (bulb), through a red lens.

      4. Martin-73 Silver badge

        Re: what colour are the lights?

        Not sure why you got the downvotes, the green is amberish, and the amber is greenish. So yeah, it's a VERY hard call to tell which is which. Not sure why this choice of colours is so dang popular

        1. KA1AXY

          Re: what colour are the lights?

          Am a colorblind hardware designer, can verify. Which is why I scrupulously avoid using bicolor LEDs, and prefer the "traffic light" green to the old yellow-green type in my designs.

    3. chris 17 Silver badge

      Knowledge vs complexity gap

      knowing how to rack a server is one thing

      troubleshooting why it won't power up once power cable(s) is/are plugged in and the lights are on is another.

      http://blog.ipspace.net/2012/03/knowledge-and-complexity.html

      http://blog.ipspace.net/2015/11/can-you-afford-to-reformat-your-data.html

      loads of out source companies relying on unskilled and inexperienced staff to provide services that on paper are cheaper but in the long run are poor value, like in this case requiring an engineer to cross a continent to power on kit.

      1. Anonymous Coward
        Anonymous Coward

        The cable monkeys at our place are very good. I wouldn't trust 90% of them to do any work with PC's let alone servers.

        Anon because apparently I am not supposed to call them cable monkeys, I am supposed to call them engineers (bollocks).

    4. phuzz Silver badge

      It's been a couple of years since I last played with a Dell server, but I'm pretty sure that the light behind the power button flashes orange when the server is off, and lights up bright blue when you turn it on.

      So even if you were colour blind the flashing should have been a give away.

      1. John Tserkezis

        "the light behind the power button flashes orange when the server is off, and lights up bright blue when you turn it on. So even if you were colour blind the flashing should have been a give away."

        It's going to be pretty much impossible to win this argument, because even if for colour blindness, there are no standards for power/running indicators - there is nothing obvious. I've seen it go every which way, even separate marked leds.

        At the end of the day, if you don't know what the bloody led does on a Dell, or any other type of box, you just don't know.

        Even checking the fans is iffy. I've seen domestic grade equipment among enterprise gear, and on those, the fans are rather subtle - I've resorted to a torch at the back to see if I can see the blades turning.

        So, as per the other post, this is a fail on multiple sides.

        1. Anonymous Coward
          Anonymous Coward

          RTFM!

          comletely agree, fail

    5. illiad

      and if color blindness is involved, ** get a Janitor *** or **guy from the street** to tell you what color it is!!!!!!

    6. maffski

      Depending on the generation of Dell server it can be more problematic - they have an ID light that can be toggled blue/amber (iirc) by a button.

  4. Anonymous Coward
    Anonymous Coward

    External IP KVM or DRAC?

    Was it an external KVM? Why didnt they have DRAC or IPMI enabled, would have been a little easier then to check the problem with the servers and power them on remotely.

    1. Philip Storry

      Re: External IP KVM or DRAC?

      Because DRAC/ILO/IPMI look very expensive, so the bean counters dislike them.

      When you're building any infrastructure that has quite a few servers, the additional cost of DRAC/ILO/IPMI soon adds up to a hefty bill.

      Everybody here knows that when you factor in the potential costs - longer outages, and time saved when called out - they're actually pretty good value for money. Not wasting time having to go to the data centre to deploy the Mk I Finger O' Doom is pretty handy. An IP KVM was a useful alternative, but the lack of the power feature made it very much an inferior solution - which was reflected in the pricing of the two technologies.

      But try telling that to the guy who doesn't understand, and is wondering why every server is more expensive by a three figure sum...

      The drive to virtualisation has often been justified solely just on the basis of shaving that cost off each server (and having standardised drivers/devices on your servers). As you scale up, it becomes a significant saving.

      1. Anonymous Coward
        Anonymous Coward

        Re: External IP KVM or DRAC?

        IIRC the basic DRAC functionality, including power control, are free or nearly. The Enterprise version with remote console and the like is to be paid separately.

        While a KVM is usally better for remote control than the ActiveX/Java "virtual KVM" of DRAC, for basic tasks and to check/control hardware, on-board management systems are very useful especially if the machines are away from you. Virtualization doesn't help, you still need a way to manage the hw the hypervisor are running on...

        Also the actual versions of on-board management system can host the server drivers and automatically install them at setup.

      2. Anonymous Coward
        Anonymous Coward

        Re: External IP KVM or DRAC?

        I've previously used an IPKVM and an IP power strip - they worked out considerably cheaper than DRAC/etc cards in each server at the time.

        1. steamrunner
          Mushroom

          Re: External IP KVM or DRAC?

          Did your IPKVM and power strip combo support remote-mounting media so that you could re-install the server from bare metal remotely? Did it alert you to hardware issues with the server? Was it able to automatically reboot your server (ASR) when it detected something was wonky or crashed? Did it provide access to the integrated management (hardware) log on the server when the server was offline? Did it allow you to power-adjust the CPUs to stay within power requirements/tolerances? Did it allow you to automatically power down servers at 'quiet' times and then start them up again when needed? I'm Hazarding a guess the answer to most of those is "no".

          So, as ever, one pays one's money for the features and things one particularly needs. Some people only need basic power control and KVM, some people need more. Price alone is an invalid comparison - we're IT people, we know this.

          Personally, I'm only interested in some, but not all, of the things mentioned above, but experience has been a hard task master and now if anyone around our place provides/specs a server without proper iLo/DRAC-level management on board they get a very stern Paddington-Bear-Stare-grade talking to. It's a philosophy that's saved our bacon (or at least saved a lot of wasted time) enough times to be warranted for us. Meanwhile, our physical KVM (where needed) does just that and nothing more. YMMV, as they say... :-)

          S.

      3. DainB Bronze badge

        Re: External IP KVM or DRAC?

        That's why I love Sun/Oracle hardware, iLOM is right there on motherboard and you can't get rid of it no matter how many cents you want to save. Makes management of lights off datacenters across the planet a breeze.

        1. Anonymous Coward
          Anonymous Coward

          Re: External IP KVM or DRAC?

          HP standard servers also...

          1. Sgt_Oddball

            Re: External IP KVM or DRAC?

            I think you even get it on the gen 8 micro servers too, which stack very neatly on each other for little towers of power (they also make a half decent workstation but that's another matter)

  5. Known Hero

    Ẁhy does nobody use skype ?

    1. Anonymous Coward
      Anonymous Coward

      Ẁhy does nobody use skype ?

      Because, since Microsoft acquired it, it is not very good any more.

      1. Jess

        Re: Ẁhy does nobody use skype ?

        > Because, since Microsoft acquired it, it is not very good any more.

        Here is a useful alternative (compatible with standard sip, though video has to be browser to browser or phone to phone)

        https://www.getonsip.com/

        1. John Tserkezis

          Re: Why does nobody use skype ?

          "https://www.getonsip.com/"

          I must be the only one who reads T&Cs. No commerical use for getonsip.

          "How can they tell" you ask?

          Yeah right, that's what's making the bulk of software writers ONLY offer rediculously expensive subscription models for everyone.

          Thanks for your attitude.

  6. CT

    I was sent to Estonia in order to fix a font-related problem. Failed to fix it on the spot, but had a lightbulb moment that resulted in days of rework.

    1. AndyS

      I had cornflakes for breakfast.

      1. Pookietoo

        I had a nutty crunchy breakfast bar.

  7. BenR

    Sadly, this barely surprises me.

    1. Quortney Fortensplibe
      WTF?

      Hold the Front Page

      AndyS: I had cornflakes for breakfast.

      BenR: Sadly, this barely surprises me.

      Well, I'm gobsmacked. I always had him down as a Wheetabix man.

      1. Pookietoo
        Headmaster

        Re: Hold the Front Page

        "Weetabix"

        1. Quortney Fortensplibe
          Unhappy

          Re: Hold the Front Page

          ..."Weetabix"...

          I stand corrected. Thank you, sir.

          I would downvote myself, for shame, but it's not allowed, unfortunately.

    2. I. Aproveofitspendingonspecificprojects

      I had a fishy serving of microwaved mackerel rolled in some sort of pepper and tasted of bones and hot burned fish. Then I had to make some soup to wash out the bloody taste of damned breakfast shit.

      Sometimes before coffee sets in I make poor life choices.

  8. Anonymous Coward
    Anonymous Coward

    Shirley...

    The first rule of helpdesk would work here...

    "have you tried turning it off and back on again?"

    1. Captain TickTock
      Joke

      Re: Shirley...

      So when they turn it on and back off again, they come back and say "Yes, but it didn't work"

      1. Anonymous Coward
        Anonymous Coward

        Re: Shirley...

        I will never forget the customer that told me the computer was *definitely* on because they "weren't allowed" to turn them off and therefore refused to even consider just turning it on as an option. The fact the screen was black and it wasn't on the network failed to shift her viewpoint. OK ticket logged, someone might get back to you tomorrow...

    2. Dave 126 Silver badge

      Re: Shirley...

      Then the Zeroth Rule would be: Have you tried turning it on?

      Also from the IT Crowd:

      - So you want me to open your laptop for you?

      - I would be beholden to you!

      1. a_yank_lurker

        Re: Shirley...

        The rules

        0th - Have you plug it in?

        1st - Is the UPS on and working?

        2nd - Did you turn it on?

        3rd - Is the monitor plugged in?

        4th - Is the monitor cable attached to both?

        5th - Is the monitor on?

        1. John Brown (no body) Silver badge
          Facepalm

          Re: Shirley...

          "0th - Have you plug it in?"

          Having returned a printer to a customer after workshop repair she phoned back to say it had arrived but now it didn't work at all, it's worse than it was before it was sent away so she'd have to use the loan one while we take it back.

          So tech takes her though obvious, starting with the zeroth law, is it plugged in. Yes, it's plugged in, it's tuned on, the head does it's normal start-up sequence, ready light is on, paper loaded, paper out light is off but nothing prints, no head movement, nada.

          So I get sent to site, arrive on site, plug in the centronics cable, demonstrate it is working and leave.

          Well, it WAS plugged in, wasn't it?

  9. Mystic Megabyte
    Pint

    Milan

    I was sent from London to Milan for the day to cable and configure some modems. When I got back to Milan airport the Customs went on strike for a couple of hours. I sat in the bar being paid an hourly rate of £mucho and was very happy :)

    P.S. When they resumed working, the custom guys queried the pin hammer and Stanley knife in my carry-on toolbox. I said that it was impossible to do any damage with those. Been proved wrong there.

    1. Hollerith 1

      Re: Milan

      Proved wrong threre...yes, yes you have been.

  10. Anonymous Coward
    Anonymous Coward

    Not very efficient

    "Can you send me a HD picture of front and rear of the racks?"

    <get pictures>

    "OK, the servers have power but are not actually on. Press the little 'On' button."

    <servers power on>

    "Happy to help! Bill is in the post."

    Time and impact of flights saved, look like a hero to the customer. Profit?

    1. Anonymous Coward
      Anonymous Coward

      Re: Not very efficient "Can you send me a HD picture of front and rear of the racks?"

      "Cameras are banned in here" - quite often.

      Back in the day when I was involved in designing industrial computers, all the little lights were labelled, either next to LEDs or actually on the lenses of larger ones. So the red LED in the bottom left corner was clearly labelled "Power" and the big green latching pushbutton with the guard against accidental shutoff was labelled "Run".

      But designers know better.

      1. Tim Jenkins

        "...designers know better..."

        "I say, chaps, on our next case thingy, why don't we put the power button just to the right of that bit, you know, the slidy-draw where the shiny spinny things go in and out?"

        "Golly, good thinking, Tarqin; now, who's up for a refreshing 'diet' Coke break?"

        1. Heironymous Coward
          Joke

          Re: "...designers know better..."

          First the cornflakes and now diet Coke? Getting a little on the side from product placement, heh??? Hint for AndyS: brand is everything....

        2. Wensleydale Cheese

          Re: "...designers know better..."

          The gazillion optical drives with the insert/eject button underneath the tray are a testament to saving 5 cents per drive.

          The manufacturers probably saved a few million over the years with that design, but every single user pays a price when pushing the button with the tray in the ejected position.

      2. John Brown (no body) Silver badge

        Re: Not very efficient "Can you send me a HD picture of front and rear of the racks?"

        "But designers know better."

        Or beancounters. Or PHBs.

        "I say chaps, how can we save money on fascia mouldings or labels having to be localised for language?"

        How about we invent some little cross-language intuitive symbols that everyone can understand no matter what language they speak or even if dyslexic. But make them different to everyone else's so we can patent the design.

        Great! Now how can we save on the cost of highlighting them in contrasting colours?

        Just emboss or raise the symbols so they are black on a black background because we just know everyone has bright lights pointing directly at the front panel and it's never dark there, no siree.

        Excellent! Anyone for choca mocha skinny latte?

        1. Wensleydale Cheese

          Re: Not very efficient "Can you send me a HD picture of front and rear of the racks?"

          "Just emboss or raise the symbols so they are black on a black background because we just know everyone has bright lights pointing directly at the front panel and it's never dark there, no siree."

          Anyone tried finding the supplied screwdriver plus drive mounting screws in an HP MicroServer in a dark corner of a server room?

          Black on black.

          And yes, mobile phones were banned in that server room.

      3. 404

        Re: Not very efficient "Can you send me a HD picture of front and rear of the racks?"

        OR the all-knowing server case designers place the power button right where your little knuckle would naturally rest while levering out a thumbdrive from the conveniently placed top USB 3.0 ports...

        Took three damn hours to resych the RAID on that bitch, ate the billable time because it was my fuck up. Spent it between staring at the percentage on the screen, praying the OS would boot, and hating on whomever genius that thought it was a clever design...

  11. RoninRodent

    I used to work on a helldesk for a financial company that supported remote branches that had no local techie. One chap landed a call about a file server that wouldn't boot and after supposedly trying all of the usual booked an engineer (which cost the company) to go out and check it. The engineer arrived on-site, pressed the eject button on the floppy drive and went home. Needless to say my colleague got thoroughly chewed out for that.

    1. Mayhem

      Floppy drives?

      The big one these days is a USB drive plugged in overnight so the pc doesn't boot the next day.

      On lenovos all you see is a black screen with white blinking cursor.

      Trivial to fix once you know about it, but with USB drives getting tinier, can be easy to overlook on a quick glance at the box. It's one of my top questions for checking out novice techies.

      1. TRT Silver badge

        Re: Floppy drives?

        I'll use that to replace my old one. That ran: two new WinBoxes, identical specs, with fresh installs of the OS and DTP software. Both have Quark Xpress parallel port dongles in place. Both are connected to the Internet. Different serial numbers for Quark on both machines, adjacent to each other on the list of serial numbers as supplied by Quark for this deployment. One machine runs Quark, the other doesn't, producing a "license failure" message. Diagnose and repair.

        1. toughluck

          Re: Floppy drives?

          @TRT: Check if both PCs have the same settings for the parallel port in BIOS/EFI. Check that Windows has parallel port with exactly the same settings on both machines. Confirm that the OS sees the dongles. If not, check that they are screwed in (or clasped) properly. Swap the dongles if it still fails. Swap the hard disks if the error is now that the serial number doesn't match.

          Almost certain it's a parallel port off or failed, or one dongle failed. Very unlikely to be any other cause.

          1. TRT Silver badge

            Re: Floppy drives?

            @toughluck.

            I'd swap the dongles over first, but you're right. Based on a real world occurrence. F***ing Dell delivered 40 machines with (parallel port settings) 30 set to ECP, 10 set to SPP in the BIOS. Took the previous tech days to discover that some didn't work. And to figure out what was wrong, they set it in the interview for the new computer tech, me. I figured it was such a stinker I'd carry on using it!

      2. phil 27
        Thumb Up

        Re: Floppy drives?

        Not just "pc" in scope for the usb key issue. A Dell 1950 does that if usb hd is enabled as a target in the bios and the bootloader on the key is borked. The first time its nearly had me napping because I thought I had video issues on some of the boxes by the time I wandered back to the kvm station on another floor before deciding to get someone else to perform the complex task of pushing the on button on the contents of a rack one by one while I stood at the station and being able to see all the perc controller crap etc before going into blinky underscore of death mode.

        To the original story, as a *owner of dell 1u hardware, there was obviously a requirement to be utterly deaf with no skin sensation of draft in addition to colour blind for the local IT support. When they first power before the environmental sensor tells the board thingy that no its not about to melt (this is a achievement...), all the considerable amount of very small high rpm fans arranged across the middle of the chassis accelerate to max speed and it has a go at making the rack move from the rearward thrust if you leave the rear doors off the cab/had to find a creative solution a too short rack cab...

        * now ex, I ripped the xeon's and ram for my workstation out the last still twitching still overly hot carcass of the last one this week, and it felt good to finally slay the last of the beasts.

        1. Sgt_Oddball

          Re: Floppy drives?

          The real fun is being in the server room with 5 or 6 doing this jet fighter take off impression when the server room is in one corner of a warehouse.

          Cue sandstorm for next 10 mins, coughing fit and waiting for the litteral and perverbial dust to settle.

      3. John Brown (no body) Silver badge

        Re: Floppy drives?

        "It's one of my top questions for checking out novice techies."

        A blank screen 'cos an extra bootable device is plugged in? In 20 years I've not seen that, so maybe it's only testing if a techy has seen a Lenovo do that rather than any real indication of skill/knowledge.

        Surely the standard boot order should NOT be USB or removable devices before the HDD?

        1. illiad

          Re: Floppy drives?

          Bootable USBs usually hold an OS that is way too large for the DVDs on most machines...IF they have them...

          also, USBs with linux, win7 images, etc, take up a fraction of pocket space compared to *scratchable* discs!!!

        2. Wensleydale Cheese

          Re: Floppy drives?

          "Surely the standard boot order should NOT be USB or removable devices before the HDD?"

          The HP MIcroServer N40L has USB Priority set to High by default, and simply plugging a new backup drive in will cause it to look at that on the next reboot, even if you have previously set the boot order to look at the internal disks first.

          Simple answer: Set the USB Priority to Low in the BIOS.

          Not quite so straigtforward when you want to use the internal USB socket for booting on a permanent basis. Net booting or a hacked BIOS seem to be the real answer here.

          1. John Brown (no body) Silver badge

            Re: Floppy drives?

            "Simple answer: Set the USB Priority to Low in the BIOS."

            Yes, exactly my point. Users desktops especially, and most servers, should be set to boot from their specific boot device as their highest (only?) priority. Only a tech or higher should ever need to boot any other device and it should never "occidentally" attempt to boot some other device. Any tech needing to boot a pendrive or other device probably has bigger problems to deal with and finding the key-press to select the boot device or changing the BIOS config should be part of that process.

            "Not quite so straigtforward when you want to use the internal USB socket for booting on a permanent basis. Net booting or a hacked BIOS seem to be the real answer here."

            Yes, there's always an exception :-)

        3. This post has been deleted by its author

    2. Pete 2 Silver badge

      Need I continue?

      > pressed the eject button on the floppy drive and went home

      The lone techie in a far away office was instructed to insert *the* one and only diagnostics CD into the drive of a rather large server box to determine what the fault was. The CD drive didn't have a tray to put the disc in, it had a slot that you pushed the CD into. sadly, there was a small gap in the chassis just under the CD drive.

    3. Andrew Moore

      I had something similar but it was a cleaner that had unplugged a computer in order to plug her hoover in. The look on the customer's face when I went to the socket and plugged it back in...

      1. Doctor Syntax Silver badge

        "it was a cleaner that had unplugged a computer"

        Ah, cleaners!

        Bomb alert persistently triggering (this was in Belfast). Eventually traced to cleaner vigorously polishing the panic buttons.

      2. Wiltshire

        Andrew, that wasn't at a call-centre in Bristol was it?

        I once worked on a project for M$ elsewhere in the UK. The Bristol call-centre systems started collapsing at 18:00 every day. Much wailing and finger-pointing at us the software devs. How could our code be so useless? Eventually, we were sent to the call-centre in Bristol. Standing looking at the main server screen at 18:00, it suddenly powered-down.

        WTF just happened?

        Still staring at the dead server, we were annoyed by the interruption. "Excuse me, dears" said a cheery voice behind. Turning round, we found the cleaner, with her hoover plugged into the one power socket in the server room. With the server's power lead dangling on the floor.

        "Err, did you just unplug that?"

        "Oh yes, dear, but I always plug it back in when I'm finished."

        1. DryBones

          The funny part is that's an infrastructure problem (only providing one outlet in a server room, what?), not the problem of the cleaner.

      3. Wensleydale Cheese

        Cleaners unplugging computers for the vacuum cleaner

        The trick is to leave the screen(s) on overnight. The cleaner will then realise what they've done when putting the plug back,

        Did it once with 5 systems coming off one socket, and it never happened again.

  12. Karl Vegar

    A while back I was between assignments, and got some ad hoc assignments from my temp agency.

    One such memorable assignment was related to a credit card terminal.

    The assigment: to set up a new terminal in a shop.

    I got a nice little documentation burn before you read and self terminate after package. And spent some time familiarizing myself with the procedure I was supposed to perform the next day. Didn't seem to complex, but there was a part with activation that needed a 25 digit code or some such. This part looked like it was a bit beyond your regular user.

    I then drove a couple of hours to the site. Took a look at the terminal I was supposed to set up and activate. It was already placed where it should be, and nicely connected. But not on or activated.

    Talked to the customer, and she'd had a bad experience last time she tried to activate a terminal.

    OK, as the connections looked fine, so I fired it up and looked at it while it self tested all OK. Including connections to the bank... basically it seemed to be activated already.

    - Hm, has this terminal been sed before somewhere? I asked.

    - Naturally, it's the same I had problem with the last time. We didn't need this POS for a while, so we've had the terminal in storage the last 6 months.

    OK, so we tested charging my card some minimal amount. Worked like supposed, both banks accepted. Had the back office test while I found some lunch. All OK. (I allways get suspicious when something look to be too easy, hence the vigorous testing.)

    Total time spent fixing the issue: 5 min

    Time spent testing: 15 min

    Invoice: 1 hour work + 4 hour drive + expenses.

    If nothing else, it makes for a good story.

    All tests turned out OK.

  13. Anonymous Coward
    Anonymous Coward

    Lack of (common) sense(s)

    I had to drive for about 2 hours to Manchester (4 hour roundtrip) once just to prove to the data centre engineer that the aircon had failed. It was the usual response, everything's green, all the units are on. My environment monitoring told me different, as did my servers which were close to their operational limits and automatic power down.

    So it was on a Sunday afternoon I drove to Manchester with a thermometer, on arrival took said engineer to cab where it was possible without the thermometer to diagnose the temperature issue, the warm draft up your trouser leg from the floor vent gave it away. So he took me to the offending air con unit and pointed out it was working - everything was green. On my request he lifted the floor panel next to it and stuck his hand in the air flow - it was blowing hot air - no **** Sherlock.

    So one Sunday afternoon ruined due to said engineer clearly having no common sense, a lack of ability to tell what temperature it is either from his own senses or utilising something as simple as a thermometer and believing what the aircon was telling him rather than his own senses or umpteen temperature sensors.

  14. Locky

    Similar

    Back in the last millennia, I worked for a company who had a server at a customer's site in a London train depot, I was based in Derby

    It "stopped working", no power to it at all. Being the lowly trainee at the time I wasn't important to speak to the client myself, but was dispatched on a train to the bright lights to try to resolve the issue, armed with spare PSU and cables.

    Arrived on site, lifted the floorbox, and turned the plug back on

    Case closed, long lunch on expenses, beers on the train home

  15. Alan Bourke

    I drove from Sheffield to Worksop

    ... to turn a monitor on, circa 1992.

    1. Anonymous Coward
      Anonymous Coward

      Re: I drove from Sheffield to Worksop

      On the upside - you got out of Sheffield for the day!

  16. Yugguy

    IP KVM??? Work of the devil!

    Can't stand those things. Give me a dedicated ILO card any day.

  17. Paul IT

    Hamilton to Lancaster

    Back in 1997, I was asked to drive from work to a travel shop to fix a printer issue.

    The staff had moved the PC and Printer and reported that the printer parallel cable would not go back in the same slot on the computer.

    Drove down in my little Fiat Panda to find the parallel cable connector had the screw holder attached.

    Two minutes later the PC was able to print again, when the screw holder was reattached to the PC.

    On my way back, I decided that this was time to move jobs. A 6 hour round trip for two screws.

    1. Aggrajag

      Re: Hamilton to Lancaster

      Back in 1997 I'd have driven 6 hours for two screws without batting an eyelid :D

      Nowadays, at my age, I CBA to climb the stairs for one.

  18. Anonymous Coward
    Anonymous Coward

    In the heady days before triage...

    I had to drive from Manchester to Aberdeen for a printer that wouldn't print. Walked in the room spotted that they were using cheap ribbons that didn't have the little piece of conductive tape at the end that marked the end of ribbon.

    Asked for a spare, fitted it and walked out, drove back home again.

    Had to get from Manchester to Leeds within the hour or "we'll cancel the contract" for a dead monitor. Walked in turned up the brightness control and walked right out again.

    1. Anonymous Coward
      Anonymous Coward

      Walked in turned up the brightness control and walked right out again.

      What difference did that make? The client surely didn't get any smarter?

  19. Anonymous Coward
    Anonymous Coward

    I've done the old trick of power cycling a network interface of a remote server...

    - Right click, select "Disable"

    - Right click, sele... right click (harder)... oh fuck!

  20. Anonymous Coward
    Anonymous Coward

    I remember about 10 years ago having to go from Crewe to a datacentre in London because the operations team couldn't follow the simple instruction power it down for a full 30 seconds. The kit in question would error and not boot if simply turned off and straight back on again.

  21. Sam Liddicott

    France

    I travelled to France to replace a 25 pin serial cable for an Iomega Ditto drive with a 25 pin parallel cable.

    I think I was due a trip abroad to see the wider company anyway, but from a technical point of view I regret that I could not diagnose that over the phone.

    1. Sam Liddicott

      Re: France

      I was travelling with a company director and we had canapés on the ferry on the way back :-)

  22. AIBailey

    Sometime around 2000 when I was a support technician, I was asked to go to see one of the directors. He's been abroad on a business trip, somewhere like Hong Kong, and had been unable to get any sound from his laptop at all. He demonstrated in his office by trying to play a CD.

    10 seconds after looking at it I rotated the volume knob at the front (Toshiba Satellite Pro) and walked out before his face became too red through embarrassment.

  23. Rory B Bellows

    can you spell IT ?

    Responding to a call out for 'network problems' I once drove 2 hours into the treacherous snow covered mountains of western Canada only to find that the machine in question was in WORKGRUOP. 2 hours there, 2 hours back, plus one hour of on-site (even though it took about 2 minutes to solve). Our troubleshooter did ask over the phone, but unfortunately there still hasn't been dyslexia for cure found.

    1. Allan George Dyer

      Re: can you spell IT ?

      Was it an HP printer? My CM1017 has the default workgroup of WORKGRPOUP

  24. Anonymous Coward
    Anonymous Coward

    You can't beat a cowboy outsourcer

    I was staring at a nice rack (ooh-err) in a server room in West Africa, a few years ago, and recorded some interesting observations about the Cowboys we'd hired, when the civil war was on. The main observation? Well, what do you use to keep a server in place, in a rack, to prevent it falling on the server below it? Erm, you use the server below it. So, one server correctly fixed in, on rails, another two on top of it, and a customer with a distinctly African sense of humour taking the piss out of my employer!

  25. Simulacra75

    Infrared printing tale

    About 15 years ago the company I worked for bought a HP Laserjet. It came with an Infrared port that was on a lead that plugged into a serial port (it looked like a small mouse). The idea being that a user could maneuver the IR port to line up with their laptop to send a print job (not that IR printing ever worked!).

    Anyhow, I was setting the printer up whilst a colleague was configuring the network queue (Novel as I recall). While I was plugging it in and filling it with paper a Senior Design Engineer saw me tinkering with the IR "mouse" and asked what it was. With tongue firmly in cheek, I said it was a voice-recognition device for the printer. She didn't believe me until I said "print test page" whilst holding the IR port up to my mouth. She didn't know my colleague had already queued up said test page on the Novel queue so, sure enough, out popped a test page once the printer had warmed up.

    She was gobsmacked.

    Absolute true story.

    1. allthecoolshortnamesweretaken

      Re: Infrared printing tale

      "(not that IR printing ever worked!)"

      Oh, I don't know... Some 15 years ago I had a MP2100 (Apple Newton, the first generation of them that actually worked and consequentially the last they ever made, had still the best address database on any mobile device ever, but I digress...). The local Staples had a HP Laserjet as a demo model on display. The customers could press a button to print a test page. The printer also hat an IR port. So every now and then whten I had to print something I'd pop into Staples and used the demo printer. Always worked like a charm.

  26. Richard Altmann

    Peavy Amp

    Working for an audio company once, i got called out because the techie on site almost went nuts. It was a small setup with just a few mics and he did not manage to get the signal out to the speakers. I´m quite skilled in phone support. So i went through the procedure step by step up to the point where the guy asked me i if thought he was an idiot. Inputs fine, output routing on the mixer fine, amp lit up but no signal indication. So i went there (didn´t have to fly to the states) to find that the bloody amp was indeed lit up by some blue LEDs onboard the PCB, shining brightly out of the cooling grill on front of the amp. Without being switched on ... Techie was quite embarressed but i don´t blame him. ISO Norm states that an electrical device has to be switched off completly by the main switch. Well, that piece of crap is no longer avaible on the market. As for servers i appreciate some blinkyblinky to indicate the state of the PSUs before i switch it on.

  27. Turbo Beholder
    Childcatcher

    Incoming!

    The first generation of the Not Left Behind. And you thought outsourcing stuff to India was shoddy?

  28. Anonymous Coward
    Anonymous Coward

    There was this one time...

    When I worked as agency staff as a field engineer, I once attended a business that had a problem with an IBM PC, that the clock kept going wrong on.

    Unknown colleagues had previously investigated this issue on multiple occasions, the last time they had replaced the motherboard - but this issue hadn't gone away...

    So, when the support request was reopened due to a "repeat issue" I was sent to site.

    When I was able to access the site, I discovered what the problem was - before I had even touched the PC.

    Approximately 5 feet to the right of this PC was a Novell Netware Server - that we didn't support

    - and it's system time was wrong by exactly the same amount...

    I think the customer would have been a lot happier if they'd received this information - and the bill - on the first ever site visit, rather than after my colleagues had already been to site on several occasions...

  29. macjules
    Facepalm

    Kind of close ..

    About 10 years ago I was asked to fix a Cisco router by my (then) employer's office in Kampala, Uganda. Not being Cisco qualified to write a config, but having a config available that they could use. I emailed them the file and informed the office that they needed a Cisco engineer to do this, which they then told me they would source locally. A few days later they emailed me to say that they had got an engineer in, that he had taken the router back to his office, zeroed the router and loaded the configuration and brought it back to them. I asked if there was still a problem and they said that it was just not working - nothing switched on or booted. Asked then to check the power source which they did and said it was all correctly powered and plugged in. As a precursor to coming out there I requested them to send me a photograph of the server cabinet.

    The cables were all correctly plugged in, exactly as they said ... just that nothing was actually connected up to the mains.

  30. Anonymous Coward
    Anonymous Coward

    travelling to site to sort out internet issues.

    When you support one of the largest resources projects in Australia, people tend to get toey when they can't access the Internet. I got a phone call on a Monday morning to say that their internet was down and they could not get it working. So I had to catch a flight at 11.30 in the morning, fly to site (2 hours), sit around for 2 hours (in 35 + degrees Celsius) while they tied up the barge and installed a walkway, get on board then find out that they had fixed the problem shortly after I left Perth. Turns out that the POE injector for the wireless link (over which the vessel connected to a satellite service) was in the bridge and someone decided they needed the power outlet to charge their mobile phone.

    Fortunately I did have to adjust the rotation of the aerials, so at least I wasn't totally wasted on site ( took 30 minutes for the paperwork and 5 minutes for the adjustment)

    Could not fly back until the following day so was there overnight.

  31. Anonymous Coward
    Anonymous Coward

    not as egregious, but...

    ...I was once nearly refused access to a satellite office.

    Having discovered that I actually lived extremely close to said office (a converted house), the duty to look after the place was duly transferred to me rather than sending a support person that lived a bus and ferry ride away. Fair enough.

    I'd been supporting them for months, gotten to know the staff over the phone, gotten friendly.

    Well, the day came when the boss wanted me to pop by and give the computers a once-over (don't ask). 'Great', I thought. 'I'll rock up there, and be home with the kettle on pronto'.

    I rang the doorbell. Admin / reception person answered with a look of incredulity on her face. 'Yes?' she asked, just barely removed from full League Of Gentlemen membership.

    'It's ******, from IT'

    'What?'

    'It's ******, from IT. We talk on the phone - I'm here today like we organised'

    'Oh no. No, you look too young'.

    'Sorry?'

    'You look too young. I thought you were a bit older. I...I don't know if I should let you...'

    'Touch the computers?'

    'No, sorry love'.

    I turn, take a couple of steps down the path.

    'That's OK, I'll just close the ticket and let my boss know I'm too young to be trusted'.

    'No..I...come back...'

    With that, her manager came out (having not heard this exchange), shook my hand, and took me inside. With age and wisdom, I don't know why I didn't clear off anyway owing to flagrant ageism - it runs both ways.

  32. ksb1972

    Who designed that font?

    A small local firm I do tech support for forwarded me an email from their ISP late on Fri 27 Nov instructing them to update their router with new DSL sign-in details by Mon the 30th Nov "or it'll stop working".

    So I go in near closing time on Saturday to minimise disruption just in case anything goes wrong. Put in the new details just to find the new credentials don't work. Try ringing the ISP only to find their support desk is shut until Mon.

    So I ring them up 1st thing on Mon to have a moan. Luckily the guy's more awake than he was when he first sent the details out checks the login attempt log at his end and noticed after pasting the password into Notepad that the two characters in the middle should be lI and not 1I...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like