back to article The dread sound of the squeaking caster in the humming data centre

Distract yourself from the come-hither finger of a Friday trip to the pub ahead of the weekend with a trolley-based tale from the On Call vaults. This week's story was related to us by a reader we will refer to as "Stan", who insisted to us that today's protagonist wasn't him. Oh no. It was a "colleague who will remain …

  1. GlenP Silver badge

    Not Me But...

    Apparently if you left a box of fanfold paper on the floor in front of certain IBM kit (probably the same controller) it was quite possible for an operator to head-butt the reload button whilst picking up the paper. As the operator realised what they'd done before standing up they were left in that position whilst an orderly shutdown was arranged.

    1. Anonymous South African Coward Bronze badge

      Re: Not Me But...

      ...whilst an orderly shutdown was arranged.

      How long did an orderly shutdown take back in the day?

      And how long did it take to get everything booted up after an orderly shutdown?

      Because I'm just curious to know, give that we have PC's and servers that can now boot up and be operating in less time to down a cuppa coffee.

      1. GlenP Silver badge

        Re: Not Me But...

        Honest answer is I don't know. I was a DEC VAX manager back then, those would take around 30 minutes.

      2. jake Silver badge

        Re: Not Me But...

        Your questions can't really be answered without specifying hardware, OS. configuration and how many ton(ne)s of peripherals were involved.

        However, as asked, one correct answer to either question could be "Anywhere from a couple minutes to well over a day".

        1. hplasm
          Devil

          Re: Not Me But...

          " However, as asked, one correct answer to either question could be "Anywhere from a couple minutes to well over a day" "

          Sounds like a Windows user...

          (I know you aren't :) )

          1. Anonymous Tribble

            Re: Not Me But...

            I used to manage a couple of WinNT3 servers with 256MB of RAM. They took approximately 45 minutes to reboot. Most of that was performing an extended memory test, and no, there wasn't an option for a fast test or to skip it.

            That was coupled with a memory leak in IIS that meant I had to reboot those servers every four weeks and that had to be done outside of service hours which meant between 2am and 4am. Nice bit of overtime for a while :)

      3. hmv

        Re: Not Me But...

        Whilst waiting for a Sun E450 to power up (with diagnostics turned on), which took a while, a Sun engineer told me that a Sun E10000 that he'd powered up with diagnostics turned on took 8 hours to get to the point where it could start the operating system.

        1. fozzy73

          Re: Not Me But...

          Yep, and more often than not it would startup with something broken (in one shutdown-startup cycle we lost a switch, 3 HDDs and a fan).

      4. VTAMguy

        Re: Not Me But...

        > And how long did it take to get everything booted up after an orderly shutdown?

        A few minutes at most, either to reload a 3705 or even to reboot an MVS or VM system. There were no Terabytes of disks to fsck, there were no Gigabytes of useless libraries to load, and remote data communications paths rarely had bandwidths greater than 9600bps. (You could monitor data trafffic by displaying it on a DataScope and watching it go by.) The size of everything was expressed in K or M. The machines were very slow but everything in the IPL path was hard coded assembler language purpose built to get things running. The mainframe OS and the 3705s were independently restarted as needed.

        Also, the 3705 was anything but a "simple beast". It was a computer with a complex OS in its own right, supporting async (aka Start/Stop), bisync (aka BSC, for older 3270 terminals), and NCP for native SNA devices like newer 3270 clusters. Larger shops had multiple person groups of systems programmers who were dedicated to the 3705s (we had about 10 scattered across various data centers). We had very imaginative names for them: I, J, K, etc. Cabling was huge and unwieldy, and duct space was at a premium in many areas.

        And yes, the 3705 load button could have been accidentally clunked by an IBM CE documentation cart, but those were big and heavy and rarely moved. Usually the CE would just go get what was needed as it was easier. I do not remember the castors on those carts being crap; if anything they were extra heavy duty like everything else IBM manufactured. More likely was the cart hit a bump in the raised floor tiles and veered off course. I could see that. CEs screwed up just like all of us other humans did.

      5. henryd

        Re: Not Me But...

        Back in the day the console of the IBM360 had a big red button to set off the fire extinguishers in case of fire.

        All operators had instructions Never Press The Button except in case of genuine need.

        It was in the 70’s when I worked in the computer Centre of a major university and, naturally, could not resist temptation..

        This caused a crash stop, not an orderly shutdown. I recall it took around 3 days to get back up.

        1. VTAMguy

          Re: Not Me But...

          Nah, that big red button (which was actually a pull switch to make it less likely to get hit accidentally) just shut off power to the system; you had to trip the fire alarm separately, usually with another big red button, ideally located by an exit door. And if the emergency power off switch was pulled, it had to be reset by an IBM CE.

    2. Captain Scarlet
      Trollface

      Re: Not Me But...

      I suspect the BOFH from giving the PFY some unexpected training

      1. Captain Scarlet

        Re: Not Me But...

        *I suspect the BOFH is giving the PFY some unexpected training

        *Damn and blast it always takes me an hour to realise what I have written contains mistakes >_<

  2. Chris G

    Gone fishing?

    A dodgy caster would be putting the bait in the wrong place, a castor however, may steer you wrong.

    1. chivo243 Silver badge
      Coat

      Re: Gone fishing?

      That's sweet too, caster...

      My coat with the caster sugar packets in the pocket

    2. Sgt_Oddball

      Re: Gone fishing?

      Also helps if you remember the castor oil on said castors. Preventative maintenance is your friend and all that.

      1. Outski
        Pint

        Re: Gone fishing?

        "Preventative maintenance is your friend and all that."

        But percussive maintenance can be more satisfying...

        1. Mark 85

          Re: Gone fishing?

          But percussive maintenance can is always be more satisfying...

          FTFY.

          1. bob, mon!

            Re: Gone fishing?

            I dunno --- these were CHAIN printers, with hammers for printheads - literally. In a percussion contest you would probably lose.

            1. StephenH

              Re: Gone fishing?

              When I was at Uni, a common challenge with chain printers was to find out the sequence of letters on the chain and then print something that would cause all the hammers to strike at once, repeatedly.

              1. Simon Reed
                Facepalm

                Re: Gone fishing?

                On our college band printer, I thought I'd be clever and edited my source code file to make every zero be a 0, backspace, then / so the numbers came out thus:

                ØØØØ1Ø IF MY-VAR > 1ØØØ THEN

                The / is the character before the 0. Instead of the paper fair hurtling through this high speed band printer, it suddenly stopped when it got to my job, requiring a full revolution of the band for every '0'. It sounded very impressive.

                B-BANG. pause. B-BANG. pause. B-BANG. pause. B-BANG. pause. B-BANG. pause. B-BANG. pause.

                It even attracted the attention of the head of department from his office at the end of the corridor.

                It was interesting to watch, but made it slower than a dot matrix. And made me very unpopular since we were running off our listings for handing in.

                1. Ken Moorhouse Silver badge

                  Re: I thought I'd be clever...

                  Were you told that that was a very noughty thing to do?

      2. JJKing
        Happy

        Re: Gone fishing?

        Preventative maintenance is your friend and all that.

        Thanks, oil remember that.

    3. jake Silver badge

      Re: Gone fishing?

      Oh, I dunno. Having incorrect caster can make your steering ponderous. In extreme cases it can prematurely wear wheel bearings ... which in turn could cause the reported squeaking.

      Source: my 1915 Case traction engine, now sorted out & happy again.

      1. Anonymous Coward
        Anonymous Coward

        Re: Gone fishing?

        Er no. Castor is steering. And Pollux

        Caster is sugar and a fishermen, or wizard, Or tricoteur.

    4. Anonymous Coward
      Anonymous Coward

      Re: Gone fishing?

      A dodgy caster would be putting the bait in the wrong place, a castor however, may steer you wrong.

      Is that castor canadensis or castor fiber? I really had no idea either had a thing about castrating bos taurus.

      I should really be more careful around the little buggers.

    5. Montreal Sean

      Re: Gone fishing?

      Beware the castors in Quebec, they bite.

      But their tails are tasty!

      https://beavertails.com/products/

      1. bpfh
        Joke

        Re: Gone fishing?

        So you really can eat a beaver?

        1. jake Silver badge

          Re: Gone fishing?

          Sure. It's a fish, ask any Catholic. Enjoy!

          Apparently, Alligator is also a fish ...

          1. Anonymous Coward
            Anonymous Coward

            Re: Gone fishing?

            Also, hungry medieval abbots were not above re-baptising geese as pike on a Friday.

            1. jake Silver badge

              Re: Gone fishing?

              It's more recent than medieval ... Apparently, it was all the way back in 2010 that the Archbishop of New Orleans decided that alligator is considered to be in the fish family.

              Seems to me that muskrat is also considered to be a fish in some quarters.

            2. Diogenes

              Re: Gone fishing?

              pelicans swans, & ducks were also regarded as fish monkish purposes -

              They are on water aren't they ?

        2. JJKing
          Facepalm

          Re: Gone fishing?

          So you really can eat a beaver?

          Yes but not recommended at certain times of each month.

          1. eionmac

            Re: Gone fishing?

            Whose month? eater's or beaver's? also why?

    6. Anonymous Coward
      Anonymous Coward

      Re: Gone fishing?

      A dodgy caster would be putting the bait in the wrong place, a castor however, may steer you wrong.

      What a load of Pollox....

  3. Evil Auditor Silver badge
    Pint

    That seems to fit rather Monday's Who, me? column. But it's Friday and any good story will do!

    1. John Brown (no body) Silver badge

      In this case, more of a Who, him? though, but yes, any port in a storm :-)

  4. Pascal Monett Silver badge

    Stan, Sid, let's make it simple, okay ?

    Instead of trying to find a new name every week, let's just convene that the week's On Call guy is Brian and the week's Who Me guy is, I don't know, George or something.

    In other words, just keep the same name every week, that way you can just start the column by "This week's Brian tells us about how . . ."

    What do you think ?

    1. Anonymous Coward
      Anonymous Coward

      Re: What do you think ?

      It's an excellent idea. But I think that "This week's Pascal tells us ..." has a nicer ring to it :-D

      1. Andrew Moore

        Re: What do you think ?

        We can't be sexists either: If we are going with Pascal, then we should alternate with Ada.

        1. phuzz Silver badge

          Re: What do you think ?

          Why not call them Alex, Chris or Sam? All unisex names.

        2. Paul Johnson 1

          Re: What do you think ?

          Well as we are dealing with secret identities the obvious names are Alice and Bob.

          1. Anonymous Custard
            Alert

            Re: What do you think ?

            Nah, it should be Charley for such dire warning cases...

    2. IanTP
      Coat

      Re: Stan, Sid, let's make it simple, okay ?

      Anyone else remember Bob from The Micro User and his monthly tales of woe?

      Mines the one with the copy of BBC Basic in the pocket...

      1. Will Godfrey Silver badge
        Happy

        Re: Stan, Sid, let's make it simple, okay ?

        Yep.

        And I also remember 'Henry' in Wireless World circa 1965

        1. Ken Moorhouse Silver badge

          Re: 'Henry' in Wireless World

          Meanwhile at Radio Constructor, Dick and Smithy were the resident troubleshooters.

    3. Anonymous Coward
      Anonymous Coward

      Re: Stan, Sid, let's make it simple, okay ?

      For this story they missed the obvious 'Stan' and, totally different colleague, 'Olly'

      1. hplasm
        Coat

        Re: Stan, Sid, let's make it simple, okay ?

        It was Stan and Trolley...

      2. Killfalcon Silver badge

        Re: Stan, Sid, let's make it simple, okay ?

        Our z-series mainframes have 4-letter names in prod/pre-prod pairs. Turns out comedy duos are excellent for this: ERIC, ERNI, STAN, OLLY, TOAD, RATY...

        1. Ken Moorhouse Silver badge

          Re: ERNI

          Can you do me a favour? If I gave you a number, could you ensure it appears on the next output run please?

    4. jmch Silver badge
      Thumb Up

      Re: Stan, Sid, let's make it simple, okay ?

      Well, "Brian" DOES have a suitably impressive theme song in 'James Bond intro' style, so there's a plus side right there

    5. Swarthy
      Thumb Up

      Re: Stan, Sid, let's make it simple, okay ?

      I like the idea, but I think that Monday's character should be Brian, because he's just a very naughty boy.

    6. still-me

      Re: Stan, Sid, let's make it simple, okay ?

      In the present climate, lets just call both of them Boris ??

    7. SoaG

      Re: Stan, Sid, let's make it simple, okay ?

      If the goal is to keep it simple we should just call them all Bruce.

      1. Charlie van Becelaere

        Re: Stan, Sid, let's make it simple, okay ?

        Bruce FTW.

      2. JJKing
        Boffin

        Re: Stan, Sid, let's make it simple, okay ?

        If the goal is to keep it simple we should just call them all Bruce.

        Then all the incidents happened at the University of Woollamalloo.

        1. Martin-73 Silver badge

          Re: Stan, Sid, let's make it simple, okay ?

          And now I am successfully earwormed, thanks to all

    8. JJKing
      Coat

      Re: Stan, Sid, let's make it simple, okay ?

      ....... let's just convene that the week's On Call guy is Brian ........

      Or maybe Bwian and Wodowich......

      Mine's the one with the large snake called Monty in the pocket

    9. the Jim bloke

      Re: Stan, Sid, let's make it simple, okay ?

      How about we call them 'Bruce', just to keep things clear..

    10. LongtimeLurkerNewbieCommentard

      Re: Stan, Sid, let's make it simple, okay ?

      I'm Brian and so's my wife...

  5. Michael Kean

    Not all serial cables are made equal...

    So, back in the'90s I was asked by a third party to check the status of the battery in a UPS at a remote location.

    I guessed a null modem cable attached to the serial port on this APC UPS would let me talk to it on 9600,N,8,1.

    I guessed wrong. Immediately upon connecting the serial cable to my laptop, the rack became much quieter.

    UPS, or OOPS?

    We were all young and naive once.

    1. Andrew Moore

      Re: Not all serial cables are made equal...

      Yeah, APC and it's propriety "serial" port.

      1. Down not across

        Re: Not all serial cables are made equal...

        Yeah, APC and it's propriety "serial" port.

        Well it is a serial port. Just happens to have non-standard pin configuration.

        As I've said before

    2. Mr Army

      Re: Not all serial cables are made equal...

      I have posted previously about this. In my case it was under the stage at Glastonbury Festival while Slash's Snakepit were playing. Off went the UPS, the lighting control network and the intercom powersupply. Behind the rack with the band playing above my head I had no idea all the lights had gone off and the operator out front was screaming into a dead comms headset.....

      1. Anonymous Coward
        Anonymous Coward

        Re: Not all serial cables are made equal...

        Glastonbury was always great for that stuff. Had a similar experience with all the video projectors in the dance tent randomly scrolling video output (all 12 of them) after the BBC decided "that one is perfect" and plugged into our power feed. Not great an hour before the Chemical Brothers go on stage.

  6. gordonmcoats

    trolleys in machine rooms..

    Not exactly similar, but in the late 80s - and also a bank, we had a file controller in doing overtime on a saturday. While pusing his lade trolley full of 12" reels he managed to trap his hand between the trolley and one of the angled metal brackets on the side of one of the tape racks that held the shelves in place.

    The sudden and unexpected blood loss was described as "impressive"

    (I should add, he did actually make it back to work on the monday - heavily bandaged..)

    1. Flocke Kroes Silver badge

      Back when I were a boy...

      My summer holiday job involved feeding a machine that turned rough planks into smooth planks and a barn full of saw dust. Inside there were lots of sharp things spinning very fast and occasionally a plank would get stuck part way through. The fix (as demonstrated by the machine's owner) was to shot-put the next plank in as hard as possible. Normally your left hand remains on the front end of the plank up to the last possible moment to accurately guide it to the input slot. Near the end of a long day I got a little careless and did not get my left hand out the way quick enough.

      By the time I got home my blackened finger tips were so painful I applied the obvious remedy: straighten a paper clip, hold it with pliers, heat it on the cooker til it glows orange and burn a hole through each black finger nail to relieve the pressure. The smell was rather bad but the pain did reduce until the nails dropped off a few days later.

    2. Anonymous Coward
      Anonymous Coward

      Re: trolleys in machine rooms..

      Seem to recall reading that when they started locking doors in the CS dept at Berkeley that Richard Stallman used trolleys to break them open on the grounds that software was meant to be free so you shouldn't keep it behind locked doors.

      1. jake Silver badge

        Re: trolleys in machine rooms..

        I saw Stallman speak at Berkeley once or twice. The doors to the "computer center" (EECS) weren't locked (the back door didn't even have a lock on it in those days!), and we actually invited him to visit. Not that there was really any one central location, mind ... We were distributed pretty much all over the campus, and beyond.

        Berkeley has had more than it's share of nutters as staff and alumni, but RMS isn't one of them. He's a Harvard/MIT rightcoastian loon.

        For Berkeley's take on FOSS, look no further than BSD.

  7. big_D Silver badge

    DEC Engineer

    We had a bank of VAX minis. The engineer came out to do a memory upgrade on a machine in the middle of the row.

    The ops moved all the jobs and users to the next machine in the line, stopped all processes and shut it down... Well, it doesn't power off, it displays a message on the master console that it has stopped.

    The engineer looked at the message, went off behind the cabinets that were the VAX and found the master power switch... And threw it.

    He then popped out again and the ops looked at him, looked at the terminal saying "stopped" and looked at the engineer again...

    Then the screams started! The ops from the next machine. You know the one? The one where all the users and jobs had been shoved! They were looking at a grey, dead ops console...

    Yep, he'd thrown the master switch for the wrong machine.

    1. Martin
      Facepalm

      Re: DEC Engineer

      Pah! only ONE server?

      A friend of mine (really!) was once decommissioning an array of servers in a machine room. He'd carefully shut them all down, and then went into the data centre, and walked along a row of server racks, powering off each server in the rack with the switch on the front.

      Only it was the wrong row....

      1. hmv

        Re: DEC Engineer

        Having done something similar (although not to a whole row of racks), I'm most insistent that servers are labelled front and back.

        1. eionmac

          Re: DEC Engineer

          I also insist that ALL wires into machines have a ID tag at both ends and also part way away from machines. When de-commissioning a server, we had a surplus Ethernet split serial cable in machine. Pulled, ah India (service desk) was disconnected.

    2. Anonymous Coward
      Anonymous Coward

      Re: DEC Engineer

      "One more turn, Delboy"

      "Brace yer self, Rodney!"

    3. MGJ

      Re: DEC Engineer

      Late 90's, about to do our disaster recovery testing one weekend. Scenario involved a power cut to the server room, which I simulated by switching off the power to the room going into the UPS, to check that it did indeed initiate safe shutdown of all the Sun, Fujitsu and HP kit in the room. It was only at that point that it became clear that the electricians had run the electrical supply in parallel to the UPS rather than in series... It all went very quiet. Except for the UPS.

      1. Anonymous Coward
        Anonymous Coward

        Re: DEC Engineer

        At a large genome sequencing organization that shall remain nameless, the line that detected loss of incoming power and fired up the diesels was connected to the wrong side of the generator changeover relays. One weekend they tested the automatic generator start by shutting down the incoming mains; the circuitry detected it, started the generator, detected power again, and switched the generator back off!

        1. Anonymous South African Coward Bronze badge

          Re: DEC Engineer

          the circuitry detected it, started the generator, detected power again, and switched the generator back off!

          Thats' what happened to us in the one building - sparky wired the genny up incorrectly.

          We found that by tripping the switch going to the genny's managerment console on the main DB let the genny continue chugging on during a power failure.

        2. Peter2 Silver badge

          Re: DEC Engineer

          On the bright side, you found out by testing it, rather than in an emergency.

    4. Anonymous Coward
      Anonymous Coward

      Re: DEC Engineer

      Not related to trolleys but mention of DEC reminded me of a story of how when a couple of decades ago when FPGAs were relativley new and expensive that one of the DEC reasearch labs had a (by standards of that time) massive FPGA setup - unforrtunately they doidn't realize one of the disks in the system controlling it was failibg and they downloaded a corrupted programming config onto it and, as doubtless at that stage designers hadn't considered possibility of such a major programming error and as a result destoryed most of the array (seem to think "quarter of a million dollars" was mentioned)

    5. SImon Hobson Bronze badge

      Re: DEC Engineer

      At a previous job, "some decades ago" (so back in the days when these devices constituted the contents of a significant chunk of rack space) a colleague told me how someone he knew had been employed by DEC testing processors prior to assembly into systems (or something like that). So he had this 'kin big PSU at one end of the bench and some very thick cables to daisy chain along the power terminals of the systems under test.

      Then one day he (literally) got his wires crossed - when he flicked the switch, he let the magic smoke escape from a whole bench-full of processors. Apparently the insurance paid out - but with a condition that said person was re-allocated to a" less expensive if he gets it wrong" role.

      But more directly related to the "switched the wrong thing off" part. At my last place one of the hell desk guys was talking a customer through power cycling a server. Consulting the notes/diagrams, the hell desk told the customer it was the bottom server in the rack - press and hold the power button. Of course it was the customer's fault for not recognising the difference between the bottom server in the rack - and the UPS below it that ran everything !

    6. Vometia Munro Silver badge

      Re: DEC Engineer

      Done a not entirely dissimilar thing myself. Shut down the computer using its console, turned the reset key and was surprised to see the console not spring back to life. But out of the corner of my eye, another one did.

      Partly my fault for not checking more thoroughly that the console's and computer's microscopic labels matched up since I'd previously admonished the ops staff weeks previously for moving stuff like consoles about and just plopping them wherever it was convenient, such as on the roof of a completely unrelated system, but no, I was sticking my nose where it wasn't wanted. Fortunately(-ish), further incidents were precluded shortly afterwards by replacing them all with a rather peculiar serial-based LAN contraption controlled by a master console programmed in COBOL.

    7. Stuart Castle Silver badge

      Re: DEC Engineer

      Friend of mine did something similar, but it wasn't a Vax or some other server hosting jobs for users. It was one of the machines involved in hosting ntl,com, and as a result, he took out the website for a while. I seem to remember El Reg mentioning it in an article at the time..

      Thankfully, the nearest I have come was a few years ago. One of our switches had crashed, so I went to the comms room to reboot it (which involved pulling the plug as I could not reach the power switch). There were a lot of switches in that room, so a lot of power cables, none of where were labelled. I traced the cable through to what I thought was the correct socket, pulled the plug and the wrong switch went off. Luckily I was able to plug it back in and get the switch booted up before anyone noticed. Surpising, considering it was a 24 port switch and every port was in use.

  8. Just Enough
    Headmaster

    I don't think that means what you think it means

    "they never did find the reason for the outage and put it down to experience."

    You can only put something down to experience if you learn something from it. As they did not, they cannot.

    Sid, on the other hand, can. He learned to be more careful with the trolley.

    1. Dave314159ggggdffsdds Silver badge

      Re: I don't think that means what you think it means

      Experience can include finding out that shit sometimes does inexplicable stuff.

  9. Dave314159ggggdffsdds Silver badge

    Brought the house down

    Changing a lightbulb in the office, standing on a swivel chair, I grabbed the burglar bars over the window for support. They came down, bringing part of the wall with them in a shower of bricks and rubble. None of the brickwork was load bearing, and the builders had skimped on cement for the mortar. The structure was a deathtrap, although the company bosses were more worried about security, given that thieves could kick their way through a wall.

  10. Shadow Systems

    I have BOFH fantasies...

    I'll have a server being a pain in my arse & needing a bit of percussive maintenence which I dutifully provide by bludgeoning it with a dull, blunt, worthless object...

    Namely my boss' pointy head.

    Ok, the server seems to be working again, but where did I leave the carpet & quicklime?

    1. Anonymous South African Coward Bronze badge

      Re: I have BOFH fantasies...

      Ok, the server seems to be working again, but where did I leave the carpet & quicklime?

      Didya checked inside the Head Beancounter's boot.... ?

      1. Shadow Systems

        Re: I have BOFH fantasies...

        There's no need to check his trunk, I already know his body is securely inside the Duct taped roll, buried like a cat turd in quick lime kitty litter, & the vehicle eroding at the bottom of the quarries acid pool.

        *Cough*

        Did I just say that out loud?

        Gotta run, I hear some noisy pounding on the server room door!

  11. Anonymous Coward
    Anonymous Coward

    Tale as old as Time...

    An Ex-Colleague of mine had a similar experience in a financial institution. Casually slid a trolley to one side only to see (too late) that it was at the perfect height to flip the key in an old Sun Server that had been up for years and years.....

    The whole place ground to a halt, overnight bulk transactions failed....chaos...

    But, he got away with it.

  12. hplasm
    Thumb Up

    Ah Pascal! What you have wrought!

    So many anonymous ops, now identify as Brian, happily Out of the Rack.

    You can be Proud!

  13. zb42

    Big lead-acid batteries, as used in large UPS systems, are really heavy.

    A person who routinely tried to cut corners, usually resulting in aggrieved customers and the rest of us having to do much more work than it would have taken to do things properly the first time, borrowed a milk trolley, a five foot tall metal cage on wheels, loaded it with about 150KG of batteries, got it going down a slope at speed then ran over his own foot.

    His broken toes caused him a lot of pain for a while but did not improve his attitude.

    1. Dave314159ggggdffsdds Silver badge

      We had a cage full of pcs that we'd just replaced. The bank's facilities guys insisted that as soon as the cargo lift doors opened on the loading bay level, moving it was something only they were allowed to do. They promptly moved it down a slight ramp and off the edge of the loading dock. I let them sweat for a few minutes before I admitted the computers were on their way to be recycled.

  14. d3vy

    Ive never done on call work as such - but Ive caused more than my fair share of call outs..

    An old client was in the process of migrating their web apps to windows 2003 and the company I was with at the time was in charge of doing the web migration, two new web servers had been purchased and had been sat at the back of our office for the last week being set up and tested - We hadn't quite finished the set up before the courier had arrived so I stayed in the office *very late* and rdp'd onto the machines to finish the setup... IIS6 needed a reboot (Cant remember what for but it did) so I dutifully pulled up the start menu and restarted the machine, while I waited for it to come back up I carried out the same process on the second machine.

    They were taking a while to come back online so I popped off to get a coffee - it was VERY late by now so I was pretty tired. When I got back they were still not responding - I then realised what I had done - I shut them down instead of rebooting them. It was a small company, the DC in Guernsey was unattended at that time of night so I had to wake some poor guy up to sort out my mess.

    Then, not having learned my lesson I did the same think the next month and switched off the skybuy* load balancer - Luckily that was hosted on site so it was just a case of running to the other end of the building.

    *skybuy used to be the site that sky used for selling their merchandise, spare remotes and that kind of stuff - quite high traffic, the kind of site that a few minutes outage would have been quite noticeable.

    1. Olivier2553

      I did something quite similar not later than yesterday evening. We had a power failure during the week-end and the systems had not restarted. So I dropped by on my way back from a remote weekend.

      The access to the server room is protected by a password operated door, linked to our LDAP. LDAP was done, door was not openable. And also the network was acting funcky for lack of proper DNS. But I have a key to override the door mechanism. So I could gain access to the server room and start doing my magic.

      When I left the server room, everything was running fine but the door system. I left and oops, I had left my bunch of keys inside the locked room.

      My first idea was to way until the morning, when the system would disengage (no password during day time) and I could walk in. Well, I started by scratching my head, then I considered waiting for the morning. But luckily I could finally SSH into the door control mechanism, disable it temporarily, go and grab my keys and re-enable the door security.

  15. john.w

    When audio mixing consoles went digital

    Launching one of the first digital mixing consoles at IBC (International Broadcasters Convention) in Brighton one year the digital processing card stopped working. Designer of said prototype card had decided to visit relatives in Scotland as he was not required at the show. Called back the 500 miles he dutifully drove down over night. The cards were 6U high and made using a new automated solder wrap system using fine wire that was wrapped and soldered directly onto the sockets of lots and lots of DIL ICs. Valiant young engineer gets out the oscilloscope, clips on probe ground wire and shorts out the 5V supply wire creating a wonderful plum of smoke all around the card. Shock turned to applause and the sales team are left to fake it with an audio console that was processing no audio.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like