back to article Software engineer sobers up to deal with 2:00 AM trouble at mill

It's Christmas Eve Eve, which means we're assuming you don't particularly want to read industry news right now and so we're instead offering an extra instalment of On-Call, our regular reader-contributed tales of nasty jobs at nasty times in nasty places. Today's yarn comes from reader “BT”, who told us that “More years ago …

  1. ForthIsNotDead Silver badge


    Dear BT

    Your boss was a twat of the highest order. And a crap coder, too! You're better out of it.

    I have a couple of cracking anecdotes to share, including a 'men in black' type moment that happened in Signapore, but I don't know where to send them.

    1. Martin Summers Silver badge

      Re: Twat!

      Link at the bottom of the story. Or you can normally click on the author name on the full fat reg site.

      1. I. Aproveofitspendingonspecificprojects

        Re: Twat!

        my definition of a Twat is a fellow who get called in more than once for the wrong shift without replacing his beeper with one from the same manufacturer as his boss.

  2. CT

    There are still steel mills in Rotherham

    see title

  3. Anonymous Coward
    Anonymous Coward

    AC for obvious reasons but...

    I was called out to fix very similar code although RPG2 not FORTRAN.

    Was 11000 lines of unstructured mess with about 120 listed modifications (and the base code was so old it was written when I was still at junior school). Auto looping, jumps, switches, subroutines calls, and yes, the jump out of loops gem...Comments of course could relate to the original code or any iteration of the modifications so were not a lot of help even if they hadn't simply repeated the code rather than hint at its purpose. Why N90 switch (which turned out to be the problem ), What is X? why do these routines nest so deeply?

    I was also rather foggy from previous night's festivities and the problem could not be replicated in a test environment.

    Jump through hoops to allow debug in production to find the 1 character change (remove an N) after about 2 hours of ticking back trace data against the source.

    1. RubberJohnny

      Re: AC for obvious reasons but...

      Coders and developers don't usually do on call do they? Seems like a good way to make sure you get reliable code.

      Except I was doing on call some years ago, and the devs were contactable as a last resort. One in particular used to write deliberate bombs into the code and then remind me that he was available if needed and to not hesitate to call, because he was on £120 remuneration if called out, and was not pleased that I figured out how to work round his "bug".

  4. Anonymous Coward
    Anonymous Coward

    I worked for a boss that tried to code...

    I replaced it...

    Although on a different job on my very first day on call ever for a system I had never supported my partner did not respond to calls or pages.

    I managed to find the solution and rectify the batch process to be told "its a great learning experience" the following day.

    1. Doctor Syntax Silver badge

      Re: I worked for a boss that tried to code...

      The code was C but not as we know it.

      The boss was a COBOL programmer and used the pre-processor to make C look as much like COBOL as possible. Some of the changes needed were in code expanded from macros but not all instances from a given macro needed changing. I just ran the whole thing through cpp and treated the result as version 1.

    2. Someone Else Silver badge

      Re: I worked for a boss that tried to code...

      I replaced it...

      Replaced the code...or the boss?

  5. Anonymous Coward
    Anonymous Coward

    One (in)famous easter bank holiday weekend...

    Ended up fielding over 50 calls calling in team members that were not on call, even had my mum answering the phone so I could get on with the fixes as it was supposed to be family time. 5 different countries down, all for different reasons.

    I did NOT get to enjoy a long weekend that year.

  6. RIBrsiq

    If not on-call, just don't answer

    Strictly-speaking, that's not the Right Way to behave, of course. And feel free to behave properly if your particular working environment warrants it -- and I am sure many here would appreciate it if you let us know which paradise you work at so they can try and join you -- But this bit here:

    "Sadly the mill wasn't willing to take BT home, so he ended up sleeping on the control room floor".

    ...shows just how little appreciation there is for someone going out of their way to effectively save the day.

    The only way crappy business practices will be fixed is if they impact the bottom line hard. And that will not happen if you answer the phone and fix the problem even when you are not on-call.

    1. Doctor Syntax Silver badge

      Re: If not on-call, just don't answer

      "The only way crappy business practices will be fixed is if they impact the bottom line hard. And that will not happen if you answer the phone and fix the problem even when you are not on-call."

      Except that when they're fixed you might not be there to take advantage.

      1. Anonymous Coward
        Anonymous Coward

        Re: If not on-call, just don't answer

        That reminded me...

        Worked on bank holiday after being called in...

        PHB: The call out rate is over my remaining budget, wouldn't it be nice if you could just take today off in lieu?

        Me: Isn't it double time on bank holiday, so that would be two days off in lieu.

        PHB: Harrumpf. I'll pay the callout

        1. Chris G Silver badge

          Re: If not on-call, just don't answer

          We have three people on call out and all three of them are off the island, I was asked 'Are you going away for Christmas?'


          'Then you won't mind coming out if we have a problem?'

          'I would be pleased to help if we can arrange suitable payment'.

          'Oh, what do you call suitable payment?'

          'Well, when I last diid call out, I was getting triple time for Christmas, at a guaranteed minimum hours, if it went over the rate went up,Oh and I got a day off in lieu as well.'

          ' Umm! We'll let you know'.

          Somehow I think I'll have a quiet weekend.

        2. Chris King Silver badge

          Re: If not on-call, just don't answer

          When leave and/or TOIL piles up and you're in a "use it or lose it" situation, that can focus management attention somewhat...

          Me: I've just checked my leave calendar, and I've worked out that if I don't start taking leave this afternoon, I will lose some of it.

          Boss: But it's the last week in August, you surely don't have over a month's worth of leave to take ?

          Me: I do, and that calculation takes the five-day carry-over into account. Here's my leave request, see you first week in October !

      2. RIBrsiq

        Re: If not on-call, just don't answer

        "Except that when they're fixed you might not be there to take advantage".

        Probably very true, that.

        But this is a bit of a meta for-the-greater-good thing, really.

        Though now that I think of it, business entities with shoddy business practices are probably not long for this world, anyway, so you're out of a job in the long run no matter what you do.

        Now, I *am* a realist, so I realize that what most people will actually do is smile and take it while looking for a better job elsewhere. Except that since everyone else is doing exactly the same thing, they may as well look for a unicorn, for all the good it will actually do.

        And so we are doomed to forever keep abandoning eternally sinking ships. Hopefully in time to avoid personal disasters.

      3. This post has been deleted by its author

    2. DiViDeD Silver badge

      Re: If not on-call, just don't answer

      "Strictly-speaking, that's not the Right Way to behave, of course"

      Don't know about that. One of the best call logs I ever read was from an engineer at BP who wasn't on call when the <ahem> call came.

      Issue: Noise coming from pager

      Base cause: Batteries discovered in pager.

      Resolution: Removed batteries from pager.

    3. herman Silver badge

      Re: If not on-call, just don't answer

      No, that was silent retribution from the transport manager for swearing at the boss...

    4. Tom 13

      Re: The only way crappy business practices will be fixed

      While I commiserate with that fact, my read on this one is that BT would have gotten canned because he didn't cover for The Boss.

  7. thedroog

    What a hero

    Trouble at Mill was it???

    1. chivo243 Silver badge

      Re: What a hero

      The Old Mill? There's trouble down at the Old Mill? No?

      1. J.G.Harston Silver badge

        Re: What a hero

        And we woulda got away with it...

    2. Alister Silver badge

      Re: What a hero

      Trouble at Mill was it???

      Aye lad, one on't cross beams gone owt askew on't treddle.

    3. Anonymous Coward

      Re: What a hero

      > trouble at mill.

      Such a faux pas should not go uncorrected.

      As all men know, the proper phrase is "trouble a' mill".

      Can we have an icon of a teacher with the other kind of flat cap please?

      1. perlcat

        Re: What a hero

        When there's trouble at the mill, you always should expect an inquisition.

  8. Martin Summers Silver badge

    Just recovering from a mini panic attack thanks to reading "it's Christmas Eve" and automatically skipping the double 'Eve'. Still safe from the in laws for another day, phew!

    1. Anonymous Coward
      Anonymous Coward

      Obligatory xkcd reference gets me through many a dull meeting with the in-laws. That, and sherry. The drink, not the horny aunt.

  9. Anonymous Coward
    Anonymous Coward


    During my 3 days off so far I've been interrupted by work every day, best one being asked for the machine administration password so they could install Spotify and listen to Christmas tunes. It was not provided naturally.

    1. chivo243 Silver badge

      Re: Christmas

      I've been on vacation since Friday, and went in yesterday for a few hours. Vacation plans changed so I wasn't in the south of France and was able help out.

      1. Anonymous Coward
        Anonymous Coward

        Re: Christmas

        I didn't see you there or even know you worked at my place Chivo but thanks for helping out. ;-)

    2. War Puppy

      Re: Christmas

      Grinch stole Crimbo.

    3. RubberJohnny

      Re: Christmas

      Tuesday was my last day this week, off all christmas until 4th Jan.

      On Tuesday afternoon something major broke and I had visions of my time off being delayed, working late and the next day.

      I found myself extremely focussed and concentrated on my task that afternoon.....

  10. jake Silver badge

    It's not Xmas eve eve ...

    ... It's the Boxing Day after Solstice.

  11. jake Silver badge

    Where have I slept when on duty?

    In the early/mid 1980s it was usually cat-napping under my desk when the marketards turned the "shipping forcast" into something resembling a hockey stick at the end of a calender quarter ... Three weeks of 18 hour days ain't exactly fun. Went on every three months for nearly two years.

    I was single, in my mid-20s, and the money and comp-time was better than good, so I put up with it ... Never again, though. That's a young man'sperson's game.

    1. I. Aproveofitspendingonspecificprojects

      Re: Where have I slept when on duty?

      I don't suppose there is an explanation of what I think I read is there?

      An hazarded guess would do.

  12. Anonymous Coward
    Anonymous Coward

    One of the big benefits of spending Christmas with the folks...

    Aside from my mother's excellent cooking, is that their house has absolutely no mobile signal at all on any network.

    Therefore, if something does go bang and I'm not on duty, I don't get to hear about it. If there is an urgent problem that needs my input, my manager has my folks' home number and he's sensible enough not to call it unless something's on fire.

  13. Destroy All Monsters Silver badge

    This should not go under "ON CALL"

    This should go under "BOSS CODE"

  14. RyokuMas Silver badge

    This sort of thing...

    ... is why my mobile number and personal email addresses are sacrosanct, and my personal laptop is not configured to connect to the company network...

  15. Your alien overlord - fear me

    My best one was post staff party, I was working (3 shift rota at a bank). I was so pi$$ed I couldn't remember leaving the party, let alone doing my entire departments nightshift work *and* the morning shift stuff. They appreiciated it being under the weather themselves and when asked about it when I came in the following night, I just told everyone the stuff was so repetitive I could literally do it unconscious.

    1. Someone Else Silver badge

      @Your alien overlord - fear me

      I was so pi$$ed I couldn't remember leaving the party, [...]

      You were angry leaving the party?

      Oh, wait...Brit usage of "pissed". Never mind....

  16. gotes


    Drunk, in the middle of the night, under intense pressure attempting to debug someone else's code. Amazing.

    Still, I'd have sent that artic round to the boss's house.

    1. Destroy All Monsters Silver badge

      Re: Heroic

      Amazing? Positively Russian.

  17. Archivist

    Non IT Callout

    I was woken by a phone call from the police to say that the company alarm was sounding. Fair enough, I lived closest.

    Because I was so close, I got there moments before the police did, and as I placed my security key in the alarm panel, a police hand grabbed me and pulled me back through the front door, which slammed.

    I was left with no keys, or ID. They didn't believe any of my protestations.

    The person next on the key-holder's list lived 40 minutes away. It was bloody freezing. Try waiting 40 minutes in winter in 1 layer.

    1. Anonymous Coward
      Anonymous Coward

      Re: Non IT Callout

      "A likely story, my lad. You look a bit terrorist, too, come to think of it. And a bit lightly dressed for this time of the year."

    2. Fatman Silver badge

      Re: Non IT Callout

      <quote>I was woken by a phone call from the police alarm company to say that the company alarm was sounding. Fair enough, I lived closest. "What, you say, I haven't worked for those assholes for more than 6 years, and I left under 'unpleasant circumstances', and YOU EXPECT ME to drive in to a former employer??? No, I don't have the asshole in charge's home phone number! Fuck off I say!!!!!"</quote>

      FTFY, and it did happen.

  18. Anonymous Coward
    Anonymous Coward

    Build room.

    So the build room is where you take boxes of kit and assemble it (take server out of box). Its in the basement and its locked to stop terrorists from opening the door.

    4pm Summer party booze up starts. By 8pm, its hot and we are all hammered, a few fallen (literally) by the wayside - heat makes you drink. More.

    10pm, security pops into pub, theres a manager and an issue that is urgent. We draw straws (who's round it is not) and me and 'Kate' head back to the office and into the basement. After plugging it back in, we spend some time in the build room plugging it back in again.

    What a tail.

    1. Destroy All Monsters Silver badge
      Paris Hilton

      Re: Build room.

      I have the feeling there some information missing. Who is Kate. Was was plugged out? Why did you plug it back in while plugging it back in? And was drunken basement sex involved?

      1. Wensleydale Cheese Silver badge

        Re: Build room.

        "I have the feeling there some information missing."

        Remote diagnostics tell me that the Christmas festivities have already started, with similar amounts of booze on offer as that summer's day,

    2. OzBob

      Re: Build room.

      When Kate said the name was "short for Bob", I think he was lying.

    3. Anonymous Coward
      Anonymous Coward

      Re: Build room.

      > What a tail

      Kate was a dog?

  19. Anonymous Coward
    Anonymous Coward

    Drunken system recovery

    Those stories are the best. In fact, many many moons ago I won a competition run by El Reg where a sysadmin story involving being too drunk to drive but having a set of recent backup tapes in my possession made for a kind of Christmas miracle. I think the star in the sky was a copy of Solaris. A great many very drunken hugs and "man, I love you"s were exchanged that night.

    In retrospect, trusting an obviously drunk person to accurate distinguish between tar xvf and tar cvf when X and C are right next to one another on the keyboard might not have been the smartest management decision.

    1. DropBear Silver badge

      Re: Drunken system recovery

      Wait, there's a difference between those two...? I always thought you just type "tar" then mash whatever you feel like on the lower half of the keyboard, and pretty much the same thing happens anyway... Well, live and learn...

  20. Keven E

    Just a coupla winks...

    Installing UPS units into a real old school server rack (no rails, threaded mount holes or square nut clip spots) meant bolting in the shelf type rails to hold these units apart and separately removable. I get to the bottom rail and the bolt/nut combos are a bit difficult to hold from both sides without really getting into the unit, so I do... out stretched a bit... 3 am job after backups finished to take the servers down... I'll just rest my neck a little... the production manager woke me up a little after 6am... he's wondering if something was wrong as he couldn't access morning reports off the (downed) POS server...

  21. FatGerman

    When not on call

    ..divert anything not from your mates to voicemail. Do you people not know how phones work? Clearly your bosses do.

  22. Someone Else Silver badge

    The old saw: You can write FORTRAN in any language...

    ...but it's easier to write in FORTRAN!

    Merry Christmas to all, and to all...a margarita!1

    1 Well, since there is no margarita icon, I'll choose the next best thing. Enjoy!

  23. Long John Brass Silver badge

    Server room snooze

    Once upon a time, when I were a young lad and dinosaurs ruled the earth

    Was the on-call to baby site the two accounting systems run by the $company

    The final job of the night was to reconcile these two systems if that failed all was lost; So then you had to figure out what went wrong where, and either fix the import data or fix the code, there was just enough time to re-run the entire batch if the problem was early in the processing. So more often than not I would sort the problem & kip behind the two prod mini-computers at the back of the hall as it was nice & toasty warm there...

    One night after curling up behind the machines; I awoke to find myself moving; my head bumping over the ridges in the raised floor tiles; A closer inspection of the situation informed me that the night operator had grabbed me by the feet and was busily dragging me out of the machine room. When I asked what the bloody hell he was up to he nearly hit the roof... Apparently he had thought I had managed to electrocute myself and had fallen behind the two big prod machines (He had just seen my feet sticking out into the main isle and had panicked)

    Thank god I woke up before he got started with the CPR :)

    1. jake Silver badge

      Re: Server room snooze

      ::heh:: Reminded me of one call ...

      Late1994, 1:30am. ... Call from a company that I upgraded the server farm for a week earlier. Nothing could communicate with the outside world. This was in the very early world of ubiquitous internet access.

      From 20ish miles away, I eyeballed the weather report (29F, possible sleat/hail)) & pulled on pants, T-shirt, thermals, wool trousers, button-down shirt (just in case), wool sweater & my racing leathers. Pop on a balaclava & helmet & good to go.

      Park bike in the CEO's space in the underground garage & convince the security dude that I'm actually supposed to be there.

      In the server room, everything was humming ... except outside connectivity.

      It wasn't running, so I unplugged the outside connectivity, and plugged the TDR into the incoming T1 line. About 250 feet between me & a break. It's MUCH more that 250 feet between between the underground garage and the server room ... I needed to head for the roof.

      I found the incoming T1 line on the roof of a 10 story building. It was bolted to the top of the elevator shaft. Trying to trace this thing from the roof down wasn't fun ... it took about three hours. I finally discovered there was a punch-down block at the TOP! of the shaft! I had to stand on the roof of the elevator to fix it. Corrosion, of course.

      Back down to verify connectivity, and back up to pick up my tools. Soft tool bag looked comfy, I was warm, I fell asleep with it as a pillow. On top of the parked elevator.

      CEO threw a shit-fit because I "stole" his parking place ... and because I hadn't reset the power to to the elevator, so he had to walk to the 10th floor ... I had the elevator on the way down about the same time he was on the 8th floor, completely oblivious that the moron was even present.

    2. I. Aproveofitspendingonspecificprojects

      Re: Server room snooze

      So carpet burns but no snogging. Is that a paid position?

  24. This post has been deleted by its author

    1. I. Aproveofitspendingonspecificprojects

      Almost tempted but not

      To see what you post looks like with paragraph spaces.

      You are not important, so that saved a call out. BRB.

  25. TVC

    Sleeping at Work

    Many moons ago in my mainframe days, I worked shift at a computer bureau in London. In the basement of the old building was the tape library and off it numerous alcoves. One of these contained the massive AC Generator, used to smooth out spikes in the power. One day the powers that be were showing some perspective client around and for some reason decided to open the AC Generator room door and show them our state of the art system.

    Inside on the floor in a sleeping bag was one of the night shift, we shall call him Frank for that was his name. Frank sat up with words to the effect “Shut the effing door some of us are trying to sleep.”

    Turned out Frank lived there quite often to save going home. But that was the last time.

    I too, with others, often slept at work overnight, when the evening trip to the pub meant we were too “tired” to get home. Each had his own space, mine was under the board room table. Unfortunately one morning no one had woken me and the whole place was in full swing and I had to escape through a full office to get to the loos to dressed etc. Happy days.

  26. Anonymous South African Coward Silver badge

    Meh to callouts

    Got a call over a weekend - a critical server was down. On a weekend, and during my leave as well.

    Assessed the situation... the dsl line was working, but somehow the server stopped responding to pings and other gropery stuff.

    Informed the one damager that I need to go to the office, damager flatly refused to pay callout charges, overtime etc.

    Phoned my direct manager, tole him the situation. He insisted I go and fix things, and assured me that my claims will be paid out.

    Just pissing me off that some damagers think nothing of IT workers, especially when they did not consulted said IT department (or workers) of new critical requirements being added to $company's servers.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019