back to article T'was the night before Christmas, and an industrial control system needed an upgrade

Welcome again to On-Call, our Friday column in which readers tell their tales of being asked to get stuff done under awkward circumstances. This week, reader “WG” shared a tale from 2010, when a client asked for some very major changes to be made to an industrial system that had been left largely untouched for 25 years. WG …

  1. Anonymous Coward
    Anonymous Coward

    I was providing on call support to an east coast US company on new year’s eve a while ago. At 3am (on new year's day their time) a manager was bored and decided to log into is work account just to see what was happening with some automated processes. He found an issue which meant that the process would have to be stopped, corrected and then would have to be monitored closely until it was completed. He proceeded to give me a call and that was me for the next twenty hours baby sitting a process that never worked correctly but we weren't allowed to fix but it needed constant tinkering to keep running. The following year I wasn’t on the team anymore but got a call from my old team manager asking for help on boxing day because the only other person in the company who knew anything about it wasn’t picking up. Told him that my knowledge was hopeless outdated, gave him what I could remember and said goodbye. When I got back to work the next day I put in my out of hours call form which for the team I was on should have been £200 (the amount we got depended on how likely a call was, more likely meant less money but more T.O.I.L.). Instead my old manager processed it as if I was still on his team and I got £10. Not working there anymore.

    1. Anonymous Coward
      Anonymous Coward

      > Instead my old manager processed

      So after you were rather less helpful than he had hoped he got his own back by paying you what he thought your help was worth? Not a stellar performance on either side.

      1. Phil W

        Re: > Instead my old manager processed

        Quite so.

        Someone who is not your boss anymore calls you out of hours about something that is not your job anymore, you willingly choose to answer, then choose to offer some limited information and the next day try to claim a large, not previously agreed, over time payment for a brief phone call that you weren't obliged to deal with anyway. Not a professional way to behave in my opinion.

        If I get called out of hours, and am not on call or in any way obliged to answer, then I either:

        a. don't answer because I'm not being paid to work, this is my time.

        b. answer, and help out voluntarily, especially if it is a quick fix or just passing on info

        c. answer, find out how big the problem is, then offer to help only if I'm getting paid for my team (N.B. unless you trust the person in question, make sure they email you their agreement to pay)

        Whether it turns out to be a,b or c depends on my mood, where I am and what I'm doing.

  2. Anonymous Custard
    Trollface

    Hurry up and wait

    WG made it home for Christmas, albeit without knowing if his work had done the trick because the client wouldn't run the whole system until the new year.

    Ah yes, the old "I need it super-duper urgent so you lot have to pull out all the stops, then I'm going to leave it sat idle for a week whilst I enjoy my holidays and other more important stuff" response from the customer.

    Where does that sound horribly familiar from?

    1. Bluto Nash

      Re: Hurry up and wait

      Where does that sound horribly familiar from?

      Ummm, everywhere?

    2. TomPhan

      I've known companies, albeit small ones, where they had to follow the opening hours of the office block they'd rented space in so no matter how much they wanted to use bank holidays for maintenance work they were out of luck.

  3. Yugguy

    “A less than really sharp manager agreed,”

    Are there any other kinds?

    1. Anonymous Coward
      Anonymous Coward

      Re: “A less than really sharp manager agreed,”

      Yes there are, most of them are "less than even slightly sharp".

    2. fajensen

      Re: “A less than really sharp manager agreed,”

      Yess - Dull!

      - Their function is to shut down all higher brain functions in any group of people so that decisions can be made.

      1. Nolveys
        Windows

        Re: “A less than really sharp manager agreed,”

        I considered a job in management, but then I learned that I would have to undergo two medical procedures. First the brain is removed, then the spine.

    3. JeffUK

      Re: “A less than really sharp manager agreed,”

      As per the Dilbert Principle.. the 'less than sharp' people are consistently promoted to management positions where they can do the least amount of damage. sounds like WG's manager is due a promotion.

      1. Anonymous Custard

        Re: “A less than really sharp manager agreed,”

        @JeffUK - it's a well tried and tested Japanese management technique too. Promote the incompetent up but out of the way, and surround them with suitably clever and competent people who can take what they say and spin, interpret or just replace it with more sensible and suitable proclamations and requests.

        1. Starlite Lemming

          Re: “A less than really sharp manager agreed,”

          @Anonymous Custard: I think someone forgot to tell the rest of the world about that second part (surrounding the incompetent with the clever).

    4. Sgt_Oddball

      Re: “A less than really sharp manager agreed,”

      The dullest blade is always the most dangerous. At least that's what I was always taught..

    5. martinusher Silver badge

      Re: “A less than really sharp manager agreed,”

      I've retired partly because after having to deal with "Less than Sharp Managers" for far too long I figured that I needed a bit of a rest. The complaint's always the same -- these people conspire with customers to set up situations where you're trying to do too much with too few resources in much too little time. If they were technical experts or at least paid you the courtesy of asking up front, fine, but invariably they're not. So I figure I'll let them handle the details -- or, more likely, let natural selection do its thing.

  4. Hugh Pumphrey

    But the loop had no test in it.

    It looks as if WG managed to discover what happened to Mel: http://www.catb.org/jargon/html/story-of-mel.html

    1. Zippy's Sausage Factory
      Happy

      Re: But the loop had no test in it.

      Ah, Mel. I used to fashion my code on the 6502 along a lot of those design principles. Those were the days... :)

  5. Ol'Peculier

    I had a client ring me up at home on Christmas Day demanding I went to his office to update something in my code. It was about mission critical as trying to find a better joke in a Christmas cracker.

  6. Anonymous Coward
    Anonymous Coward

    Which movie was this?

    I'm sure I've seen this software-wizard-reprograms-complex-system-on-the-fly thing in some Hollywood movie or another, but just can't remember which one. Real life, not so much.

    1. Dr Dan Holdsworth

      Re: Which movie was this?

      I've seen it in some Perl code written by a chap with the moniker of "Random John". This was for the now-happily defunct ISP Netline UK, and ran some of their internal reporting systems, AKA Lies for Managers.

      Sometimes now I really do wish I had kept a copy of the original code, simply as a terrible example of how utterly shite Perl can be and still limp along being vaguely functional. The script ran as a CGI program called from a web site, and as it started up grabbed a huge chunk of required info from a separate script of "useful variables".

      Then it lurched into action, and grabbed data from various RRDtool databases, web servers, and the in-house database server. That doesn't sound too bad, but all variables were declared global and "to save memory" variables got re-used in the code, so something like $start_date would change to holding a completely unrelated piece of information, and do this several times over.

      The entire script could be broken entirely with the simple declaration: use strict ;

      All of this was written in a coding style best described as "hurl tin of alphabet soup at wall", with minimal commenting (and most of that wrong or deliberately misleading), no indentation and precious little attention to readability. A complete dog's dinner of a mess, in other words, and I got tasked with sorting the thing because web browsers were timing out before its hideously unoptimised code could respond.

      1. DJV Silver badge

        Re: Which movie was this?

        There was a major Perl script running at a company I used to be at. The original programmer, who had departed some months earlier, had made the script as terse and unreadable as possible. The only comment in the whole thing was next to some code whose purpose no one had any idea about.

        The comment was extremely useful - it said: "This is a skanky hack"

        1. Swarthy

          Re: "This is a skanky hack"

          I have left that comment.

          I have also left 3 lines of comment per line of code, just so that I could understand WTF I was smokingdoing when I wrote it, should I ever have to return to the scene of the crime.

          1. Vic

            Re: "This is a skanky hack"

            I have also left 3 lines of comment per line of code, just so that I could understand

            I once wrote some meta-python to take a WSDL file and produce a set of python methods respresenting everything in the WSDL.

            The code without comments was a dozen lines. The file I committed was nearly 100 - I was pretty sure whoever was going to touch it next wouldn't have appreciated it if I'd just left terse commenting...

            Vic.

            1. Michael Wojcik Silver badge

              Re: "This is a skanky hack"

              The code without comments was a dozen lines. The file I committed was nearly 100 - I was pretty sure whoever was going to touch it next wouldn't have appreciated it if I'd just left terse commenting...

              Good man.

              I've committed source files with more comments than code, for much the same reason - implementing some non-obvious algorithm, so I describe it in detail (often with some examples) for maintainers. Including myself, a year (or a week) later.

              Real literate programming would be even better, but the systems for doing so are mostly terrible. Knuth's a brilliant guy, but his idea of a domain-specific language tends more toward line noise than even Perl does. And documentation-comment systems, while useful, are not LP.

        2. Anonymous Coward
          Anonymous Coward

          Re: Which movie was this?

          I was an on-call mainframe analyst programmer. I was asked to look at a job on a callout that wasn't my normal area of expertise. It was a direct debit run - huge company so the job ran most nights with the customer base divided into 28. That meant that if it couldn't be run before batch processing ebbed for the night those direct debits wouldn't be taken for a month.

          Every batch job had an associated text file detailing triggers, inputs, outputs, memory allocation, and how to nurse it if it was poorly. The file for this job was compact and to the point; "Put some shit in here".

        3. Vic

          Re: Which movie was this?

          The comment was extremely useful - it said: "This is a skanky hack"

          I once left a comment that said "This next bit is evil. Look away now". I had to bless a perl structure because the CGI library I was using didn't return what the documentation said it would...

          Vic.

      2. Mark 85

        Re: Which movie was this?

        That sounds like a programmer I worked with except he was using C on a Unix box. No line separation, no comments, just one long bit of code run together for 40 printed pages. Seems he was trying to save memory and someone told him to remove "whitespace'. Which he did.

      3. T. F. M. Reader

        "hurl tin of alphabet soup at wall"

        Larry Wall?

  7. Doctor Syntax Silver badge

    I've seen the opposite. Vendor's support ending at the end of the year so we proposed to cut over to a replacement during the Christmas break. Top management said no. Subsequently discovered that external accountants were sneaked in over the break to value the business for acquisition.

  8. This post has been deleted by its author

  9. Anonymous South African Coward Bronze badge

    Updates/work in the silly season

    Short and sweet - it is not a good idea, unless planned for well ahead.

    Our support department recently did an update on a remote site - I'm still waiting to see said updates/upgrades coming back to bite their a$$es, especially over the 24/25/26th...

  10. wolfetone Silver badge
    Pint

    I Tip My Hat To You Sir

    I'm all for ignoring the opinions of others when taking on jobs. Sometimes they can be right, sometimes the work will be far more complicated than thought, and the documentation might never be there. But at the end of it all, you can walk away from it and know you did something no one else was brave enough to do.

    For Christmas, have a pint. On me.

  11. Spaceman Spiff

    I hope they got a 3x overtime pay package for this, plus two weeks in the Bahamas on the company dime! They deserve it! And fire that manager!

  12. This post has been deleted by its author

    1. Stevie

      Re: copy editor already on holiday?

      There's a right way to report typos and markup problems at El Reg, and there's your way.

      Do you seriously think copy editors have the time to read these comments?

      1. Anonymous Coward
        Anonymous Coward

        Re: copy editor already on holiday?

        El Reg have copy editors?!

      2. Ugotta B. Kiddingme

        Re: @ Stevie - copy editor already on holiday?

        You are, of course, correct.

        Withdrawn and submitted appropriately.

  13. kmac499

    Trivial by comparison

    We had a PLC rack in the control room and one particular flap valve ( dry goods ) in the factory would not flip over.

    The LED tell tale on the relay unit in the PLC cabinet showed it was commanded to open but no joy.

    After the usual rounds of swearing and fiddling, by almost closing the cabinet door against the control room lights, we could see the PLC telltale was actually flickering not solid, and on each flicker it reset a 30 second count down timer to give the valve time to flip over.. D'Oh....

  14. Anonymous Coward
    Anonymous Coward

    Er...

    "WG and his colleague “realised the guy had discovered a way of bulk updating large blocks of data, which he'd preceded to use enthusiastically, blissfully unaware that this also corrupted variables in the 'gaps' that he wasn't using but we were trying to!”"

    How inept of the original programmer to not realize that memory he wasn't using would be needed 25 years later by WG. In other words, the guy was a better programmer than WG, and WG & his colleague didn't understand the memory structure of the 25 year old controller's program.

    1. Peter2 Silver badge

      Re: Er...

      I must admit, this was my first thought when I read it.

      If I had to work on them today, 'm not sure if I could understand the creative hackery I used in my own autoexecute.bat & config.sys files in the DOS days getting that one extra device running in a tiny memory footprint.

      How i'd deal with the creative use of memory in ancient PLC's with code I didn't write from yesteryear doesn't bear thinking about, but it's certainly not the original programmers fault or problem that later modifications are a problem.

      1. SteveK

        Re: Er...

        If I had to work on them today, 'm not sure if I could understand the creative hackery I used in my own autoexecute.bat & config.sys files in the DOS days getting that one extra device running in a tiny memory footprint.

        I agree... I used to pride myself on being able to outperform memmaker on getting the optimal sequence for loading DOS drivers and TSRs and create multiple boot sequences loading different combinations. Although it did take me longer!

        At the time, worked for a games company and the QA department PCs tended to have at least 3 completely different ISA sound cards, a variety of VGA (and later additional 3DFX or PowerVR) cards, random network cards and CDRom drives - all of which needed different drivers loaded, not to mention just dual booting between DOS and Windows (with or without the drivers for connecting to Netware via IPX)...

        A lost art. For which I am very grateful!

        1. Peter2 Silver badge

          Re: Er...

          Hah, everybody could outperform memmaker! You bloody well had to, because it never was able to load sound cards, graphics cards, network cards, cd roms, plus the optional stuff like mouse drivers, joysticks etc. Unless I was just systematically unlucky.

          Ah, the dimly remembered dark arts of what to put in higher memory to clear working space and when. And if and when to do it at all, because loading the higher memory could use more memory than it saved.

          *shudder*

          A lost art. For which I am very grateful!

          Oh yes. Now I might bemoan programmers solving software problems with hardware, but this is one set of problems i'm quite happy to have ditched.

    2. TomS_

      Re: Er...

      How inept of a programmer to assume that memory addresses he wasnt using wouldnt be used by someone else later down the track, more like it...

  15. wyatt

    Reminds me of a 24hr customer who had a hardware usb dongle fail the week before xmas. I diagnosed it, raised it with the manufacturer for replacement and was told it'd be a few weeks. Que various managers demanding I double check that this was the failed part to satisfy the customer.

    Loads of OT for our on calls guys as we rebooted the system every few hours to reduce the effects of the failures. Replacement part was DOA, so I looked like a knob until the replacement resolved the issue.

  16. Alan Bourke

    Ah, PLCs ...

    Rotherham. 1992. On placement for 6 months from a college course. Siemens PLC controlling a machine that put steel bands around flat reels of steel 'tape' to hold it together during transport. A similar situation saw me doing a 24-hour straight shift to get a new ladder program working. When I got home the Freddie Mercury tribute concert had just started.

  17. Anonymous Coward
    Anonymous Coward

    This makes me glad that I work in a place where absolutely nothing happens from Dec. 22 until after the new year. I kid you not. We had a PDC die at 4:55pm on Dec. 22 one year and the decision was made to live without it until Jan. 2. As long as the public facing website was up to tell people Merry Christmas (yes, Merry Christmas, not happy holidays. It's the Bible Belt, ya'll.) no one cares over the holiday break.

  18. Anonymous Coward
    Anonymous Coward

    Heh

    The one and only server a county elections committee had died on the second day of a national election and I received a panicked early morning phone call because without the Windows XP 'file/program server', it was back to pen & paper voting (which imo is not a bad idea but no one asked me).

    I drove all over the east end of the state collecting hardware and rebuilt the machine. The amusing irony was that rebuilding the machine, reinstalling the state software wasn't really that big a deal, shit happens, I had already advised them of the fragile nature of their network. It was that they got their data back on over a million voters because the only thing they did listen about, was the backup system I had just installed two months before. Had them back pecking at their keyboards by the end of the day.

  19. Anonymous Coward
    Anonymous Coward

    I'm currently on the phone

    Trying to talk someone out of a major change on Xmas eve that they already tried to put in once and didn't work. It's someone else's problem if they do put it in but I still don't want to come back to it in the new year.

  20. Palpy
    Facepalm

    Skanky hack -- must use that.

    Not the night before Christmas, but for one job we had a skinny communication link between a vendor-programmed PLC and the plant-programmed automation system, and a fat number of control variables to shove down the link. Asynchronous, with a minimum scan speed of 1 second on one end. (Yes, dear reader, that's as fast is it will go.) The resulting multiplex code is written in a control-scripting lingo and, in action, resembles an early Boris Karloff monster: stumbling and groping and occasionally spasming. If it had salivary glands it would drool. It seems these things always must be programmed at the rush-fast-now end of the project. I am not proud.

    But it was one of "those" projects all around -- an outdoors installation on which the engineers specced indoor-quality electronics, flow-control valves unsuited to purpose, and so on. Many change orders. Some of the valve actuators corroded out in the first 18 months.

    Fortunately the whole thing remains mothballed most of the year. It rises from the dead periodically and the operators make it work, then they clap a chloroform-saturated towel over its face and put it to sleep again. Skanky hack indeed.

  21. JQW

    Myself and a couple of colleagues once had a site visit between Christmas and New Year, upgrading and patching the obscure OS on a couple of dozen file servers, and then upgrading the client workstations. My servers upgraded fine, but my colleagues had serious problems getting the upgrade to take on other servers, caused by OS changing the order it allocated storage controllers between versions without documenting it. As our supplier's UK support centre was closed for the holidays, and with this being pre-internet days, getting a fix was tricky. Still, with some playing with EISA configs we got it working.

  22. LesC
    Stop

    Was working in Northern Ireland on point of sale stuff for a well known agency a few years ago on Christmas Eve after finally winding up at 6pm installing some point of sale system in Belfast headed off home to NE England in worsening weather. At Belfast Ferry Terminal the RUC took exception to my green Skoda Octavia estate full of boxes of decrepit IT which I intended to drop off in Watford on the 27th. Went something like this.

    "Random Search, sonny"

    "Why"

    "Terrorism Act do you have any ID"

    Experience meant I carried my passport in NI as my driving licence at that time was the old green paper one - one with a couple of speeding tickets - "why the passport, son?"

    "It has a photo in it"

    "You have an Irish name" - and a bright green car which would be ignored in the uk "empty your car out NOW. Do it now. You have no rights" said RUC officer with an increasingly itchy trigger finger over his holster.

    Emptied car of old Cisco hubs, 486 Compaq Presarios, isdn bridges and HP4 laserjets in the car park.

    "Nothing here son. Move along"

    I caught the ferry with 5 minutes to spare it turned out it was the last one as the wind went to Force 9. I never went back in the New Year and haven't been there since.

    Got home at 2355 to one very agitated wife and sleepless daughter #1.

    :( .

    LC

  23. timhowarduk
    Pint

    Party on!

    About 15 years ago me and a colleague were doing a NT4 to Win2k domain migration at a small firm. They didn't want to buy new servers so the plan was to use a virtual NT4 BDC to upgrade the domain and put new bigger disks in their main servers, then rebuild the main servers as win2k and then blow away the virtual temporary DC.

    It was going well until about 3pm when the virtual NT4 had been upgraded to win2k but somehow became corrupted before it had done a full GC replication to the real servers running on the new disks. This we actually a show stopper but it was the first time we'd done this in the wild and we didn't really understand that it was 'game over'.

    It also turned out this was the day of the office Christmas party and everyone around us was getting merrier and merrier, particularly as no one could log on. This didn't affect anything until someone had the bright idea of bringing a bottle of wine and two glasses to the server room for us. By about 6pm we had finished that bottle and still had an unholy impossible mess on our hands as someone came down with another bottle for us and some crisps. We dutifully carried on fighting the impossible and hopeless fight, with our decisions and perspective by now severely hampered by having had a bottle of wine each. At around 7pm the boss came and said "We're all going to an Italian reasturant now, you guys are working very hard, are you nearly finished and do you want to come?"

    This was a difficult decision. We couldn't really walk away and leave them with no functioning domain, we also didn't want to admit we were in a right fix, but it did seem too good an offer to refuse. Then the shelf caught my eye with the old disks that had been in the servers that morning before we had started. I motioned to my colleague in the direction of the old disks all carefully labelled and his face lit up. I said "We'll be 10 minutes" and the boss said "fine, I'll wait upstairs and we'll get a taxi".

    So we threw the original disks back in, taking things back to how things were before we had turned up in the first place, checked a client PC could log on, then went to the restaurant and on to some club afterwards.

    The official line was that we returned two days later still nursing the mother of all hangovers for "a bit of optimisation and fine tuning". The reality was of course that we did the entire job again with no hiccups this time.

  24. jake Silver badge

    Uh ...

    ... Industrial control systems ALWAYS get upgraded when the users are offline. Why pay 'em when they can't actually do their job? Force 'em to to use their "days off". It's just the way it's done.

    (Note that I don't actually agree with this practice.)

  25. Starlite Lemming

    I hope that "less than really sharp manager" was disciplined.

    The prevailing attitude -- that, if it works, there was never a problem -- only breeds problems (and incompetence) down the track.

  26. Starlite Lemming

    Re: But the loop had no test in it.

    Loving some of the comments.

    My first home computer was a Commodore 128.

    Wrote my first programme in BASIC 7.0 (it had graphics commands!).

    Wrote my second programme in the Machine Language Monitor -- think Assembler without labels (= variable names). My grandpa was actually able to supply me with a specially-printed pad of assembly programming paper. Made it a lot easier to precalculate jump addresses (which were predominantly relative in 6502 assembler).

    But my favourite thing about the Commodore 64 (there was a lot more material published for the 64, so I inevitably found myself in C-64 mode), was that you could load an entire screen of graphics into a blank bit of memory, then display it on the screen in the blink of an eye by simply changing the memory address of where the graphics chip looked for screen memory. I think there may have been as many as 16 different locations (4 bit flag -- seems likely), so you could have that many completely different screens at your fingertips.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like