back to article How an over-zealous yank took down the trading floor of a US bank

Good Monday morning, Reg readers, and welcome once more to Who, Me? – our regular trip down memory lane for those with something to get off their chest. This week, we meet “Alan” who once took out an entire trading floor at an arm of a US bank. Back in 1996, Alan had just graduated from a software engineering degree and had …

Page:

  1. PerlyKing
    Facepalm

    DevOps?

    I'm willing to bet my 1996 bonus that Alan did not get "a job in devops" back then.

    1. David Roberts
      Pint

      Re: DevOps?

      Damn, beat me to it!

      Never too early for a pint; have one on me.

      1. Aladdin Sane

        Re: DevOps?

        A rose, by any other name...

        1. jake Silver badge

          Re: DevOps?

          A rose by any other name ... would stink of rotting garlic.

    2. Velv
      Headmaster

      Re: DevOps?

      While it might be the buzzword of the CIO, CTO, et al these days, the concept has been around a long time, long before 1996.

      1. Anonymous Coward
        Anonymous Coward

        Re: DevOps?

        While it might be the buzzword of the CIO, CTO, et al these days, the concept has been around a long time, long before 1996.

        No such thing as a new thing. I remember when people were getting all excited about the "new" thin client model a few years ago...."oh, you mean like the mini/mainframe I was using at university a couple of decades back?"

        1. StargateSg7

          Re: DevOps?

          I think we called that a Client/Server Model of Computing like BACK IN THE DAY of 1980's era IBM AS-400 or even the 1960's/1970's era VAX-780 and IBM 360 mainframe. DevOps is basically Top-Down development and VERY BASIC 1950's era Systems Analysis.

          a) Define the business problem you're trying to solve.

          b) Define the FUNCTIONALITY of current or new components needed to solve a part of the overall business problem. (i.e. Do we need hardcopy printing, email, calendar functions, streaming videophones, SMS/instant messaging, calendar input, etc)

          c) Define the TYPES of processing modules NEEDED to solve each require functionality as defined above. (i.e. print PDF reports, Enter data via touch screen or keyboard, send email to clients on POP server, use RTSP for video, etc)

          d) Define the functionality of sub-components needed within each large-scale component

          (buy RTSP DLL from Datastead for video streaming, use Thunderbird components for email,

          code custom data entry screen in-house, use PDF generator component in compiler, etc)

          e) Code, Link and Combine components into test runtime folder

          f) test on SEPARATE MACHINE with SEPARATE TEST DATA ONLY!!!!!!!

          DO NOT USE LIVE DATABASE !!!

          g) test 1000 times with local admin department heads and team of each component and data entry and processing module over 3 months.

          h) Do bug report log and send back for recode and retesting of 'E" and "F" and "G" over another 60 days.

          i) Get Go-ahead for One-Branch-Office-at-a-Time deployment with initial two weeks allocated for in-house training and another two weeks of real-time use per office for real-world feedback and fixes.

          If no major issues, then go ahead and use same two-week training and two week real world test schedule for next three months at another three branch offices.

          j) If no major problems at the four branch offices, then roll out to ABOTHER four branch offices at the same time again with two week training and two weeks in-office testing until all offices update to new version of application. Update ONLY four branch offices at a time and train/test for 30 days!

          h) Shut down last database from old system ONLY AFTER ALL OFFICES are converted but still keep on online version of the old software on ONE machine in every branch office connected to a separate head office server for at least one year as backups just in case something BAD happens to new system!

          i) do a 3 month, 6-month and 1 year review of each office interviewing certain users personally on fixes or updates required for next app update version. Send responses back to coders and management for review and add-in and THEN DO ANOTHER office-by-office rollout but with an accelerated one week of in-house training and two weeks of testing for NEW bugs.

          THAT is how you do a PROPER Roll-out of a new app using Top-Down Modeling and Deployment!

          1. Anonymous Coward
            Anonymous Coward

            Re: DevOps?

            Amen, brother. After 12 years of dealing with that same scenario on the same application, for each update, I can attest to the absolute desire to chuck a bunch of flaming bags of dog/cat/horse/human/elephant/cow poo at the doors of anybody of that vendor that has a 3-letter-acronym in their title. Why haven't I moved on? The pay is fantastic, I determine my own (over 40 hours) workweek, and I know enough about the software that I commonly correct their developers on their mistakes. Sometimes, you find a tech job that actually is worth the frustration, irritation, and insanity that comes with it. I got lucky on that.

            1. Alan Brown Silver badge

              Re: DevOps?

              " I can attest to the absolute desire to chuck a bunch of flaming bags of dog/cat/horse/human/elephant/cow poo at the doors of anybody of that vendor that has a 3-letter-acronym in their title"

              Or to visit George (of the jungle) and ask to borrow Rex for a while.

    3. Anonymous Coward
      Anonymous Coward

      Re: DevOps?

      It may not have been called "DevOps", but we most certainly used the same concepts and methods in the mid 90s. Of course, back then, buzzwords were real buzzwords....

      1. Doctor Syntax Silver badge

        Re: DevOps?

        "It may not have been called "DevOps", but we most certainly used the same concepts and methods in the mid 90s."

        One of the joys of being a grey-beard is that you can watch all the young folk [re]discovering so much stuff. All we need is a buzzy name and we can get waterfall development back in fashion.

        1. onefang
          Coat

          Re: DevOps?

          "All we need is a buzzy name and we can get waterfall development back in fashion."

          Golden Showers?

          Gonna need a good rain coat now.

        2. Rich 11

          Re: DevOps?

          All we need is a buzzy name and we can get waterfall development back in fashion.

          TQAIASSM (t'-kwai-as-'m)

          Total Quality Artificial Intelligence Administered Stateful Software Management.

          I can feel the dollars rolling in already. I'll write it once I've been funded enough to retire to a distant tropical island (volcano lair optional).

          1. IceC0ld

            Re: DevOps?

            All we need is a buzzy name and we can get waterfall development back in fashion.

            ===

            TITSUP

            Total Inability To Support Usual Processes ....................

            well, why the heck not, it's a great acronym :oP

          2. onefang

            Re: DevOps?

            "Total Quality Artificial Intelligence Administered Stateful Software Management."

            You left off blockchain.

        3. EarthDog

          Re: DevOps?

          It's called SAFe.

      2. Joe Gurman

        Re: DevOps?

        .... or buzzfeed.

    4. Anonymous Coward
      Anonymous Coward

      Re: DevOps?

      I'm willing to bet that "screaming traders assaulted the frame room baying for my blood” didn't happen either. From the traders I met when I worked in the city most of them couldn't tell you where their phone was plugged in, never mind where the server room was and even if they did find it , the door would have been auto locking even back then so unless 1 of them had the combination or he jammed the door open its total BS.

    5. wayne 8

      Re: DevOps?

      I was doing DevOps in 1978. One person computer shop. I added peripheral maintenance to DevOps when I replaced the print chain on a GE line printer. Fun times. Start a batch job, go over to Times Square to down one or two liter of Dinkelacker. BDevOpFH.

    6. Fruit and Nutcase Silver badge
      Pint

      Re: DevOps?

      Must be some El-Reg editorial policy or bonus scheme (extra beer?) for the journos. Thou shalt strive to mention "DevOps" in as many articles as possible

  2. Anonymous Coward
    Facepalm

    Unplugging the keyboard = kernel panic ?

    My god... what were people thinking back then...

    1. Aladdin Sane

      Re: Unplugging the keyboard = kernel panic ?

      Came off in my hands guv'.

      1. Rich 11

        Re: Unplugging the keyboard = kernel panic ?

        Came off in my hands guv'.

        Said the actress to the bishop.

    2. David Roberts
      WTF?

      Re: Unplugging the keyboard = kernel panic ?

      A long, long time since I used IPX or Sparc but I can't remember that being a feature.

      Perhaps some misguided "security" precaution?

      1. Chris King
        Coffee/keyboard

        Re: Unplugging the keyboard = kernel panic ?

        I remember someone unplugging the keyboard on a Sun box to tidy up the cabling.

        The reason I remember this is that my boss at the time - a normally placid man - started screaming obscenities at the offender, as he happened to be in the machine room working on another system.

        I thought I heard the words "force-fed", "floor tiles" and "without salt" somewhere in that rant.

        Icon, but don't get another one - just plug the old one back in and step well away, there's a good chap. Or as the boss-man put it "GET OUT AND LEAVE MY KIT THE FUCK ALONE !"

      2. Denarius
        Unhappy

        Re: Unplugging the keyboard = kernel panic ?

        David, this was a non-delightful feature of SPARC hardware for a long time. Probably still has it. Bitten by same "feature" two decades ago. Which reminds me, must dump the Solaris 8 pizza boxes lurking in shed so I have more room for something useful.

        At least AIX, HPUX and PCs hardware was not so scream inducing. I loved the old PARA-RISC HP hardware. One could do a full backup without starting the OS using embeded firmware. Great for major OS upgrades and raw disk databases.

        1. dedmonst

          Re: Unplugging the keyboard = kernel panic ?

          >> I loved the old PARA-RISC HP hardware. One could do a full backup without starting the OS using embeded firmware.

          COPYUTIL FTW!

      3. Alan Brown Silver badge

        Re: Unplugging the keyboard = kernel panic ?

        "A long, long time since I used IPX or Sparc but I can't remember that being a feature."

        They don't panic, but they DO stop - at least IPX and SLCs did.

        They also had this wonderful feature of stopping the entire machine to scroll one line of text on the screen, so the first thing you did after installing them was to install and start X

      4. Anonymous Coward
        Anonymous Coward

        Re: Unplugging the keyboard = kernel panic ?

        My recollection is that messing about with the console like this tended to cause the firmware to suspend the OS and throw you back out to the OpenPROM prompt.

        Thing is, this isn't a kernel panic. If you kept calm, it was possibly to simply 'resume' the OS and life would return to normal. Only damage done would be that anyone's X11 session would have frozen for the duration, and mysteriously come back to life a short time later.

        I remember these days well. We used to have stacks of Sparcstation20 and Ultra60 workstations with 10-bay SCSI disk enclosures chained off them running under desks in the dev area. Flipping noisy, but kept the office comfortably warm in winter.

      5. This is my handle
        Go

        Re: Unplugging the keyboard = kernel panic ?

        No, I remember well that you could take down certain SPARCstations by unplugging the keyboard. Weird, but true. I was working for a trading firm where the Traders were in fact quite smart, and did have access to the glassed in room with the raised floors, so nothing about this story sounds at all improbable to me (except the duly noted anachronism of the buzzword 'DevOps'). About that same time, working late I accidentally did an rm -rf ./src in the wrong place, deleting all the source code staged for our current build. As I frantically called my boss (for visibility), I remembered that our NAS took frequent snapshots, and was able to restore the company's tens of millions of lines of proprietary source code, and kicked off a validating build that came up clean. This was way before "git" so source control systems were centralized, and our developers were not overly dedicated to committing code frequently, so this no doubt saved my job, if not the company. Ah, the good old days...

    3. Crypto Monad Silver badge

      Re: Unplugging the keyboard = kernel panic ?

      Almost as good as the infamous IBM PC boot error:

      "Keyboard not found. Press F1 to continue"

      1. Field Commander A9

        "Keyboard not found. Press F1 to continue"

        Because...serial console?

      2. imanidiot Silver badge

        Re: Unplugging the keyboard = kernel panic ?

        Almost as good as the infamous IBM PC boot error:

        "Keyboard not found. Press F1 to continue"

        I don't get why people think that is so stupid or funny.

        Plug in a keyboard and press F1 to continue booting... What else were they supposed to do? Shut the system back down immediately? It's a clear, concise error message with an easy resolution.

        1. Steve K

          Re: Unplugging the keyboard = kernel panic ?

          "Plug in a keyboard and press F1 to continue booting" would therefore have been a better and unambiguous message.....

          1. jake Silver badge

            Re: Unplugging the keyboard = kernel panic ?

            Unambiguous error messages from the hardware would have to wait until there was enough ROM to store them. Sadly, that doesn't seem to have happened yet.

            1. Steve the Cynic

              Re: Unplugging the keyboard = kernel panic ?

              Sadly, that doesn't seem to have happened yet.

              And yet the UEFI firmware on my new PC's motherboard has room for a mouse-driven GUI for the UEFI equivalent of BIOS Setup.

              1. GrumpenKraut
                Devil

                Re: Unplugging the keyboard = kernel panic ?

                > .. .a mouse-driven GUI ...

                And still, for a menu item "Set XRLZP" with selectable settings nup/zup/gargl the help text reads "XRLZP settings". Correct?

                BIOS programs ------->

                1. Steve the Cynic

                  Re: Unplugging the keyboard = kernel panic ?

                  And still, for a menu item "Set XRLZP" with selectable settings nup/zup/gargl the help text reads "XRLZP settings". Correct?

                  It's been a while since I looked at it, but that does ring a bell.

                  1. onefang

                    Re: Unplugging the keyboard = kernel panic ?

                    I've always wondered which setting turns off that horrid bell sound. Thanks.

            2. G.Y.

              Multi-GB Re: Unplugging the keyboard = kernel panic ?

              When goody-bag USBs are multi-GB, there is no room for decent error messages???

              1. JK63

                Re: Multi-GB Unplugging the keyboard = kernel panic ?

                Not in 1996.

                He would have been at the OBP (aka the OK> ) Prompt. Resume would have done the unthinkable, resume the still in core OS.

                Yeah, Sun hardware circa 1996 had its quirks.

              2. ITS Retired

                Re: Multi-GB Unplugging the keyboard = kernel panic ?

                Computer side USB drivers can be full of bloat too. Remember Microsoft recognizes USB stuff and even sets them up to work; most of the time.

          2. Nick Kew

            Re: Unplugging the keyboard = kernel panic ?

            "Plug in a keyboard and press F1 to continue booting" would therefore have been a better and unambiguous message.....

            I think the reason that message was "funny" is precisely that it generally happened when a perfectly good keyboard was indeed correctly plugged in all the time. The problem lay elsewhere.

            I was using mostly sparc workstations around the time of the story. But I don't recollect ever yanking a keyboard out, so I can't say one way or t'other whether anything bad happens. I suspect it depends on what is listening to the keyboard, and how it reacts on losing it, hence some seeing huge overreaction while jake saw no problem.

            1. Anonymous Coward
              Anonymous Coward

              Re: Unplugging the keyboard = kernel panic ?

              My one and only experience with SPARC workstations was when visiting a VLSI lab and the tech there was proudly showing off the multi-user, remote desktop capabilities. I noticed the mouse buttons didn't work and he calmly pointed out that numlock was on. I calmly pointed out that overpriced crap is still crap. The look of hurt on his face still haunts me and I have since learnt to temper my honest opinions when not on home turf.

              Anonymous because the hard working tech guy could very well be reading this and still holding a grudge, or wielding a type4 keyboard.

            2. Missing Semicolon Silver badge
              Windows

              Re: Unplugging the keyboard = kernel panic ?

              Didn't a PS/2 keyboard only get recognised on power-up? So plugging it in post-boot did not initialise the keyboard, as the entire PS/2 interface had disabled itself. Only power off, plug in, power on worked.

        2. Anonymous Coward
          Anonymous Coward

          Re: Unplugging the keyboard = kernel panic ?

          I don't get why people think that is so stupid or funny.

          Because when confronted with a message on a screen, people's understanding becomes astonishingly literal. The stupidity really isn't as one sided as it seems at all.

          (I wonder how many have headed for the exits when seeing the lpd "printer on fire" message)

          1. Steve K

            Re: Unplugging the keyboard = kernel panic ?

            On that note, how about "Jam in printer"...

            1. M E H
              Trollface

              Re: Unplugging the keyboard = kernel panic ?

              When I was on a hell desk and users reported that the printer was jamming I used to calmly and professionally ask if it was jazz or reggae.

            2. Saruman the White Silver badge

              Re: Unplugging the keyboard = kernel panic ?

              I've always preferred "Printer on fire..."

          2. Crypto Monad Silver badge

            Re: Unplugging the keyboard = kernel panic ?

            Because when confronted with a message on a screen, people's understanding becomes astonishingly literal.

            Like people who phone the helpdesk saying that they can't find the "Any" key.

            https://www.theregister.co.uk/2003/09/25/compaq_faq_explains_the_any/

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like