back to article How an over-zealous yank took down the trading floor of a US bank

Good Monday morning, Reg readers, and welcome once more to Who, Me? – our regular trip down memory lane for those with something to get off their chest. This week, we meet “Alan” who once took out an entire trading floor at an arm of a US bank. Back in 1996, Alan had just graduated from a software engineering degree and had …

  1. MacroRodent Silver badge

    Console

    At my first IT job as the operator trainee tending a Honeywell mainframe in early 1980's, we were told to be quick about adding paper to the console printer (which collected status messages, rather like syslog or journald), as the system would supposedly crash if the printer was offline for more than a minute or so. Luckily I never found out if this is true.

  2. jake Silver badge

    Re: Console

    'twas true for some kit, back in the day. When the printer ran out of paper, the computer buffered the output until more paper was loaded. If the buffer filled, you crashed. Try to remember, though, this was back in the day when a meg of core memory was considered extremely extravagant ... and I don't think that Honeywell had that problem, although I vaguely remember some people thinking it was a normal problem with computers in general.

  3. Alistair Silver badge
    Windows

    Re: Console

    I was on a DRP exercise once where our Z mainframe ran into this issue as they'd used a FF printer as the 0 console....... Even mainframe's can run out of memory eventually.

  4. stiine
    Happy

    Re: Console

    What model? The DPS-8/47 wouldn't, neither would a dual DPS-90 or dual DPS-8000.

    Ah, memories.

  5. ricardian

    Re: Consoleer

    Back in the 1980s I worked for a large Government department in Gloucestershire. The local tech college (not Gloucester) ran a course on "C" programming so our boss signed us up for it. The "notes" looked as though they'd been written in the pub the night before and the machines were on a flaky network where you had to quit the editor in the "proper" way otherwise the whole network crashed. We reported this to our boss who managed to get the course fees returned to his acount.

  6. tony trolle

    Re: Console

    ICL let you dump the console printer to tape. I think; been a while.

  7. Anonymous Coward
    Anonymous Coward

    I can trump that !

    This one has to be anon.

    About twenty-five years ago, I was working for a large bank on their market data systems. The vast majority of the Reuters market data was page based, each page being 17 rows x 64 columns of data. However Reuters had just released a new set of page data (in beta) which was 25x80.

    I typed in the new page name (I think it was BP.L or something like that) and lo and behold, up came a new page, 25x80, with lots of useful data. I showed this to a friend of mine (really, it was a friend of mine), who said "I've got a mate at XXX bank which runs YYY system (different to ours) - I'll tell him about it"

    So he rang him up, and this is what I heard.

    "Hi there"

    ...

    "Have you seen these new large pages from Reuters?"

    ...

    "Well, try typing in BP.L"

    ...

    "Ah. Well, I'd better let you go then."

    He put down with the phone, turned to me with a grin and said "We've just brought down the whole trading floor at XXX bank."

  8. Doctor Syntax Silver badge

    Re: I can trump that !

    He put down with the phone, turned to me with a grin and said "We've just brought down the whole trading floor at XXX bank."

    Does that still work? It could explain quite a lot.

  9. Herby Silver badge

    Console? What console..

    Yes, sometimes your machine won't work if the system teletype (it was the 70's) is on the fritz. Luckly, we transported the inner workings of the ASR35 (EBCDIC) off to get fixed. On the way back from doing the transport, I thought a bit, and hooked up an idle computer (HP 2114) to take the place. A little software later (translation from EBCDIC to ASCII) and it worked. Good thing as well. The repair was to take a day, and ended up taking three as I remember it. When returned, it snapped back in, and all was well.

    Of course things like this are hard to forget. I needed to a more urgent repair a year or so later, and took out the old hardware and slapped it together again. We had it that way for about 5 days as I recall. Somehow those "temporary" lashups have a life of their own.

  10. H in The Hague Silver badge

    Re: Console? What console..

    "Luckly, we transported the inner workings of the ASR35 ..."

    Just looked that up - made me feel v sentimental. At school we had occasional access to a PDP12 (I think) with mark-sense card reader and one of those Teletype terminals with tape punch. Regret not keeping a few of those cards and a bit of paper tape - those were the days when you could hold bits and bytes in your hand and look at them.

    Thanks!

  11. Antron Argaiv Silver badge

    Re: Console? What console..

    Luckly, we transported the inner workings of the ASR35 (EBCDIC) off to get fixed.

    I worked on Teletypes when in uni. It's extremely unlikely that the ASR-35 was EBCDIC.

    The amount of mechanical redesign to make that happen would be prohibitive. I'm not saying it's impossible, but if true, it would require an impressive amount of hardware bodging.

    Now, I *might* believe that, squirreled away somewhere in the voluminous base of the '35, there was an electronic ASCII-EBCDIC converter board...

  12. Alan Brown Silver badge

    Re: Console? What console..

    "Now, I *might* believe that, squirreled away somewhere in the voluminous base of the '35, there was an electronic ASCII-EBCDIC converter board..."

    Somewhere in a junkbox at one of my parents places there's a bit of perfboard with an ASCII<->BAUDOT converter on it. The (entirely mechanical) Creed Model 7A teletype it drove is long-gone.

  13. Antron Argaiv Silver badge
    Thumb Up

    Re: Console? What console..

    perfboard with an ASCII<->BAUDOT converter

    Built one of those myself, and it would copy the shortwave Baudot RTTY transmissions on an ASCII 33 machine. I had loads of fun with it, still have it...somewhere in the attic.

    The UARTs today don't support 5-level, so you'll need to bit-bang the Baudot to drive whatever.

    I have a TT-253/UG "typing reperforator" which could stand to see some data...must get to work on that.

  14. Anonymous Coward
    Anonymous Coward

    Re: Console? What console..

    "Just looked that up - made me feel v sentimental."

    Remember seeing KSR33 in the "really old computers" section of a technology timeline at the Bradford Media Museum and similarly feeling very nostalgic ... my first computing was done on one of those connected to a Data General Nova at school (had similar waves of nostalgia at the Computer History Museum on a visit to Mountain View where they had a Nova ... took great temptation not to pretend to load the bootstrap on the front panel switches!). My abiding memory of teletypes is that, unlike virtually every other keyboard, if you hit it in anger you came off worse!

  15. gotes

    Chess?

    Back in 1996 I was playing Duke Nukem 3D on my PC.

  16. jmch Silver badge

    Re: Chess?

    Given that it took a supercomputer quite later than 1996 to defeat a human chess champion, the ability to play Duke Nukem 3D doesn't indicate a particularly high aptitude for chess

  17. Paul Shirley

    Re: Chess?

    You could play a passable port of Sargon Chess on the original Gameboy by 1992! Wouldn't want to crank the search strength up much compared to PC ports though.

  18. Alien8n Silver badge

    Finance stories

    I recall a tale told to me by someone who allegedly worked in a trading bank in the UK many years ago. Apparently one trader was so incensed by him taking more than 2 seconds to diagnose what was wrong with his PC (something along the lines of cables kicked out by user) that the user actually started kicking him and screaming abuse. His response was "in that case fix it yourself", followed by "I earn more than you, I'll get your boss to fire you" and the bosses reply of "you can fix it yourself, you don't earn more than him."

    Not sure how true the tale is but every job I've seen for the financial sector has stipulated experience within the financial sector or the job is specifically for a graduate with no real life job experience. Something to do with not knowing that you're supposed to be treated with respect by other employees apparently...

  19. OssianScotland

    "over-zealous yank"

    Am I the only person who read that title totally differently to the way the author intended?

  20. Admiral Grace Hopper

    Re: "over-zealous yank"

    No, you aren't. I immediately started to compile a mental list of former colleagues who fitted the description, being careful, of course, to omit the Good 'Ole Boys from Texas who didn't appreciate being called Yankees.

  21. Alien8n Silver badge

    Re: "over-zealous yank"

    They really need to stop just tossing these headlines out there..

  22. Aladdin Sane Silver badge

    Re: "over-zealous yank"

    Seeing as it was a US bank, I think the headline is fair enough.

  23. Doctor Syntax Silver badge

    Re: "over-zealous yank"

    "Am I the only person who read that title totally differently to the way the author intended?"

    What way did you think the author intended you to read it.? This is el Reg, not /.

  24. jmch Silver badge
    Trollface

    yank?

    anyone else first thought 'yank' referred to a 'yankee'?

  25. disgruntled yank Silver badge

    Re: yank?

    Yes, indeed.

  26. David Nash Silver badge

    Re: yank?

    @jmch

    Yes that's obviously what @OssianScotland was referring to. No need to spell it out.

  27. DropBear Silver badge

    Re: yank?

    Is that even a word people still actually use...? Or have we stopped some time around Mark Twain?

  28. DougMac

    Sun IPX "server"?

    The Sun IPX was never meant to be a server, but a tiny workstation. Typical configuration was something like 40MHz CPU, and 16MB or 32MB of RAM. It came in a tiny "lunchbox" case. (about 1 foot by 1 foot by 8 inches tall).

    I doubt a PC era hardware was more powerful, but almost certainly, SunOS was 1000 times more stable and capable then anything running on a PC at the time.

  29. Alan Brown Silver badge

    Re: Sun IPX "server"?

    "The Sun IPX was never meant to be a server, but a tiny workstation. "

    Perhaps, but it packed 16MIPS and 16MB RAM when PCs were lucky to push 1MIPS and have 2MB.

    It took several years for PeeCee technology to outclass them and those PCs needed substantially more memory and CPU MHz to do so, which was surprising given that SPARCs are RISC and Intel are CISC, so it's supposed to be the other way around.

    My first one cost $2500 (secondhand) in 1993, and another $1000 for the staggeringly large 1GB drive to go with it. It was the core server of my network for about 6 years (and that keyboard thing was annoying)

  30. Anonymous Coward
    Anonymous Coward

    Re: Sun IPX "server"?

    "Perhaps, but it packed 16MIPS and 16MB RAM when PCs were lucky to push 1MIPS and have 2MB"

    Remember at work when we maxed out one SparcStation 4 with an insanely large 64MB of RAM!

  31. TobyDog
    Stop

    Not a Kernel Panic, but no boot either

    I used to support a lot of video editing and effects systems running on various Silicon Graphics workstations. The Tezro was particularly good in that it would display a GFX error on the front panel LCD if the mouse was unplugged. What it was trying to tell you was that it couldn't move a pointer in X11 without a mouse, but it halted the entire system and made it look like an expensive hardware fault.

  32. Bandikoto

    Maybe not 1996 PCs, but 1987 Macs did it well

    It was 1987 and although the start-up I was working for had just flamed out, the one client we had committed to deploying our system was building their new trading floor and were not about to fill it with Quotron terminals. Instead, we deployed a fleet of Macs, both color and black&white, both ethernet and localtalk, to handle the traffic. I did the transport layer, both server-side and client-side. Riding up on the train one day, inspiration hit me and I came up with a semi-reliable (reliable enough, as measurements proved) UDP broadcast protocol that solved our "TCP can't possibly handle hundreds of machines" - at that point, with the Mac II being the most powerful machine we could buy. And yes, it handled Black Monday later that year with aplomb.

  33. Anonymous Coward
    Anonymous Coward

    IBM memory and keyboards

    Dunno if it's true, but heard years ago that if you put in memory tin plated SIMMs into a IBM computer of some description, instead of the gold plated ones - that the keyboard will stop working?

  34. It's just me

    Re: IBM memory and keyboards

    Using different metals in the contacts would cause galvanic corrosion which would cause memory errors.

  35. tony trolle

    Re: IBM memory and keyboards

    seen the tin plated ones get dirty and error, fixed with a hard rubber/eraser over the contacts

  36. disgruntled yank Silver badge

    lockup, yes

    I do remember freezing a Sun box or two by pulling the keyboard. Reboot, no.

  37. Paul Hovnanian Silver badge

    The guy who set up that rack

    should be taken out and shot. Lesson #1: All cords to hinged or sliding equipment need to be secured. Most rack mount keyboard/display units have a hinged or flexible arm just for this.

    I did work for an electric utility where rack mounted relays had to be slid out for calibration. Before s substation had been commissioned, the relay techs found that the leads to the racks had been made too short by the electricians. They proceeded to take side cutters and lop off every cable to every racked relay. Then called the electricians to come back and try it again. Correctly this time.

  38. the hatter

    Re: The guy who set up that rack

    > Most rack mount keyboard/display units have a hinged or flexible arm just for this.

    It did take manufacturers several decades to get to this point, and neither low-end kit, nor reassuringly expensive boxes were necessarily quick to join this trend. Not necessarily feasible to have everything secured too, when a rack's keyboard would be moved between servers at the top/middle//bottom of the cab. Especially in places which didn't have the budget for expensive serial console servers (only for a moderately expensive proprietary keyboard).

  39. Stevie Silver badge

    Bah!

    While I have never seen a kernel panic from a yanked keyboard cable I wouldn't write it off as BS. Pulling the plug cleanly would have caused a stop-a I think (in-head memory is faulting when I try and access long-term storage for useless facts) but if the plug deformed and the pins separated from he motherboard connector in haphazard order, who knows?

    I once worked at a place that issued their staff those neato IBM Thinkpads with the butterfly keyboard - the one that sprang out all steampunk-like and assembled itself into a full-sized keyboard when you opened the lid of the decidedly tiny computer.

    The way IBM managed the miniaturization trick was to move all the peripherals normally fitted into the case (floppy, CD etc) out into external units that connected via a parallel or serial port as appropriate.

    Problem was, the back of the lappy had a proprietary port the width of the laptop designed to dock into a "workstation" frame, and to break out the pins of this enormous harmonica-like port you needed to snap in a special accessory that hooked over one side of the laptop and snapped onto a spring-loaded pawl on the other. Still with me?

    A colleague was madly typing away one Wednesday afternoon when the plastic pawl broke and the port adaptor thingy ejected. But it did so in an arc, starting on the left and not completely disengaging from the right side when the laptop reacted.

    The bizarre unplanned unplug resulted in the lappy hardware sending all sorts of interrupts to Windows 95 in rapid succession, which decided that the laptop obviously needed to load a new profile as it was clearly undocking. From something. It didn't have a profile to load, but didn't let that get in the way of an attempt to do so.

    It got about halfway through this before the laptop hung (or was turned off in a panic; I was suspicious but couldn't prove anything).

    There were lots of company-revenue-generating ideas on that machine's hard-drive, but because of the new-fangled RIDs 'n' SIDs 'n' Bellz 'n' whistlez all now were unavailable without recourse to advanced low-level digital spanners and hammers.

    They were still recovering the machine when I left work on Friday afternoon.

  40. Kernel Silver badge

    My small cock-up

    Many years ago, back when I worked for a telco, I was given the job of replacing a noisy fan tray in the processor module of a NEAX61E-VS (very small) digital exchange, the purpose of which was to provide the 0800 service for a smallish country.

    Unlike its big brother, the NEAX 61E, the VS version had both of the hot-standby processors in a single module, rather than two separate modules.

    What wasn't made very clear in the procedural documentation was that the 50v power feed to processor modules was via relays which were held operated by a signal from the module's fan tray - obviously to shut the processor down should the fans ever fail. As it turned out, this was a fine concept for the 61E, but not so clever for the VS version.

    So, having undone the retaining screws I carefully removed the fan tray, only to be greeted by a distinct lack of blinky lights on the processor consoles and the cards which filled the processor module.

    By the time I worked through the convoluted boot process (by modern standards) and then loaded the core software followed by the latest data backup (all from tape cartridge) people had most certainly become aware of my activities.

    No point in posting as an AC for this, it was all documented and acknowledged at the time.

  41. Alan Brown Silver badge

    Re: My small cock-up

    "By the time I worked through the convoluted boot process (by modern standards) and then loaded the core software followed by the latest data backup (all from tape cartridge) people had most certainly become aware of my activities."

    Not as aware as they became when someone futzed up PMR switch.

  42. Anonymous Coward
    Anonymous Coward

    That reminds me of the time in the mid 90's that i went to clean the old vax/vms consoles keyboard.

    In came the the boss and everyone into the computer room, to find me, cleaning cloth in hand, proud of the shiny keyboard which had been a little crusty before.

    Apparently, the pause/break key really did what it said.

    But you just had to press resume and the system started where it left off..... phew(ish)

  43. jms222 Bronze badge

    Sparcs

    I do remember that Sparcs of about twenty years ago would indeed drop you into the Open Firmware prompt if the keybioard was removed where you could type c to continue (having replaced it of course).

  44. Crazy Operations Guy

    Ah, the early Sun days

    I remember those days where there desktop machines were really just mainframes / mini-computers that were trimmed down with the standard TTY and printer were replaced with a video controller and a keyboard. They'd go all weird weird when one of those wasn't working because the system never expected those things to not be available (what with them supposed to be soldered on...).

    I also remember some of the models that tried to be smart and if a keyboard and/or monitor was missing at boot, the system assumed you are wanting to use Serial 0 as the console. So if you accidentally knocked out the keyboard at boot, the system would work just fine (OS would boot, daemons would start and begin doing their thing, etc), but nothing would be displayed on the console, nor would keyboard input do anything (Since the keyboard input is now routed to TTY1, but TTY0 is attached to the kernel).

    A lot of it was just really teething issues and programmers needing to unlearn a bunch of assumptions from before the beginning of the transition from computers having their own rooms to them being out in the office.

  45. fixit_f

    I worked for a large investment bank in the late 2000's with multiple trading desks and about 2000 employees, plus two huge server farms spread across two datacentres for DR purposes. The whole trading infrastructure was an absolute rat's nest of duplicated systems and people's vanity projects which all had to be kept in sync with a sprawl of horrible middleware and half-arsed trade feeds, all built completely differently and inconsistently. One day one of the tech team managed to issue a remote shutdown with some kind of crazy wildcard that rebooted every Windows desktop and server across the whole organisation. The time it takes the machines to come back up is trivial, but in a normal (for example post patching) environment the machines would normally all be brought up in tiers in a particular order - domain controllers first, then things like email servers, then servers housing SQL Server databases, then servers housing application engines, then finally things like desktops. This was important as a lot of the services were set to auto-start and they'd get confused if things they depended on weren't already up. Once all these machines had been brought up the application teams would start bringing up remaining services, synchronising FIX sequence numbers for trade feeds, resolving issues with things that are partially processed or transactions rolled back as part of the shut down etc etc. So resumption of trading varied desk by desk depending on their systems, but in many cases they weren't able to trade for several hours and ironing out every single issue and making sure all trade and pricing data across all systems was correct and synched up took days. On the desk I worked for, what was the impact of not being able to trade? Well - after hours of the traders screaming at us that they had positions to close out, markets had moved unexpectedly and the P+L on the positions they'd been unable to close out was actually substantially up. That one IT guy literally made millions for the bank - he still works there now in fact.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018