back to article Software bug flattens NYSE trader

An algorithmic trading software bug is being blamed for a day of wild swings at the New York Stock Exchange – and has resulted in the trader placing the dodgy orders reporting a $US440 million pre-tax loss. What’s been called a “mini flash crash” by Forbes saw 150 NYSE-traded stocks, from General Electric down to minnows, …

COMMENTS

This topic is closed for new posts.
  1. Dodgy Geezer Silver badge
    FAIL

    Zerohedge reports....

    ...that their automated trading software (which performs arbitrage trades at millisecond intervals) was buying at the Sell price, and selling at the Buy price - the opposite of what it should be doing...

    Quick way to lose a company....

    1. John Smith 19 Gold badge
      WTF?

      Re: Zerohedge reports....

      I understand your description.

      I'm just have trouble processing how anything *that* stupid could get through testing.

      1. Anonymous Coward
        Anonymous Coward

        Re: John Smith 19

        The software has probably been running for a lengthy period of time without any problems.

        This means that the bug probably only occurs after an unlikely series of events.

        Your comment tells me that you do not develop, or play any part in developing (including testing), complex software for a living.

        1. Platelet

          Re: John Smith 19

          "The software has probably been running for a lengthy period of time without any problems.

          This means that the bug probably only occurs after an unlikely series of events."

          The software with the bug in went live Tuesday night according to Knight Capital Chief Executive Officer Tom Joyce

        2. Seán

          Re: John Smith 19

          Actually your comment reveals your incompetence and sloppy nature.

    2. stanimir

      Re: Zerohedge reports....

      Not all (or most) 'trading systems' are based on arbitrage. I do not think it's just buying at the ask, selling at the bid case.

      There are many ways for shit to hit the fan.

  2. cosmo the enlightened
    FAIL

    Yup

    The quest for profit in micro second deals is one of the wild west frontiers that needs to be more regulated. These systems are all over the markets; particularly in the FX arena.

    It's nice to see that their system went through traditional dev test and production cycles that SOX compliance would call for.

    When will these companies finally realise they are tech houses with a bank or trading license and organise acordingly.

    1. stanimir

      Re: Yup

      @cosmo

      they are tech houses with a bank or trading license and organise acordingly.

      It aint happening ever. The chiefs are just suits and the computers are the necessary evil. I realize it's too simplified and any pointers to real-world stuff is strictly banned.

      These systems are all over the markets; particularly in the FX arena.

      Overall FX is unregulated, a lot of FX is retail and the retail is bound to lose its hard earned. The retail FX players don't really have micro-seconds strategies or possibility to execute in such manners. With that regard, the big players/brokers can effectively trade against their own clients

  3. Herby

    Interesting stuff this "software"

    It seems to have flaws and not tested well. Not a good thing. Was it produced in Redmond? It sure isn't open source, that's for sure!

    1. Anonymous Coward
      Anonymous Coward

      Re: Interesting stuff this "software"

      '... Was it produced in Redmond? It sure isn't open source, that's for sure!'

      Much as I'd like to slag off MS, open source isn't somehow magically immune from 'monkey' coders and their stupid errors.

      Yes, in the open source world they *may* get picked up faster, but only for a given value of *may*. Having delved through the source code of a number of open source projects in my time, I could probably have regaled you with tales of coding howlers in some projects, however, the application of liberal amounts of alcohol over the intervening years has had it's legendary effect on the little grey cells, most of that chapter of my life is thankfully gone.

      All you're going to get is this story, after investigating a recurring 'issue' we had at one job, I once had to 'correct' 214 lines of code (that particular number stays in my head, as it was a mind numbing three weeks of getting to know the original code, spotting the flaws and making the changes and a further fortnight of debugging the code with those frigging 214 changed lines) and further excised about 300 or so lines of code from a (then) popular open source IMAP server (I don't know if it still is popular, don't really care). Yes, the changes worked (and did so flawlessly for another three years-they then switched platforms), no, the maintainers of the software were never told about the issues or my changes to the code, as I wasn't allowed to under the terms of my contract.

    2. Anonymous Coward
      Anonymous Coward

      Re: Interesting stuff this "software"

      I love open source software and dislike MS as much as the next hater, but in this case even I think you're pissing up the wrong tree bashing MS! What have mistakes in trading algorithms got to do with MS or indeed any O/S software?

      Muppet!

    3. Anonymous Coward
      FAIL

      Re: Interesting stuff this "software"

      Sigh, summer holidays are here again I take it.

    4. Anonymous Coward
      Anonymous Coward

      Re: Interesting stuff this "software"

      I hate to tell you this, but most of the *really* quick stuff runs on customised Linux kernels because every clock cycle counts. I happen to know one of the maybe 20 people who are actually smart enough to do the math underlying this stuff, and I know for a fact he would not have made such a mistake because he is close to anal about testing - and you just had a nice public demonstration why.

      From what I recall from my time at one of those large companies, the trading floor may have MS Windows, but a trading engine which can recognise a trend and spit out an order before you have even seen the trade on a screen is NOT going to run anything Microsoft has ever come up with. It would be the costliest BSOD ever.

    5. Rovindi
      Windows

      Re: Interesting stuff this "software"

      Wow. That has to be one of the daftest posts on here in a long, long time.

      Do you really imagine that something like this would be an open source project? I mean, really.

  4. amanfromMars 1 Silver badge

    A Quick Way to Lose Everything in a Great Flash Crash of Quant Trading Machines*

    "and the embarrassing crashes that plagued the Facebook IPO, preventing the stock from plunging anywhere as quickly as it probably should have on listing." ….. Ouch, that hit the target bullseye, RC. And how refreshing to read some real reporting on dodgy shenanigans.

    And is market-rigging exactly the same as market-making only quite different because no one much ventures opinion on that qubit like territory ….. "Reuters states that the only comment to come from the company is a terse statement that “a technology issue occurred in Knight's market-making unit related to the routing of shares of approximately 150 stocks to the NYSE”."

    You do know about quant trading, don't you?

    You know, whenever something can be one thing and another at the same time as being something different altogether …….. "A qubit has some similarities to a classical bit, but is overall very different. Like a bit, a qubit can have two possible values—normally a 0 or a 1. The difference is that whereas a bit must be either 0 or 1, a qubit can be 0, 1, or a superposition of both.

    And is it undoubtedly revealed there, that lightning quick machines and not slow pedestrian man dictate terms on the NYSE and are responsible for US business prices? Wow, that can easily be, and therefore will probably definitely surely be, unless there is something available to divert it, a virtual trading disaster to monitor because there would be practically no one able/enabled to effectively mentor the activity in what would not be dissimilar to a NEUKlearer HyperRadioProActive IT System with Command and Control of Virtual Machine Units and SMARTR IntelAIgently Designed Entities for the Phantom Trading of Novel Stock/Options/Derivatives/Futures/Hedges.

    Is pulling the plug on the NYSE, an available option to consider, to avert a market trading disaster/natural free market correction of an anomalous perversion?

    *And whoever would have thought that Title would ever have made sense and is able to destroy ponzi empires in a virtual instant and practically overnight, or not as the case may be if things are radically changed to reflect and accommodate the New Realities of Life in LOVE. ...... which Conquers All, Every Time Everywhere.

    1. NumptyScrub

      Re: A Quick Way to Lose Everything in a Great Flash Crash of Quant Trading Machines*

      quote: "And is market-rigging exactly the same as market-making only quite different because no one much ventures opinion on that qubit like territory ….. "

      Came here to post this :) I have had a few "financial professionals" tell me how the market cannot be controlled or predicted by one or more players (it is a magical organic thing, and apparently immune to malicious manipulation), yet here a "software glitch" at one trader can rape the value of many companies, in a surprisingly short space of time. Looks like it's perfectly vulnerable to accidental manipulation, eh?

      Mind you, these are the same buffoons that think short selling is a good thing, because they are incapable of seeing the carnage beyond the immediate profit. And that's tells you pretty much all you need to know about the group who hold the fiscal strings of every publicly listed company worldwide. If you want stability, don't IPO... going public is simply a tool for CEOs to take the money and run :)

      1. Vic

        Re: A Quick Way to Lose Everything in a Great Flash Crash of Quant Trading Machines*

        > these are the same buffoons that think short selling is a good thing

        Short selling is an *essential* thing.

        Without shorting, a share price can stay artificially high long after the value of the company has decreased - those who own the shares just don't part with them. Short-selling stops this problem; it allows those who own the shares (and expect their value to go up) to loan them to shorters (who expect their value to go down). If the value goes up, the short-seller loses. If it goes down, the share owner loses. But importantly, the value does not stagnate.

        Naked short-selling, on the other hand, ...

        Vic.

    2. Anonymous Coward
      Anonymous Coward

      Re: A Quick Way to Lose Everything in a Great Flash Crash of Quant Trading Machines*

      amanfromMars "1" - what happened, lost your password or did you get cloned?

      1. amanfromMars 1 Silver badge

        Re: A Quick Way to Lose Everything in a Great Flash Crash of Quant Trading Machines*

        amanfromMars "1" - what happened, lost your password or did you get cloned? .... Anonymous Coward Posted Friday 3rd August 2012 08:07 GMT

        If I remember correctly, AC, El Reg did a bit of tinkering with the comments board, ... an upgrade ...... and hey presto, amfM1, who is also amanfromMars too. It was all pretty painless.

  5. Herby

    Another thought...

    Is that to buy high and sell low, or buy low and sell high? Let me see!

    Buy, Sell, Low, High, it is some combination there I need to do.

    Money? Just fake it! I understand that it is printed somewhere, or stored as digits in another place...

  6. Peter 39

    if not now, when?

    NYSE has been seeing this stuff for a while and has not done anything about it..

    Computers can do things much faster than any human, and screw them up much faster than any human.

    It's time that NYSE and other exchanges established limits so that the volatility swings from a single company do not exceed the net worth of the company. That is, if a company has "middling" net worth, then massive trades should not be accepted. If it has "low" net worth, then the limit should be lower. If it has a lot to lose, then the limit should be higher.

    Having a small company generate a real-quick loss of a half-billion dollars is just stupid

    1. Seán

      Re: if not now, when?

      Box of chocolates

  7. PT

    The answer seems obvious

    This sort of thing is a good justification for a small financial transactions tax, perhaps waived if the stock is held for 24 hours or more. While bankers don't seem to give a damn about losing millions of investors' money in speculation, they tremble with fear and outrage at the thought of having to pay a few pennies tax. It would make them pay a lot more attention.

    1. Anonymous Coward
      Anonymous Coward

      Re: The answer seems obvious

      Err, no that is a bad idea. It just makes trading more expensive and ends up with multinationals delisting and relisting overseas thus your stock market goes from having a FTSE 100 as the main index to having AIM. The yanks won't put a tax on transactions as they are anti-market intervention and taxation in general. It has been mooted as a plan in Europe but then the Europeans seem to just be seeking ever more creative and theatrical ways of ending up in the stone age.

      The error was buying high and selling low which is the opposite of what the market maker is supposed to do. A Tobin tax will not have any effect on stupidity...

      http://www.zerohedge.com/news/what-happens-when-hft-algo-goes-totally-berserk-and-serves-knight-capital-bill

      NB Nanex are extremely good and forensically examining market action for anomalies and then decyphering what happened.

      1. Anonymous Coward
        Anonymous Coward

        Re: The answer seems obvious

        The easy fix to that is to have taxes paid on the earning of all money earned by every company in the place that money was earned. Any company not wanting to pay said tax will have it's assests in that country seized and not be allowed to do business in that country.

        I don't think it would take too many asset seizures to bring the business community around.

        1. Anonymous Coward
          Anonymous Coward

          Re: The answer seems obvious

          Your ignorance of how global financial markets, and global finance in general, work is heartwarming yet nonetheless astounding.

  8. William Boyle
    FAIL

    No due diligence here

    All program and high-speed trading organizations should be REQUIRED to run their software in a simulated trading environment and certified "stable" and non-disruptive of the market before it is allowed to trade in the actual market. The fact that these boneheads have screwed up the equities markets so thoroughly in the past couple of years must result in new rules regarding how quantitative trading software can be used, or this sort of crud is just going to continue to rape our investments and retirement funds. Companies that cause this sort of disruption should

    1. Eat their own losses, or if they made a profit, disgorge it back to those who did lose as a result.

    2. Be required to make good any and all losses of others caused by their trading.

    3. Be fined at least 2x the value of the market disruption that they caused.

    4. Have adequate insurance to make good on these "fines" should they not have adequate funds of their own.

    These sort of regulations/requirements would put an end to this cruft in short order I would think (hope)...

    1. Rob 21
      IT Angle

      Re: No due diligence here

      The rape of our investment funds and pensions is not a bug, it's a feature. How else are GS going to continue to earn their money in an environment where there are no true earnings, only transfers?

      1. Anonymous Coward
        Anonymous Coward

        Re: No due diligence here

        I agree. I've become so cynical that I wonder if we followed money would we find that this was no accident at all. Perhaps a test run for a bigger incident to come?

    2. This post has been deleted by its author

    3. Anonymous Coward
      Anonymous Coward

      Re: No due diligence here

      Agree with your sentiments but unfortunately its not possible. As to why see my comment below: "What Cantor’s e-Speed taught me about the frailties of HFT!".

      Basically information in the market in not created equally in terms of inside knowledge of how an EXCHANGE’S SYSTEMS actually operate. All the usual suspects use privileged information here to game the system. When you add this to the fact that the same privileged few also get to locate their servers beside the exchange’s servers, well it all screams to me that HFT has to be banned!

  9. Anonymous Custard
    Facepalm

    As the saying goes...

    To err is human, to really foul things up you need a computer...

  10. Duffaboy
    Joke

    Finally Software Bods fess up

    Well at least its not "a hardware problem"

  11. Purlieu

    Financial institutions and computers

    What could possibly go wrong

  12. Shagbag

    Zero sum game.

    Knight Capital's $440m pre-tax loss = someone else's $440m pre-tax gain.

    1. johnnymotel

      it's all funny money anyway

      sure a loss here is a gain there, but when the DOW drops 10% in one session, I think just about everyone loses something.

      Then they bemoan the exit of the retail investor....and the markets become a computer game. Game Over.

  13. John Smith 19 Gold badge
    Meh

    So "mini" flash crash stopped Facebook sliding as fast as it should have?

    How very "fortunate" for them.

    And of course *completely* unpredictable.

  14. Magnus_Pym

    About tax?

    Short selling is gambling. Gambling is taxed.

  15. This post has been deleted by its author

  16. Anonymous Coward
    Anonymous Coward

    What Cantor’s e-Speed taught me about the frailties of HFT!

    What Cantor’s e-Speed taught me about the frailties of HFT!

    Speaking from experience here's three more reasons why HTF's mess up. I was working at a large bank for US interest rate traders once. As an example, the traders wanted a mechanism to move large blocks of 30 year bonds without moving the market. So the goal was to design a black box (BB) to move small parcels throughout the day, trading on defined limits of the market at that time. The latter phrase proving crucial here!

    The problem is: whenever its a busy morning the BB is reliant on timely receipt of MESSAGES and the timely order of those messages from the exchange’s trading systems, which is itself a kind of BB. And all of this withstanding the usual glitches like network latency, bottlenecks, outages etc What I was discovered was :-

    #1. You can't recreate a live market in Beta. Why not? A. Because the beta system is not anything near as liquid, B. the beta is often not even running the same Exchange software version, and C. your black box is playing with other robots-- and not humans and robots and real-world actualities.

    #2. There are significant subtleties in the way a busy morning can affect the ORDER in which your black box receives messages into its queue! I found cantor's e-Speed system sent messages out of order making it tricky to pair up last trade prices with current market pricing. It meant trade confirms with the actual SIZE moved were delayed well beyond where the market was now. In short, my BB frequently found itself in ill-defined state.. What to do next.…? Where was the market now? How much size was actually executed? What should the next BID / ASK posting be?

    #3. The exchange provider only shares some technical subtleties with its clients, that is-- unless your are one of its darlings that generate the most fees: hedge funds and Goldman, Deutsche et al... What this means is that you are often operating in the dark. In addition the Exchanges systems can be the actual cause of the glitch. Without warning Cantor re-numerated trader operator ID’s overnight. Anyone who hadn’t re-logged on was unwittingly now using the wrong trader ID. Boy, that was messy! Traders getting another traders confirms, trades executing in the wrong books etc.

    1. Snark
      Alert

      Re: What Cantor’s e-Speed taught me about the frailties of HFT!

      It's been nearly 17 years since I was in that business (and I don't miss it a bit) but I can tick off everyone one of your three points as being the bane of my life at the time.

      In the particular exchanges I was dealing with, their "test" market were so dead often I'd be the only price in there. I'd have to use two different "test" trader accounts to trade against myself to get anything moving at all. Liquidity, zero, volatility, zero, That just can't compete against the speed of a swiftly moving market. In the end the best way of testing was to record the live feeds from the exchange of market movement then play them back through our test systems, bypassing the exchange itself completely. I'd like to have split off the feed live into our real trading system and dummy trading system to compare them, but I always decided physically disparate systems was the ultimate firewall "just in case" something went wrong.

      I hope the exchange test systems have got better.... but knowing how secretive everyone was about revealing what their systems could do... I suspect everyone will still be holding back.

      1. John Smith 19 Gold badge

        Re: What Cantor’s e-Speed taught me about the frailties of HFT!

        " In the end the best way of testing was to record the live feeds from the exchange of market movement then play them back through our test systems, bypassing the exchange itself completely."

        This sounds like the most likely way to build a system that is robust. Tough to do, as you have to catch *every* little nuance in the stream you recorded, it might not show all the most awkward behaviors on that day etc. But still the closest thing to hooking your development system to the live exchange with live consequences. It seems the exchange *expects* a human user and your S/W has to mimic one, even down to logging out at the end of a trading day to keep things on track.

        Forgive my ignorance but I did not realize that the exchanges even *operated* a test environment, weather or not it was any good. I'd guess *every* house was on their own when it came to developing this stuff.

        Interesting stuff but sounds quite stressful.

    2. Magnus_Pym

      Re: What Cantor’s e-Speed taught me about the frailties of HFT!

      So essentially the problem is how to model systems so complex that the emergent behaviour cannot be modelled in order to test how something reacts to an unpredicted event. What could go wrong?

  17. Sundiata
    Stop

    Change, Configuration, Environment and Test Management

    Anyone who works in IT knows that when you hit a situation where "it was working yesterday and it's not working today", the first question you ask is "what has changed?"

    If this was a change in Pre-Prod or Test environments, Knight Capital's Configuration & Environment management functions should have ensured full logical (and ideally physical) separation of environments so no "real" trades were made.

    If this was a change in the production environment it should never have been allowed to go live without extensive functional, regression and user acceptance testing.

    The IT Change and Control function should have enforced both points above.

    Simples.....

This topic is closed for new posts.