back to article Millennium bugs hit stock exchange

The London Stock Exchange suffered a second day of problems yesterday as its new trading platform struggled to function again. On Tuesday the market failed to close correctly, causing confusion over closing prices for brokers and traders. Yesterday the MillenniumIT system displayed zeroes against some bid and ask prices, …

COMMENTS

This topic is closed for new posts.
  1. James Le Cuirot
    Dead Vulture

    "Millennium bugs"

    I know El Reg likes to have fun with their headlines but I actually thought it was the millennium bug at fault here until I got half-way through the article.

    1. Anonymous Coward
      Unhappy

      re : millennium bug

      I expect it won't be long before some ignorant twat turns up to tell me that all the code fixing and testing I did was just a big IT conspiracy to make money.

      1. Anonymous Coward
        Black Helicopters

        But it was all just a big IT conspiracy to make money.

        I know, because I was in the room when it was planned and by the time the gag order expired, it didn't seem to be worth outing them any more.

      2. maclovinz
        Happy

        RE:AC

        It WAS...How'd you KNOW!?!!?!!!

  2. Anonymous Coward
    WTF?

    You what!?!?!

    Since when has SUSE Linux Enterprise Server been equivalent to Microsoft .NET framework? Will El Reg be telling us that a new car based on the Internal Combustion Engine is the equivalent to a new car based on the VW Golf platform next?

    You are supposed to be an IT rag after all...

    1. Anonymous Coward
      WTF?

      I'm not sure..

      just exactly what you are trying to be detrimental about, the VW Golf, or any brand new ICE based automobile?

      And,

      Does this mean the FTSE is looking for new Linux Admins?

    2. John 62

      Mono

      Seems like it's contracted the nuceoleosis, though! Maybe from kissing the MS boys.

  3. Anonymous Coward
    FAIL

    "The low latency system is based on... Linux"

    'nuff said.

    1. Anonymous Coward
      Happy

      AAAAAAA++++++++ excellent commentator

      would down-vote again.

      1. Version 1.0 Silver badge
        Happy

        *nix-tards

        *nix based systems are not real-time systems. Well, not if you've actually used a real-time system. Yes you can make them *look* like real-time if you throw enough horsepower at them but it's rather like watching that guy with the spinning plates ... as the number of plates rises ... he's going to start dropping them.

        1. Anonymous Coward
          FAIL

          FAIL man

          You can get real-time libraries for debian, etc and YES IT WILL BE A REAL TIME SYSTEM. Of course, a traditional real-time system might be something more people would be comfortable with, but not necessary.

    2. Anonymous Coward
      FAIL

      Commentard!

      I'm betting on the fact that the Linux-bashing commentard is a BSDite!

  4. Tzael

    Warning: Pure speculation

    See this is what happens when you continually replace good coders with fresh uni graduates - the lessons learned don't get passed on.

    1. Anonymous Coward
      Anonymous Coward

      RBS may well be in the same boat

      Royal Bank of Scotland is about to replace a whole load of its Infrastructure engineers with guys in India. Over the next year or so 10s of thousands of man-years of experience will be made redundant in the UK and picked up in India.

      It'll be interesting to see how that experience goes for them...

      1. Anonymous Coward
        FAIL

        It isn't about to happen ...

        It is already fully underway. Its pathetic watching it.

        AC for obvious reasons ...

    2. Ian Michael Gumby
      WTF?

      Uni graduates?

      Try off shore resources!

      I was just at a local user group last night. We live and work in an industry where the software developer is de-evolving.

      'We are Devo!'

  5. Anonymous Coward
    Anonymous Coward

    let's shut it down for a week or two

    see if we can't get on with our lives without worrying about money every second of the day.

    maybe people would even find the time to be happy again.

    1. The Commenter formally known as Matt
      Coat

      re: let's shut it down for a week or two

      let's shut down your job for a week or two

      see if you can't get on with you live without worrying about money every second of the day.

      maybe you would even find the time to be happy again.

    2. Apocalypse Later

      Yeah, the problem is money

      People who bad-mouth money don't get what it is. You can't have lives without it. It is how resources get where they are needed and how injustices are healed. There are also a lot of sharp types who are constantly looking for a way to divert some of the flow into their own pockets- that's how we got flush toilets and mobile phones.

      You think you could be happy without thinking about money? Stop thinking about it then. Let us know how you get on.

      1. Anonymous Coward
        Troll

        Re: Yeah, the problem is money

        I said shut the stock exchange down for a fortnight, not ban money forever and retrospectively so as to remove all toilets too.

        Maybe you need to consider the subtle but enormous difference between what I wrote, and what you think I wrote.

      2. The Fuzzy Wotnot
        Pint

        Christ's sake!

        Lighten up a bit!

        I think the original poster was making a thing called a JOKE. You see a JOKE is a funny thing people say to provoke a reaction to a situation that is becoming tense. JOKEs or HUMOUR can relieve the stress and tension and allow people to relax a little!

        Now why don't you relax a little and take the comment in the vein it was so obviously intended you muppet!

        1. Disco-Legend-Zeke

          When You Are...

          ...a venture capitalist, or an insider getting the IPO, you are buying a piece of the horse.

          The Stock Market is just the sports book or the pari-mutual window. Closing the OTB parlor will not stop the races.

      3. codemonkey
        Go

        RBE or RBOSE

        Google it...Money is not the root of all evil...of course...but the antagonistic competition for profit is...it pits me against you...money is only the oil...still, we can, as a species, live without either...You lot have just not worked it out yet..one day :)

        Time for the brightest minds on the planet to waken up I'd say...

    3. Anonymous Coward
      Anonymous Coward

      Brilliant Idea

      You remind me of the guys at the G20 protests who, outside the Bank of England, had a massive banner saying

      "Abolish Money"

      I think you'll find it's a bit more complicated than that.

      1. Alan Firminger

        & , more seriously

        Capitalism isn't working

    4. Anonymous Coward
      Happy

      This is perverse.

      If it was necessary a workaround would be found just as they did in this case. Critical bits would continue. 'Keep Calm and Carry on"!

      Don't even consider talking about real time systems and their relative structure and merits until you've considered something 'mission critical' like train or EMU control, nuclear station or explosive chemical reactor plant control.... That is real 'real time' control. Everything else is just control. I suppose timeliness of information in the stock exchange starts to become important were you have automatic trades happening - but that is just automated greed...

  6. Anonymous Coward
    Anonymous Coward

    Ok...

    Obligatory comment:

    I seem to recall loads of people slagging off Windows, when the previous system was having problems, saying how Windows should never be used for mission critical etc. etc. etc. Now I presume that we'll see similar people saying the same about Linux...

    In reality, hows about this:

    Both OSes are perfectly capable of running high up time/high load systems provided:

    1) The software is designed and written properly

    2) The OS builds are competently designed and implemented

    3) The infrastructure is designed and implemented properly

    4) The operators are competent

    And finally as a tedious PS, to show I've not left anyone out:

    Why aren't they using mainframe?

    1. Anonymous Coward
      Anonymous Coward

      Ok... - er, no

      Trading systems require real time features - that's the "low latency" bit. .NET runs on top of the CLR, which isolates it from the critical timing facilities it needs. Java would have been another awful choice for similar reasons.

      The SUSE system will be based on the Linux kernel's real time extensions, and therefore makes much more sense as a platform.

      1. Shakje

        Ah I see

        so presumably you can point us in the direction of some metrics to prove your point?

        The JIT compiler only compiles CLR code into native code ONCE, then simply does an address lookup from then on in. It may be very, very slightly slower on the first run of stuff, but I very much doubt there's a critical difference in the timings (unless there's a difference in resolution of the timings, .NET is limited to 10ms IIRC, but I've run tests on it before and it has performed very well). If you have some figures to back it up, instead of making assumptions, I'll happily apologise and retract. I also can't speak for Java, but I would expect that it does suffer from some slowdown on real time-critical systems because that extra layer does actually exist.

        1. Anonymous Coward
          Anonymous Coward

          Ah, you don't see

          > so presumably you can point us in the direction of some metrics to prove your point?

          It doesn't require metrics as such, just a better understanding. The issue isn't about a "slowdown" it's about a "slowdown just when you don't want it." You're not quite right about the JIT - although it does only do it's thing once, it doesn't do it all in advance. It does it when it first encounters a new code path, which can happen at arbitrary times. Same with garbage collection - as described here:

          http://msdn.microsoft.com/en-us/magazine/bb985011.aspx

          "When a collection starts, the collector suspends all application threads..." If this happens just when you don't want it to, you get a delay and the trade happens outside the required time window. Things like memory paging also cause little hiccups. You won't notice them except in timing critical applications.

          In Java-land there's the Realtime Specification for Java modification which solves some of these problems (ahead of time JIT, controllable GC, pinned memory, etc). Maybe there's something similar in .NET-land?

      2. Anonymous Coward
        Anonymous Coward

        @AC 1153

        Trading systems don't need real time facilities, what they need is to be sized correctly to process X volume of transactions in Y secs as general load, allowing for peaks, which will also be specified. This is not realtime processing. The actual response times don't seem to have been the problem with the old or new systems, rather the availability.

      3. Anonymous Coward
        Anonymous Coward

        Not quite

        The Low Latency in this context actually refers to how quickly the exchange can receive an order and send an acknowledgement.

        Anonymous in case I'm wrong - but I dont think I am.

        1. Tom 13

          You are correct.

          And the big hedge firms can gain or lose millions on a few ms of timing. It's a big issue for stock exchanges all over the place, and the narrower they can clip those margins, the better they serve the people who trade in the largest dollar volumes.

      4. Anonymous Coward
        Anonymous Coward

        Not Real Time

        Trading systems don't need real time features - they just need to be fast enough that most of the time you can't tell the difference.

    2. Tom 13

      You get an up vote for the last tow lines,

      not the four before them.

  7. Andyroid5

    Bits bytes and b@ll@cks

    Linux..Windows..???

    Whatever fits your agenda..

    Watch out for the incoming solar flares guys..might have to make do without money..the digital kind anyway.. :)

    1. Anonymous Coward
      Stop

      Anyone else...

      ... read that as ballacks?

      1. CD001

        More like

        batllatcks as I refuse to read @ as anything other than "at" :P

        1. Joe 3

          Are you sure

          Are you sure it doesn't mean "about"?

  8. Turtle
    Happy

    Very enjoyable article

    Very enjoyable article. Am looking forward to reading *many* more like it in the future.

  9. James 5
    Joke

    Bonus?

    "But that was blamed on "human error", which the exchange claimed was likely sabotage"

    Just shows what happens when you don't pay the Greedy *ankers their bonuses.....

  10. amanfromMars 1 Silver badge

    The Rise of the Machines..... ? Advanced Intelligent Robots awakening .......*

    "But that was blamed on "human error", which the exchange claimed was likely sabotage." ..... Is there any other kind of malfunction if we are not to assume that there is mechanical life too?

    *And their Stealthy TakeOver of Systems and MakeOver of Humanity is Sublimely Supplied by your Thinking that such Thinking, and therefore Life Phorm, is Preposterous and Most Unlikely even whenever IT Floats the Program in front of you.

    1. Anonymous Coward
      Anonymous Coward

      Wow..

      I actually got to the second paragraph before I realised this comment was from amanfromMars 1

      1. amanfromMars 1 Silver badge
        Grenade

        Alien Modus Operandi scores a Perfect 10 .... Tory Toffs Tweet of a Temporal Tottering ?

        "Wow .. I actually got to the second paragraph before I realised this comment was from amanfromMars 1" .... Anonymous Coward Posted Thursday 17th February 2011 12:54 GMT

        I think that then positively proves the second paragraph rather nicely, AC. Thanks for the confirmation of the efficacy of the methodology.

        I like to think that you have been advancing, AC, rather than suggest that things are being made much easier for more general understanding and virtual comprehension, although, as is quite natural and therefore to be fully expected in the quantum communications world, is it probably a bit of one and a lot of the other, for travel towards the Singularity of Omniscience Shared for a Better Beta Operating System in Big AI Societies and SMARTer Enabled Civilizations.

  11. Chika
    Grenade

    Hangonamo!

    Why are we suddenly blaming SuSE or Windows for this? Do we know that this is an underlying OS fault? The article doesn't say. Just because I can't get OpenOffice to run on my own main Linux system, I'm not throwing out the OS, any more than I would throw out my Windows 7 system because it doesn't run Pinnacle Studio 10. Neither is the fault of the OS (in one case it's a setting problem, in the other it's the age of the application).

    I'd want to see a lot more information about the cause and effect of this fault before I'd pass judgement.

    1. Anonymous Coward
      Anonymous Coward

      I think that's the point...

      When Windows was the OS hosting the platform lots of "the usual suspects" were queuing up to blame Windows/MS. Now there are a few people blaming Linux, but really it seems to be apparent that there is nothing like enough information available to blame either. In reality it's the infrastructure, support and software that are far more likely to blame.

  12. amanfromMars 1 Silver badge
    Grenade

    Tick, Tick, Tick ...... Boom goes the Turing Bombe when Nothing Changes Fundamentally

    "let's shut it down for a week or two .... see if we can't get on with our lives without worrying about money every second of the day.

    maybe people would even find the time to be happy again." ..... Anonymous Coward Posted Thursday 17th February 2011 11:03 GMT

    Please let it be wwwidely known, that that is the present abiding problem for the likes of the Markets, which isn't going away and grows ever bigger and more strident every day.

    In a nutshell it is thus and this ..... There are those who are Ostensibly Anonymous who have hacked the money system protocols and would share what they have discovered, which removes forever the worry of not having enough money for one's needs. And of course, that then removes the need for those who would control currency supply, and really fcuks up their virtual command and control game, which has them living in luxury for free, off the backs of those who are providing them with their needs and feeds and seeds.

    Now who would have that information and intelligence and Most Valuable Intellectual Property, is something which ensures the mystery insures and assures its holders and stock stakeholders, and thus is necessarily invariably always generally unknown as it is known and passed through Networks to SMARTer Keyholders and Trigger Guards for further Sleeper Awakenings for the Additional Protection of Greater Force Numbers.

  13. Anonymous Coward
    Megaphone

    There is no bug...

    That really is Nokia's share price!

    1. Anonymous Coward
      Anonymous Coward

      Actually, the bug is real,

      there is no spoon.

  14. Anonymous Coward
    Anonymous Coward

    back in the day...

    The London stock exchange system used to run nice & reliably on VMS until Accenture took over and no doubt charged an arm and a leg to move it all to a "more modern platform". IIRC they moved or sacked all the LSE employees who knew how the existing system worked and then said it had become uneconomic to maintain. Management consultancy 101.

    1. Anonymous Coward
      Anonymous Coward

      "legacy" = "stuff that works"

      And therefore isn't written about very much, it just sits and hums quietly and invisibly.

  15. Ocular Sinister

    Hmm...

    Looks like the data from a third party is not in the expected format. Far from ideal, but not a sign that the world is about to collapse. You have to wonder what integration testing they did though.

  16. 88mm a.k.a. Minister for Misbehaviour
    Paris Hilton

    Linux is balls

    I understand the ball that goes up at the start of trading is controlled by Ubuntu. I guess that's a form of real-time system if it has knack all else to do but at least once it failed due to a full log partition/poor build. Funny how often that happens. Frankly for my money any financial trading system should be on Solaris with ZFS but what do I know, I never worked at any exchange and was too stupid to bother learning mainframe when I had the chance.

    - Paris... because she knows where the money is.

  17. Radelix

    man made systems

    Last I checked human-made systems go titsup every now and again. Naturally we don't want them to but it happens.

  18. John Smith 19 Gold badge
    Headmaster

    *lots* of terminology.

    Let's try unscrambling some of it

    *Millennium bug". Failure cased by incorrect processing of change from 1999 to 2000. The article shows *no* evidence for this class of bugs being the cause of the problem.

    Business critical. System fails company goes down pan. That *would* have described any major *offline* accounts app EG LEO (Lyons electronic Office). More recently would be the code that did calculations for a medical radiation source to decide, given various factors how much to dose them with. Not real time at all (except in the loosest sense) but *absolutely* critical to delivering the machines function. The code had bugs and several patients got 10x the dose they should have had.

    Latency tolerant soft real time. Describes lots of stuff on *any* GUI. You clicked on "My Documents" and it showed you the folder. It opened fast *enough* that you're not bothered. If it takes a *bit* more (or less) time to act you're *still* not bothered. The cursor followed you mouse *well* enough to keep you feeling in control.

    Latency intolerant soft real time. Here the *consistency* of the response is the important thing but I can't think of a situation where this applies.

    After this things get *tough*. Typically the computer system is connected to "stuff" and if the computer can't work out the *right* response in a *guaranteed* amount of time *every* time *very* bad things happen. The "stuff" might be another computer on the network (Skype's latency and *consistency* of that latency are both pretty important for it to work well). If the system is an airliner, jet engine or Uranium enrichment centrifuge things can go badly wrong very quickly.

    *nix's were never designed to support *hard* real time constraints but various upgrades to standard functions and the *long* background in building architectures that run such tasks have built up a *lot* of expertise in what to do and how to do it.

    Armadillo Aerospace runs such a box to to fly the lunar lander which won the NASA challenge prize.

    Some of the usual features for the work include a high resolution clock for the scheduler, software modules locked in main memory (no writing to disk) and either locking module scheduling priorities *permanently* or only allowing them to be changed under *very* carefully controlled criteria.

    The developer and admin mindset is important here. While it looks to the apps and the dev tools like a normal *nix box it is not *run* like one.

    Of course while the knowledge is out there and developers *with* the knowledge are out there as well it's not clear if *these* developers have it.

    BTW Be *very* careful of the phrase "real time." Real time relative to *what*? A human can't control an unstable aircraft without computer assistance (watch the control surfaces of an F16 if you ever see one but *can* respond fast enough to simulate an ABS (c5 brake presses a second IIRC). OTOH a chemical plant with a time constant of 1 hour can be controlled by a human with a chemistry set, some graph paper and a calculator in "real time."

    That's my pedantry done.

    1. Anonymous Coward
      Anonymous Coward

      these guys should get you to write a column

      "Latency intolerant soft real time. Here the *consistency* of the response is the important thing but I can't think of a situation where this applies."

      Flight simulator or similar? It's not necessarily a disaster if you miss a deadline slightly occasionally, it is useless if things are sufficiently unpredictable as to make them unrepresentative.

      "*nix's were never designed to support *hard* real time constraints"

      In general true, but (if you'll pardon the expression) there were exceptions of various kinds. DEC OSF/1 was one I was familiar with, though again it rather depends on what is meant by "hard". You wouldn't fly an aircraft, you might run an air traffic management system (and indeed Eurocontrol did).

  19. Anonymous Coward
    FAIL

    crippled rubbish

    I gave suse version anything a wide berth long ago, just after the time they sold out to those Novel scum. SDE was the most miserable pile of crud I ever had the misfortune to install on my machine, possibly even including Corel linux. Trying to run a business with SUSE you need to have your head slapped and to have your servers go titsup. Well, look what just happened! And this is me, the biggest linux fanboi since 1991. Serve them glad.

    How do I make this FAIL icon bigger?

  20. Wile E. Veteran
    Coat

    'nix in Real Time

    Don't use it. Period. Instead, use something like QNX Neutrino or RTEMS which layers a POSIX API over a hard real-time core to allow normal 'nix functions to run when desired (such as during development) but also provides the hard real-time API's necessary for time-critical tasks.

    Use Ada, too, so somebody can maintain the bloody code 10 years after you've departed the project.

    Mine's the one with the QNX demo floppy in the pocket.

  21. Nameless Faceless Computer User
    FAIL

    Y3K?

    Y3K?

This topic is closed for new posts.