back to article How City IT is under attack from politicians, diesel bugs, HR

The stupidest thing I’ve ever said was “if it was a jet, the tower would have collapsed” on September 11th and I feel the same about RBS. As I pass it most days, part of me expects to see crowds outside, perhaps including the police and TV camera crews, because I can’t understand why it still functions. The Reg has covered in …

COMMENTS

This topic is closed for new posts.
  1. James 51

    I only wish I was surprised by anything this piece.

  2. hugo tyson
    Coat

    Diesel Bug

    It's not just biodiesel, common diesel can grow the slime/mould/bug at an interface with water. As there's often condensation in fuel tanks, that's where it grows. But that only matters if you stir it up, so it blocks filters (before blocking the injectors).

    Does the biodiesel one thrive throughout the fuel, or is there water dissolved in the fuel in biodiesel? That would happen if there are any emulsifiers (eg. mustard in salad dressing, detergents) present?

    1. Dominic Connor, Quant Headhunter

      Re: Diesel Bug

      I'm told there is water in biodiesel, space didn't permit a full discussion of the subject, but it is going to come as a nasty shock to some people.

      1. Yet Another Hierachial Anonynmous Coward

        Re: Diesel Bug

        Diesel bugs exits in both mineral diesel and biodiesel (which is, roughly speaking, a blend of mineral diesel and vegetable oils).

        It's not a critical issue with cars and general transport as the fuel is pretty fresh, ie. from refinery to your tank in a week or two, and re-filled in your tank every few days/weeks. It can be a problem with standby generators as the fuel may have been sitting in the tank for years. And I mean years. If you have a standby genset at work, when was the last time anyone ran the tank to the red-line or until the little light came on, and refilled it? Sure, you might "top up" a few hudred litres each year after routine testing, but the bulk of the fuel may have been there 5 or 10 years or more.

        It's not unusual for a genset to fire up during a power failure, run for a hour or so, then cough and splutter to a halt as the sludge and stuff get drawn through the filters.

        1. Alan Brown Silver badge

          Re: Diesel Bug

          "It's not unusual for a genset to fire up during a power failure, run for a hour or so, then cough and splutter to a halt as the sludge and stuff get drawn through the filters."

          One of the reasons to run your gennies for at least a couple of hours every tuesday AND to have an alternate feeder/filter set plus spares - at least that way you can clean one set while the other's still in service.

          At least one setup I know of continually cycles fuel through the filters and back to the tank in order to ensure there's as little contamination sitting there as possible.

          (Both of these setups are common in Telcos, they know a few things about both batteries and backup systems - how loud do you think the screaming would be if you lost dial tone along with your power?)

      2. Frederic Bloggs

        Re: Diesel Bug

        Specifically, there is a problem with biodiesel in that it is hygroscopic, thus providing more house room for bugs. Another *major* problem with biodiesel is that is much more corrosive to the sorts of "rubber" glands, seals and pipework that "traditionally" are used with mineral diesel guzzling engines. This is why many diesel cars specify what percentage of biodiesel is allowed in the the road fuel that cars burn.

        The impending road crash(es) that will occur very soon when the EU start enforcing much higher biodiesel content for road cars is for another article...

    2. Nigel 11
      FAIL

      Re: Diesel Bug

      I already knew that bacteria can thrive on petrodiesel as well as biodiesel.

      Some think that the bacteria that cause problems kilometers underground for the oil industry, are not always introduced into the crude oil by the drilling process. It's possible that life has been surviving there, since the oil was organic-rich sludge at the bottom of a pre-historic sea.

      So why do emergency generators run on diesel, not petrol or (best of all? ) LPG?

      1. Alan Brown Silver badge

        Re: Diesel Bug

        "So why do emergency generators run on diesel, not petrol or (best of all? ) LPG?"

        Some do. It's also possible to run them on mains gas - but it assumes pressure will be maintained.

        BTW calling them emergency generators may be a misnomer. When power draws get high and utilites are squeaking, they'll often pay large consumers more than the cost of running the generators simply to get off the grid for a while - it's in situations like this that using gas-fed gensets shines (again, this has happened a lot in Telcos)

      2. This post has been deleted by its author

      3. Anonymous Coward
        Anonymous Coward

        Re: Diesel Bug

        Aside from the fuel contamination issue Diesel engines are intrinsically more reliable than petrol ones due to not having the need for an ignition system. Also, the availability of large, reliable petrol engines is very short, as these days most industrial power applications use diesel. Running a small petrol engine at high speed is also inefficient from a fuel consumption POV.

        Most heavy generators use truck engines (some very big units will even use Scania V8s) which are designed for long duty cycles at a near constant RPM, perfect for generator work.

        1. Nigel 11

          Re: Diesel Bug

          One last thought. You can run a diesel engine on any fuel, if the compression ratio is right. Diesel (the inventor) patented one that ran on coal dust, though don't ask me how he got it into the cylinders. In the old days before fuel injection etc. you could run a diesel car or lorry on petrol with nothing short-term worse than loss of power. I'd hope that the army's diesels still can run on anything: diesel, kero, petrol, cooking oil ... In a war your fuel supplies may be adversely affected by the enemy.

          So emergency generators should use specialized diesels running on LPG, I think. Of course a bog-standard lorry engine will be cheaper, but at a higher likelyhood of failing when an emergency happens.

          1. Anonymous Coward
            Anonymous Coward

            Re: Diesel Bug

            Diesel IIRC never actually got a coal dust engine working, he merely suggested the concept to get MAN (yes, they of wagon fame) to finance his research.

            Your suggestions of specialised LPG running diesel engine falls down on that first word "specialised". Do you really want to depend on an engine for which parts are hard to find? Or one that's proven over thousands of hours of work in all conditions and where standardised parts are available from a dealer 24hrs a day? (HGV dealers often work night shifts in workshops to provide round the clock support).

          2. xj25vm

            Re: Diesel Bug

            "In the old days before fuel injection etc. you could run a diesel car or lorry on petrol with nothing short-term worse than loss of power."

            As far as I know, diesels always used injectors and fuel injection. I assume you mean, the old days when diesels used mechanical pumps and injectors, as opposed to electronic common rail pumps and electronic piezo injectors.

          3. Anonymous Coward
            Anonymous Coward

            Re: Diesel Bug

            It is dead easy getting coal dust into the cylinders of a traditional Diesel engine. They used blast injection in which the fuel is injected into the cylinder by a blast of compressed air, and could burn just about anything that was at least fine particulate and ignitable. You could probably run one for a while on flour. At the speed of these things (75-90rpm) a mechanical system could be used to deposit slugs of power into the blast chamber - there is nearly a second available in which to do with a 4 stroke engine. Some marine engines ran on what was more or less bitumen that had to be heated just to get it to flow enough, and contained actual carbon particles.

            But LPG...no joy, I'm afraid. Its ignition temperature is far too high for spontaneous ignition, and if you use an igniter (as with a spark ignition engine) in such a big engine, you will get destructive explosions, not the nearly constant pressure power cycle that distinguishes a Diesel.

        2. xj25vm

          Re: Diesel Bug

          "Aside from the fuel contamination issue Diesel engines are intrinsically more reliable than petrol ones due to not having the need for an ignition system. "

          Well - that was pretty much the case in the good ol' days. Having a simple engine, which once started didn't actually need electrical power as it doesn't need the sparks of a petrol engine - and could continue working even if the alternator (in a car) would break down (some of them started with manual power, so didn't even need electrical power for starting) - was indeed a major point of reliability. However, after roughly 2000 when most manufacturers have moved away from the mechanical diesel injection pumps to the common rail units, which use electrical power and an onboard computer to control the pump and the piezzo injectors - the overall reliability from the point of view of old fashioned engines has gone down the pots. Don't get me wrong, I'm not saying that modern diesel engines are overall less reliable than their ancient counterparts - but at least in principle, you don't have anymore an engine which can continue to run if its electrics fail (no to mention that it also relies on the onboard computer as well, now) - and the new common rail, high pressure pumps are certainly far more picky about the quality of the fuel. On the upshot, the newer diesels are far smoother, far more fuel efficient and quieter.

          Unless manufacturers of industrial diesel engines (as opposed to the ones specifically designed for vehicles) still use old fashioned mechanical injection pumps?

          1. rhydian

            Re: Diesel Bug

            @xj25vrm

            Most of the generator units I've seen recently have standard style rotary injector pumps. Remember a generator doesn't have to meet the same emissions standards as a car, and truck/plant engines are expected to be pretty much totally reliable.

    3. Denarius
      Thumb Up

      Re: Diesel Bug

      not quite. Cladisporum loves kero, using an electrolytic reaction to break down the hydrocarbons. No, I dont know how. The current does horrible things to aircraft fuel tanks and will slowly erode anything else if the fuel is left long enough. Diesel oxidises and one gets gums which block or restrict flow in injectors. Lean mixtures damage engines of any kind, except ramjets maybe. Perhaps a commentard chemist can estimate how much faster biodiesel ages faster than hydrocarbon ?

      And hydrocarbon fuels seem to extract water from atmosphere which slowly rusts out containers even without bugs.

      Dominic, good articles. I now feel worried by something threatening and known to exist. Bank PHBs.

    4. Tim99 Silver badge

      Re: Diesel Bug

      The problem of fungi and oleophilic bacteria in fuels has been known for decades. Even clean dry fuel will become contaminated from bugs in the air. Changes in temperature, humidity and pressure cause contaminated water vapour to condense in the tank where it slowly settles to the bottom. The bugs gennerally grow on the fuel/water interface near the bottom of the tank.

      The problem can be ameliorated by using a suitable fuel biocide and a desiccant cartridge on the fuel filler vent. Well designed systems generally have the fuel outlet above the bottom of the tank so that any crud stays below the outlet. Some systems also have a drain-cock at the bottom of the tank so that any water/crud can be drained off periodically.

  3. Sergiu Panaite
    Thumb Up

    Good

    Good Comment, I get the feeling that Mr Connor is mellowing a bit for some reason - less snappy (for lack of a better word right now) than previous articles.

    I concur, it's frightening how much relies on old tech - whether that means old hardware, old code or just plain old ideas/processes that barely work nowadays, for whatever reason. The trouble however is exactly what you highlighted in your previous article about the RBS follow-on: you can of course start from scratch, but that will take a very long time, it's bound to hit numerous issues and it will require a lot of resources because you're still supporting your current stuff (as a side note, "stuff" and "it's f'd" should be officially made into acceptable technical terms).

    I don't have a solution, for two main reasons: I don't have anywhere near enough experience and I'm not paid anywhere near enough to have that kind of weight on my shoulders. Whichever way it goes however, it keeps me busy.

    1. Dominic Connor, Quant Headhunter

      Re: Good

      Mellowing ?

      That's fighting talk ;)

      I agree there is no quick fix, its a long term management issue, but RBS may not make it to the long term.

    2. Irongut

      Re: Good

      I'm pretty sure "stuff" and "its fucked" are official technical terms.

  4. Jediben
    Stop

    "One of the larger transformer buildings in the heart of the City almost next to where the Stock Exchange servers live is surprisingly warm to the touch. Yes that’s a lot of power, and no I’m not saying where."

    Please do say where, and contact your local distribution company to advise them. Electrical safety is very important and the distribution networks are massive. Don't fart about when public safety could be at risk.

    If it's London, tell UK Power Networks - http://www.ukpowernetworks.co.uk/internet/en/contact-us/

    1. Captain Scarlet Silver badge

      Free Heating!

      But they might be using it for free heating!

    2. Dunstan Vavasour

      He's just grandstanding for effect, these places are designed to get warm. The building being warm to the touch probably shows that the heat is being removed effectively from the transformer.

  5. proto-robbie
    Pirate

    Smalltalk, Perl and Cobol; I haz them allz

    Bring on your data transfers. Just not in XML, please.

  6. John 98

    We could ban Excel for a start

    Top of my list for when i become dictator - it does have a few legitimate uses but it sits at the heart of not a few hopeless muddles. Perhaps the banks should be required to seek regulatory approval for any and every spreadsheet?

    On thestate of the UK grid - a decade of craven surrender to Nimbies and Luddites has left us staring disaster in the face. i suggest ordering a billion petrol generators from China and stacking them round parliament to keep things going - costs to be deducted from MP's pay until they sort things.

    1. Irongut

      Re: We could ban Excel for a start

      Top of the list should be banning Powerpoint. As I regularly tell my users "there is no good reason for using Powerpoint, ever."

      Usually followed by "Excel is not a database."

      1. Nigel 11

        Re: We could ban Excel for a start

        Powerpoint is a far lesser crime than using Excel as a database. It's a lesser crime even than using Access as a database!

        1. h3

          Re: We could ban Excel for a start

          Most of the investment banks use mainly Sybase as far as I know unless it has changed quite recently.

          Powerpoint is bad because it is mostly used by people who don't have anything remotely worthwhile or interesting to say and the way they use it makes at least me switch off completely.

          People doing public speaking should do it without prompts and actually understand and know from memory properly what they are speaking about.

          One Powerpoint maybe at the end with the main points. (Might help to remind the speaker that he hasn't missed out anything).

          If other people want to use Excel or Access I don't have a problem with it. (I use Octave or Matlab or Mathematica for simple calculations when I need to. Usually octave these days.) If it works for them it is fine by me.

          What isn't fine is Powerpoint users systematically wasting the time of every single person in a room day in day out.

    2. Anonymous Coward
      Anonymous Coward

      Re: We could ban Excel for a start

      I want nuclear generators in container lorries. Easy, green and the military has a proven design used in subs ...

      In one case we were approached about whether in a power failure our back up generator could be used by local authority sites ...

      1. Alan Brown Silver badge

        Re: We could ban Excel for a start

        "In one case we were approached about whether in a power failure our back up generator could be used by local authority sites ..."

        Even if you said no, they can and will requisition the units if they want them (eminent domain) and leave you to argue the finer points in court later.

        The fact you were asked means that someone's probably got plans filed away somewhere to do just that.

      2. Anonymous Coward
        Anonymous Coward

        Re: safety margins on the grid

        "we were approached about whether in a power failure our back up generator could be used by [the grid]"

        Expect more of that in the UK before too long, for two reasons:

        1) We're perilously close to grid demand exceeding grid capacity, and there is no practical fix in the next five years or so

        2) The currently proposed workaround for this in general is not to substantially increase grid capacity but to provide "better management" ie operate the existing grid closer to its ultimate failure level by means of using "better management" to actually *reduce* operating margins. Excellent, can't fault that logic can you. That's to be supplemented by using otherwise-idle standby generator capacity to feed into the grid at times of high demand. Shouldn't cost too much, and after all what could possibly go wrong?

        Anyone who's regularly used a "traffic managed" city road network, or the motorway equivalent, knows that all traffic management does is defer the chaos a bit, and make the chaos worse on the days when it actually happens, because there is no longer a relatively smooth transition between normal operation, small overload, and large overload.

        As it is with traffic management, so it will be with demand and supply management on the grid.

        LED torches, UPSes, and gennys. Buy now, while stocks last. And say no to smart meters.

        1. Anonymous Coward
          Anonymous Coward

          Re: safety margins on the grid

          "1) We're perilously close to grid demand exceeding grid capacity, and there is no practical fix in the next five years or so"

          Not grid capacity. Peak demand has fallen by around 6 GW since 2006. Generation capacity is dropping as LCPD closures bite, but we've still got some spacre capacity (and mothballed plant). Obviously if demand leapt back up to 2006 we'd be looking at blackouts, but where the economic growth for that? Certainly we've got less reserve margin that we did have, but it's probably post 2016 that the real risks bite.

          "Demand side response" is the name given to trying to manage peak demand, whether through co-opting stand by gennies, or paying people to shed load (like turning the freezers off at a refidgerated warehouse for an hour or two). The main flaw here is that the very limited hours that you'd call on commercial standby means that the rewards are pathetic, even at very high unit rates, and most facility managers rightly suppose that the complexity, investment needs, and risks outweigh the modest potential upsides.

          Centrally generated and despatched electricity is reliable, clean, and cheap. Moving to the use of crappy diesle gennies to avoid spending quite paltry sums on new CCGT (£0.5bn for 2GW of capacity) is madness.

          1. Anonymous Coward
            Anonymous Coward

            Re: safety margins on the grid

            To be fair, the stuff I've seen as an outsider (IET stuff?) was referring to *distribution* capacity as well as generation capacity.

            But yes it does seem a bit unnecessary and a bit ineffective and a bit like a poor RoI for all concerned.

            "most facility managers rightly suppose that the complexity, investment needs, and risks outweigh the modest potential upsides."

            What kind of motivation is being offered to folks with a few MW to perhaps contribute? Who is picking up the integration costs (presumably you can't just parallel an existing 2MW standby system straight onto the grid without sparks flying?)

            Demand management and energy efficiency would seem to be more rewarding, except it doesn't immediately lead to profits for corporates. UK overnight minimum demand is not far off 30GW (peak is 50+). Who's using 30GW overnight?

        2. xj25vm

          Re: safety margins on the grid

          "Anyone who's regularly used a "traffic managed" city road network"

          I wonder how much do you really know about traffic management. Maybe we should all give up those pesky traffic lights. Or at least put them in old fashioned fixed timed mode - none of this malarky with fancy traffic management algorithms to make urban traffic more efficient. It is clearly pure quackery!

          1. Anonymous Coward
            Anonymous Coward

            Re: safety margins on the grid

            "I wonder how much do you really know about traffic management"

            "none of this malarky with fancy traffic management algorithms to make urban traffic more efficient."

            I'm not in the trade but I used to follow things like SCOOT which did computerised traffic management stuff. I'm not talking about a few traffic lights with sensors linked so the basic stupid effects of fixed-timing operation are avoided.

            In terms of the effects I see, It's like the difference between analogue TV and digital TV. Analogue TV degrades slowly as it approaches its limits. Digital TV is better most of the time [1] but when things get worse it eventually falls off a rather unpleasant cliff, without much warning.

            Traffic managed routes may work better much of the time but they do that by working with much less contingency/headroom in the overall system, and when something goes wrong, sometimes all hell breaks loose. Advanced traffic management might save commuters a few minutes a day, but leave them faced with a multi hour delay when something inevitably does go wrong.

            Systems are often like that. Gridco seem to want to run the grid more that way (lots of "advanced management" to increase utilisation without actually increasing capacity) rather than the traditional "lots of headroom" mode. That'll be fun.

            [1] Digital TV in reality is often crap because of excessive compression but that's another story.

            1. Dominic Connor, Quant Headhunter

              Re: safety margins on the grid

              I'll be honest and say I know nothing about modern road traffic management systems, if anyone would like to enlighten me, feel free.

    3. Anonymous Coward
      Anonymous Coward

      Re: We could ban Excel for a start

      The FSA - God rest their souls - did indeed chase after Investment banks who ran their trading books in Excel. A number of firms then migrated from them to custom-built apps, needing to use lovely Middleware like Tib RV. Some more successfully than others.

      There's nothing quite like sitting in the middle of a trading floor in the middle of a multicast storm.

    4. Anonymous Coward
      Anonymous Coward

      'Seek regulatory approval for any and every spreadsheet?'

      Understandable knee-jerk reaction but it wouldn't work. I've seen traders work out arb trades on beer-mats to calendar girls. Without Excel it would be even worse.... What's needed is a time-freeze or snapshot facility for compliance officers, so they can audit Trader spreadsheets after the fact.

      Very often there are so many add-ins, so many linked spreadsheets, and so many real time feeds that there is no way to predict how the spreadsheet will behave in any given market. i.e. Bad tic data. This AC gets it: "nothing quite like sitting in the middle of a trading floor in the middle of a multicast storm.'

    5. Anonymous Coward
      Anonymous Coward

      Re: We could ban Excel for a start

      Problem is, every time you get in one of the big four accountants into a bank because of a regulatory change, they will provide a set of spreadsheets as the 'solution'

  7. S4qFBxkFFg
    Meh

    Punish Failure?

    I used to think that bank failures (the big ones that make the papers, at least) could be avoided by simply making it illegal to simultaneously possess:

    - a banking licence

    - limited liability

    Now I'm not so sure, does possession of oodles of shares/options by execs really change their behaviour for the better?

    It might remove the need to bail them out though...

  8. Anonymous Coward
    Anonymous Coward

    Smalltalk

    I know of a very major bank in Europe whose clearing system is (or was) written in Smalltalk. I had an interview there ten years ago for second level support on it, but I reckoned that the stress levels would be too much when it failed with tens of billions of dollars not being cleared.

    No, I don't know Smalltalk.

  9. Mike 102

    Smalltalk

    Is there anyone other than JPMorgan still jusing it?

    I loved coding in Smalltalk - I miss it.

    1. John H Woods Silver badge

      Re: Smalltalk

      "Me too!"

      ^ (Queue new for: 'Smalltalk Jobs') add: 'Mike 102'; add: self.

  10. Anonymous Coward
    Anonymous Coward

    Keep on writing

    Your articles are refreshingly honest. Keep them going.

    Here's what I want to know: why don't companies recruit contractors directly? Given that most recruiters don't know f@#$ all about tech they continue to pass over talented people and put forward poor quality candidates that have purely fictional CVs.

    Their 10 or 20 or 30% cut isn't deserved, is what I think, but what do you think?

    1. Anonymous Coward
      Anonymous Coward

      Re: Keep on writing

      Couldn't agree more. I dread to think how much the IT services company that put me in my current position are making from me every day.

      1. Sir Runcible Spoon

        Re: why don't they..?

        Admin overhead

        IR35 issues

  11. fruitoftheloon
    WTF?

    Irongut.... Ban powerpoint?

    It's a tool, if your 'users' produce a load of visual ordure that doesn't actually help them to get their message across, perhaps they some help with:

    - figuring out what their message is

    - determining how best to convey it

    - using the [insert name of relevant bit of software] properly

    - making fewer presentations

    Just a thought...

  12. Anonymous Coward
    Anonymous Coward

    Not just their trading systems

    For a while I consulted in quite a few banks and even bigger institutions in the City (LIFFE and the interbank clearing system) installing network monitoring systems - since quite a few of them had literally no idea if networks were up or not until then.

    One I can mention is that I installed (and was paid for) a rather complex HP Openview Network Node Manager system into Enron ... just before they went down. Before then they didn't know if their network nodes were up except by pinging them. :) Lovely offices though - free food and drink for all developers and operations.

    Considering this was more than 10 years ago I hope that the banks that still exist have moved on since then - but it's debatable.

    I am reminded of this (anonymous) IT story about someone almost destabilising the Euro with shell scripts: http://www.b3ta.com/questions/expensivemistakes/post95910

    From what I hear now - the entire economy is *still* held together by shell scripts. Keeping your money in the mattress doesn't sound so bad now does it?

    1. Anonymous Coward
      Anonymous Coward

      Re: Not just their trading systems

      Your link to the Shell Scripts story is behind a paywall or something. Could be an interesting read, can you elaborate or offer another link please?

      1. Loki23

        Re: Not just their trading systems

        Nope - b3ta doesn't have a paywall - your connection must be censored (some of the other B3TA image manipulations can be NSFW).

        Here is the text:

        I.T. is a minefield for expensive mistakes

        There's so many different ways to screw up. The best you can hope for in a support role is to be invisible. If anyone notices your support team at all, you can rest assured it's because someone has made a mistake. I've worked for three major investment banks, but at the first place I witnessed one of the most impressive mistakes I'm ever likely to see in my career. I was part of the sales and trading production support team, but thankfully it wasn't me who made this grave error of judgement...

        (I'll delve into obnoxious levels of detail here to add colour and context if you're interested. If not, just skip to the next chunk, you impatient git)

        This bank had pioneered a process called straight-through processing (STP) which removes the normal manual processes of placement, checking, settling and clearing of trades. Trades done in the global marketplace typically have a 5-day clearing period to allow for all the paperwork and book-keeping to be done. This elaborate system allowed same-day settlement, something never previously possible. The bank had achieved this over a period of six years by developing a computer system with a degree of complexity that rivalled SkyNet. By 2006 it also probably had enough processing power to become self-aware, and the storage requirements were absolutely colossal. It consisted of hundreds of bleeding edge compute-farm blade servers, several £multi-million top-end database servers and the project had over 300 staff just to keep it running. To put that into perspective, the storage for this one system (one of about 500 major trading systems at the bank) represented over 80% of the total storage used within the company. The equivalent of 100 DVD's worth of raw data entered the databases each day as it handled over a million inter-bank trades, each ranging in value from a few hundred thousand dollars to multi-billion dollar equity deals. This thing was BIG.

        You'd think such a critically important and expensive system would run on the finest, fault-tolerant hardware and software. Unfortunately, it had grown somewhat organically over the years, with bits being added here, there and everywhere. There were parts of this system that no-one understood any more, as the original, lazy developers had moved company, emigrated or *died* without documenting their work. I doubt they ever predicted the monster it would eventually become.

        A colleague of mine one day decided to perform a change during the day without authorisation, which was foolish, but not uncommon. It was a trivial change to add yet more storage and he'd done it many times before so he was confident about it. The guy was only trying to be helpful to the besieged developers, who were constantly under pressure to keep the wretched thing moving as it got more bloated each day, like an electronic ‘Mr Creosote’.

        As my friend applied his change that morning, he triggered a bug in a notoriously crap script responsible for bringing new data disks online. The script had been coded in-house as this saved the bank about £300 per year on licensing fees for the official ‘storage agents’ provided by the vendor. Money that, in hindsight, would perhaps have been better spent instead of pocketed. The homebrew code took one look at the new configuration and immediately spazzed out. This monged scrap of pisspoor geek-scribble had decided the best course of action was to bring down the production end of the system and bring online the disaster recovery (DR) end, which is normal behaviour when it detects a catastrophic 'failure'. It’s designed to bring up the working side of the setup as quickly as possible. Sadly, what with this system being fully-replicated at both sites (to [cough] ensure seamless recovery), the exact same bug was almost instantly triggered on the DR end, so in under a minute, the hateful script had taken offline the entire system in much the same manner as chucking a spanner into a running engine might stop a car. The databases, as always, were flushing their precious data onto many different disks as this happened, so massive, irreversible data corruption occurred. That was it, the biggest computer system in the bank, maybe even the world, was down.

        And it wasn't coming back up again quickly.

        (OK, detail over. Calm down)

        At the time this failure occurred there was more than $12 TRILLION of trades at various stages of the settlement process in the system. This represented around 20% of ALL trades on the global stock market, as other banks had started to plug into this behemoth and use its capabilities themselves. If those trades were not settled within the agreed timeframe, the bank would be liable for penalties on each and every one, the resulting fines would eclipse the market capital of the company, and so it would go out of business. Just like that.

        My team dropped everything it was doing and spent 4 solid, brutal hours recovering each component of the system in a desperate effort to coax the stubborn silicon back online. After a short time, the head of the European Central Bank (ECB) was on a crisis call with our company CEO, demanding status updates as to why so many trades were failing that day. Allegedly (as we were later told), the volume of financial goodies contained within this beast was so great that failure to clear the trades would have had a significant negative effect on the value of the Euro currency. This one fuckup almost started a global economic crisis on a scale similar to the recent (and ongoing) sub-prime credit crash. With two hours to spare before the ECB would be forced to go public by adjusting the Euro exchange rate to compensate, the system was up and running, but barely. We each manned a critical sub-component and diverted all resources into the clearing engines. The developers set the system to prioritise trades on value. Everything else on those servers was switched off to ensure every available CPU cycle and disk operation could be utilised. It saturated those machines with processing while we watched in silence, unable to influence the outcome at all.

        Incredibly, the largest proportion of the high-value transactions had cleared by the close of business deadline, and disaster was averted by the most "wafer-thin" margin. Despite this, the outstanding lower-value trades still cost the bank more than $100m in fines. Amazingly, to this day only a handful of people actually understand the true source of those penalties on the end-of-year shareholder report. Reputation is king in the world of banking (see Northern Rock for details!) and all concerned --including me-- were instructed quite explicitly to keep schtum. Naturally, I *can’t* identify the bank in question, but if you’re still curious, gaz me and I’ll point you in the right direction…

        Epilogue… The bank stumped up for proper scripts pretty quickly but the poor sap who started this ball of shit rolling was fired in a pompous ceremony of blame the next day, which was rather unfair as it was dodgy coding which had really caused the problem. The company rationale was that every blaze needs a spark to start it, and he was going to be the one they would scapegoat. That was one of the major reasons I chose to leave the company (but not before giving the global head of technology a dressing down at our Christmas party… that’s another QOTW altogether). Even today my errant mate is one of the only people who properly understands most of that preposterous computer system, so he had his job back within six months -- but at a higher rate than before :-)

        Conclusion: most banks are insane and they never do anything to fix problems until *after* it costs them uber-money. Did I hear you mention length? 100 million dollar bills in fines laid end-to-end is about 9,500 miles long according to Google calculator.

        1. Iain Griffiths

          That sounds like ...

          The guy who writes the ITShambles blog ....

          well worth a read ... in fact get him on teh Reg

          http://itshambles.wordpress.com/

          1. OzBob
            Thumb Up

            Re: That sounds like ...

            Iain, is Dominic the author of itshambles? I started reading at work but had to stop because I was laughing so hard (which clued the boss up that I was not working). Its Friday night and I'm reading it and wearing incontinence pants. A++ recommended.

          2. That Awful Puppy
            Thumb Up

            Re: That sounds like ...

            Good grief, whoever is writing this is brilliant. I'm not even in IT and I find it hilarious. Thank you very much for pointing me in this direction.

        2. Anonymous Coward
          Anonymous Coward

          Re: Not just their trading systems

          Thanks Loki23! Had similar experiences.... I posted this in last weeks thread about HFTs...

          ============

          Regulators Beware: Inadequate testing or Impossible Simulation...

          Having worked on HFT systems, you would be amazed at how many changes are made on the fly without care and consideration to investors. Think FIAT: Fix It Again Tomorrow as a working mantra. Part of the problem is this. You can't recreate a live complex organic trading system in beta. Often the beta isn't even running the same release as the live system. So it isn't just a question of numbers i.e. only testing 10% of the orders. Its also a question of real-world complexity in the way orders are placed and in the complex interaction between all the different players.

          Complexity is exponentially increased with automated market makers (AMM) and their interconnected exchanges, HFT systems, hardwired bank and institutional screens, retail systems, and legacy phone or pit orders etc. When you sprinkle in AMM stock-pinging, liquidity-rebate-trading, front-running, fat-finger trades, legitimate cancelled orders, and competing orders from co-located servers versus those at a distance... life can get very messy.

          Its very difficult to build a good simulation. I wish the regulators would acknowledge this and herald it as warning.... Instead they continue to see these events as one-off problems in tunnel vision fashion. So when-is the next flash-crash or IPO non-event...? ...FIAT!

        3. Fatman

          Re: 'my errant mate'

          Even today my errant mate is one of the only people who properly understands most of that preposterous computer system, so he had his job back within six months -- but at a higher rate than before :-)

          I would have placed as a condition of re-joining that fucked up outfit, the dimwit that fired me got sacked, otherwise, they could go and fuck themselves.

          1. Dominic Connor, Quant Headhunter

            Re: 'my errant mate'

            This reminds me of the debate about the death penalty, the dimwit could have a quick clean execution, or he could be made to linger for years being tormented, which would be better punishment ?

  13. Petrea Mitchell
    Boffin

    Smalltalk: We've heard of it

    I graduated in 2000, and I heard about Smalltalk and its use in the banking industry at school. In fact, I took a class in it! So some of us younger 'uns do know what it is.

    (Note that I do not claim I *know* Smalltalk: the guy teaching the class was clearly enthusiastic about the language and made most of his living doing contract work in it, but he was hopeless at actually communicating any of his knowledge or making the language look in any way useful. For instance, he would demonstrate examples by hopping around between a dozen scratchpad windows, highlighting random sections of code and "inspecting" (running) them to make something happen. One day a classmate tried to ask him how to make a standalone program, and after several iterations it became clear that he just could not conceive of doing such a thing.)

  14. Anonymous Coward
    Anonymous Coward

    Why waste money on keeping things working?

    I'm a contractor upgrading a local authority's computer systems to WIndows 7. The politicians keep saying: the government has cut our funding again, why are we wasting our money on this, we should cancel it. The answer "Windows XP will stop working in a few months" cuts no ice. Interestingly, Windows XP support vanishes the same time the council tax bills are supposed to be issued. I will await my post with interest next year...

    1. xj25vm

      Re: Why waste money on keeping things working?

      "The answer "Windows XP will stop working in a few months" cuts no ice."

      Isn't that a bit like BS? Windows XP *support* from Microsoft will be cut off in a few months. That doesn't exactly mean the sky will fall, or the OS will stop working the next day. Plenty of large and small organisations will carefully keep it working for years to come - and save money in the process. It is self-serving, exaggerated and at least partially misleading statements like this from some actors in the IT industry that makes decision people in the business treat what we say with suspicion.

  15. goats in pajamas
    FAIL

    scribblers

    The stupidest thing I’ve ever said was “if it was a jet, the tower would have collapsed” on September 11th

    Wow, yup, you're right, you're as thick as pigshit.

    You haven't understood what happened on 9/11 yet and that was 12 years ago. And now you cite your utter failure to understand what happened that day as some sort of supporting credibility for your journalistic efforts. It would be funny if it wasn't so fucking sad.

  16. The Vociferous Time Waster
    FAIL

    @goats in pyjamas

    You didn't understand that sentence did you.

    Either that or you are a conspiracy freak. Both quite pitiful.

    1. Dominic Connor, Quant Headhunter

      Re: @goats in pyjamas

      Actually, the thing that is clear to me about 9/11 is that very few if anyone understands it.

      Feel free to insert your own conspiracy theory, but to me the most interesting part is that the systems didn't work and their failure is too wide for a conspiracy because a conspiracy that powerful wouldn't need to conspire.

      Critical parts of NY's emergency response control were under a tower which which known from previous attacks to be a target.

      Helicopters which were supposed to rescue people weren't sent, most banks DR was itself a disaster, there was a major risk managers conference in a tower that day (one I might have been at), the fire department bravely rushed up the stairs because the NYFD has no proper plan for dealing with skyscrapers in a city that has more than any other.

      The US secret service didn't take proper care of Bush, instead of bundling him to somewhere safe they left him exposed in a widely known place. He then hid in a cave for a while, such was the chaos at the top of the US administration that Chelsea Clinton who was at that time a student found herself acting as a spokesman and doing a pretty good job.

      If you want a good conspiracy, you have one to make her President.

      The the US did nothing.

      Despite knowing who had done it and where they were, they asked a hostile regime politely if they could please hand them over and not only was the Afghan regism not American friends, recall that in Islamic culture you have a strong obligation to defend your guests.

      So the US let the commanders of their enemies get away.

      The search of the WTC ruins looked more like the courageous incompetence you see in Pakistan or Bangladesh after a disaster and the utter disregard for hazardous materials like asbestos dust for both the rescuers and citizens of NY looks like it will kill more people than the attacks themselves.

      1. J.G.Harston Silver badge

        Re: @goats in pyjamas

        "Critical parts of NY's emergency response control were under a tower which which known from previous attacks to be a target."

        Just before the Sheffield Floods of 2005, the emergency response control centre had just moved to new premises.... next to the river.

      2. Anonymous Coward
        Anonymous Coward

        Re: "the NYFD has no proper plan"

        "the NYFD has no proper plan for dealing with skyscrapers in a city that has more than any other."

        Should anybody be surprised?

        Is London (or any other 'civilised' city) any better off e.g. due to different planning regulations, better inspection+enforcement of existing building regulations for e.g. fire-resistant insulation around load bearing structures?

        Any plan for dealing sensibly with big trouble in one or more skyscrapers would surely make it very clear that there is no practical and affordable answer, and therefore they might stop being built. And we can't have that can we.

        Best not to even think about it, and pretend it can't (won't) happen, at least not in our lifetimes. And what happens in the lifetimes of future generations isn't important (that's how markets work, right?).

  17. Dominic Connor, Quant Headhunter
    Angel

    Just got an email from UK Power Networks

    ...who didn't get what I was saying. The dear little thing thinks The Register is *my* blog, basic PR professionalism would have stopped her looking so foolish.

    This building has been warm for (I believe) about 20 years, so I don't see it as a risk and yes I do rather think I understand some of the physics of transformers, in fact I was using the heat to illustrate how serious the power usage was.

    The other point I was making about the location is that this critical bit of IT infrastructure can be found simply by walking around the City on a cold day because although the absolute temperature isn't all that high, a large wall radiating heat are quite noticeable to a bad person with teenage physics.

    It's also not near the Stock Exchange I said " near where Stock Exchange servers live ", not the Stock itself which not only is visible but has a small TV studio in it to make sure you know where it is.

    That''s why I explicitly didn't mention the location, note the name of the article is "How City IT Is under Attack", since that info might be useful to the bad guys. It is a sensitive enough location that friends of mine have flatly refused to confirm or deny it from my own research. Since they're competent (and irritatingly loyal), I assume that the servers have multiple power feeds as well as generators so screwing with this building wouldn't affect them, but still would be messy.

    I was also illustrating that the City uses serious amounts of power, not just as large office blocks but having grids and clouds on a serious scale together with the aircon to cool them down.

    I would have explained this to the arrogant "Media Relations Manager" at UK Power Networks who seemed to be under the impression that her words would be so wise that I'd make a premium rate phone call to hear them from her.

    I didn't.

    Note to PRs, if you want me to call you, give either a land line or a mobile, I'm not paying 0.50 a minute , ever. There are no variables in this equation, the best you can hope for is

    a) Me mocking you in an open forum

    b) The following email response, if you can't be bothered to work out my direct email address (it's not hard, yes it is what you think it might be and yes your second and third guesses are right as well, I'm famously easy to contact)

    to: UK Power networks media relations

    Thanks for that, I'd call you but as policy I never speak to people who require that I use premium rate lines to do so.

    You will note that I do not include my phone number in this message, take that as a hint.

    1. Alan Brown Silver badge

      Re: Just got an email from UK Power Networks

      Publish for our entertainment. Pretty please?

  18. kimosabe

    Aside from that Mrs. Lincoln...

    The world is too complicated for primates that evolved to deal with small clans.

    http://www.wired.com/wired/archive/8.04/joy.html. Bill Joy has a great saying, "The smartest people in the world don't work for your company."

This topic is closed for new posts.

Other stories you might like