back to article Finance bods probe RBS over bank-crippling IT cock-up

A banking watchdog has launched an investigation into the IT meltdown that crippled bank accounts held by millions of NatWest, RBS and Ulster Bank customers in June last year. The Financial Conduct Authority (FCA) has begun a probe into the disaster, which saw the RBS group unable to process payments following a catastrophic …

COMMENTS

This topic is closed for new posts.
  1. alain williams Silver badge

    Investigation preview

    ''It was all the fault of some lowly paid minion, he should be flailed alive and never allowed to work in IT again. The top executives cannot be held to blame in any way at all; they could not know what was happening since they were too busy counting the bonus given to them as a reward for having cut costs by sacking long time experienced staff and instead employing untrained and inexperienced contractors in India.''

    1. Velv
      Big Brother

      Re: Investigation preview

      Maybe.

      Or maybe the new FCA will want to make its mark early. It might just go the other way and make wide sweeping revelations that people with time served SITE KNOWLEDGE cannot be replaced with "cheaper" people who on paper have similar technical qualifications. It might even go as far as to say that replacing onshore people from your own culture with offshore people from another culture leads to a substantial additional risk if you don't fully manage those culture differences.

      One can only hope (he says even more cynically than the expected whitewash)

      1. Anonymous Coward
        Anonymous Coward

        Re: Investigation preview

        Possibly, but I would imagine it is more along the lines of a small fine and committing to take appropriate actions to prevent any future occurrences.

  2. Anonymous Coward
    Anonymous Coward

    This is what happens when you outsource business critical systems to numpties in India. RBS got everything they deserved, I hope the money they saved on firing their IT guys over her made up or that £174 million fine, I'm betting not though. You hire shite, you get shite.

    1. Anonymous Coward
      Anonymous Coward

      It's nothing to do with India, it doesn't matter where the numpties are, that they are Indian is irrelevant.

      1. Nigel 11
        FAIL

        Loyalty and experience

        It's to do with two things - loyalty and experience.

        If you pay well for local staff, they'll stay in the job long enough to acquire a large knowledge base specific to your operations, stored where it's most needed -- in their heads. A lot of this stuff can't ever be written down, because the people who know it don't necessarily know what they know until a problem arises that has to be solved asap.

        Then someone arrives and cuts costs by sacking the experienced staff and relying on a bunch of outsourced (or even locally contracted) mercenaries who'll leave as soon as someone offers them more money. For that reason, staff turnover is high and the in-head knowledge base is destroyed. (Loyalty, of course, is long gone). The old system ticks over on auto-pilot for a large amount of time, and management pronounce it a success (and make further cuts?). Then the roof falls in.

        If it's tangential to the primary purpose of the organisation, outsource it. If it's mission-critical, don't. Is there really anything else needs to be said?

        1. Anonymous Dutch Coward
          Coat

          Re: Loyalty and experience: more to be said?

          "Is there really anything else needs to be said?"

          Yes: don't pay managers for short term cost savings/revenue increase.

          (Hopeless optimist here, yes)

        2. Seanmon

          Re: Loyalty and experience

          "If it's tangential to the primary purpose of the organisation, outsource it."

          Unfortunately, to most business types, this is exactly what IT support is.

        3. Annihilator
          Meh

          Re: Loyalty and experience

          "If it's tangential to the primary purpose of the organisation, outsource it. If it's mission-critical, don't. Is there really anything else needs to be said?"

          Except in this instance they didn't "outsource it", they hired cheaper, inexperienced staff - and offloaded more expensive, experienced staff.

          Presumably the Reg (and any other online rag) is failing your test too? Their mission critical things that I can tell are "host web site" and "generate ad revenue". Their solution? Rackspace and DoubleClick.

        4. Stoneshop

          Re: Loyalty and experience

          A lot of this stuff can't ever be written down, because the people who know it don't necessarily know what they know until a problem arises that has to be solved asap.

          I recently wrote a document describing the steps in bringing up a particular set of machines after a total power outage. There's (obviously) a specific sequence of switching on gear, and checking whether minimum system functionality, connectivity requirements etc. are met before proceeding to the next step in the sequence.

          The product manager for the platform complained that a) I hadn't put a 'shutdown' chapter in the document (err, this is about powering up after a catastrophic power failure. Meaning. There. Is. No. Graceful. Shutdown. Sequence.) and b) that my examples showing the results of the checks whether or not to proceed were trimmed down to show the essentials ("you're supposed to see something like [few lines of output] and NOT [some other lines of output]").

          Sorry, dude, this stuff is supposed to be used by people WHO KNOW THOSE SYSTEMS, not the first PFY you grab from wherever it is they're hatched. If the command used to check actually outputs 50 lines of info, they don't need to have all 50 lines of info present in the document, just a handful that show things are OK, and a few that show errors, with a comment on the severity of those errors and whether you can still proceed or not. The full output would be different from the document for any of the 15 clusters anyway, so apart from cluttering up the document and obscuring the essentials, it would be of no actual use whatsoever

          This document isn't so much an instruction sheet telling people in detail what to do for every step, but to help them remember the right sequence, and the checks, in a situation when things have gone utterly haywire and several levels of managers may be breathing down their neck. And there are more circumstances that just can't get solved by a PFY with reams of documentation (presuming it is all complete and up to date in the first place), but only by someone who can correctly interpret the (lack of) messages a system is spewing.

      2. Equitas
        Paris Hilton

        Probability ....

        of getting numpties in India seems to be rather higher. Indian call centres have done absolutely nothing positive for the reputation of India but rather led to an expectation of very poor service.

        Paris Hilton, because even Paris would do better!

      3. Anonymous Coward
        Anonymous Coward

        Yes, I don't think this is at all to do with nationality. It is do to with short term employees working on a contract while looking for the next job or project are never going to be as good as long term employees who built the system. It would be interesting to see the amount of time the employee in question, or all of the employees on the contract, had working on this system or similar systems. CA-7 for mainframe batch is not a skill that is easily found. If you just hand over a tremendously complex financial system to someone who has never seen it before and give them a few weeks of training, mistakes are going to happen. If you have a system where mistakes cannot happen ever, it is best not to let it out of your sight or supervision.

    2. Anonymous Coward
      Anonymous Coward

      I've worked at RBS. I was in a one of the teams that had the very first round of offshore members.

      Their technical knowledge could best be described as competent - on a par with some of the candidates we got in the UK.

      95% of the job was not technical. The actual installations and fault finding took no time at all. It was the business interactions, the Change Management, the documentation, the talking to people that took all the time and SKILL.

      So to everyone out there who needs to recruit staff, remember its not just about the pure technical knowledge, its about how that staff member will fit with the team and your business.

  3. Anonymous Coward
    Anonymous Coward

    Not outsourced

    This is not a failure of outsourcing - these people were/are RBS employees, just in India - it's a failure of off-shoring.

    It's a failure of kicking out all your experienced staff (I'm one who was made redundant at RBS) and replacing them with inexperienced staff who cost far less. Nothing to do with where they are or who they work for, their experience was inappropriate.

    This is what's enabled the RBS management to keep saying that it's not about outsourcing, because everyone has the opinion that this sort of problem is caused by external companies being cheapskate, rather than the company itself.

    1. Richard Wharram
      Devil

      Re: Not outsourced

      That can't be true. A techie is just a techie. If you can get a brand-new one for a very low wage then that is much better than paying a higher wage for an old one. Simple economics. You need to do some totals on a spreadsheet to prove this is correct.

      500 x £50,000 = £25,000,000

      500 x £20,000 = £10,000,000

      That's 60% cheaper annually. A 15 million pound saving per annum!

      Honestly, can you techies not do maths? No wonder you never got promoted to management!

      1. Anonymous Coward
        FAIL

        Re: Not outsourced

        And how many 'techies' employed in the UK previously by RBS caused a 3 week outage of services due to rank incompetence? 'Old techies' usually cost more because they know what they are doing.

        1. DanDanDan
          Facepalm

          Re: Not outsourced

          Did you miss that seminar on sarcasm last month?

      2. TeeCee Gold badge
        Coat

        Re: Not outsourced

        Ah, you forgot to factor in the 175 million it cost to fix the cockup caused by the cheapshit versions.

        So that's actually 12 years to break even, assuming no further fuckups and that the regulators don't levy a whopping fine.

        Management mathematics. It's like technical mathematics, only performed in fairytale land.

      3. Anonymous IV
        Thumb Down

        Re: Not outsourced

        "No wonder you never got promoted to management!"

        The correct word should be "demoted"...

        1. Fatman
          FAIL

          Re: The correct word should be "demoted"...

          NO The correct word should be "condemned".

      4. Magister

        Re: Not outsourced

        >>A techie is just a techie<<

        And that dear children, is the root of the problem. An assumption that "all animals are equal" but when it comes to management, "some are more equal than others"

        We all know what happens when you assume? (it makes an ass out of u and me)

        Perhaps we need to have a piglet icon?

        1. Anonymous Coward
          Anonymous Coward

          Re: Not outsourced

          >>A techie is just a techie<<

          That's the bean counters view.

          I am also ex-RBS. I saw 2 project teams totally off-shored as part of the panic to save management bonuses, both teams were 10-12 people, most of them with 20+ years experience each and all of them working on the same systems for at least 7 years (longer in some cases).

          One project team with the equivalent of over 200 man-years of collective mainframe experience and over 100 collective man-years of specific system experience were expected to transfer all that knowledge to 4, just-out-of-college, graduates in just 4 weeks.

          What could possibly go wrong?

          1. Richard Wharram

            Re: Not outsourced

            Should I have used <sarcasm> tags?

            1. BlueGreen

              Re: Not outsourced

              > Should I have used <sarcasm> tags?

              I thought mozilla were removing them.

              (Actual answer: no, maybe consider it unintended pro-bono trolling. Evidently they'd have missed a steamroller going over their legs, so it wouldn't help anyway).

      5. Juillen 1
        Holmes

        Re: Not outsourced

        Honestly, a management or financial task is just a manipulation of numbers. Here in techie land, we automate those and make them reliable and repeatable. What could be more efficient than replacing all the management number crunching with automation and saving shed loads more, removing most of the management? And CEOs? Hell, you can pick them up bargain basement straight off the end of a degree course. Big savings in the millions!

        Do beware, that <insert role here> is just an <insert role here> and thus replaceable cheaper works for every profession. And some are automatable.

        1. Anonymous Coward
          Anonymous Coward

          Re: Not outsourced

          I think it was Roger Needham who said words to the effect that administration can be automated, but management is that which requires human input. His phrasing was more elegant but I'm on a mobile and cba to google it.

      6. LoneStranger

        Re: Not outsourced

        So, this "15 Million per annum" costs 174 Million.

        On that basis, I'm glad I've never been promoted to management.

      7. Solmyr ibn Wali Barad

        Re: Not outsourced

        Ah, yes. Curse of the Mythical Man-Month.

        So many times has this beast been unleashed by the overconfident fools, who have just learned to put some simple numbers together, and got a feeling like they have now mastered Great Secrets of the Universe.

        Always fun to watch. With a hint of guilt and sadness.

      8. Anonymous Coward
        Anonymous Coward

        Re: Not outsourced

        +1 superb trolling.

      9. Tom Maddox Silver badge
        Meh

        Re: Not outsourced

        @Richard Wharram:

        I see what you did there, but I'm not going to upvote you because you're probably hitting too close to home with regard to what upper management thinks.

    2. Annihilator
      Headmaster

      Re: Not outsourced

      "This is not a failure of outsourcing - these people were/are RBS employees, just in India - it's a failure of off-shoring."

      Yup, you'd think an IT rag would understand the difference between offshoring and outsourcing. Think I've made a similar comment in pretty much all the RBS outage stories on here.

      If anything I'd wager that part of the problem is that RBS *didn't* outsource their offshore work. Outsourcing can work, offshoring can work. For a company to decide to work out its own recruitment policies across two continents and put the management practices in place to cope with it is pretty brave/foolish. It's cheaper than getting someone to manage it for you, but it means you have to develop your own expertise in managing it.

      Never though I'd say it, but TCS et al exist for a reason and as such cost more than hiring directly. But you also have the fact that *they* would be on the hook for any such fines and as such have a vested interest in getting it right.

      Outsourcing is generalised as a "bad" thing (this article being no exception) despite pretty much *every* company doing it to some extent (from cleaners to IT host). IT is often cited as a core service for banks and as such should remain inhouse. By the same token you could argue that IT is crucial to any web news site (el reg being an example) but I'd bet heavy money that the hosting is done by Rackspace, the search by Google CS and the Ads by DoubleClick.

      1. Chris Harden
        Angel

        Re: Not outsourced

        "I'd bet heavy money that the hosting is done by Rackspace"

        EIther that or they plastered El Reg with RackSpace adverts because they thought they looked pretty - not sure that's a good bet to take.

  4. JimmyPage Silver badge
    FAIL

    Moving away slighly

    does anyone know how far the consequential losses claimed can go ? ISTR at the time there were stories of house purchases falling through, and businesses being forced into bankruptcy.

    1. This post has been deleted by its author

      1. JimmyPage Silver badge
        Mushroom

        Re: Moving away slighly

        not really - T&Cs are subject to civil law, and can be struck out if unreasonable or unfair.

        Anyway, RBS T&Cs only apply to RBS customers.

  5. Toothpick
    Flame

    All these comments....

    ... should be forwarded to the RBS board & the FCA. Perhaps then they may see the error of their ways. The multi-million pound bonuses though may still make them a bit deaf.

    1. Anonymous Coward
      Anonymous Coward

      Re: All these comments....

      The thing is that they were doing the bidding of Gordon Brown - everyone knew what their task was when they were put in. Nice Labour policy that - save the company by making higher rate UK tax payers redundant and shipping their jobs off-shore. No wonder Margaret Thatcher said that her proudest achievement was Tony Blair. This is something that (like him or loathe him) would not have been done under Fred Goodwin.

  6. sugerbear

    Mike Errington's head on a plate?

    Is Mike Errington still in charge of their IT?

    If so he needs to be renamed Teflon, because the brown stuff certainly isn't sticking to him.

  7. Electric Panda

    Didn't the original balls-up at RBS happen in an Edinburgh data centre by an onshore employee? I can't recall where India came into it.

    1. Anonymous Coward
      Anonymous Coward

      No, it was an RBS datacentre in Edinburgh - that's where the two main datacentres are - but the RBS spokesperson said it was an RBS employee, when asked if they were outsourced.

      The implication of the question was - is this person offshore, the answer was technically perfectly correct.

  8. Captain Scarlet Silver badge
    WTF?

    This investigation is a bit late isnt it?

    I would have thought it would be started almost immediatly, otherwise it tends to get harder and drawn out.

  9. Shagbag

    One happy Indian

    "According to a whistleblower who worked with the bank for several years, an inexperienced, outsourced employee unwittingly made a huge error while carrying out the relatively straightforward tasking of backing up an upgrade to the CA-7 software."

    If my experience with off-shore Indian IT is anything to go by, I bet you THAT employee who pushed the button just went home at 5pm, came back the next day at 9am, said 'oh, sorry' (while wobbling his head) to his boss and sat down at his desk expecting just another days' work.

  10. John H Woods Silver badge

    Not a single person error...

    Whether a noob made a mistake or not is largely irrelevant ... if a single person can do this, whether accidentally or deliberately, whether remote or inside the datacentre, the system is already a failure.

    1. Anonymous Coward
      Anonymous Coward

      Re: Not a single person error...

      Not really, they were backing out an update which had gone wrong, at this point at RBS there is a peer reviewed SOE document (Schedule of Events). You follow the SOE and you don't go off-piste, because you've usually got highly elevated access which allows you to do things like backing out software updates etc. There is no getting away from the fact that to do this kind of work you need something akin to root access.

      Don't run with scissors boys and girls.

      1. VinnyR

        Re: Not a single person error...

        OK then the processes that allowed a junior member of staff to do this task is at fault.

        1. Tom Maddox Silver badge
          Flame

          Re: Not a single person error...

          OK then the processes that allowed a junior member of staff to do this task is at fault.

          The process that allowed a junior member of staff to do this task without senior supervision is at fault. Of course, all the senior people had been fired . . .

  11. PeterM42
    FAIL

    That'll teach them to outsource

    Hopefully the fines imposed wipe out any cost saving through outsourcing.

This topic is closed for new posts.