back to article Half the team at the heart of the RBS disaster WERE in India

Cost-cutting RBS management had halved the team within which the banking group's recent data disaster happened, sources have told The Register. The sacked British employees were replaced by staff in India, and there had been concerns about the quality of the work done in India for a lengthy period prior to last week's …

COMMENTS

This topic is closed for new posts.
  1. Chris Miller

    Beware of the weasel

    "The management and execution of batch processing is carried out in Edinburgh"

    Yes, I'm sure the actual execution of the code took place on a computer located in Edinburgh. And there's probably a manager for the process in Edinburgh, too (he'll be the poor sod with red eyes who's just got back after a 12-hour flight from Chandigarh). But the question that has not been answered is whether the staff who actually made the changes and cocked it all up were in Edinburgh. I wonder why that would be?

    1. Anonymous Coward
      Anonymous Coward

      Re: Beware of the weasel

      I guess you could say that the management are the SysOps, all based in Edi, unless they've gone too since I left.

    2. Anonymous Coward
      Thumb Up

      Re: Beware of the weasel

      Or indeed the statement -

      "The management and execution of the batch process is based in Edinburgh at Fettes Row, as is all the current work to resolve the problem." From yesterdays Guardian Q&A.

      Which was part of an answer to a question about Indian involvement with the bank problems. Note the use of the word 'current'. So it is just possible that there was some work that wasn't undertaken by staff located at either 'Fettes Row', Edinburgh, or Scotland? Who knows.

      1. Anonymous Coward
        Anonymous Coward

        Re: Beware of the weasel

        There do seem to be some definitive statements in this one -

        http://economictimes.indiatimes.com/news/news-by-company/corporate-trends/offshoring-blamed-for-rbs-onshore-glitch-bank-clarifies-fault-was-local/articleshow/14447599.cms

        It would seem 'The unsubstantiated allegations, denied by RBS, reprise a familiar pattern of noise against offshore outsourcing from developed nations to India.'. Also this has happened at a delicate moment when the Indian Outsourcing business is being hampered by slow growth.

        1. Anonymous Coward
          Anonymous Coward

          Re: Beware of the weasel

          There may be no evidence (yet) for it being a mistake by an outsourced third party, but there's no evidence that it was the onshore team either. RBS aren't saying, even though they certainly know. As they know, you'd think that they would say "here's the proof it wasn't due to outsourcing" if it wasn't, but they aren't, hence the reasonable assumption that they are trying to cover up what happened.

          For the benefit of any Indian readers, the hostility towards low paid Indian contractors is based on the fact that an awful lot of us are losing our livelihoods by our jobs being moved to dirt-cheap developing countries (where the quality of the code/support is piss-poor), whereas such a move would be illegal if they did it wholly within the UK.

          In other words, they're putting us out of work to save money in the short term, then when it goes wrong they are trying to pretend it was nothing to do with their stupid and short-sighted decision to use outsourcing.

          1. Anonymous Coward
            FAIL

            Re: Beware of the weasel

            I find the, albeit circumstantial evidence, in the phrasing of the denials that outsourcing wasn't to blame makes me tend to think that the opposite is the truth, and caused at least part of the problem.

            Also the continuous mantra that it is all hands to the pumps to fix the problem. I mean this problem happened getting on for two weeks ago, and RBS have stated that it is 'mainly fixed'. How can they make that statement without having a good grasp (names, places and so on) of what happened in the first place. Either they really are that bad (*) at management, or by now they must know what happened.

            If the failure was caused by a poorly trained CA7 operator in either of Edinburgh or India surely they must know? If nothing else part and parcel of the fix should be ensuring that this operator, and anybody else with a similar level of training, should by now be sitting in a classroom. At least as an absolute minimum these folks should be nowhere near a keyboard connected to the live systems.

            (*) for 'bad' above read appalling, intelligence of pond weed etc etc.

            1. Anonymous Dutch Coward
              Devil

              Re: Beware of the weasel

              Agreed with you on facts re timing, but why not spin out the "we're helping our customers, no time for pointing fingers" excuse as long as even the press will take it?

              Perhaps the furor will die down by itself and the "managers" can breathe a sigh of relief and get on with cost cutting as usual....

          2. Slabfondler
            Coat

            @AC 12:13 Re: Beware of the weasel

            Downvote for not reading the article - its NOT outsourcing, its off-shoring. Subtle difference, but different none the less. The rest is pretty spot on though!

            Mines the one with the "thanks for all your years of service, now get back to training your off shore replacement." card.

          3. Anonymous Coward
            Anonymous Coward

            Re: Beware of the weasel

            I can confirm from personal experience when I used to work there, that work from their TSI branch is often of shocking quality. One 'resource' we were allocated didn't even speak English past a level so basic that we couldn't use him at all. Supplied work in other areas was patchy at best and took as long for local staff to review and/or fix as it would have taken for them to do in the first place. The pattern of failure and cover up permeates the business at this point. When these failures are investigated the situation gets wrapped up into ping-pong style discussions and eventual denial that there's a problem, or yet more control processes that result in no change at all.

            They're starting to learn that you can't run a large IT estate properly when you get rid of all your staff.

      2. Anonymous Coward
        Anonymous Coward

        Re: Beware of the weasel

        The RBS SysOps are based at Fettes Row, as are (were) large proportions of the IT staff. In my more than five years at the bank, I never knew the sysops to take part in an upgrade or recovery of any software, except for a few secondary activities, such as stopping batches running, starting them again, etc.

        It is possible that the upgrade/downgrade work was done from the same building by others, of course.

    3. Anonymous Coward
      Anonymous Coward

      Re: Beware of the weasel

      Upgrades such as this taking several months of planning and involve technical resources and managers from several teams. So while the actual button monkey may have been in Edinburgh on this occasion, the Schedule of Events will have been worked on by multiple people in multiple locations with potentially varying degrees of diligence.

      Even if the diligence was good on the implementation planning, the backout planning either lacked some detail or was poorly followed.

    4. Chris Miller

      Weasel II

      The other weasel claim is that 'these were not outsourced staff'. This could be true - if the operation were jointly owned by Infosys and RBS, senior management could claim that they were still RBS staff, just based overseas. The interesting question would be: "did you get rid of most/all of your experienced staff who knew how this software worked and were deeply familiar with the intricacies of your bespoke systems and replace them with the cheapest option you could find?". Whether the result of this was staff based in Chandigarh or Auchtermuchty is irrelevant to the discussion.

      1. Anonymous Coward
        Anonymous Coward

        @ Chris Miller

        RBS are correct in saying it had nothing to so with outsourcing as they effectively stopped using that route (networks to Accenture excepted) at least 3 years ago. Instead they started their own IT centre in Gurgoan (near Delhi), so strictly speaking most the offsore staff are RBS and those who aren't are cotnractors working for RBS TSI.

        1. Daggersedge

          Re: @ Chris Miller

          This job advert supports that:

          http://jobsearch.naukri.com/job-listings-Batch-Analyst-RBS-India-Development-Centre-P-Ltd--Chennai-Delhi-NCR-5-to-10-years-160112001195

          However they try to wriggle out of it, though, RBS sacked experienced people in the UK and gave the work to cheap labourers in India.

      2. Anonymous Coward
        Thumb Down

        Re: Beware of the weasel

        @AC 13:11 GMT

        > Racist

        Would like to qualify that comment? Nothing I have said to my mind is 'racist' ?

        1. Anonymous Dutch Coward
          Coat

          Re: Beware of the weasel

          Don't talk to your mind. It might get ideas.

          Otherwise, couldn't find anything racist either...

      3. Blitterbug
        Unhappy

        Who is the downvoter?

        Anyone else noticed the single (sometimes double) downvote theme here? Odd... well, not really - someone has obviously mistaken the outrage and concern for raging xenophobia.

    5. Arctic fox
      Flame

      Re: Beware of the weasel

      The one thing that this proves (whatever else it may highlight) is that the "Directorati" are so fucking ignorant that they do not understand that (in modern industry, whichever type of industry it is) you cannot cut IT support in the same way that you can (if you are very bloody careful) in other areas of your business. The domino effect of the wrong decision in that area of your activities can have utterly disastrous effects - as we are now seeing.

    6. Anonymous Coward
      Anonymous Coward

      Re: Beware of the weasel

      "Never believe anything until it has been officially denied". - Claud Cockburn

    7. Charles Manning

      xenophobic witch hunt

      Half the team were in India.

      So half the team were in Blighty then. What were they doing?

  2. Shane8

    India v China

    I thought all (most) IT jobs were being outsourced to China rather than India ?

    1. Anonymous Coward
      Anonymous Coward

      Re: India v China

      No and certainly not for banks. RBS did have some dealings with Bank of China, but the servers ran from Singapore because it's fairly well known that half the staff of BoC are actually government staff, if you see what I'm getting at...

    2. ukgnome

      Re: India v China

      Too difficult with the great fire wall and some rather stringent data rules

  3. Forget It
    Alert

    A couple of typos Anna

    1) This created the backlog [that] has taken so long to clear.

    2( because he [had] been loyal to the company for years

    1. IronSteve

      Re: A couple of typos Anna

      *2) ...the irony ;)

    2. Jean-Luc
      Facepalm

      Re: A couple of typos Anna

      Re: A couple of typos[,] Anna

      Now, why would I be thinking of glass houses this fine morning?

      1. Anonymous Coward
        Anonymous Coward

        Re: A couple of typos Anna

        Nothing wrong with pointing them out, although nowadays I believe there is a separate link for that.

  4. Anonymous Coward
    Thumb Down

    RBS Management

    On a personally level I would think long and hard either if I was an existing customer, or thinking about, banking with RBS.

    Also on a personally level I have always wondered why the management of organisations such as RBS ignore the advice of IT staff on these sort of issues.

    http://www.channel4.com/news/rbs-warned-of-it-outsourcing-risk-in-2010

    As a tax payer (and assuming the lot of them don't end up in prison for manipulating LIBOR) the management of RBS have a lot of explaining.

    1. Anonymous Coward
      Anonymous Coward

      Re: RBS Management

      It's fairly simple - they ignore warnings from a self interest group about doing something which may damage that group. There doesn't seem to be thinking that the professional staff in that group may know what they're talking about.

    2. sugerbear

      Re: RBS Management

      Because come hell or highwater the head of IT was going to reduce the staff costs. And he based that decision on the fact that UK workers cost x and India workers cost y. And because Y is about 1/4 the cost of X that made perfect sense. They both do the same job after all. Anyone can be taught to programme, anyone can be taught to run a batch schedule. and so on.

      By reducing staff costs he would have been seen to have made a massive dent in expenditure and would no doubt have given him a nice leg up in his accent to the boardroom and his bonus is very likely to be linked to cost reduction and increased output. Manday cost were about £400 for a UK and about 1/4 of that for non-uk workers. Imagine reducing your costs by 75% and with ZERO impact on your business. What could possibly go wrong.*cough*

      I am sure that execs rationalise the negative internal comments by linking them to the UK staff fear for their jobs. So anyone that suggests the outsourcing is a bad idea either gets sidelined or kicked out early.

      1. Anonymous Coward
        Anonymous Coward

        Re: RBS Management

        "They both do the same job after all. Anyone can be taught to programme, anyone can be taught to run a batch schedule. and so on."

        The biggest irony of management thinking in banks is that any monkey can be taught to fuck up and lose millions too, yet only IT people get outsourced/offshored.

        1. Just An Engineer

          Re: RBS Management

          Or in the case of JPMC Billions...

      2. Fatman
        FAIL

        Re: ...anyone that suggests the outsourcing is a bad idea..

        BUT, if you do 'write them off as `in fear for their jobs`'; and it turns out that they were indeed 'correct'; then how do you explain a clusterfuck like this one?

        1. Shaun 1

          Re: ...anyone that suggests the outsourcing is a bad idea..

          "BUT, if you do 'write them off as `in fear for their jobs`'; and it turns out that they were indeed 'correct'; then how do you explain a clusterfuck like this one?"

          Coincidence

      3. Anonymous Coward
        Anonymous Coward

        Re: RBS Management

        I don't know how the financial picture looks in the UK, but in the US the financial case for outsourcing has a ton of caveats and asterisks attached. 15 years ago the financial case for outsourcing to India was obvious (financial case, morality and national interest aside). Over the last five years, the salaries for IT and engineers have been increasing by 20% per year in India while the US rates have basically been stagnate or decreasing. The current predictions, which were cautious, from the BLS are that it will be more costly, in raw terms, to hire an Indian programmer than a US programmer by 2017. That does not factor in the high overhead on outsourcing contracts, such as sales, PM, various liaisons which are for some reason required, travel, government costs, administration, communication tech, etc.

        The other headwind against outsourcing in the US, and even more so in Europe, is that the currencies are still strong vs. Rupee. That should swing in the other direction pretty furiously as the Chinese, and to a lesser extent, the Indians have been sterilizing their currency by parking their profits in US treasuries. They cannot go on parking their profits in US treasuries until the end of time... eventually they are going to get concerned that the US cannot pay it off, even with the US's giant economy. When they have to start spending their cash in non-dollar denominated bonds (e.g. buying stuff), the Eastern currencies will rise substantially and the export led economies will lose their cost advantage. It has to happen... It is getting very close to the end of offshoring, at least in India. The corporations have, of course, been going to even weaker economies such as the Philippines and China, but they do not have the talent or educational system that India has and the same principles apply. The Western governments and populations, with the exceptions at top, are flat out broke. When one group of people has all of the money and another group of people has none, the group with none cannot go on buying stuff from the group with all the money on credit from the group with all the money ad infinitum.

  5. Anonymous Coward
    Go

    One, two, three, jump! But where?

    Are any of the other banks any better? Do any of them have well managed IT departments staffed by long-term, experienced people? (UK or India or wherever)

    Where should a RBS customer take their overdraft?

    1. Anonymous Coward
      Anonymous Coward

      Re: One, two, three, jump! But where?

      We moved to Nationwide about 2 years ago (OK it's a Building Society not a bank, who cares). Very helpful service, transitioning the account went soothly with frequent progess reports, and absolutely no problems since then.

      1. Anonymous Coward
        Anonymous Coward

        Re: One, two, three, jump! But where?

        Should have mentioned, we had previously banked with RBS for about 25 years.

        1. John Arthur
          Thumb Up

          Re: One, two, three, jump! But where?

          I too jumped from Natwest who I had been with for over 50 years only about 3 months ago. I too went to Nationwide who have been perfect so far. They even have somebody in the bank I can go and talk to, unlike Natwest where every contact seems to have to go via a call centre. The whole Natwest "customer care" system which involved 4 months of mucking me around for a simple change of account name got me so livid I moved. And now I get free travel insurance too! Well done Natwest, you have save me about £100 a year on that one. Idiots!

      2. JimmyPage Silver badge
        Thumb Up

        Who cares ?

        Well, because it's a building society, it's only shareholders are *you*.

        Given that Nationwide appear to have charted their way through the credit crunch with no fuss at all (possibly because they can't play fast and loose with *members* cash, unlike a bank) I would say it matters a great deal.

        We moved to Nationwide after HBOS messed us around - can't fault them so far. Pleased we also have our mortgage with them.

      3. richard 7

        Re: One, two, three, jump! But where?

        Nationwide. That would be the bunch of asshats that stranded me on the wrong side of the world with no money because of an 'inputting error' 'Yes sir, I apreciate you are in the middle of no-where* and need to buy fuel with the money in your account but it will be at least 3 days to get it sorted'

        LLoyds have been allright for us thus far. Seems the ones to really stay away from WERE Halicraps and HSBC, the list seems to be growing at an alarming rate...

        * North Carolina, its scary out there.

        1. Anonymous Coward
          Anonymous Coward

          Re: One, two, three, jump! But where?

          I'm not a Nationwide fanboi - all service providers can screw up.

          But at least, with Nationwide, you can be guaranteed that anything done to benefit the shareholders will benefit you, since you are the one at the same.

          At the end of the day, HSBC, HBOS, RBS, Lloyds, Barclays et al have to keep the sahreholders happy - and damn the customers.

          1. Anonymous Coward
            Anonymous Coward

            Re: One, two, three, jump! But where?

            "But at least, with Nationwide, you can be guaranteed that anything done to benefit the shareholders will benefit you"

            Yeah, but who says they're interested in benefits for the shareholders:

            http://blogs.thisismoney.co.uk/2010/06/vote-against-nationwide-bumper-pay-packets.html

            I had a bank account, credit card and mortgage with Nationwide this time last year and they managed to cock 2 of those up, now I've closed them all.

            Mortgage was on a rented out house I wanted to change the terms on, I went to great lengths to ensure it was ok, talked to many people in many departments and on the day of change I was told it wasn't possible - as the house was rented out. Their excuse? The guy was new. A different guy in every department I spoke to was new?...

            They dropped my credit limit to £500 on the credit card. No one would say why, made many complaints and got nowhere. I paid the balance off completely every month without fail for years. My credit rating was perfect, in fact I took out a Halifax card in replacement and got a £12,000 limit.

            Not to mention dropping their commission free foreign currency withdrawals...

            1. Anonymous Coward
              Anonymous Coward

              commission free foreign currency withdrawals...

              Yes, I didn't like that either. In other respects —modulo all bankers are bastards I would trust futher than I could throw 'em— Nationwide havn't been bad.

              But commission free foreign currency withdrawals were worth having for me. And don't try to persuade me that 'free' travel insurance is compensation, a) it overlaps with the same I get on the Alliance and Leicester account, b) it dosn't cover me for the kind of travel I do; if I need something I buy the right tool for the job not rely on free toys.

          2. Anonymous Coward
            Anonymous Coward

            Re: One, two, three, jump! But where?

            Always Co-operative Bank... no Share Holders.

          3. Anonymous Coward
            Anonymous Coward

            Re: One, two, three, jump! But where?

            HSBC, HBOS, RBS, Lloyds, Barclays et al have to keep the sahreholders happy

            Which is probably a lot of us here, P45'd away over the years and left with PAYE sharesave schemes.

            When RBS reaches £20/share, even after the recent share consolidation, only then will I break even.

            So yes, more for the share holders, please.

        2. Anonymous Coward
          Anonymous Coward

          Re: One, two, three, jump! But where?

          Lloyds? As in Lloyds TSB?

          The bank where "for the journey" actually means a nice smooth straight middle class path, and not a graduate path on a low salary saddled with debt, with them sending out threatening letters?

          They can go jump.

          They're all as bad as each other.

          RBS/Ulster Bank seemed to believe a student account meant a basic upgrade from a Henry Hippo account. No cheque book to pay rent, no debit card to buy books off amazon. "Computer says no" type of staff.

      4. Anonymous Coward
        Anonymous Coward

        Re: One, two, three, jump! But where?

        My Dad had continual problems with RBS - has now closed all his accounts. Personally I never used them, lots of organisations have a lot of problems stored up for the future - hopefully RBS off-shoring blunder will make others think twice about risking core-systems.

        I bank with FirstDirect whose systems are all based out of Leeds and AFAIK managed from there too - FD's telephone-based customer service is 2nd to none. My simple re-payment Mortgage (almost paid off 10yrs early - how's your endowment doing?) is with the Derbyshire / Nationwide BS because as a Building Society they work in my interests -not- the shareholders (like ALL banks).

        Remember only your first £85K savings (£170K for a joint account) is guaranteed per institution and RBS is made up of lots of familiar names http://opencorporates.com/corporate_groupings/royal%20bank%20of%20scotland

        1. Matthew 3

          Re: One, two, three, jump! But where?

          I second the call to go to First Direct. OK so, yes, they're officially part of HSBC but they're managed independently. I doubt that'd help much if this kind of midden hits the windmill but their day-to-day interaction just works. The phones are answered by a real person - in the UK - and that one person can deal with 90% of things without having to transfer you.

          I'm one of those rare types who does actually move my account between banks - I've tried them all at some point - and only FD have managed to keep this awkward cuss impressed over a long haul.

    2. Monkey head
      Thumb Down

      Re: One, two, three, jump! But where?

      Nationwide??? The company who has already outsourced the support of their main banking engine to India, and is working to outsource as much of the rest of their systems as well? The company who has spent hundreds of millions of pounds on a replacement banking engine based on SAP to replace their old legacy system, but finds it still can't do all the processes so the old system still does a lot of the work (and the SAP system still doesn't work yet)? Good luck with them.

      1. Anonymous Coward
        Anonymous Coward

        Re: One, two, three, jump! But where?

        Not arguing with the offshoring and outsourcing concerns but as for the replacement of their core banking platform with SAP: that project is not complete yet nor live so in fact the old mainframe still does *all* of the work.

        The other banks have to go through their core ledger replacement at some point in the future. As others have said, time will tell with Nationwide's transformations, but you can't say they haven't been warned...

        1. Monkey head
          FAIL

          Re: One, two, three, jump! But where?

          Even after the SAP 'replacement' banking engine goes live the old legacy system will still do a lot of the core processing as it was all descoped after the SAP project ran late, and over budget without delivering. The new platform which was supposed to replace the legacy system to reduce costs will now run in parallel, meaning increased support costs.

    3. Anonymous Coward
      Anonymous Coward

      Re: One, two, three, jump! But where?

      "Are any of the other banks any better?"

      To the best of my knowledge, no.

      Barclays? Hahahahahahahahahahahahahahaha.

      Lloyds/Halifax/Bank of Scotland? I have been a BoS customer for decades - since it was a proper bank, in fact - and I too have been asking "are any of the other banks any better?" Clue: when I instructed them to pay off the balance of my mortgage without fail on a certain day, they failed to do so. So then I reminded them, and next day they paid it twice. Then they sent me a letter telling me I was overdrawn, "perhaps through some oversight".

      Santander? One word: "Spain".

      HSBC? Maybe one can trust the Chinese.

      I think you will find that all the banks, and other financial institutions, have found ways of offering equally abysmal service to customers. There is no escape.

    4. Jason Hindle

      Re: One, two, three, jump! But where?

      Like others here, I jumped to Nationwide a while back (just after the run on Northern Rock) but I'd hardly call them perfect. Their current account is free and competent with decent enough internet access. And of course, they weathered the crisis with relative ease, taking in a number of smaller building societies who were not so strong.

      Incidentally, I jumped from Barclays, the one major British bank who didn't go cap in hand to the government for a bail out. So, they may also be a good bet for a jump (in spite of current controversies).

  6. John G Imrie

    I have always wondered why the management of organisations such as RBS ignore the advice of IT staff on these sort of issues.

    Because management are brought up on cost benefit analysis, and can't see the benefit of spending all that money that IT demands for something that has such a low probability of happening.

    Of cause what they fail to understand is that the low value of a probability of a catastrophic failure is due to maintaining the current spending and that reducing the spend will increase he chance of such a failure.

    1. Anonymous Coward
      Anonymous Coward

      It's a two way problem

      The management only think in cost-benefit analysis.

      The IT guys can't quantify what they do in management terms.

      1. Mike Smith
        Mushroom

        Re: It's a two way problem

        True, but... management do indeed think only about cost-benefit, but surely that's an admission of culpability. It is utterly inexcusable for managers not to understand what they're dealing with. If an IT end-to-end service delivery assurance implementation deploymant compliance whatever manager's IT skills start and end at Microsoft Office when they're supposed to be responsible for enterprise-level systems and platforms, then that's a clear indication that they're not fit to do their job, no matter how nice an MBA they've got.

        Why should technical people have to quantify what they do in idiotic management speak? When you're in management, you're in charge; and you should make the bloody effort to understand what you're dealing with. Expecting the reverse is slipshod, incompetent and downright lazy.

        OK, I know that ain't gonna change overnight, but still...

      2. Anonymous Coward
        Anonymous Coward

        Re: It's a two way problem

        now they can quantify it - "If you eliminate your UK IT folk who are familiar with your system, your bank will do an RBS within 24 months." 1 RBS = 4 days outage = x billions in damages to bottom line.

        1. Mostly_Harmless Silver badge
          Boffin

          Re: It's a two way problem

          Is there a new Register Unit waiting to be defined? The RBS - a measure of system downtime, where 1 RBS = 4 days of system outage

          1. Velv

            Re: It's a two way problem

            Pedantry, but there was not a four day outage.

            There's been 10 days of massive impact and inconvenience, but they were still taking money out of your account on time, therefore there wasn't four days of "outage".

            1. Anonymous Coward
              Thumb Down

              Re: It's a two way problem

              The duration of any technical or systemic problems is hardly of concern to those who have been affected. Call it what you will, but there was a >4 day loss of service to many retail clients. Trying to argue otherwise flies in the face of any argument here about IT people not being listened to.

            2. Jabber 44
              Go

              Re: It's a two way problem

              I'd say that its an outage of between 2 and three RBS's then :)

          2. Fatman

            Re: new `Register Unit`

            Allow me to piggyback on that one, please.

            If you accept the "RBS" as a system downtime measure, then it would follow, that "RB$" could represent the value of that lost downtime.

      3. Fatman
        Flame

        Re: The IT guys can't quantify what they do in management terms.

        That's not always true.

        A few years ago, one of the newly minted PHB's had ideas of heading "toward the cloud"; which was not ever in the CIO's vision for the company.

        The day for the PHB's C level presentation came, and IT came "loaded for bear". Newly minted PHB put on his 'dog and pony show', and got a few C level types interest up. Then it was IT's turn.

        Metaphorically speaking, our CIO "blasted that turkey" as if she was carrying a 10 gauge shotgun loaded with 4/0 buck. His "plan" ended up so 'full of holes', it resembled Swiss Cheese. Needless to say, newly minted PHB was promoted to a new position with one of our competitors.

        Flames, because it was fun watching him "crash and burn".

      4. hplasm
        Devil

        Re: in management terms

        But IT guys can talk shit too!

    2. relpy
      Pint

      The answer lies in the phrase: "IT teams in areas considered non-critical had suffered redundancies of 50-70 per cent".

      They simply don't understand that so much of their "banking" industry is now IT.

      IT People aren't Banking People so they're not "core business". They don't know about banking.

      Beer. Need Beer.

      1. Anonymous Coward
        Anonymous Coward

        "IT People aren't Banking People..."

        It's worse than that at my place of employment. Not only are IT people not Banking People, but they're frequently not seen by anyone as Management People either.

        Management are simply required to be able to deal effectively with people; outside of IT techies are seen to not have any interpersonal communication skills.

        Some of the most creative, forward-thinking people I know have been techies yet once they reach a certain level within our organisation they face hitting a glass ceiling. HR are desperate to remove glass ceilings based on sex, race, religion or sexuality yet don't seem to care about the one that causes techies to get bored doing the same job at the same level for the same pay year after year before they eventually quit and go work elsewhere.

        Despite, as you say, so much of the banking industry now being IT, there are very few (i.e. none) senior managers or executives who either still are or were once techies; they simply know enough IT buzzwords to bluff their way through the recruitment process.

        Anonymous Coward because I'm already in enough trouble at work as it is.

        1. Anonymous Coward
          Anonymous Coward

          Re: "IT People aren't Banking People..."

          In my place of work, and I'm being very anonymous about it here, it seems that management don't give a stuff about the average hard working IT person. We are just "resources" to be used as they see fit.

          We've just had a department re-shuffle after a series of redundancies, you know the osrt of think, a bit of bum shifting at the top table which means bugger all to averyone else but it's hyped up as a big and important change. The boss of my team adimitted that he didn't know anything technical. Well, I'm sorry, what the f**k are you doing in IT then?

          I was involved in a crisis meeting a few days ago, you know, get 30 people on a conference call to try and found out what's gone wrong with a system and how to fix it. The person running the call (let's call him an Incident Maanger), had to also be on an executive call, so they could give us advice on how to proceed. WTF? I understand giving the execs an update and looking for guidance, but getting advice from them? Most of them can't see over the pyle of their executive carpet.

          ...oh and the problem....

          Caused by an Indian person installing a piece of software that was still meant to be in test....took six hours to find....

          Goes to show, you can have all the procedures, maangement control and signoff, red tape, but an implementation can still stuff up because the person doing it is inexperienced, and because everything is so complex now, impossible to find when it goes wrong.

          1. The Jase

            Re: "IT People aren't Banking People..."

            "The boss of my team adimitted that he didn't know anything technical. Well, I'm sorry, what the f**k are you doing in IT then?"

            He needs to be able to manage. People, budgets, resources, not be a techie. His job isn't tech, its management. He needs to be good enough that people can be honest about the tech with him, so he can make the right decisions.

          2. Anonymous Coward
            Anonymous Coward

            Re: "IT People aren't Banking People..."

            Was this an incident manager organising a non-technical call and then asking the techie what the problem is and how it's being fixed? "But we don't want to hear technical details". Well like duhhh... you have a technical problem that requires a technical solution...

    3. Andyb@B5

      They always have, traditionally IT department heads have not been good at communicating with the main business, especially at the higher levels. A few times when I've seen good communication the main board did *not* want to hear the message....... go figure.

      Sadly proven competence and relevant business experience and knowledge don't appear on balance sheets anywhere.

    4. naw

      Because the management get an idea about how they make savings/get bonus, then they ask their in-house experts who disagree, so they get into a cycle of asking the same question to people of decreasing expertise and increasing vested interest (eg outsourcing supplier) until they the answer they want to hear. Delighted because they've finally been proven right they blunder on and sign a deal as fast as possible before someone *higher up* changes their mind. Forget about *Due Diligence* - sign the deal Kerching!

      Seems like the spirit of Fred the Shred lives on in RBS

  7. Anonymous Coward
    Anonymous Coward

    Anyone want to cast the runes ?

    Already the government has started asking awkward questions.

    The FSA has wobbled into action, and will not doubt start asking other banks to demonstrate that this couldn't happen to them.

    As the court cases start piling in (I have read of more than one house sale that could not be completed due to this, so a few hotel bills will start to come in) will the board come under attack ? If it can be proven their actions lead to this (on a balance of probablities) ?

    Already my companys directors have started asking questions and seeking reassurances. I'm sure a fair few posters here will have been tasked with some sort of review, to settle their directors.

    1. Anonymous Coward
      Anonymous Coward

      Re: Anyone want to cast the runes ?

      "have read of more than one house sale that could not be completed" ... "hotel bills"

      ... that might not just be a case of hotel bills, that could presumably be a case of "loss of deposit", and therefore in the 10k (and up) range.

      1. Anonymous Coward
        Anonymous Coward

        Re: Anyone want to cast the runes ?

        House sale, loss of deposit for removals / van rental, having to rebook this (possibly with a more expensive price), storage for all of your belongings (and an associated '2nd' move from there to the new house, as I had to do when a form was held up, ironically due to IT issues...), possible loss of earnings (taking more time off work, moving mid week instead of weekend etc.), heck the sheer stress of it all!

    2. Velv

      Re: Anyone want to cast the runes ?

      "The FSA has wobbled into action"

      The FSA is being wound down and will not exist by the end of the year.

      1. Anonymous Coward
        Anonymous Coward

        Re: Anyone want to cast the runes ?

        Which doesn't change the fact they have started asking questions - even if they won't be here to get the answers

      2. Anonymous Coward
        Big Brother

        Re: Anyone want to cast the runes ?

        the harper government (right-wing fascists more inline with America's republicans and not far from the tea-party) here in Canada, is doing the samething to government services and oversight. Winding them down and closing them, because we all know that corporations can be trusted to police themselves.

        I chose the Big Brother icon because although the harper government ran on a platform of transparency and openness they have, blatantly, been the most secrective and dishonest bunch of politicians we've seen in the years. Who knew you can't trust politicians?

  8. Anonymous Coward
    FAIL

    It's abhorent that a governent owned organisation outsources anything to any other country imho.

    1. Anonymous Coward
      Anonymous Coward

      Part of the deal when RBS was given all that money was that the government would not interfere with the running of the bank.

      1. frank ly

        re. 'deal'

        So, RBS agreed to let the government pour money into them, provided the government didn't interfere with the future running of RBS? That doesn't sound like a 'deal' to me; it's sounds like a charitable donation.

        The situation was probably more complicated than that, I'm sure.

        1. Anonymous Coward
          Anonymous Coward

          Re: re. 'deal'

          It was more part of the deal that the public were given - remember that there isn't a single board member still at RBS from the Fred Goodwin days - if the government own a bank, they certainly shouldn't be messing with it for short term political gain.

      2. Anonymous Coward
        Anonymous Coward

        Just a cotton pickin' moment.

        Part of the DEAL the RBS was GIVEN.

        Isn't that like throwing a drowning man a lifebelt of his choosing and design ?

        The sheer arrogance ...

    2. Anonymous Coward
      Anonymous Coward

      Part of the deal RBS was forced to take when the Government appointed THEIR preferred Chief Executive and board to run RBS, a group of executives who had a track record of outsourcing and off-shoring British jobs.

      And the Government can't interfere, knowing that the then Government wouldn't be the Government much longer.

      Way to go Gordon Brown, Alistair Darling and the whole New Labour movement.

    3. Destroy All Monsters Silver badge
      Big Brother

      "It's abhorent that a governent owned organisation outsources..."

      It's abhorent that government-owned organisations even exist.

    4. Anonymous Coward
      Anonymous Coward

      @ Paul 161

      Forget a government owned bank - what about a local government body (multiple)?

      Certain local government activities are supplied by thrid parties who have been allowed by Boris among others to send to teams offshore, resulting in the lay off no permanent staff, but numerous short term contracts have been cancelled who pay (at least some) UK tax.. Assuming they're not doing the full Jimmy Carr.

      AC as extremely close to that issue!

  9. Thomas 18
    Go

    Good journalism

    Lots of digging, that's what I like to see. Keep going until your spade hits a skeleton.

  10. TrishaD

    Many years ago, I operated mainframes and we ran UCC7. Which was a very effective batch job scheduler. Then they were taken over by CA and it became CA-7. Still effective but to be frank, over the years, it became a complete camel to work with.

    I can understand outsourcing to India - I've worked with developers from India for quite a while, on and off, and they can be very good indeed.

    But CA-7 IS a camel and there cant be that many people around with the skills to run it, so to get rid of those who do and to replace them with those who dont seems like complete idiocy to me.

    The sort of idiocy that led RBS to appoint a marketeer like Fred Goodwin to run the bank, and the sort of idiocy that Hester has sadly failed to get rid of.

    There's no getting away from the fact that IT requires skills, and mainframe IT requires specialist skills. And that, in the end, you get what you pay for,

  11. Mephistro
    Joke

    " we will fully investigate the causes of this incident. We hope people will understand that right now our complete focus is on destroying the proofs fixing this problem and helping our customers ."

  12. Callum
    Meh

    Is it legal?

    I was wondering whether it was legal to make a role redundant and replace it with an offshore/recently onshore individual?

    * I'm sure we've all seen it happen; but anyone know the legality of this and is this why RBS are being very careful in their wording of the events that took place?

    1. Anonymous Coward
      Anonymous Coward

      Re: Is it legal?

      Yes, provided it's outside the EU.

    2. Anonymous Coward
      Anonymous Coward

      Re: Is it legal?

      Also - many redundancies are in fact voluntary separations, i.e. the role is NOT made redundant, I believe that the employee leaves of their own volition... How this gets by the tax law I'm not too sure ??

      1. Anonymous Coward
        Anonymous Coward

        Re: Is it legal?

        "Also - many redundancies are in fact voluntary separations, i.e. the role is NOT made redundant, I believe that the employee leaves of their own volition... How this gets by the tax law I'm not too sure ??"

        Correct. At our place, a compromise agreement is given to the (soon to be former) employee to sign (hence we call being made redundant "being compromised") rather than actual redundancy as such.

        AC: Do I need to explain why?

    3. sugerbear

      Re: Is it legal?

      And RBS set it all up nicely.

      First they had an arbitary downgrade of all grades at the end of the year. They also had the bell curve, what this did was to put a lot of staff that came in and did their day to day work but didn't want to get involved in office politics or career advancement or working stupid hours into the "failing" category. It also put people that would normally have got a bonus into the "no bonus" category. We were also told that anyone getting two consecutive two's would be "managed out". (good bye payoff). No one understood the rating system either as they were given behind closed doors.

      A lot of staff in their fifties were entitled to full pensions with no penalty for leaving early (ie get the pension at 50 odd that you would have got at 65). We also got a payoff of something like 3.5 x number of years service = number of weeks pay. If you were retiring you could put that lump sum into your pension and then claim 25% of your pension pot tax free. So a bloody good deal.

      So generally anyone with long service who was over 50 jumped at the chance, the first wave was fully subscribed.

      Anyone with with long service and a decent work history also went, it's just a case of incentivising people of else making it so miserable to work there people volunteer to leave.

      It looks good for RBS because very few people have to be made compulsory redundant and they can spin the line that everyone "wanted" to go where as if they gave everyone their notice then there would be headlines in the national paper.

      Glad I got out in the first wave ;-)

      1. Anonymous Coward
        Anonymous Coward

        Re: Is it legal?

        That is a great deal for those who were able to retire early with a full pension. Now they can bring you or those interested back as expensive contractors to run the system. I hope that works out for you. For the others not so good but that's becoming a common corporate prectice.

  13. Anonymous Coward
    Anonymous Coward

    It seems clear to me that this was messed up in the UK and the blame seems to be trying to be shifted to India. Saying that, I don't think this is an ideal setup at all, having the teams separated by continents and cultures.

    1. Anonymous Coward
      Anonymous Coward

      what seems clear to me

      is that you've typed something before you've read everything.

    2. Cokezero

      I dont think theres any question that the CA7 Upgrade was performed in the UK. Of course it was. with 20 years ca7 experience I'm still scratching my head trying to work out how an Operator or Batch Administrator could clear the entire CA7 jobqueue by mistake , assuming thats what happened. Offhosring has done me no favours, but lets be honest here. Lets assume an offshore member of staff was responsible for the mess. Are RBS and all the other banks going to stop offshoring ? Nope. It's just a witch hunt with ulterior motives.

      1. Peter2 Silver badge

        No, there is no question that the CA7 upgrade was performed on hardware in the UK at all. Of course it was. RBS has confirmed as much, and they have also confirmed that the staff performing the action were not, through the language that they are using. When they start sounding like Sir Humphrey from Yes Minister and put substantial amounts of effort into dodging the question it's pretty obvious what's going on.

        Presumably they are hoping that the uproar will go away, the public will forget and they can carry on as usual. The funny bit is that the mainstream media are taking it hook line and sinker, only to be pulled along by el reg. I expect that the MSM is going to get upset when they eventually realise they are being misled, and then it's going to start getting really ugly for RBS.

  14. Armando 123

    Cheaper?

    Well, it may cost you less up front, BUT ...

    Last summer I took a contract (while looking for a permanent position) with a former employer who had been forced by corporate to offshore because it was cheaper. I was not permitted to touch the code ... read permission only ... and could have made the changes in two days since I'd written it five years before. However, I was not permitted to do this, but had to document what was to be done.

    And by "document", I mean write the code so the Tata people could cut and paste it. And EVEN THEN they screwed it up.

    1. Anonymous Coward
      Anonymous Coward

      Re: Cheaper?

      Did you get files back which had "CUT" written at the top of one, and "PASTE" on top of the other ?

      Yes, I have seen it. I kept on expecting a call from India, asking where the "any" key was.

  15. EvilGav 1

    In response to the varied comments of "why didn't management listen to staff?" - because they never do. Especially, it seems, not to IT staff.

    I remember personally writing a report, explaining changes that needed to be made to one of our information systems, which got ignored for a number of years, right until management paid for an external "consultant" to come and decide what we needed. They more or less re-headed my report and charged £50,000 or so for the months work (this was over a decade ago).

    Today, in the same company, in a different part of IT, warnings regarding off-shoring are being ignored and the cracks are beginning to show - in about 6 - 12 months quite a lot of our senior management will be out of a job (no doubt with golden parachutes) and leave behind a mess that will cost multiple times the cost to fix.

    1. Anonymous Coward
      Flame

      Which Would Be Good

      "in about 6 - 12 months quite a lot of our senior management will be out of a job (no doubt with golden parachutes) and leave behind a mess that will cost multiple times the cost to fix."

      That is how the free enterprise system works - stupid ideas are tested until they work or not work. Those which do not work are abandoned. Sometimes, people who don't work are being abandoned, rather physically. That's what happened in the 1930s.

      Seeing how incredibly stupid and greedy our "elite" was in the 2010s, I fully expect history to repeat itself in some way.

      1. Anonymous Coward
        Anonymous Coward

        Re: Which Would Be Good

        Actually in the 20xxs and 201xs.

  16. Parax
    Alert

    missing a point?

    Does it matter where they are based? we still don't know for certain who fucked up yet. maybe the fuckup was a guy in Edinburgh... then, so what if half the team are in India?

    The thing that concerns me most is that the whole system goes down from 1 point of failure.

    And how many single points of failure 'COULD' be deliberately exploited by one operative working abroad?

    Just imagine if Visa goes down during the olympics.. who needs a plane these days? when you can inconvenience several million people for a week or more...

  17. Annihilator
    Meh

    Three things

    1) It seems that people are conflating offshoring and outsourcing as the same thing. From what I can tell, RBS are employing people in India? This is offshoring, not outsourcing.

    2) To me this smacks of not testing a backout plan, regardless of where the work is done (and I would have a strong assumption that the implementation and ballsed up backout was done in the UK)

    3) The story reads as if having Bob-with-30-years-experience is a brilliant thing, instead of the reality that having all your process and system knowledge lodged in Bob's head is an inherently bad thing. Bob-with-30-years-experience is 5 years from retirement / contracting at an exuberant rate due to no other option.

    The whole thing smells a bit funny to me, including the instant reaction of "it's India's fault" based on anon ex-RBS staff. Lack of proper governance around implementation planning/rehearsal is the first place I'd be looking at, and I've done this for a living. A low-paid bod shouldn't have the ability to cause this much carnage on his own, and so points to shoddy systems, not (just) shoddy operators. Offshoring and/or outsourcing only works in a high process-driven environment, which many firms don't have to begin with.

    1. Anonymous Coward
      Flame

      So, You Say The Same

      ..just more complicated worded ??

      Apparently RBS did a royal fuckup in the years to 2008/9, by participating in the Big Casino (All those funny "investment banking" schemes). They got rescued by the government injecting dozens of billions Pound.

      Then they sought to "reduce cost at all cost" and got rid of their best mainframe people. They simply ignored all the warnings that their new staff was badly trained. Unix != Mainframe ! So hiring a Unix guy for mainframe is as good as hiring a car mechanic to fix an aircraft engine.

      Even if they got some mainframe people, did they assure they had all the proper knowledge ? Of course not, as it was about "quick results". There was no such thing as a well-thought out and controlled transition plan to move all the knowledge to India. That is what we can infer from all the transpirations.

      Of course that can all be done, but it first costs more money than it saves (for, say, two years), it must be rigorously checked for success (has the Indian guy really acquired all the essential knowledge at project milestone X ?) and it does not produce quick cost reductions.

      So the MBA idiots fucked it up. That's all.

    2. DragonLord

      Re: Three things

      re: 3) Yes, having bob with 30 years experience is a good thing, because he has all of that system knowledge in his head, however you should also have Charlie (25 years experience), Alice (20 years experience) as well as Joe (2 years experience), Simon (5 years experience), and fred (5 years experience). Bob's job will be to maintain the systems and bring everyone else up to the level that the department doesn't die if 1 person is hit by a bus.

      Replacing Bob (30 YE) with 3 months notice is a bad thing because he can't ensure that everything that needs to be transferred has been transferred.

      1. Thomas 4
        IT Angle

        Re: Three things

        Especially if Bob doesn't have much incentive to explain it properly and lot of incentive to watch the system go titsup.

        1. Anonymous Coward
          Thumb Down

          Re: Three things

          That diminished sense of responsibility, associated perhaps with worrying about where the future mortgage payments are coming from, and explaining stuff to somebody who's command of English is almost but not totally non-existent?

          How can 'Old Bob' be so totally and utterly disloyal?

      2. Annihilator

        Re: Three things

        "Replacing Bob (30 YE) with 3 months notice is a bad thing because he can't ensure that everything that needs to be transferred has been transferred."

        Yes, 30 years experience of CA-7 is a good thing, but a system that relies on 30 years experience of *that particular* CA-7 environment is a very bad thing as it implies there is zero documentation or processes around it. And to be clear, I'm talking about more than commented code, I'm talking about design docs of various levels.

        The problem is that there is no Joe, Simon and Fred willing to move into these roles and stay there for 30 years. The long-term model doesn't work, there is no job for life any more.

    3. Chris Miller

      Re: Three things

      Another problem - all the clearing banks and many more large (mainly financial services) institutions are dependent on mainframe systems. The folk that have spent a lifetime working with them (like 'Bob') are nearing retirement - they need to be replaced. But how do you get bright young graduates to work on mainframes. They don't want to work with CA7 and CICS/Cobol, they want to work with Python and Javascript - which will look far better on their CV. Migrating from these systems isn't a realistic option, at least not with a timescale that isn't measured in decades.

      I'm not sure that there's a solution to this problem, and if there is, what it would look like. But if it isn't addressed pretty soon, we're going to be seeing more such incidents.

      1. Destroy All Monsters Silver badge
        Coat

        Life is uncool

        Military - it's not about fighting and shooting, it's about logistics [and state violence]

        Banking Informatics - It's not about Python, Javascript and Web 2.0, it's about CA7 and CICS/Cobol [and state bailouts]

      2. Anonymous Coward
        Trollface

        Re: Three things

        "...Migrating from these systems isn't a realistic option..."

        Dude - just move it all into the Cloud - problem solved. I'm actually just waiting to hear the first bank suggest it.

  18. MrPatrick
    Stop

    According to this article

    According to this article :

    - The issue was caused by someone backing out of the CA7 upgrade.

    - CA7 is wholly supported in the UK

    I don't see what the offshore angle is. I can see the 'reduced numbers' angle. Although perhaps the offshore team aren't very experienced in restoring the batch jobs after they were wiped by an onshore engineer.

    I worked with (went over and trained) a large offshore team so I can confirm whole heartedly and completely that the quality of work can be... patchy? Despite enthusiasm and individual talent, but I do wonder if perhaps people need to take a breath and wait for facts (or press for more facts before finger pointing).

    1. Keith 17

      Re: According to this article

      > I don't see what the offshore angle is.

      In the first instance, it's a "replace experienced staff with cheap inexperienced staff" angle. However the way the way that is done more often than not is by offshoring.

      If the cheapest way of doing that it was to outsource it to a local playgroup and let the 3 year olds do it, then we'd be talking about the playgroup angle.

      But no-one's thought of doing that yet, they offshore instead. Hence there is an offshoring angle.

    2. ajcee

      Re: According to this article

      That's not the way I read it. Whilst it would appear that the CA7 upgrade and the subsequent backout a couple of days later were handled by the UK-based support team, I've not read anything to suggest that either procedure was problematic (apart from the fact that the upgrade clearly introduced issues that then necessitated that it be backed out.) However, the batch processing is done from India and - if The Reg's story yesterday about someone clearing the batch queues and deleting the schedule is on the money - you can start to hypothesize that the Indian-based staff halted the batch processing, following which the UK-based staff backed out the change. Thereafter, the Indian-based staff were meant to kick the processing off again, in co-ordance with their checklists used in such instances. Except that there was nothing to be kicked off, the queues were unpopulated and one of the staff had bolted... I'm sure it's nowhere near as simple as I think it is so - as I've bugger-all experience with IBM mainframes and CA7 - I'll await comment from someone qualified...

  19. GreyWolf
    FAIL

    India and mainframes

    I worked for Big Blue for many, many years. As far as I know, India had very few mainframes - Bank of India (AFAIK a central bank, not a retail bank) was the only one I ever heard of.. Late 70s- early 80s a change of government policy pushed many Western firms out of India, including IBM, just at a time when the role of the mainframe was being questioned. So India probably never had more than a tiny number of CA-7 installations, if any at all.

    And that's why it made perfect sense (but only in the RBS dream world) to go to India to recruit a CA-7 scheduler with 5 - 7 years experience.

  20. GreyWolf
    Unhappy

    Mainframe Forum: Where Indians Cavort

    Check out this place http://ibmmainframes.com/index.php

    Pick any forum on a subject you understand.

    Check the poster's names, and the quality of their questions.

    I wonder how many of these people are being billed out at full rate to western firms. People who cannot read the manual, look up an error message, or use the utilities that have been in the IBM environment for 50 years. My favourite is the guy in the CLIST & REXX forum who has written a REXX program to copy a file. Jesus wept.

    1. QuiteEvilGraham
      Thumb Up

      Re: Mainframe Forum: Where Indians Cavort

      Had a look at that: I particularly like the assembler and PL/1 forum where people are posting their homework from some college course in the hope that they will get stuff explained, then getting flamed when they start arguing the toss.

    2. Anonymous Coward
      Anonymous Coward

      Re: Mainframe Forum: Where Indians Cavort

      Did you mean http://ibmmainframes.com/about57821-0-asc-0.html ?

      Made me smile, thanks.

      Completely off-topic

      The site brought up a pop-up with this message :-

      Your Adblocker prevents this site from functioning correctly. To stop this alert, please disable Adblock for this site from Firefox Menu -> Tools (ALT+T) -> Adblock Plus -> Disable on ibmmainframes.com

      Never seen that before !

  21. Anonymous Coward
    Anonymous Coward

    the perfect storm

    Two types of cultural weaknesses could be at play. Could be people who have no concept of the risk they are taking on (remember india has 200000 road fatalities a year) or somebody in the UK dropping the ball due to Euro2012 distractions. (everyone watching the footy)

  22. Alan Brown Silver badge

    Whose fault?

    The issue is that of using outsourcing to replace expensive, experienced staff with cheap script monkeys. That works fine until the shit hits the fan.

    Accents don't matter. A good chunk of the monkeys are in Edinburgh. What matters is the bank dumped its experienced staff overboard in order to cut costs and got bitten on the ass as a result.

    In a fully accountable company structure the managers responsible would already have been escorted to the door by security. If it does get to Select Committee there will be a LOT of selective amnesia.

  23. JaitcH
    FAIL

    Where is the Immigration / Foreign Worker Control?

    Seems to me that the government needs to tighten up on who and why foreign workers are allowed to work in Britain.

    I would be very suspicious of, in this case, Indian companies trying to bring workers in to the UK.

    This whole fiasco highlights the shortsightedness that British jobs can simply be exported to other countries where the pay is minimal. I suspect any money saved has long been burned up by management trying to rescue the operation.

    Only bankers could interpret this as implementing economies.

    1. Destroy All Monsters Silver badge

      Re: Where is the Immigration / Foreign Worker Control?

      That's why the free market idea has been floated, once upon a time.

      It might work out. One should try it.

  24. Ian Michael Gumby
    Boffin

    Missing numbers...

    Part of the story that is missing is how much this cock up will cost RBS.

    Clearly the downtime can be measured in terms of dollars/pounds per hour.

    Then there are the intangibles in terms of good will...

    The issue is that while the bean counters talk about off shoring as a revenue saver and point to the lower costs, they never reconcile the costs associated with an actual disaster. Sure they saved $$$ over the years, but this type of disaster just caused them $$$$.

    When you put things in perspective... Off shoring jobs increases risks that could wipe out any savings.

    Not to mention that the numbers are skewed to begin with...

    1. JimmyPage Silver badge
      Alert

      It will cost then Dear

      that's dear with a capital "D".

      Quite aside from the fact that no one in Britain would let them near running a fete stall, let alone their bank account, is the fact that they face a deluge of court cases for *consequential* losses. Not only from their customers, but people who were relying on them to transfer money in a contracted timescale. Each case will need to be decided on it's merits, which means NatWest need to defend each one individually. We've already heard of cases where house sales were affected - what if some of those sales were cancelled, due to the lack of funds transfer ? Everybody so far seems to think in terms of a few quids interest on late payments, or overdrafts. I'm thinking of cases which could run into tens of thousands. Enough incentive for both sides to take to court,

      1. DragonLord

        Re: It will cost then Dear

        What about the person that was remanded in gaol for a couple of days extra due to the fact that his bail money couldn't be verified?

        1. JimmyPage Silver badge
          Alert

          that's a fascinating legal minefield ...

          because I read of *two* prisoners who were affected. The one you mention, who was held over a weekend, because despite his bail being posted, the Prison Service couldn't verify it.

          However, *another* prisoner in a similar situation did get out, because his solicitors did some ringing around, and managed to either stand surety themselves, or convice the PS that it was a genuine hiccup, and the money was there.

          So, as I originally said, *every* case will have to be fought individually. How much do you think that will cost NatWest - even if they sucessfully defend every claim. Especially since they can't be guaranteed of recouping any costs - even in the high court.

          1. Daggersedge

            Re: that's a fascinating legal minefield ...

            Each case having to be fought individually actually favours RBS group. This is because taking a case to court, even the small claims track, can be a very tiring process. RBS group, as the defendant, can do all manner of things to slow down the process so that claimants get tired of the whole thing and settle for some small amount of money.

            What can the bank do? If the registered office is in Scotland, it might be able to say that all claims have to be made in Scotland. If I recall correctly, the Scottish small claims track has a much lower limit than the one in England. That would mean that many cases would have to be heard at a more expensive level, if, that is, they were heard at all, because many people in England just wouldn't be able to afford to bring a case in Scotland.

            But let's say that RBS has a registered office in England. That doesn't mean that everything will work well for the claimants, not at all. First of all, if a claim is filed using Money Claim on-line, it can have the claim struck out for inadequate particulars. This is because there is a limit of how many characters one can use to describe the claim. This is an old trick, but, as far as I know, the courts are still accepting it. OK, move on to claiming the old-fashioned way at the local court. RBS' legal types will then apply to have the claim moved to wherever the registered office is in England and, as the defendant, it has that right, although the court can show discretion here. Even if the claimant manages to get the claim heard in his area, or is determined enough to go to wherever RBS has registered its office, RBS can then slow down the procedure by asking for the case to be delayed because of lack of staff. That will kick the case three to six months into the future. Meanwhile, RBS will make the claimant a small offer of compensation 'Without prejudice'); by this time, many people will take it just to be done with the matter. There are other tricks, but you get the picture... oh, some cases will be heard, but most people will be put off. They will also be on their own as far as costs are concerned. Costs have to be paid up front. Even if a claimant wins, he will still have to collect the money. RBS can also make that difficult. In the end, most people will be worn down, you can bet.

            I know for a fact how difficult it can be to sue someone in the UK as I successfully did so, but was a very wearing process and, if a not-so-bright person like the person I was claiming against can game the system so easily, just think what highly-paid lawyers can do.

  25. Custard Fridge

    Soooo.....

    Soooo someone in Edinburgh wiped the batches off, then the inexperienced team in India took an age to recreate / restore it all?

  26. -tim
    Pint

    Where did the money go? You don't have that level of cash sitting around mid-transfer without it going to a suspense account for a bit and even those have to balance.

  27. Anonymous Coward
    Anonymous Coward

    Reeks of Daily Fail

    Seriously? We're bringing race into this now are we?

    Ok here's one for you. Last week two very white British staff members brought down the PAC's system for all of XXXshire (many IT staff members at the trust read this and said trust don't llike negative leaked news stories). Eight hours without any X-Ray's or whatever it is the system does.

    Obviously it's not made front page national news, or local news for that matter, probably due to the fantastic job of the risk team keeping tabs on such an instance.

    My point is, lets say for instance the down time hit days rather than hours. Let's say the down time resulted in deaths, which it could of. Would the reg then run an anti human story? Perhaps selling the benefit of giving jobs to robots?

    1. Tom 38

      Re: Reeks of Daily Fail

      We're not down on the offshored workers because they are Indian, but because they are incompetent and inexperienced, and managed to burn a bank. If we weren't in the middle of a huge financial crisis already, we'd be in the middle of a huge financial crisis because of this.

      Offshoring is a nonsense plan, just like Chinese development teams - unless you are trying to develop a Chinese product. China is growing at 7-9% annually - which means prices go up by that much each year. Chinese dev teams are increasingly similar in price to a UK dev team, particularly when you take into account (as we already do) their poor productivity, which mainly comes down to not being native English speakers (and thus not understanding the nuances) and not asking appropriate questions rather than any lack of application or intelligence.

      1. Destroy All Monsters Silver badge
        Holmes

        Re: Reeks of Daily Fail

        > China is growing at 7-9% annually - which means prices go up by that much each year.

        Err no. Either that growth is actual, in which case prices stay the same or even go DOWN, or it's mainly inflation (increasing monetary mass which looks like growth on paper) in which case prices go indeed up. Wealth not so much.

      2. Anonymous Coward
        Anonymous Coward

        Re: Reeks of Daily Fail

        "We're not down on the offshored workers because they are Indian, but because they are incompetent and inexperienced, and managed to burn a bank"

        But (and this is important), you don't know this to be true. And even if it were true, the blame would either be a) piss poor outsourcing constructs (not by default offshoring) which would be covered by liquidated damages if their procurement team was worth their salt or b) really poor recruitment policies (if in this case they're hiring directly in India) in which case, yes, hang 'em out to dry.

    2. GreyWolf
      FAIL

      Re: Reeks of Daily Fail

      Nothing to do with race, shipmate, and everything to do with management thinking that people, you know, human beings, are plug-compatible. Thinking that they can get shot of 1500 experienced long-term people, and get EXACTLY THE SAME OUTPUT from 800 inexperienced short-term people.

      And the RBS management not paying enough attention to see whether the outsourcing company actually delivers on any of their promises regarding the quality and experience of the supplied contractors.

      The really big problem with outsourcing is that the customer (RBS or whoever) has absolutely no effective control when the outsourcing company fails.

      I've seen similar things happening elsewhere (BT, anybody?)

      1. hplasm
        Mushroom

        Re: Reeks of Daily Fail

        The problem being is that the Execs ARE plug-compatible empty suits. That's why they can fuck up, fuck off and wander into a new position and carry on.

        Kill them all with fire.

  28. Anonymous Coward
    Anonymous Coward

    8 Hours...

    ... well I'm glad techies weren't based in India as there probably would have been deaths!

  29. Anonymous Coward
    Anonymous Coward

    It's utterly disgraceful

    That you're posting copies of that guy's CV in news stories. Regardless of whether you've blanked his name out, it will be very easy for potential employers to work out who it is and avoid him like the plague. And you've got no proof that the poor sod was even involved in this whole fiasco.

    It's fucking comicaly that you're putting "This picture cannot be reprinted without permission" ON A SCREENSHOT OF SOMEBODY'S CV THAT YOU'VE NICKED FROM LINKEDIN

    This whole "IT WAS OUTSOURCING WOT DUN IT" approach is very Daily Mail and is more than a little unsavory. Yes we all know that outsourcing - when it's done badly - can have maddening effects to what used to be simple processes. And yes the same can be said for offshoring, when it's done badly. But why are you incessantly trying to pin this particular incident on an offshore team? What point is it you're trying to make? Outsourcing is bad? Well fuck-a-doodle-doo, that's a new one.

  30. Anonymous Coward
    Anonymous Coward

    It's utterly disgraceful

    Oh yeah..

    Googling "a hard-working, dedicated and technically sound professional with expertise of almost 5 years" brings his name up in the Google cache.

    Though his linked-in page has been pulled. Poor sausage...

  31. Anonymous Coward
    Anonymous Coward

    What happened?

    I don't know CA7, just a little OPC/TWS but did someone flush the JES queues and clobber the JES checkpoint while CA-7 was down? As it's a backed out upgrade the other possibility is that they had to convert CA-7s database format/schema from 11.1 to 11.3 then going back they screwed up their backup copy of the 11.1 version? But these are things the CA-7 sysprogs should be doing.

    Or the guys setting up the jobs were doing something deprecated but which worked then the upgrade screwed it up? My ignorance of CA-7 may be leading me up the garden path but interesting to speculate.

    I do know RBS are big CICS users implying they run MVS not VM/CS.

  32. Stephen Channell
    Facepalm

    Whether the staff were Indian (onshore or offshore) is irrelevant

    There are good and bad people everywhere, a Scottish graduate is just (if not more) likely to cock-it-up as an Indian. The issue is about cost, experience, orientation, training and commitment.

    Mainframe schedulers like CA-7 & OPC/A have the concept of “current-plan” that is a pre-calculated optimisation to avoid resource contention and release jobs very quickly irrespective of the size of the schedule. If you’ve got 18k jobs “Current-plan” is vastly better than the ponderous dispatch you get from Unix schedulers that do things dynamically.

    My hunch would be that it’s the “current-plan” that was lost; and the extended delay was caused by some of the jobs being run before the problem was found. CA saying the problem was unique to RBS, is probably not a polite version of “they’ve got an idiot there”, but simply that CA-7 had local customisation (perhaps additional resource dimension of running three banks off one schedule).

    My guess is that they’ve replaced older mainframe-era staff with younger staff lacking the orientation to understand architectural differences; younger staff looking for a stepping stone; managers without the relevant experience.. but the real problem is cost.

    Scheduling an upgrade mid-week is always about cost; Mr Hester should take it on the chin and not hide behind some poor soul, but state what he’s going to do better to deserve the trust we place in him.

    1. Cokezero

      Re: Whether the staff were Indian (onshore or offshore) is irrelevant

      Sounds like you work in IT. Sometimes people just F* up and there comes a point in Problem Management where the analysis just becomes a witch hunt. The trigger (if you pardon the CA7 pun) for this was backing out a 11.3 upgrade to CA7(performed at the weekend) on the Tuesday. Presumably there was little choice in the matter after it had been identified Monday. Either the backout is the problem , or a batch administrator cancelled a job they should have force completed in CA7, completely unrelated to the backout with ensuing chaos . One or the other, and it sounds like the former. Its very difficult for an Op or Batch Admin to wipe the queue by mistake. I repeat, sometimes people just f* up. Solution ? Two sysprogs on each software change ? You cant automate version upgrades. And i didnt even mention offshoring. Oops.

  33. jason 7
    Facepalm

    Nationwide and their Mortgagre Quotes System

    Back in 1996 I was looking to buy my first flat. As I had some accounts with Nationwide I went in and got three mortgage quotes from them.

    I got home and that evening my Dad and I went through them. We couldnt get them to add up.

    Just wouldnt work. So I went in the next day and asked the Mortgage lady. She just looked at me as though I was from Mars and then tried to work them out. Three attempts and a lot of odd looks she goes to the Manager. He sits down and just keeps starting at his calculator as he tries to work them out.

    He then quickly makes a call.

    Seems that their quote software was all wrong. All the quotes they dished out were incorrect. It seems my Dad and I were the only people ever to check them.

  34. Anonymous Coward
    Anonymous Coward

    Youtube

    NSFW lyrics https://www.youtube.com/watch?v=-o1tHVa7VV4

  35. Anonymous Coward
    Anonymous Coward

    Speaking from experience

    People "down in the trenches" know who/what caused it. However, by the time it gets to management for any kind of decision making, the actual information will have long been lost. It's sad, but it's the truth of big buisness. Midle-management, in an effort to cover it's own idiotic ass, will make it somehow the more expencive guy's fault and claim it was right all along to fire experienced people and hire cheaper monkeys.

    That's the really sad part in all this. I've seen it before, more then once.

  36. Anonymous Coward
    Anonymous Coward

    I have two issues with this article

    Number one: What are your "sources" (plural)? Your article byline says "exclusive" and going by the content I cannot but help to think that you have one single "source" consisting of a UK-based (I would go as far as to say white and middle aged) technical employee who of course would have no axe to grind whatsoever.

    Number two: Regardless of whose fault this is, and how incompetent or otherwise people working in one place or another might be, a major incident is never caused by a single factor (in my industry we are taught about the "Swiss cheese" theory of many holes lining up)--this is touched upon by somebody with the nick "Annhilator" in the comments above, who makes some very pertinent questions, none of which are even hinted at in this article. Personally, I would like to know the whole story.

    For the above reasons, I consider this article substantially deficient. Perhaps The Reg would care to properly check and contrast its source(s) and offer a wider analysis of the probable causes in due time?

    In the meanwhile, since your heading reads "Half the team at the heart of the RBS disaster WERE in India", perhaps you would also care to explain where the other half was and what were they doing. What do your source(s) have to report about that?

    Thank you.

    1. Anonymous Coward
      Anonymous Coward

      Re: I have two issues with this article

      I think the pertinence of headline is that in the first(?) article on the The Reg about this issue RBS suggested that there was no outside of the UK, offshored/outsourced, involvement related to this issue. Whether it was the fault of outside of the UK staff may not have been determined, yet. Although, it does appear that there was outside of the UK involvement.

  37. Anonymous Coward
    Anonymous Coward

    It's the business model

    I'm sorry I have to post anonymously. We're living that particular dream. Outsourced IT appears to be comprised of a few very sharp people who work as sales engineers for the outsourced sales team and as high level admins, tackling the problems that the first string can't handle.

    The problem appears to come down to the business model used by companies offering outsourced services. It goes something like this:

    In order to provide savings (and reap a reasonable profit margin), the rank and file, the first level, the people who most often touch your issues, will not be highly trained or experienced. There is no money in the business model to provide for training, and expertise in the first level is also a casualty of the business model (more on that later).

    In order to keep margins thin, the first level relies heavily on written procedures, trying to match the error, issue, or task to a document describing what to type or what buttons to press. The operator has no understanding of what the procedure actually does.

    When, in time, the operator gains experience and/or expertise, that makes them valuable in a different position, so they either move to a new job or quit and get rehired at a different company for an incremental increase in pay. Hey, who wouldn't? We've had admins brag that they've finally gotten enough experience to get off night shift (our day shift) and then we never hear from them again.

    In turn, the vacating operator is replaced by another former taxi driver, again to keep margins low.

    And so, the problem never gets better. You can do all the training you want, and all you achieve is people moving on to better jobs at a faster rate. The solution is for the outsource company to pay first level employees enough to keep qualified employees, but that breaks the business model. You fired all of your competent employees in the first place to save money. So now, you're saving money. Aren't you glad?

    1. GreyWolf
      Thumb Up

      Re: It's the business model

      Bang on, especially about the staff turnover. This is a hidden cost for the customer - I've been in the situation of being asked to update printed documentation, only to find when I asked for the original Word doc that it had been written by an offshore person, who had subsequently left the outsourcer, So the customer had to pay me to repeat the creation of the doc from scratch, as if none of the work they had paid for had ever been done.

      1. Anonymous Coward
        Anonymous Coward

        Re: It's the business model

        Confronted with absolutely horrible first level response from our outsourced admins, a group of architects at my company traveled to India and created a "university" there, which was a training program specifically to provide incoming first level admins with enough expertise to provide what we (the customer) considered adequate support. The cost was justified to our own management as "teaching first responders our company culture and proper ways to communicate". -- where it was actually a series of bare bones technical administration courses tailored to the area of responsibility.

        The program was very successful, but not in the manner intended. The locals flocked to those entry level positions, completed their training, and... quit to find better paying jobs. Given the opportunity to get paid while being trained for your next job, who wouldn't jump at that?

        And so, on top of the outsourcing cost, we have the added cost of maintaining our "university" there, and we still get "support" which consists of responding "yes" to everything we say.

        In the most recent employee satisfaction survey, outsourcing was the absolute top issue by a wide margin. Upper management passed this off as a perception issue. I guess in some Alice mirror world it could be. If you lower your expectations enough, anything you get could be called sufficient.

  38. Anonymous Coward
    Anonymous Coward

    It seems...

    ...None of you worked in a Bank.

    As a consultant I have been having this pleasure only to came to the conclusion that banks should go back to Hollerith cards, abacus and pencil.

  39. Anonymous Coward
    Anonymous Coward

    Management vs techie

    I think the following descriptions are quite apt in this context...

    Groupthink is a psychological phenomenon that occurs within groups of people. It is the mode of thinking that happens when the desire for harmony in a decision-making group overrides a realistic appraisal of alternatives. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative ideas or viewpoints. Antecedent factors such as group cohesiveness, structural faults, and situational context play into the likelihood of whether or not groupthink will impact the decision-making process.

    The primary socially negative cost of groupthink is the loss of individual creativity, uniqueness, and independent thinking. As a social science model, groupthink has an extensive reach and influences literature in the fields of communication studies, political science, social psychology, management, organizational theory, and information technology.

    http://en.wikipedia.org/wiki/Groupthink

    Tacit knowledge (as opposed to formal or explicit knowledge) was first introduced into philosophy by Michael Polanyi in 1958. Tacit knowledge is defined as work related practical knowledge. It is that which is neither expressed nor declared openly but rather implied or simply understood and is often associated with intuition [1]. This kind of knowledge is difficult to transfer to another person by means of writing it down or verbalising it. For example, stating to someone that London is in the United Kingdom is a piece of explicit knowledge that can be written down, transmitted, and understood by a recipient. However, the ability to speak a language, use algebra,[2] or design and use complex equipment requires all sorts of knowledge that is not always known explicitly, even by expert practitioners, and which is difficult to explicitly transfer to users.

    While tacit knowledge appears to be simple, it has far reaching consequences and is not widely understood.

    http://en.wikipedia.org/wiki/Tacit_knowledge

  40. Anonymous Coward
    Anonymous Coward

    Offshore

    I work for a traitor that offshores in India and in China.

    I can only talk for India and the telco, i worked before. There are a lot of nice guys there.

    But you can only say one thing concerning them. They are total assholes in this field. Whatever company you are talking accenture or others whatever.

    Replacing one local guy with 3 indians which are cleaning their noses while being explained how works a solution during several video conferences was a total loose of time. Understood nothing about NT/Unix/DOS whatever.

    Doing some PL/SQL dev and invoicing 5 days in Europe instead of 20 days in India was a fucking funny experience about cheap work.

    Telling to some high cast in india that he is a politics/asshole was fun too.

    But more fun is the client who is totally clueless and only sees the bill at the end of the month. "We Want 60% offshore....". Virtual factory. If only shit could be offshored.

    So what now? Nuclear bombs on india? China? Could be nice.... But the culprits are not there they are here.

  41. Wakjob

    Mr. Wak

    Companies ruined or almost ruined by imported Indian labor

    Adaptec - Indian CEO Subramanian Sundaresh fired.

    AIG (signed outsourcing deal in 2007 in Europe with Accenture Indian frauds, collapsed in 2009)

    AirBus (Qantas plane plunged 650 feet injuring passengers when its computer system written by India disengaged the auto-pilot).

    Apple - R&D CLOSED in India in 2006.

    Apple - Indian national and former Goldman Sachs board member Rajat Gupta charged with leaking Intel and Apple secrets over the phone.

    Australia's National Australia Bank (Outsourced jobs to India in 2007, nationwide ATM and account failure in late 2010).

    Bell Labs (Arun Netravalli took over, closed, turned into a shopping mall)

    Boeing Dreamliner ES software (written by HCL, banned by FAA)

    Bristol-Myers-Squibb (Trade Secrets and documents stolen in U.S. by Indian national guest worker)

    Caymas - Startup run by Indian CEO, French director of dev, Chinese tech lead. Closed after 5 years of sucking VC out of America.

    Caterpillar misses earnings a mere 4 months after outsourcing to India, Inc.

    Circuit City - Outsourced all IT to Indian-run IBM and went bankrupt shortly thereafter.

    Cisco - destroyed by Indian labor, laid off 55,000 in 2012, going down the drain.

    ComAir crew system run by 100% Indian IT workers caused the 12/25/05 U.S. airport shutdown when they used a short int instead of a long int

    Computer Associates - Former CEO Sanjay Kumar, an Indian national, sentenced to 12 years in federal prison for accounting fraud.

    Deloitte - 2010 - this Indian-packed consulting company is being sued under RICO fraud charges by Marin Country, California for a failed solution.

    Dell - call center (closed in India)

    Delta call centers (closed in India)

    Duke University - Massive scientific fraud by Indian national Dr. Anil Potti discovered in 2012.

    Enron, WorldCom, Qwest, and Tyco all hired large numbers of foreign workers from India before their scandals.

    Fannie Mae - Hired large numbers of Indians, had to be bailed out. Indian logic bomb creator found guilty and sent to prison.

    Goldman Sachs - Kunil Shah, VP & Managing Director - GS had to be bailed out by US taxpayers for $550 BILLION.

    GM - Was booming in 2006, signed $300 million outsourcing deal with Wipro that same year, went bankrupt 3 years later

    HP - Got out of the PC hardware business in 2011 and can't compete with Apple's tablets. HP was taken over by Indians and Chinese in 2001. So much for 'Asian' talent!

    HSBC ATMs (software taken over by Indians, failed in 2006)

    IBM bill collecting system for Austin, TX failed in 2012 written by Indians at IBM

    Intel Whitefield processor project (cancelled, Indian staff canned)

    Intel - Trade secret stolen by Indian national Biswamohan Pani in 2012.

    JetStar Airways computer failure brings down Christchurch airport on 9/17/11. JetStar is owned by Quantas - which is know to have outsourced to India, Inc.

    JP Morgan - Outsourced subsidiary & IT integration to India in 2009 for $400 million, lost $2 billion in 2012.

    Kodak: Outsourced to India in 2006, filed for bankruptcy in Jan, 2012.

    Lehman (Jasjit Bhattal ruined the company. Spectramind software bought by Wipro, ruined, trashed by Indian programmers)

    Medicare - Defrauded by Indian national doctor Arun Sharma & wife in the U.S.

    Microsoft - Employs over 35,000 H-1Bs. Stock used to be $100. Today it's lucky to be over $25. Not to mention that Vista thing.

    MIPS - Taken over by Indian national Sandeep Vij in 2010, being sold off in 2012.

    MIT Media Lab Asia (canceled)

    MyNines - A startup founded and run by Indian national Apar Kothari went belly up after throwing millions of America's VC $ down the drain.

    Nomura Securities - (In 2011 "struggling to compete on the world stage"). No wonder because Jasjit Bhattal formerly of failed Lehman ran it. See Lehman above.

    PeopleSoft (Taken over by Indians in 2000, collapsed).

    PepsiCo - Slides from #1 to #3 during Indian CEO Indra Nooyi' watch.

    Polycom - Former senior executive Sunil Bhalla charged with insider trading.

    Qantas - See AirBus above

    Quark (Alukah Kamar CEO, fired, lost 60% of its customers to Adobe because Indian-written QuarkExpress 6 was a failure)

    Reebok - Massive fraud and theft in India second in size only to Satyam fraud

    Rolls Royce (Sent aircraft engine work to India in 2006, engines delayed for Boeing 787, and failed on at least 2 Quantas planes in 2010, cost Rolls $500m).

    SAP - Same as Deloitte above in 2010.

    Singapore airlines (IT functions taken over in 2009 by TCS, website trashed in August, 2011)

    Skype (Madhu Yarlagadda fired)

    State of Indiana $867 million FAILED IBM project, IBM being sued

    State of Texas failed IBM project.

    Sun Micro (Taken over by Indian and Chinese workers in 2001, collapsed, had to be sold off to Oracle).

    UK's NHS outsourced numerous jobs including health records to India in mid-2000 resulting in $26 billion over budget.

    Union Bank of California - Cancelled Finacle project run by India's InfoSys in 2011.

    United - call center (closed in India)

    US Navy F-18 jet crashes into Virginia apartment building on 4/6/12 after outsourcing F-18 work to India's Tata.

    Victorian Order of Nurses, Canada (Payroll system screwed up by SAP/IBM in mid-2011)

    Virgin Atlantic (software written in India caused cloud IT failure)

    World Bank (Indian fraudsters BANNED for 3 years because they stole data).

    I could post the whole list here but I don't want to crash any servers.

    1. ThomH

      Re: Mr. Wak

      The issues related to outsourcing, and highlighted in this article, tend to be about communication and oversight — stemming from the physical distance between one set of staff and the other, and the way that decisions about how to distribute work are often money-oriented and made by people that can largely ignore the consequences.

      Conversely, you seem just to dislike Indian people.

      It's a nation with a population of 1.3bn so you'd expect a few bad eggs, and if everything else were equal you'd expect failed companies to be significantly more likely to have CEOs and other employees from India than from almost any other individual country. Similarly individual wrongdoers should be much more likely to be Indian, just like people who scratched their left ear yesterday are much more likely to be Indian.

      What doesn't follow is that Indians are much more likely to be individual wrongdoers. To argue that is to fall for a trivial logical fallacy — a statement does not imply its inverse, e.g. "thieves are likely to earn less than £1m a year" does not mean that "those that earn less than £1m a year are likely to be thieves".

      1. This post has been deleted by its author

      2. ajcee

        Re: Mr. Wak

        No: the biggest issues related to outsourcing are dealing with the lack of real-world sysadmin or dev experience. Communication/oversight are indeed issues but it's the lack of experience that's the killer.

        I used to work for a UK-based ISP and racked up six year's experience dealing with offshore teams in India. Every single one of them churned staff at dizzying rates of speed and after four years of banging our heads against a wall and getting absolutely nowhere, my peers and I petitioned our CTO to go to bat for us to the Board and bring the India-based techs over to the UK to work with us in the office and under close supervision. It worked a charm: just a few months after arriving the majority of them hit 5th gear and became damn good techies. Their experience started to rack up and - by the time I left two years later - they were pretty solid. Of course, it still sucks that the jobs were being sourced outside the UK, depriving any locals of employment, but - as the Indian techs were paid approximately one-third of what I'd have considered a decent UK rate - we could never get the bean counters to acquiese. We simply had to take what was on offer. The Board still saw their all-important cost savings and, after some intensive training and supervision, we got a team of very capable techies.

        None of this is (or should be) a rant against Indians. If UK companies had all targeted China, South America or Eastern Europe 15 years ago for their outsourcing requirements, we'd be bitching about those people. Bottom line: it makes no difference what nationality, race, class, gender, creed, sexual orientation, etc. you're dealing with. If you're looking to cut costs and you have senior managers trying to ensure that they pay the lowest wages possible, you're going to end up entrusting your operations to people who have no business sitting behind a computer. Whether that person is sitting in Pune, Prague, Palm Springs, Panama City, Perth or Pyongyang is irrelevant.

  42. Jonas Taylor

    So, which bank should I move to?

    I used to bank with The Woolwich until Barclays bought them out and then scrapped my debit card - when I went to reapply I was told my credit rating wasn't good enough (as I refused an overdraft facility). That meant while I was previously able to make online purchases with my Visa Electron card, the Barclays replacement Savings card was literally useless to me. Needless to say I will never go with Barclays ever again.

    I then applied to Halifax, again turning down an overdraft facility - I was rejected. Finally I applied for a Natwest account and ticked the box for a £1400 overdraft facility and suddenly I was accepted with open arms. However, over the years I've been unimpressed. It started with an iPhone banking app being heavily hyped, yet with no Android equivalent in sight. It took years for them to finally support Android, despite it having a much larger user base - it was more important for them to be trendy that address the needs of their customers. Then recently I spoke to a financial adviser at Natwest and asked about setting up an ISA, only to be told that as an existing customer I couldn't get a decent interest rate - they then advised me to go to another bank, set up an ISA and then come back to them to transfer it, at which point I could get the highest rate. Treating your customers like dirt is a bad way to do business.

    The latest fiasco is just too much, as it was entirely self-inflicted. If a company keeps cutting its ICT staff - especially with the severity talked about here - it will almost inevitably end in disaster at some point down the road. I appreciate that banking systems are incredibly complicated but aggressive cost savings clearly took things too far. We have plenty of qualified ICT workers in this country, yet RBS would rather sack them and save a little bit of money by shipping everything off to India. It's super-capitalism gone mad.

    I'm looking for a bank that has online banking, a mobile app, that wasn't part of the banking collective that destroyed the economy and that treats its customers with a modicum of respect. That's probably asking too much but what are the real options? I'm interested to see what Virgin Money does when their current account is launched next year.

  43. Charles Smith

    Somewhere in RBS there is a list of names of the people who decided to offshore the daily batch control work to the cheapest operations staff. These people didn't ensure these cheap offshore operators were properly trained and had proper controls in place.

    These people should be fired including the senior managers at the top of the chain who insisted on unsafe cost cutting.

    The cost to the bank in terms of reputation loss and recovery will be enormous, far exceeding any savings made by going cheap. Has RBS properly evaluated the real cost impact of laying waste to its technology human capital in the UK?

    Has its managers changed the support teams from people who care about their work to those who just regard it as a job to earn low wages while working awful hours? That might be okay at the counter of a fast food store but not in a key business team.

  44. Lusty

    staff cuts

    Given that it was one person who caused the error I have no idea why everyone is blaming staff cuts and outsourcing. The people in the onshore and offshore departments were all fully trained, and had there been more staff around that guy would STILL have pressed the wrong button.

    IT departments are some of the most overstaffed I'm aware of, and the sheer number of comments on Reg articles posted during the day is proof of that. Most people I know in IT average 1-2 hours of actual work a day. The real problem is that there are too many staff in IT which is keeping wages for the 10% of actual competant IT workers down. And I know I'm not alone in thinking that if the 90% were laid off, I would have less work to do rather than more!

    1. Anonymous Coward
      Anonymous Coward

      Re: staff cuts

      "IT departments are some of the most overstaffed I'm aware of, and the sheer number of comments on Reg articles posted during the day is proof of that"

      Funny you should say that. I have been having those exact same thoughts recently.

    2. GreyWolf
      Holmes

      Re: staff cuts

      >The people in the onshore and offshore departments were all fully trained<

      Where on earth do you get that from? Rather obviously that's not true at RBS, just from the fact that they got the recovery wrong several times over.

      1. Anonymous Coward
        Anonymous Coward

        Re: staff cuts

        "The people in the onshore and offshore departments were all fully trained"

        Might have been fully trained, but doesn't mean they fully learned. Doesn't mean that those fully trained individuals are still working on the same project or even for the same company.

        My experience of off-shored sys-management is that they have a _very high_ staff turnover (measured in months not years). As they acquire skills (on paper) and additional (limited) experience, they seek better paid jobs - nothing wrong with that as they're paid peanuts (with no chance of a pay increase since that would increase the off-shoring costs and that can never be allowed to happen). So, handovers fail to happen and the half-learned skills are repeatedly diluted until the systems are running on autopilot.

        But that's ok - the bean counters who initiated the off-shoring, have reduced costs, they got their bonus. It's not their fault that the systems are running at risk (not that anyone knows that anyway) - that is someone else's problem.

        Any company that uses off-shoring to maintain profitability has run out of ideas. They have given up on innovation, and are relying on the bean-counters to keep the wolves at bay while they wring every last penny out until they leave / retire. They are advertising themselves as failing and that is why off-shoring is so often kept a secret.

    3. The Axe

      Re: staff cuts

      It might only be one person, but that person was working for RBS on the basis of selecting the cheapest and not necessarily the most competent. The staff cuts led to a lot of experienced staff losing their jobs. With them went a huge amount of business knowledge. Knlowedge that they had picked up and fine tuned over many years. Replacing them with someone who might know CA7 but who is not experienced in all it's nuances is the fault of the RBS management who dictated that everything should be outsourced.

      Bean counter mentality. Look at the figures, but not at the overall picture. Outsourcing only works when you outsource non-key parts of your business or stuff that can use economics of scale. A bank's key part is it's computer systems and software. Banks MUST NOT outsource that. They can outsource the printing of statements, and alll the other letter stuff for instance as that is not a key part.

  45. Anonymous Coward
    Anonymous Coward

    RBS management culture summed up by Millhouse off The Simpsons:

    "This is great! Not only am I not learning anything new, I'm forgetting stuff I used to know!"

  46. Anonymous Coward 15
    Holmes

    Lacking in excrement, Mr Holmes.

  47. daveeff

    Re: One, two, three, jump! But where?

    Co-op for me dudes!

  48. daveeff
    Megaphone

    pay peanuts...

    The fact that FTSE 100 companies directors salaries have risen 12% in the last 12 months because "we need to pay the market rate to get good people" would imply these people know that that paying the least possible tends not to get the best work.

    Doesn't apply to people who actually do something apparently.

    I've worked with a few (developers) from China - all young, keen, pleasant but very little experience and some with poor english / communications. Sounds like the RBS / India experience was similar.

    Being UK based doesn't make you a better worker (and certainly not a better human being!!!) but 20 or 30 years experience and working in your native language makes a difference.

  49. Ed_UK

    Waiting for the spam..http://www.theregister.co.uk/Design/graphics/icons/comment/meh_32.png

    Dear <username>,

    I was a loyal RBS IT employee for 27 years, until they decided to offshore my job to India, making me redundant. <Link to genuine El Reg coverage>

    Now, I face personal bankruptcy and my children will have to leave college. My house will be re-possessed and my wife is leaving me for an RBS manager.

    Fortunately, before I left the office here in Lagos, I was able to use my IT skills to transfer over $132million to a secret account. This account does not even appear on official audits because I have made it 'invisible.'

    Are you a trustworthy person who can help me get some revenge on the stupid managers at RBS? Please send me your personal details and I will give you 50%.

    Yada yada.

    p.s. Sorry - forgot the typos and Caps-lock.

This topic is closed for new posts.