back to article RBS collapse details revealed: Arrow points to defective part

The tech problems at the RBS banking group that left millions of people unable to access money for four days last week were caused by a failure in a piece of batch scheduling software, sources have told The Register. And at least some of the support staff for that software have been outsourced to India - as recently as …

COMMENTS

This topic is closed for new posts.
  1. Anonymous Coward
    FAIL

    Thou shall

    reap what you sow.

    1. Thomas 4
      Meh

      If only.....

      I just saw this on another news site:

      "The boss of Royal Bank of Scotland and NatWest has told Sky News that senior management will face "proper accountability" over the computer chaos when deciding on bonuses."

      When popped into Google Translate (Bullshit -> English):

      "We're only going to award ourselves a 1 million pound bonus instead of the 2 million we originally planned. The poor shmucks we've been running ragged with overtime this past week will receive fuck all and we'd really like our ex-IT staff to stop laughing at us now please."

      1. sugerbear

        Re: If only.....

        [quote]"I have no evidence of that. The area, if you like, the UK backbone, has received substantial investment."[/quote]

        what UK backbone ? It was all outsourced to Chennai. Unless the investment is the money they used to pay the IT staff off (or maybe the money they pay monetise to manage their mobile applications).

        Here is a CEO that doesn't have a fucking clue or else his reports are lying to him. If I were him (thankfully I am not) I would be looking for a new CTO because the ones he has aren't going to finger themselves in any blame.

        Utterly clueless. And very likely to be found out when reporters start digging.

        1. Anonymous Coward
          Anonymous Coward

          Re: "Here is a CEO that doesn't have a fucking clue or else his reports are lying to him"

          Congratulations on having independently reinvented the Snafu Principle!

        2. Anonymous Coward
          Anonymous Coward

          Re: If only.....

          Maybe he's referring to the network work Accenture did for RBS, oh no, hold on, they made a 'James Hunt' of that as well...

        3. Inventor of the Marmite Laser Silver badge

          Re: If only.....

          Bank? Backbone?

          Oxymoron.

    2. Anonymous Coward
      Anonymous Coward

      How much

      Would the equivalent job be in the UK?

      1. Martin 37
        Unhappy

        Re: How much

        There are no equivalent jobs in the UK. That's the problem.

      2. Anonymous Coward
        Anonymous Coward

        Re: How much

        The calculation we used was one good UK IT guy would need to be replaced by 2 in India plus maybe 20% input from a good UK IT guy to ensure quality. Common problem with Indian culture seems to be a kind of "can't lose face" issue, very reluctant to seek help when necessary.

        Future problem for UK is how do we grow our own "good IT guys" without having them work lower down the team to build experience, skill, knowledge, insight.

    3. Anonymous Coward
      Anonymous Coward

      Re: Thou shall

      Indeed. I wonder how much money outsourcing has actually saved them? If it weren't for the taxpayer backstop it'd be f*cking hilarious.

      It may serve one purpose however - to highlight the risk to the financial marketplace of outsourcing your IT staff and perhaps any planning to will hold off in fear of being seen as more risky by the wholesale funding markets.

  2. Captain Scarlet Silver badge
    Meh

    And the finger pointing continues

    I doubt they will ever publicly say what the issue was, as they would probably be sued.

    1. Anonymous Coward
      Anonymous Coward

      Re: And the finger pointing continues

      Possibly sued by Infosys perhaps ?

      anon for legal reasons :-)

  3. Anonymous Coward
    Anonymous Coward

    I knew there would be problems if you got Bob Shawadiwadi in, plenty cheapness!

    http://shawadiwadi.blogspot.co.uk/

  4. Just_this_guy

    This article reads like unwarranted finger-pointing. 'Outsourced' and 'India' are not synonyms for 'incompetent', and if RBS hired someone with "4-7 years experience" of the relevant software then hopefully that person will know what s/he's doing! I just hope that insulting salary goes further over there than it does here.

    1. Anonymous Coward
      Anonymous Coward

      Sort of...

      No, employing staff in India isn't a synonym for incompetent, but getting rid of staff who have worked on the same systems for 20-30 years and replacing them with someone who has 4-7 years experience with the software you need doesn't compare.

      Also remember: When you offer voluntary redundancy, the people who go first are the ones who are most confident of getting a new job: ie: Your most highly skilled staff.

      1. Anonymous Coward
        Anonymous Coward

        Re: Sort of...

        "No, employing staff in India isn't a synonym for incompetent, but getting rid of"

        Yes, that's of course a fair observation, but not one particularly emphasised by the article writer, which I believe is our fellow commenter's point above.

        The article repeatedly mentions the outsourcing to India, but fails to comment on the relevance of it, if any, as regards the mishap.

      2. Anonymous Coward
        Anonymous Coward

        Re: Sort of...

        "...Also remember: When you offer voluntary redundancy, the people who go first are the ones who are most confident of getting a new job: ie: Your most highly skilled staff...."

        Spot on. Voluntary redundancies in this sort of case are generally a really good filter for keeping hold of the dregs and losing the cream. Doesn't stop some management doing it again and again...

        As they sowed...

      3. Peter Simpson 1

        Re: Sort of...

        Getting rid of staff who have a vested interest in getting it right, and replacing them with outsourced staff in a foreign country, who have no proprietary interest in the organization to which they are contracted *might* be part of the problem here.

      4. Anonymous Coward
        Anonymous Coward

        Re: Sort of...

        Or 4-7 years experience in SOMETHING (different from what you need), which is what you normally get.

    2. Anonymous Coward
      Anonymous Coward

      market forces

      Lets be blunt: the best brains aren't in India. The reason? because the really excellent Indian IT people have been hired and work in the US and are being paid lots (and lots) of money. The next best are in the UK, the next best are in the Middle East or Singapore. Add onto this that IT people are paid huge salaries by Indian standards and there is an massive IT boom going on there and you get a situation where there are lots of not-very-good people in the market.

      Why work in India for 10 laks (about a million rupees, or about 9K) when you can work in the US for 100K USD or in the UK for a comparable amount?

      I spent weeks interviewing devs in India. I started off with 1250 CVs, did about 400 phone interviews, and 200 face-to-face interviews. I hired 15 people. And some of them were simply "warm bodies".

    3. Anonymous Coward
      Anonymous Coward

      'Outsourced' and 'India' are not synonyms for 'incompetent',

      True, but as an IT professional with nearly 30 years experience who was replaced by an Indian graduate they are synonyms for inexperienced, naive and clueless, and the last one does not only apply to India.

    4. Keith 17

      @Just_this_guy

      and if RBS hired someone with "4-7 years experience"

      That's a big "if" there. That's what they were looking for, but it doesn't mean that's what they got.

      1. Anonymous Coward
        Anonymous Coward

        Re: @Just_this_guy

        More likely it will be 0-3 years in reality or in fact 0 years which is what some Big US/EU Outsoucers are proudly advertising openly for new "developers" in certain locations.

    5. Anonymous Coward
      Anonymous Coward

      @Just This Guy

      >'Outsourced' and 'India' are not synonyms for 'incompetent'

      No they aren't but the reason for outsourcing is to cut costs so you have to add in "cheap" and when you do so you get incompetent. It's not the bods fault he's incompetent, he's probably been employed purely because he can speak reasonable Enlgish. Problems with outsourcing lay squarely at the feet of the management who instigated it and then to a lesser degree with the outsourcing supplier who recruits based on cost not ability.

      1. Anonymous Coward
        Anonymous Coward

        Re: @Just This Guy

        Excuse me for being thick, but 11K is not cheap. In fact, if it costs 11k there, there is _NO_ real saving.

        Let's factor in all factors in the equation.

        1. The overlap between UK and India business hours is ~ 4h. This means that you need both a development head and a support head for every UK person you replace. In general based on experience from previous projects the "coefficient of inefficiency" here is roughly ~ 2x. Still better than India-US which tends to go into 3x because of the zero overlap between timezones.

        2. You have to pay for the infrastructure there, infrastructure here and telecoms infrastructure in-between. Ballpark for that moves the number to 3x+. It is worse than India-US as you pretty much need to buy capacity on Flag/Reliance's network which is booked to the hilt and costs an arm, a leg and a prosthetic.

        3. You still have to maintain a "head" in the UK either by rotating people from India or keeping PM, Arch, requirement gathering staff locally. This increases the "coefficient of inefficiency" to ~4x +

        4. You now have to employ extra compliance resources to keep DP and various financial regs happy. That 4x just became 5x at the very least.

        5... 6... 7....

        So all in all we are talking what? The real price is at least 4-6x times the posted salary of that guy. So in reality, replacing one person in UK on a 60K salary with the one imaginatory person in India on 11K salary (in reality you are looking at 2+ and some resource on-shore to feed them) actually ends up costing 60K so no savings. At least. More like loss.

        At that price you might as well go to Eastern Europe and save on compliance costs and TZ ineffiency or go even further east where labor is still cheaper and you have cheaper comms by going across USA+Pacific (at the expense of some extra compliance costs).

        1. Anonymous Coward
          Thumb Down

          Re: @Just This Guy

          If you are hiring one person, that true, however, if your hiring, say, 800, Economies of scale kick in.

          Also,

          1), they are usually prepared to work UK hours for the money they are getting.

          2), Lots of companies seem to manage fine.

          3) Again, economies of scale. 1 head in the UK for every, say, 10 in India?

          4) See (3)

          If the numbers didnt work, people wouldnt be doing it.

          1. Nuke
            FAIL

            Re: @spodula @Just This Guy

            Wrote :- [re economics of outsourcing ] "If the numbers didnt work, people wouldnt be doing it."

            Well, they haven't worked, have they? People cannot get to their money. I won't ever be banking with RBS. Is that (and many like me) in your equation?

          2. Hans 1
            Mushroom

            Re: @Just This Guy

            "If the numbers didnt work, people wouldnt be doing it." is the worst argument I have read so far - great work, mate! It looks great on paper, but it does not work, we see it all the time. Remember: in theory, theory and practice are the same, in practice, they ain't.

            Theory is, we can replace our 10y+ experienced staff by some Indian graduates, who cannot mumble proper English if you trained 'em to (no offence, guyz, the same goes for Scott's etc). You put knowledge transfer in place and hope that knowledge/1000 = knowledge + experience. Then, when you have a whole team being replaced, no amount of "knowledge transfer" is gonna work, especially when you have the guyz who get the boot delivering the knowledge transfer. Hello??, think these guyz are stupid, or what? Who really, honestly thinks a bunch of experts who have been asked to leave with a special bonus will really help those that will replace them?

            Then you have the "management cretins" practice. In some companies, cretins are promoted to management positions to get them out of the way and stop them causing harm - we see this a lot in France, for example, but not only. This has a very bad effect on outsourcing.

            Outsourcing will always fail, the more complex the environment, the more likely the failure AND the more complex the environment, the more costly the failure.

            Note, I do not work in the UK, am not and have not directly been confronted with outsourcing ... I have been asked to clean up the mess after a failed attempt, that is all ....

        2. Anonymous Coward
          Anonymous Coward

          @AC 13:11

          Actually while I was at RBS it was widely known that each seat in India (Gurgoan, not Chennai) was costing RBS under £15,000 per year, so considerably less than the UK roles they replaced.

          Oh and just because some one over here is on, say, £50k per year, does not mean they cost the employer £50k. by the time you add on their side of tax, NI, insurance, liability, medical, pension (which incidentally RBS was still ratehr good for even after ending the non-contrib final salalrey scheme), training, cost of holidays and sickness, you can add 50%+ on to the sallery. Oh and another 50% ro their first several years if they are a new starter (inductions costs, payments to agencies, cost of training up, ...)

          AC because while not bitching about RBS and the fact that didn't just off-shore to India, they off-shored to cheap by local standards Indians, I am ex RBS staff.

          1. Anonymous Coward
            Anonymous Coward

            Re: @AC 13:11

            And by outsourcing abroad we loose the benefits of the tax and insurance that would have been collected to pay for our services here.

            We lose jobs

            We lose income of tax and insurance

            Lose, lose.

      2. bristolmoose
        Facepalm

        Re: @Just This Guy

        "It's not the bods fault he's incompetent, he's probably been employed purely because he can speak reasonable Enlgish."

        Oops!

        1. Anonymous Coward
          Anonymous Coward

          Re: @Just This Guy @bristolmoose

          So I won't get a job in India and where I do work I don't speak English.

      3. Gregsy

        Re: @Just This Guy

        >'Outsourced' and 'India' are not synonyms for 'incompetent'

        This is true, but they have got rid of the 20-30 year 'lifers' who have invested blood sweat and tears into an organisations IT; the ones that take ownership of problems and add value in that commitment to their work and the invaluable insight they provide into complex system operations.

        You cannot and never will be able to put a price on that.

        If I wasn't an RBS customer with no access to my account, I would be laughing at them.

    6. Anonymous Coward
      Anonymous Coward

      In my experience

      Indian staff that are based in India tend to be very junior and have a high turnover rate.

      Understanding a particular software package is one thing, but knowing how it is used in a particular company environment is a very different thing. The result is that you can use them for day-to-day running of a system, providing you have an up-to-date set of docs describing in detail the process and what to do, but getting them to make all but the most basic changes is asking for trouble.

      Indian staff working in the UK tend to have much better communication skills and more experience. On the other hand they are much more expensive so the bean counters (who know the price of everything and the value of nothing) don't like to employ them.

    7. Anonymous Coward
      Anonymous Coward

      While I am sure there are plenty of talented people in India, the outsourcing companies are often terrible, I have seen companies ripped off with work so shoddy they have had to hire in local experts to fix the mess left.

      You have to look at local costs before you compare salaries, the UK is very expensive, so comparing like for like is wrong.

      I just know I avoid outsourcing to India now, the few times I have tried it, the results were terrible, long term its often better to hire someone locally, even if the cost is higher!

    8. Gordon Stewart
      FAIL

      You've never tried recruiting anyone from India then?

      At least 90% of the CVs that land on my desk - from organisations like Capgemini and InfoSys - are utter lies.

      You read a CV that appears to be a perfect candidate, with 7 years experience working on .NET projects, but upon interviewing it very quickly becomes apparent that they haven't got a clue.

    9. Anonymous Coward
      Anonymous Coward

      Does anyone honestly thing there are "unemployed" gangs of experienced CA-7 specialists roaming the streets of Chennai or Bengaluru? If you taser them do they start shouting random COBOL code?

  5. Anonymous Coward
    Anonymous Coward

    Of course

    people that were recently laid off in favor of Indian alternatives would buy into this story.

    Not sure it's that impartial.

    1. Anonymous Coward
      Anonymous Coward

      Re: Of course

      Replace "laid off" with "chose to leave when offered the opportunity".

      1. Anonymous Coward
        Anonymous Coward

        Re: Of course @AC 12:18

        chose to leave when offered the opportunity

        The term used by spineless RBS HR was "workstack reduction"

        1. Anonymous Coward
          Anonymous Coward

          Re: Of course @AC 12:18

          They actually called it 'staff augmentation' where I work :)

          Remind me of the Borg for some reason...

          Anon for obvious reasons

          1. Anonymous Coward
            Anonymous Coward

            Re: Of course @AC 12:18

            "Augmentation" reminds me of hilariously coy 1980s Harley Street adverts for surgical bosom-boosting in the Daily Heil.

    2. Anonymous Coward
      Anonymous Coward

      Re: Of course

      You never had the "opportunity" of working with them.

      Look at landrover/jaguar, now indian owned and run and doing very nicely despite the economy. Nothing wrong with offshoring, but having seen the way it was done at RBS I can only say that it was ONLY cost that was factored in.

      QUALITY went out the window.

      1. Anonymous Coward
        Anonymous Coward

        Re: Of course

        Actually, I did... I worked there for seven or so years, I left when I was offered the opportunity of a big wedge of redundancy payment. I worked with Indian staff for the last couple of years, some of whom were pretty good, but most didn't stay long enough to become pretty good or learn the systems.

        1. Anonymous Coward
          Anonymous Coward

          Re: Of course

          @AC 12:31 GMT

          "I worked with Indian staff for the last couple of years, some of whom were pretty good, but most didn't stay long enough to become pretty good or learn the systems."

          That is exactly the problem I found in other developing countries. The bright ones you train up to take over from you soon get better offers elsewhere.

    3. Anonymous Coward
      Anonymous Coward

      Re: Of course

      Entirely correct. And who proved to be have such an entirely unsuitable experience type and length that they completely closed down the freshly purchased Indian company in <= 2 years. Of course that still leaves them with them problem of having closed the first office that had the experienced Euro product developers in it.

      1. Anonymous Coward
        Anonymous Coward

        Re: Of course

        But the time zone was the real killer, for W. Coast USA -> India,

        or UK-> India

        or even better W. Coast USA + UK + India developer team meetings on videoconf.

        Like, forget it.

        Who wants to get up middle of the night every week maybe for 2 days every week to talk to their team?

        Clue- Silicon Valley is approx 12.5 hrs behind India.

  6. Anonymous Coward
    Facepalm

    Well, if you pay peanuts.....

    1. petur
      Meh

      That's not even the problem... the real problem is that when another employer pays 0.001% more, they are gone by the end of the week.

  7. Ol'Peculier
    FAIL

    UK based

    "UK-based piece of software" does not mean it's overseen by UK-based staff.

    Just saying...

    1. teebie

      Re: UK based

      It is a suspiciously specific piece of PR

      1. Anonymous Coward
        Anonymous Coward

        Re: UK based

        Yes it's spin designed to give the impression that outsourcing had noting to to with the problem.

    2. GitMeMyShootinIrons

      Re: UK based

      When a company makes a comment that specifically refers to "UK-based piece of software", I start smelling a rat. Why would the company's PR-drones be so specific, unless they were paranoid about media comments regarding the risks of off-shoring?

      Also, it's a suitably vague PR-drone written comment - the software may well be based in the UK, but it could be written by an Indian, in Somalia, using a Brazilian keyboard and uploaded using a proxy in Thailand, after being QA'd by a kid in Poland. Where it's based is somewhat meaningless in this context.

      1. James Micallef Silver badge
        Facepalm

        Re: UK based

        Also, what the hell does "UK-based piece of software" mean? This is software we're talking about, it isn't "based" anywhere. If he meant "it's running on a physical server in the UK", even that means nothing, because it still could have been coded anywhere in the world, and it presumably still could be remotely executed by RBS staff at a remote location.

        So it's nothing but PR-bullshitting meant for the consumption of people who don't have a clue about software.

    3. HappyInTheValley
      Angel

      Re: UK based

      I saw this posted by http://twitter.com/#!/YamalDodgyData over at The Telegraph, it certainly sounds plausible!

      "RBS PR has specifically referred to a "UK-based piece of software" as the fault.

      Why would the PR-drones at RBS be so specific, unless they were paranoid about media comments regarding the risks of cheap third world off-shoring ?

      Perhaps a few questions you should ask RBS are:

      Was it an Infosys employee on a Tier 2 visa at the mainframe site in Edinburgh who bungled the customer accounts batch job, or

      Was it an Infosys employee working remotely from Chennai that bungled the batch job in Edinburgh ?

      Does Mr Hester regret making the last British mainframe specialist at RBS redundant three weeks ago ?

      and does he think the savings from sacking 1,800 British IT professionals and third world off shoring will pay off ?"

      1. Anonymous Coward
        Anonymous Coward

        @HappyInTheValley

        It's RBS Gurgoan, so a mix of cheap permies, contractors and some 3rd parties. RBS gave Infosys the boot ~3 years ago for being sh*t.

        Also Hester just told the head of IT to reduce costs, it is they who decided to go off-shoring despite two previous failed attempts.

        I knew many of the mainframe batch guys and they were a good bunch of long termers who were all showing a lack of morale when I left, but there again senior IT management had removed much of the desire to do a decent job and company loyalty with their announcements and actions.

        Oh and as to sources areas to lay blame on for this fiasco. What's the bets RBS Incident management did their usual block on any repair/back out actions for amny hours while they searched their own navels and those of senior management (oh the joys of working support there).

    4. Callum
      FAIL

      Re: UK based

      I saw Stephen Hester say it was "UK based" on the news last night - my immediate reaction was that they are already trying to dodge the resulting FSA fine. Especially since the largest fine ever levied was to Zurich regarding an out of control outsourcer to their outsourcer.

  8. Anonymous Coward
    Anonymous Coward

    Not convinced

    Think there must be more to the story than this.

    Whilst I've been personally responsible for scheduling issues that clobbered the Month End of a bank - I'm struggling to understand how that on its own could have caused a weeks worth of problems.

    Think there is more to come on this.

    Anon obviously.

    1. sugerbear

      Re: Not convinced

      It is easy. If on the first night of your batch run you miss a few input file and it isn't picked up until say the Tuesday afternoon (and let us say for example your support staff who specialise in CA7 had left for the day), you would be into Wednesday before you could be anywhere near identifying what the problem is and what caused it and what was missed the previous night. Then you have to fix the problem and work out how you are going to get the missing transaction into your batch file (they would then most likely reject because the batch header didn't have the correct day on it .. and so on..)

      Of course all the time its getting escalated and people want answers, so that delays the fix because they dont want the fix screwing up something else.

    2. Anonymous Coward
      Anonymous Coward

      Re: Not convinced

      One possible reason is the lack of knowledge of the RBS configuration as opposed to the product.

      When the first batch failure occured, the "old" staff who knew RBS would probably have found the error quickly and could back-out the issue to the known good position.

      The "new" staff however, who probably know the product, but not the RBS configuration, didn't find the issue quickly enough, further batches had completed over the top, and it was no longer possible to return to the known good position. You're then into a fix-forward scenario to apply the missing batches to an updated position which is then out of transactional order.

      Think of it like an old fashioned engine - any old mechanic can probably fix it eventually, but old Bob who's looked after it for 30 years knows just where to hit it with a hammer when it stutters.

      The lesson for the accounts therefore is to consider a persons salary not just as a book cost this year, but also on the basis of the investment you've already made in that resource. You can get a YTS mechanic on minimum wage, but would you really trust them to service your Aston Martin.

    3. Anonymous Coward
      Anonymous Coward

      Re: Not convinced

      "Think there must be more to the story than this."

      I think so as well, as sugarbear points out it should be a simple enough task to add the missing transactions to the next days run, I'm quite positive that there must also be some sort maintainance application what can be used to apply ad-hoc transaction to accounts and that the experienced maintainence programmers would know what to do to recalculate interest, refund fees etc.

      wait a minute, they got rid of the experienced people didn't they

  9. Anonymous Coward
    Anonymous Coward

    £10k costcutting job

    costs RBS, sorry - the UK public how much money?

  10. Nash_Equilibrium
    IT Angle

    I have not used CA-7 or anything like it, god I would hate to be left alone with a mainframe.

    But after a bot of googling, I assume you don't want people like this either

    http://ibmmainframeforum.com/viewforum.php?f=38

    "cann any one please explain the commands like PF(page forward),PB (page backward) and SR(save Replace)

    Also TSO CA7cLIST"

    1. sofaofthedamned
      FAIL

      Blimey...

      ... I found the post you're on about - go Google for the guys name on linkedin, He's apparently been a 'mainframe programmer' for 15 months.

      Then go to his companies home page, and they're providing mainframe guys for HSBC!

      Frankly, i'm going to keep my cash in my house from now on :)

      1. irish donkey
        Happy

        Re: Blimey...

        Outstanding analysis.

        LMFAO 15 mths dribble

  11. Anonymous Coward
    Anonymous Coward

    Freedom of Information

    Since "we" own RBS, what about a Freedom of Information request to find out what really happened :-)

    1. nichomach
      Big Brother

      Re: Freedom of Information

      Upvoted the idea, but I can see the phrase "commercially sensitive" popping up like a flipping meerkat on speed in the rejection letter...

      1. Stuart Halliday
        WTF?

        Re: Freedom of Information

        Fair enough.

        So ask your local MP why a UK Government owned Bank is using non-UK staff?

        1. Anonymous Coward
          Anonymous Coward

          Re: Freedom of Information

          >So ask your local MP why a UK Government owned Bank is using non-UK staff?

          My MP asked a question in Parliament about fines over dog shit once, but that's her sole contribution to governance so far. May as well ask the cat.

          I've seen loads of horrible tales of woe...but has anyone else noticed that international payments which normally sits in the system for a couple of days - notably Google Adsense - have arrived on the day of payment this month? I've never understood why this doesn't happen all the time - it does the other way.

  12. Anonymous Coward
    Anonymous Coward

    UK-based, sort of....

    Ha Ha The country continues to reap the results of citizen goodwin's ego purchase of abn-amro.

    Yes CA-7 sits on the cluster of mainframes in Gogarburn, the idiots fucking it up are in India.

    No, I take that back, the idiots who fucked it up were Gogarburn based as well, fred 'the pleb' goodwin, mckillop et al. 8K a year to manage CA7 Pay peanuts etc. No wonder hester thinks he earned his £1M bonus.

    1. Anonymous Coward
      Anonymous Coward

      Re: UK-based, sort of....

      There are no mainframes at Gogarburn.

      While Goodwin's ego over ABN-AMRO may have been the major cause of the financial crash, it was Hester et al (appointed by the LABOUR government) who aggresively outsourced the IT services to India and Accenture.

      If Goodwin had backed off from ABN-AMRO, Barclays would have one the deal, which was a share swap. Come the crash, Alistair Darling would have approached RBS to take over Barclays, much as they did Lloyds for HBOS.

      1. Anonymous Coward
        Anonymous Coward

        Re: UK-based, sort of....

        AC @ 12:46 - not necessarily. I worked for both during that time.

        RBS was outsourcing to India way, way before the ABN farce. I know as I ran teams there (and here in the UK, Chicago, Singapore, etc). Some of those in India were good, some poor, some very bad. Sometimes the cultural issues made it worse.

        As for the RBS vs. Barclays issue. When RBS "purchased" ABN they really didn't look very closely. Having seen both sides, I think I can say that Barclays would've either backed off or made a better job of the purchase.

    2. Anonymous Coward
      Anonymous Coward

      Re: UK-based, sort of....

      @AC12:31 - Sorry, nothing to do with Goodwin's purchase of ABN, RBS used z and CA-7 for decades before ABN was purchased. Also, Gogaburn is the de-facto group head office (technically RBS St Andrew's Square is head office), they don't run the datacentres from a group head office.

      Now take your name calling of an easy media hate figure and work out something better to do with your day.

      1. Anonymous Coward
        Anonymous Coward

        Re: UK-based, sort of....

        Fred was "asset-sweating" local staff long before this kicked off and long before Gogarburn and the India-outsoucing mania. So, it is his fault, actually, for permitting this kind of behaviour to start under his watch.

        The opposition across the road were much less shitty. You know, the ones without the word "Royal".

        Cheers Fred, was that a fried potato of some sort on your shoulder BTW?

        Here's a clue, buying things then breaking them by trashing inside processes to make this years bottom line look good isn't actually "business".

  13. Anonymous Coward
    Anonymous Coward

    Oh dear

    Hope TATA didn't get the contract.

    From my experience TATA systems admins, sorry got to laugh at that, wouldn't know how to recognize a computer even if they'd read "Peter And Jane Meet A Computer" or "Identifying a Computer for Dummies"

  14. Anonymous Coward
    Anonymous Coward

    Nice job advert

    Using a google mail & the indian version of gumgree for job adverts? Yes I'm sure that is very legitimate.

  15. Anonymous Coward
    Anonymous Coward

    Shame on El Reg

    It looks like El Reg has recruited editors from the Daily Mail for this article and the comments also reflect the type of readership of that publication. Shame.

    1. Anonymous Coward
      Anonymous Coward

      Re: Shame on El Reg

      Well, if by Mail-readership, you mean racist, sexist, climate-change-denying, Euro-sceptic, hanging's-too-good-for'em, illogical, knee-jerk-reacting, warmongering carnivores, then - welcome to El Reg!

      1. Anonymous Coward
        Anonymous Coward

        @FatsBrannigan

        Who the fuck are YOU calling illogical?

    2. Yamal Dodgy Data
      Thumb Down

      Re: Shame on El Reg

      The most catastrophic IT failure in banking history, on a supposedly failsafe system with NO explanation or answers coming from RBS.

      And AC at 12.48GMT decides no one should know why... because YOU think the answer is rawwwwwcist.

      you wanker

    3. Blitterbug
      FAIL

      Re: Shame on El Reg

      You utter twat. So having a pop at the banking establishment makes us Daily Fail readers now? Do you even know what a right-wing rag is? Have a guess who Mail readers are more likely to support; fat work-shy tax dodgers like ex-Sir Fred & Mr Pester, or hard working IT grunts like us?

      You muppet.

  16. Anonymous Coward
    Anonymous Coward

    ouch

    If proven (can be, will be?), the cost of this minor (cough) glitch might offset all those fabulous offshoring / outsourcing savings they have been creaming their pants about over the last... what, 10 years?

    now, how to recover this cost, so that the punters don't notice they foot the bill again....mmmm... how about bringing back the Fred?

    1. Anonymous Coward
      Anonymous Coward

      Re: ouch

      Or maybe they could hire a "manager". You know, there are plenty of nice, honest, qualified, young folks in UK and EU. Some even have experience. Oh and they might be qualified in a thing called "banking", or even better, also in "business". Did I mention that you can get qualifications in "banking" these day at all? Apparently they don't do courses in "funny handshakes", though.

  17. Anonymous Coward
    Anonymous Coward

    Well I'm closing my accounts at RBS.

    Yep! Bye Bye RBS.

    Thank god I had an account an another Bank that didn't need bailing out originally.

    1. Anonymous Coward
      Anonymous Coward

      Re: Well I'm closing my accounts at RBS.

      Go for Barclay's. Every other bank calls it's cashiers 'sales' staff and gives them sales-based performance targets; which involves flogging credit cards to grannies and worse.

      1. JimmyPage Silver badge

        Re: Well I'm closing my accounts at RBS.

        Go for Nationwide:

        1) It can't play casino games with your money

        2) It doesn't worry about shareholders

        3) It hasn't had a penny of government cash

        Is it just me who thinks if we had lost fewer building societies, we might not be quite as deep in this mess as we are ?

        1. Anonymous Coward
          Anonymous Coward

          Re: Well I'm closing my accounts at RBS.

          But you've still got the same sort of city fat cats on the board of Nationwide, with the CEO helping himself to over £2m last year. For an organisation that isn't particularly complex, supposedly works for its members, and allegedly doesn't engage in risky trading activities, I can't see much justification for that (other than of the "because I can" sort).

        2. Tharglet

          Re: Well I'm closing my accounts at RBS.

          I have a bank account with Nationwide. Had a problem with my account and their customer service left a lot to be desired. Not entirely trusting of their backend.

      2. Blitterbug
        Happy

        Re: Go for Barclay's

        Nah, try Nationwide - ethical banking (if there is such a thing) and all that.

    2. Anonymous Coward
      Anonymous Coward

      Re: Well I'm closing my accounts at RBS.

      COOP banking, not the best customer service, but has all the services and as ethical as banking can get without going for a credit union (which often just bank at a high street branch anyway).

  18. Anonymous Coward
    Anonymous Coward

    @12:48 How is this racist ? (which I infer from your "Daily Mail" editor line).

    At no point has anyone said the chaps now looking after the NW systems are incompetent because they are Indian, which would be discriminating due to race. The point being made is that NW have outsourced to inexperienced staff to save costs, and that inexperience has possibly caused the problem and contributed to the length of time taken to fix. The same thing would happen if they outsourced to less experienced staff anywhere in the world. By screaming "Racist" over this shows what little grasp you have of the word, devalues any genuine attempts to eradicate racism and is simply an annoying and uneccessary input to this story .

  19. Fred M

    Been there, done it

    Having seen first-hand what happened to bits of RBS as they were outsourced to India, this comes as no surprise at all. Incompetent doesn't even come close.

  20. heyrick Silver badge
    WTF?

    Unwanted redirect?

    Reading this on my Android phone via Orange France (WiFi). Why did it, the first two times, point me at http://www.windowsphone.com/en-GB/apps/7f8072c8-57a1-4dc7-af48-5776e16c28e0 ???

  21. Anonymous Coward
    Anonymous Coward

    Dealing with offshore (Indian/Pakistani) devs is an art from in itself

    Which can end up costing more than the saving. I managed 3 teams over 2 years, and by the end of it could pretty much know how to word a spec so they would deliver something that vaguely resembled it. However, this meant you had to spec so tightly you may as well have written the code yourself.

    Also, they have a worrying tendency to say "yes" without understanding the issue.

    1. Anonymous Coward
      Anonymous Coward

      Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

      "Also, they have a worrying tendency to say "yes" without understanding the issue."

      It's a cultural thing, as anyone familiar with Norwich Union's cross-culture training CBTs now knows.

      In my experience, Indian staff are not comfortable disagreeing with superiors, so say "yes" when they mean "not a snowball's chance in hell", out of politeness. I got caught out a few times (prior to my NU cultural training, of course!) before I realised the issue. And knowing is half the battle!

      Also beware "Insh'Allah" in the Middle East. In the UK you can't get away with saying "with a bit of luck" after giving formal deadlines in senior management meetings, but in the ME, you can add a cast iron excuse, and never be taken to task for it!

      1. Anonymous Coward
        Anonymous Coward

        Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

        Perhaps there is a similar English phrase to inshAllah with a meaning closer to "providing all goes well" than "with luck"

        1. Anonymous Coward
          Anonymous Coward

          Re: Perhaps there is a similar English phrase to inshAllah

          I can personally guarantee it

        2. Anonymous Coward
          Anonymous Coward

          Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

          "Perhaps there is a similar English phrase to inshAllah with a meaning closer to "providing all goes well" than "with luck" "

          You'd still be taken to task for it not all going well, though. Whereas in the ME, several times I walked out of crash meetings scot-free, having said "well, I did say that it would only work God-willing" and everyone pretty much nodding and saying "Fair enough!".

          1. Keep Refrigerated
            Alert

            Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

            Well the ME is not India, but I can attest through a number of adventure-laden taxi rides that Indians have a strong aversion to saying "No" (seen as a negative) to anyone they are providing a service for - regardless of truth or consequence!

      2. keithpeter Silver badge
        Windows

        Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

        Would that be the Hofstede thing?

        http://www.kwintessential.co.uk/intercultural/power-distance-index.html

    2. Anonymous Coward
      Anonymous Coward

      Re: Also, they have a worrying tendency to say "yes" without understanding the issue.

      I think that's everyone at some time

    3. Anonymous Coward
      Anonymous Coward

      Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

      That may not be incompetance at work.

      One of my colleagues spent quite some time working closely with the lads from a certain large Indian outfit we use. One of the things they let slip was that they were incentivised on profit over quote. There's no money to be had in the fixed-price quote, so any gravy has to be found in the T&M Change Requests, which can be charged at "got you by the balls now" rates.

      Thus, when reading a spec, any ambiguity will be read the wrong way, if the bloke reading it's got the slightest clue what the intention was, as his livelyhood depends on it. Anyone not making money tends to find themselves out the door in short order. Thus the whole system's rigged so that the idea of a quick phone call or email to resolve questions is avoided like the plague.

      What you may see as incompetance, could just as easily be a deliberate and quite cunning attempt to rip you off.

      1. JimmyPage Silver badge
        Stop

        Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

        When I started managing an offshore team, they were pretty much working on retainer. Mysteriously code never worked first time, things weren't tested, etc etc. It should have been apparent to anyone that we were paying for their incomptence.

        First think I insisted on, was that they set up a test server (we were dealing in a web app) their end, that I could test *before* they checked it in. Much squealing over that one (in the end I just said if you can't set a simple server up, how can you be expected to be any good at coding).

        Then we started using a workflow system for all communications, rather than ad hoc emails. This let us label a fix/change and track it's progress.

        Thirdly, we switched to a T&M model, where *we* suggested the initial timeframe. They were free to discuss it with us - maybe we'd revise upwards if they could show why they needed more time. Otherwise, they got paid for the quoted time - regardless of how long they actually spent. I tweaked the workflow to have 2 timescales - worked time, and due time.

        Once we got to that state, things were workable. However, despite the firm showering me with the cv's of their team (was there *anything* they hadn't got an MSCE in ?) I would still have to value one good UK contractor=4 offshore devs. And the firm still needed me to liaise. I dread to think how things would have gone if a BA, or worse still, customer, had to deal with them.

        Having dealt with an MS employee based in India, whose English was impeccable, I think it's pretty obvious how things are. IBM had first pick. MS had second. Other big name firms third. And *then* your team got formed. So you're stuck with the developers who weren't good enough for the first 3 cuts.

        The other problem people have correctly identified is churn. As soon as a dev gets some experience in your team, they'll up and off. My current employer has a 25 man team in India, who have been with the firm for 5 years now, with no one leaving. However, they are paid a *lot* more than they were 5 years ago. Couple things like that with falling UK wages, and outsourcing really doesn't have the same shine it did 10 years ago. It's a shame no one suggested that could happen 10 years ago - oh hang on, we did. But who listens to techies ?

  22. Alan Brown Silver badge

    Outsourcing

    Is sometimes done, not for reasons of cheapness, but to restructure locally and then ckaw it all back when things are working the way you want (and the incompetent/overpriced/impossible to fire staff have been shown the door)

    What gets to be "interesting" in chinese proverb terms, is when the outsourcers then outsource the job to a 3rd party. The more steps removed the people doing the work are from the people wanting the work done, the greater the potential for all hell to break loose (and remember, those people doing the work answer to the middleman, not the company hiring the middleman)

  23. Anonymous Coward
    Anonymous Coward

    indian call centres fail our children and our banks!

    This fubar from RBS effected my child tax credits

    Won't somebody think of the children.

    *there you go, now it's fully fledged daily fail fodder

    1. Anonymous Coward
      Anonymous Coward

      effected

      Glad to hear it worked out ok for you.

      </grammarnazi>

    2. Nick G
      Joke

      Re: indian call centres fail our children and our banks!

      It could have some bearing on the Lady Di conspiracy too!

      There - that's the Torygraph taken care of...

  24. Anonymous Coward
    Anonymous Coward

    RBS are great at cost saving - really

    An associate of mine used to work for RBS until he was offered voluntary redundancy. He graciously accepted their offer of 6 months paid gardening leave, an enhanced final salary pension which would start immediately and a big fat lump sum.

    Within months they realised they actually needed his skill set so he is now working for them, full time, on a consultancy basis for even more money than he was on before.

    With such compentant management I'm really surprised they got in this mess...

    1. Anonymous Coward
      Anonymous Coward

      Re: RBS are great at cost saving - really

      - Totally anon here too... seen the exact same thing happen, for the same company!

  25. Anonymous Coward
    Anonymous Coward

    Outsourcing

    My experience of outsourcing things to India as both someone who has worked for banking/oil company clients and someone who's a consumer who has to deal with offshore support (eg 3, Lovefilm, Barclaycard) is that it tends to be pretty dire.

    Having worked with people from India in the UK, I can say that those people have proved themselves entirely capable, so it would appear that the offshoring process itself which is at fault - usually a result of trying to do stuff on the cheap and hiring brainless morons for the lowest amount of money possible at the other end. I've also noticed a culture problem where it is sometimes seen as okay to tell you things which simply aren't true.

    Outsourcing on the cheap to some poor country seems to be the quickest way to mess things up in my experience - though I'm sure outsourcing companies would disagree. As long as there are enough stupid people to put up with bad customer service and MDs who think they can save a packet on their IT budgets, mistakenly thinking a developer can be easily swapped out for another one, this practice will long continue.

    1. Anonymous Coward
      Anonymous Coward

      Re: Outsourcing

      I would aggree but it is worse than you think. I worked along side a few Indian engineers and got an inside look at the out sourcing companies. The biggest problem was staff churn, the companies pay them peanuts treat them badly and like to turn over all their staff within 2 years to keep the wage bills down.

      The only way to make out sourcing work is out source some incidental part of you business build in punishing SLA so when the make a mess up you never have to pay them. After a couple of years bring it back in house. Result 2 years free IT support.

    2. Anonymous Coward
      Anonymous Coward

      it is sometimes seen as okay to tell you things which simply aren't true.

      I have to agree, having worked with outsourcing whilst working in a foreign-owned bank that used to be an English building society. (work it out!)

      One of the biggest problems I had was knowing whether or not they understood what needed done or whether or not they thought they could do it, because the answer was "yes" to just about anything.

      More than once during conversations I'd ask them "can you do such-and-such" and get the answer "yes". Unconvinced, I'd start to expand it to things that I knew damn well they couldn't do, e.g. "These alerts we need to send out through the browser - if the customer isn't using their browser at the time of the alert, can you start up their browser, log them in, and show the alert? It's all got to be through the browser, we can't install anything on their machine." and I'd get a "yes".

      So I'd basically say "No you can't. Now, we'll go back and start at the beginning again and be realistic this time". They were just as happy to put it down in writing too.

      I'd really have to work hard at convincing them it was ok to tell me they couldn't or didn't know how to do something.

      I don't think it's being racist - and I'm not sure I can express sufficiently that it's absolutely not intended as such - but there is definitely a cultural issue where it would appear they wish to please to the extent that they are very reluctant to say "No" to just about anything.

      1. Anonymous Coward
        Anonymous Coward

        Re: it is sometimes seen as okay to tell you things which simply aren't true.

        I was chatting to a south african a few years back who'd spent three years working in India. Keen to make friends, he kept on inviting people over to dinner. They graciously said yes but never turned up. Happened four or five times until someone told him that they had no intention of going but didn't want to upset him by saying no.

        Helped me a great deal at the time, because I started phrasing questions open ended, rather as yes/no responses when dealing with Indian clients.

      2. Muscleguy
        Boffin

        Re: it is sometimes seen as okay to tell you things which simply aren't true.

        It's a cultural thing and it happens in a lot of Asian cultures where the concept of 'losing face' operates. They lie to you so they are not seen as losing face which is a much worse thing, for them, than being caught in a lie. I've seen it when working with Japanese, Chinese and Indian colleagues and face to face you can learn to read the signs of when they are saying 'yes' out of politeness/fear. On the phone or via email/skype that is harder to do.

        Understanding that it's a cultural thing makes it easier to deal with, unless someone who doesn't understand the difference through being too PC intervenes.

  26. JaitcH
    WTF?

    India, you say, must be just down the road from the HSBC sweatshop call centres

    HSBC has got all this stuff covered.

    They call it Unscheduled Maintenance.

    Then, when their long suffering customers actually try calling their branch, they get shuffled off to Sweatshop India where totally inept people, who speak a form of English that is totally incomprehensible to the average English person, try to confuse you.

    By the time you have wasted your time trying to communicate with Customer Support (India) for ten or fifteen minutes you will find the Unscheduled Maintenance is over.

    Can't beat exporting British jobs.

    1. Boris the Cockroach Silver badge
      Happy

      Re: India, you say, must be just down the road from the HSBC sweatshop call centres

      Thats a bit unfair

      Every time I've called the HSBC customer support , I've been put over to their India based support centers and had the problems resolved in no time.

      Because they dont understand my accent when they fail to understand my password for the 5th time and put a security lock on the account .... which is usually what I'm after since I wont use their on-line banking system :-)

      Boris the luddite

  27. MH Media
    FAIL

    Should've

    .. stuck to CRON (cron)

    1. AOD
      WTF?

      Re: Should've

      Cron, really?

      Have you ever had to use a pukka enterprise grade scheduler such as CA7, Autosys, UC4 or Control-M in a production landscape? I could be taking a stab in the dark here but I'm guessing not.

      Cron bears as much resemblance to any of these as does say Windows Task scheduler.

      They are very different in their modes of operation, resilience (if configured properly) and capabilities.

      For one thing, they support the concept of dependencies (eg only run this job once the previous one has completed successfully) and alerting when things don't go to plan.

      1. Anonymous Coward
        Anonymous Coward

        Re: Cron, really?

        Should've stuck to Make. :-)

      2. Anonymous Coward
        Anonymous Coward

        Re: Should've

        @AOD, while I agree with you, I'd point out that Windows Scheduler has dependencies, event and time driven schedules, alerting and outcome based decision making (if I fail X times, do A, if I fail Y times, do B, etc). It;s not just at/winat any more.

        1. AOD
          Boffin

          Re: Should've

          Hi AC,

          Have you got a link for any of these fancy new features in Windows Scheduled Tasks? Last time I looked the Windows scheduling facilities were no more advanced than those back in Windows 2000 as detailed here:

          http://technet.microsoft.com/en-us/library/bb726974.aspx

          Sure the snap in under Win 7 and the latest server versions for managing the tasks looks snazzier, but I bet if you export the tasks via the command line to .CSV they'll look the same, eg:

          schtasks /query /FO CSV /V > schtasklist.csv

      3. Steve I

        Re: Should've stuck to cron..

        and bunged in a few shell variables for the dependancies.

    2. Anonymous Coward
      Anonymous Coward

      Re: Should've

      CRON???

      I'd like to see you schedule tens of thousands of mainframe jobs, with complex interdependencies, using CRON. CA7 (along with some other systems) is a very well established and reliable scheduling package used by some of the biggest companies in the world.

      1. Anonymous Coward
        Anonymous Coward

        Re: Should've

        And It Just Failed.

  28. Derek Kingscote
    FAIL

    Crasharama

    Where the software is and where the staff are is irrelevant.

    Experience and competence are essential.

    The head honchos at all companies are paid inflated salaries because you have to have "World Class People" to run the business.

    All board members should only ever be bonussed on the customer experience and NOTHING ELSE!

    They think they are running the company, but it is everyone below them that are doing it.

    When I was working I always said, I'd settle for annual salary increase on the same basis as the chairman in either cash or percentage terms.

    This is a massive fail. Hester and the person responsible for the Banks IT should fall on their swords.

    Customer Experience = No bonus for any of the board this year!

    [The person responsible for the Banks IT or may not be Ron Teerlink. He is Chief Administrative Officer of Royal Bank of Scotland Group Plc. He joined the RBS Group in April 2008 as Chief Executive of Business Services, becoming the Group Chief Administrative Officer in February 2009. At the same time he was re-appointed to the Managing Board of ABN AMRO to oversee the integration programme. Ron started his career with ABN Bank in 1986 as an IT/Systems analyst and held various functional positions before becoming Chief Operating Officer of the Wholesale Clients Business in 2002. He was appointed Chief Executive Officer of Group Shared Services in 2004 and joined ABN AMRO’s Managing Board in January 2006, where he was responsible for Services and Market Infrastructure. Ron holds a Masters degree in Economics from Amsterdam’s Vrije Universiteit.]

    Anyway, Teerlink is the only board member that has any IT experience listed.

  29. Cliff

    Offshoring...

    I used to contract for a big OS and Office software company - they sent loads of work to India. After 2 years they brought a lot of it back to Europe - offshoring was no cheaper (in fact it worked out quite similar cost-wise once you factor in everything) and it's not that the quality was bad, it was more that it meant trying to discuss complex issues over VOIP when having 6 people in a room together with a printed schema and some sharpies was just more efficient.

    Installation instruction scripts were followed to the letter, everything needed to be spelled out as you never knew who you would get doing an install or update - so you had to try to pre-empt all the failure modes and explain in bullet points how to roll back etc for each case. Sure, this is good practice by and large, but it meant UK engineers were spending hours writing these instructions each week when the patch was 2 minutes work and could only really be rolled back by those same engineers anyway as it required deep business knowledge.

    They were also hit by the vendor creating and billing phantom employees, which really did not help, but is very hard to monitor.

  30. Professor Falken
    Stop

    I remember ringing Barclays a few years ago...

    ...and having issues with understanding the person on the other end of the phone.

    Turned out the person I'd been speaking to in Bangalore (whom I understood perfectly) had put me through to a call centre in Merseyside for "local service" - I couldn't understand a word...

  31. Snowy Silver badge
    Mushroom

    So the bank 80% owned by the UK goverment to save cash offshored work to India. If they had kept the work in the UK the money spent atleast would have stayed in the UK to help the UK economy get going rather than India's. In the long run that would have helped the UK more ( and cost the UK less). If all the banks that have taken government cash offshore then all we have done is help everyone esle out at the expense of the UK economy.

  32. Danny 14

    heh

    " perhaps exacerbated by poor documentation"

    documentation is usually one step above training. I have comments in my code, does that count as documentation?

  33. Melanie Winiger

    As others have said

    "The RBS spokesman stated that "the software error occurred on a UK-based piece of software" but declined to give details on where the staff overseeing the software were based."

    "UK-based piece of software"

    FFS!

    I obviously don't work in marketing because only a Spin Doctor could come up with nonsense like that.

    1. Steve I
      Facepalm

      Re: As others have said

      The software was also written using letters from the English alphabet.

  34. Blobster
    Happy

    Wouldn't it be funny if they have actually been robbed ,just like the film Swordfish

    A script was ran from India ,Billions in cash was vanishing from the bank and the only way to stop it was to turn cancel the script in progress corrupting all the data.

    1. 4ecks

      The words "Mainframe" & "Bank" make me think of...

      Salami slicing & logic bombs, but I was a very prolific reader at age 10.

      Look after the pennies and the pounds will look after themselves.

      http://books.google.co.uk/books/about/Computer_capers.html?id=SbkiAAAAMAAJ&redir_esc=y

  35. Iain Leadley
    FAIL

    Why

    Oh why do the UK government bail out a UK bank then let it sack a load of UK people and then replace their skills with skills sourced in another country?

    So we are now paying a load more redundancy and then effectivly importing stuff when we need to be exporting more?

    1. Anonymous Coward
      Anonymous Coward

      Re: Why

      Because the long-tem costs don't matter in the slightest to them. Someone (let's call them an advisor - the government use loads of these) has told them that they will save a load of dosh by not having to pay loads of people any more. The long term costs will hopefully be someone else's problem to deal with.

      And it's not just limited to government. It happens everywhere, and not just in IT. Quarter-end profit and share prices matter more than anything, even if it means running the company/country into the ground. As long as it happens after those reaping the benefits have retired/are dead then it doesn't matter.

      Running a company should come with responsibility. If you fuck it up you should lose everything.

  36. Anonymous Coward
    Anonymous Coward

    Every person who comments on dealing with offshore co-workers seems to have the same experience? Either everyone commenting here works for the same company as me, or the issues are consistent across all offshored companies. The other bit we were fobbed off with is that the offshore staff will work shifts and do all the on-call work and work that needs to be done outside UK business hours. Well all the bits they want to do, which is basically nothing beyond a few very simple menial tasks, and as soon as they have to make a decision or find out why the documented procedure didn't work then they're screwed and leave it for us to sort out the mess they've left behind. Oh wait... hang on... that's where we started with this mess wasn't it?

    1. Blitterbug
      Unhappy

      Re: the issues are consistent across all offshored companies

      I think they are. Terrible, terrible time at Logica working with outsourced pieces of the business (won't say which sector) except for one awesome guy who Bangalore sent to us for hands-on training. Sharp as a knife, brilliant ASP.Net skills and a thoroughly decent chap. All the rest: rubbish.

    2. MnM

      and then they call me at 3am...

    3. This post has been deleted by its author

      1. Anonymous Coward
        Anonymous Coward

        The isue seems to be that non-technical managers believe that every issue has a simple, easy to follow 'technote' to fix the problem. No concept of investigation to get to the point where a solution - documented or otherwise can be implemented.

  37. toadwarrior

    The problem is most of these big companies don't outsource because they need talent but they do it to cut costs and in some cases they don't even pay well in the country they're outsourcing too so save even more money so it's no surprise things go wrong with outsourcing for some companies.

    Bad management always leads to a bad job no matter where the developers are from.

  38. Chris Miller

    The thing about outsourcing

    (which may or may not be offshore) is that it may be a good idea if some or all of the following criteria apply:

    (a) your internal operation isn't big enough to gain economies of scale;

    (b) the activity being outsourced is not a core part of your business;

    (c) your requirements are quite similar to many other businesses;

    (d) you can produce a clear and unambiguous specification for the services you need that is unlikely to change for the duration of the contract.

    So catering and night-watchmen - OK to outsource. IT may be a possibility if you're a small to mid-size manufacturer (say) using COTS systems. But if you're a high street bank whose product is (to a first approximation) your bespoke IT systems, it doesn't take a genius to realise it will all end in tears.

    1. Anonymous Coward
      Anonymous Coward

      Re: The thing about outsourcing

      +1 I heard a bank IT manager saying "we're in the banking business not the IT business."

      Wrong! Your IT is everything to your business. Unlike the local car mechanic who can probably get by without IT for a while.

      1. BorkedAgain
        Thumb Up

        Re: The thing about outsourcing @ AC

        I wonder if we met the same bank IT manager; I've heard the same argument.

        My reply was, essentially, "No, you're in the IT business, you just happen to work at a bank."

        I guess we now have a powerful example of what happens when the IT folks lose sight of what business they're in...

  39. Anonymous Coward
    Anonymous Coward

    Radio 4 news

    I just heard Stephen Hester on Radio four news, being very careful to say that this wasn't an outsourcing problem. He's right, the staff in India are technically RBS employees, the problem is employing cheap labour in a foreign country who don't have the experience of the staff they are replacing.

    Beware of semantic, qualified answers to questions about this situation...

  40. sugerbear

    Failure linked to the policy of Mike "offshore" Errington

    If anyone is to blame its the man named above who started the whole process about five years ago.

    Wouldn't listen, didn't care, you would think he was only interested in securing his own bonus (or he had shares in infosys!)

    I wonder if he is feeling the heat now....

  41. Anonymous Coward
    Anonymous Coward

    Hell yeah!

    This seems to be a ISV (independent software vendor) Product (CA7) upgrade issue, nothing to do with the person's job role you 'googled' and posted in the news!!! or even his nationality. Any Tom or any Ram might run to this issue if your systems are way too 'obsolete' and refuse to pay for updated s/w.

    Please have your facts clear before you post something like this, shame on you.

    http://www.computerworlduk.com/news/it-business/3366112/ca-helps-rbs-resolve-tech-problem-that-led-massive-outage/

    1. David Neil
      Trollface

      Re: Hell yeah!

      Three exclamation marks in the middle of a sentence, serious business.

    2. Anonymous Coward
      Anonymous Coward

      Re: Hell yeah!

      This is not a race issue, it's a skills issue. CA7 is a current piece of software and not antiquated.

    3. Anonymous Coward
      Anonymous Coward

      Re: Hell yeah!

      This seems to be a ISV (independent software vendor) Product (CA7) upgrade issue

      and where does it say that in the article you quote?

    4. Steve I

      Re: Hell yeah!

      "This seems to be a ISV (independent software vendor) Product (CA7) upgrade issue, ... Any Tom or any Ram might run to this issue if your systems are way too 'obsolete' and refuse to pay for updated s/w."

      This seems to be a failed s/w upgrade - which happens all the time - followed by a botched backout to a known good restore point.

    5. Anonymous Coward
      Anonymous Coward

      Re: Hell yeah!

      You'ŗe looking a bit foolish now, mate. Just sayin'·

  42. Scotty

    the fault lays at the door of the guy who was hiring over there .............called Krash :D

  43. The Alpha Klutz

    uk based stalling tactics

    one of the big buckets of money fell over and got spilled and now they crawl on the floor putting the coins back in the right accounts

  44. Anonymous Coward
    Megaphone

    Let's have the "Royal" back!

    Given the systematic incompetence that RBS have demonstrated, their cavalier disregard for either private or state shareholder, their snivelling unfelt apologies for pissing off millions of customers, their well dodgy corporate banking arm, isn't it about time that the government did what they did for Fred?

    Sir Fred is now plain Fred. Let's follow the pattern, and whip the "Royal" out of Royal Bank of Scotland. Obviously HBOS own the name "Bank of Scotland", but RBS could perhaps investigate other possible names. A quick check on Companies House shows that "Bank of Thieving Bungling Twats" doesn't yet appear to be taken.

  45. Anonymous Coward
    Anonymous Coward

    Lets's have the "Royal" Back

    "A quick check on Companies House shows that "Bank of Thieving Bungling Twats" doesn't yet appear to be taken."

    Sounds like a sound move to me - keeps them out of trouble with Trading Standards as well!

    AC

  46. Anonymous Coward
    Anonymous Coward

    Seen it before at a bank outsourced to the subcontintent

    batch failed... I'll run it again

    and again

    and again

    until it was truly stuffed.

  47. Steven Burn

    Hmmm ...

    Anyone else find it highly suspect that the advert for the job, shows a Gmail address? Perhaps because I've been investigating Indian based scammers for years, I've be come overly skeptical, but no bank or financial company, should be dealing with ANY company that uses a Gmail address as a contact spot, especially for something as important as this.

    1. Anonymous Coward
      Anonymous Coward

      Re: Hmmm ...

      Maybe their real email systems were down, because they had been outsourced ....... ;)

  48. Anonymous Coward
    Anonymous Coward

    Deja Vu

    Makes you wonder how many ex employees of other large companies are saying "that could be XXXX". I know I am.

    Skilled UK people bailing at the sight of a reasonable redundancy - check

    Skilled Indians leaving almost before they had finished company specific training - check

    Morale diving in the UK as what was once a relatively smooth efficient machine starts making grinding noises - check

    Management proclaiming how successful it all is while everyone else wonders what they are smoking - check

    Cultural differences causing problems as the two ways of working clash - check

    The cost savings justifications being eroded over time as the cost of the Indian work force rises - check

    The cost savings justifications being destroyed as one or more easily avoidable blunders eat up cash like crazy - check

    My own poison was Tesco and while I have many fond memories of the company I still shake my head at how gung-ho they were about the off-shoring process. After the accidental deletion of the entire UK user tree in AD followed by the not so accidental loss of the relevant log files and the discovery that the backup procedures hadn't been followed that THEN they might actually have learned something but no..... that was the signal for me that I'd be better off elsewhere.

    1. Anonymous Coward
      Anonymous Coward

      Re: Deja Vu

      Mumbai lads took over English Support, then, not 6 months later, we get asked to fix the backlog .... 1 month of backlog because the Mumbai call centre boss grabbed the cash and left ... Mumbai office closed "unexpectedly", took 1 month to figure out it had happened... LOL.

  49. Anonymous Coward
    Anonymous Coward

    Offshored email

    The reason for the gmail address is that the email systems in India are unreliable and, therefore, offshored to Google... which is , of course, very reliable....

  50. aussieroof
    FAIL

    CA7 not at fault

    CA7 has been around for many years - i was using it in the 90's. It works and performs very well, provided you know the tool and you know the environment. it is uniquely mainframe.

    Not only did they give it to people who dont know how to use it nor do they know what their environment is, but they scheduled this upgrade on a tuesday. Had they done this on a saturday after midnight they would have at least had the sunday to back it out.

    This is incompetence at its grandest.

    It also proves... you get what you pay for.

  51. Anonymous Coward
    Anonymous Coward

    I used to work for them too....

    I was at Direct Line, Croydon when RBS shifted it all off up to Gogarburn, c2003/4

    Relocate to Edinburgh? No thanks, next stop probably Mumbai, could see it coming even back then.

    Initial redundancy offer was the state minimum - from a company (DL) making over £1M a day in profit!

    Whereas RBS staff T+Cs = far more lucrative.

    Make us RBS staff first then? NO.

    And they wondered why we had no enthusiasm for knowledge transfer.

    No disrespect to the Edinburgh lads - they were a decent bunch and our situation wasn't of their making.

    And re RBS hardware investment - Can we have STK tape robot/silos please? NO. But do have this unreliable hunk of junk instead. Whatever it was, nasty thing. Probably cheap though.

  52. Anonymous Coward
    Anonymous Coward

    CA-7 ?

    CA-7..?

    It was Control-M when I worked for one of the RBS companies.

  53. Alien Doctor 1.1

    Did anyone else...

    read the contact name on the job advert as "krash me live jobs"?

  54. Anonymous Coward
    Anonymous Coward

    Too many Ops staff

    TSB

    Mid 1990's, some mid-management fool asked why we have 24 shift ops staff.

    Well, dimwit, it's 4 shifts of six staff.

    But why four shifts?

    Well, dimwit, would you want to work more than an average of 8hrs a day, covering 24x365?

    24hrs = three 8 hour shifts, plus 8 hours off, which = 4 shifts.

    1. Anonymous Coward
      Anonymous Coward

      Re: Too many Ops staff

      >> Well, dimwit, would you want to work more than an average of 8hrs a day, covering 24x365?

      >>

      >> 24hrs = three 8 hour shifts, plus 8 hours off, which = 4 shifts.

      What pattern did you work 3 on 1 off? I work 12 hour shifts, so we have 2 x 12 hour shifts plus 2 x 12 off - still works out as 4 shifts. We work for days (or nights) on and 4 off. Means longer days, but fewer trips to work (so less wasted on travel), also we avoid rush hour traffic.

      BTW, I'm not with NatWest or RBS, however my wife's employer is - and she was due to have been paid her monthly salary last Friday.

      1. Richard 41

        Re: Too many Ops staff

        What pattern did you work 3 on 1 off?

        Eight week pattern, meaning, by average, only one weekend a month.

        WK1

        Sat + Sun 20:00 - 08:00

        Mon-Fri 00:00 - 08:00

        WK2

        Sat+Sun OFF

        Mon-Fri 16:00 - 20:00

        WK3

        Sat+Sun OFF

        Mon + Tue OFF

        Wed-Fri 08:00 - 1600

        WK4

        Sat+Sun 08:00 - 20:00

        Mon+Tue 08:00 - 16:00

        Wed-Fri OFF

        WK5

        Sat+Sun OFF

        Mon-Fri 00:00 - 08:00

        WK6

        Sat+Sun OFF

        Mon-Fri 16:00 - 20:00

        WK7

        Sat+Sun OFF

        Mon + Tue OFF

        Wed-Fri 08:00 - 1600

        WK8

        Sat+Sun OFF

        Mon+Tue 08:00 - 16:00

        Wed-Fri OFF

        ....and repeat, for very nearly a decade until Lloyds took it all away, redundancy #1

  55. Revelationman

    Why outsource in the first place saving money give me a break

    How many on you on this discussion love calling a overseas call centre love giving your details or knowing in this case your sensitive banking information is somewhere thousands of miles away.

    The Lord of the Banks say we are saving money, yes you are but now this will cost you millions of pounds to fix the potential loss of customers could damage the bank for years. The common sense approach is to bring back all IT services back to the UK were they belong, bring back the jobs that were taken,

  56. FreeTard
    FAIL

    Still not fixed

    I'm with ulsterbank (natwest/RBS) and my wife's pay has not arrived at all.

    My pay is supposed to arrive on the 28th but I'm not convinced it will, as none of te other transactions are working. I have four accounts with ulsterbank and none of them seem to be working as before this fiasco.

    I bet the feckers charge me for late fees on my ulsterbank credit card, let them try.... I've never failed to pay it in full before!

  57. Anonymous Coward
    Anonymous Coward

    jObScHeDuLinG

    You know, it is better to use one brand of software in scheduling across the landscape, especially if you can have one central point of control. I guess the sales guyz of the other schedulers will be knocking on RBS's door soon. I want run my jobs, total control, no maintenance costs, no downtime ... one of them must have the solution ...

  58. Anonymous Coward
    Anonymous Coward

    https://en.wikipedia.org/wiki/CA7

    responsible for (among other things) the production of hot air and spit.

  59. Nick G
    Flame

    And another thing...

    Whilst it's all very fascinating hearing about peoples issues with outsourcing (& I have my own experiences...) nobody seems to have touched on the other major show of incompetence, namely - who, in the name of all that's holy, schedules an upgrade to a major finance system in the middle of the week?

    I'd love to hear the reasoning behind that!

  60. Anonymous Coward
    Anonymous Coward

    This sounds more like the real story...

    I didn't believe they could be so dumb as to do an upgrade midweek. These comments tally with what their spokeperson said on Moneybox last Saturday: basically it was the backout on Tuesday that caused the problems. These posts spell out why they needed to do the backout, following an upgrade at the weekend that caused problems on Monday.

    The following is taken from a number of posts on the westhamonline dot net forum by a poster called CTM who is currently working on the Natwest/RBS fiasco and who was due to be made redundant 2 weeks ago. The most worrying comment is that 'The backout caused the "queue" of jobs for that batch to be unexpectedly deleted. '

    "Ok, since I work on the mainframe at RBSG, I'll let you know some facts - I'm currently on my 5th 12 hour shift in 4 days (and am quite busy so won't be back here for a while)

    It is a mainframe problem and not a virus It was not deliberate by the bank

    It was a third party product that was upgrading that caused a big whole in the batch schedule

    RBS wasn't actually affected directly as the schedule is on a different LPAR - however focus was on NWB & UB and normal everyday errors on RBS side were not getting dealt with

    COBOL and Assembler on the Mainframe, like most big banks. But it's not the application code that is the problem as I stated below. CA7 was updated to a new level last weekend. It was causing problems with the schedules on Monday so was backed out on Tuesday. The backout caused the "queue" of jobs for that batch to be unexpectedly deleted.

    Not sure why DR wasn't invoked to be honest, but it wouldn't have made much difference as we'd still have been hours behind on our batches.

    All transactions made on Tuesday and since WILL be processed eventually.

    As for the account thats being sold to Santander - you can just phone the number on your letter and you won't be moved, so stop fucking whining like you've been made to do it. We could have done without the work and expense that was foisted on us! Anyway, there's a pot of gold at the end of the RAINBOW. SWT

    Also, in my previous posts, the "not sure" was my own, not the banks - I'm sure that the discussions were had about DR and there must have been a good reason why it wasn't invoked. In truth DR is great in certain circumstances if it works, but in this one it wouldn't have. Obviously i can't go into too much detail here, but although the outsourcing to India did not cause the intial problem, the recovery has prompted questions to be asked as to who thought dumping over 50% of the UK support staff was a good idea - indeed people made redundant less tha a month ago have now been offered short term contracts to come back and help. Unfortunately, the balance sheet and pressure from the government to become independent again hasn't helped in the outsourcing/offshoring decision and I speak as a someone who was due to have been made redundant two weeks ago."

    1. Chilky

      Re: This sounds more like the real story...

      Of everything that I have read so far, this seems to be by far the most plausible explanation. Combination of upgrade not working and then the decision to roll back causing a serious error.

      From (admittedly back in the 90's) experience, you can recover if you lose the CA7 data. But only if you have a bunch of guys in the computer room who have worked on those batch schedules for years and know them off by heart. You can't simply re-create without them. However, they are generally the first in the chopping line as on a resourcing spreadsheet you simply see very expensive resources doing something that the management consultant doesn't understand.

      Again from experience of working in IT in retail banks, the business / management neither understand nor respect IT. As such they are dealt a crap hand every budgetary cycle and simply have to make do. Until the business understand that they are dead in the water without IT and start giving it the money it requires then this event is going to be repeated over and over again.

  61. Anonymous Coward
    Anonymous Coward

    Outsourcing...

    A few years ago I was made redundant when my job role was offshored to Chennai, India. It was a nice investment management company with circa 500 people and visions of an IPO. I had been at this company for 14 years. The knowledge transfer guys were pleasent and nice guys with a little, but not too much knowledge on the subject (the subject being, for me Information Security). Their production of Run Books for their offshore team, however, wasnt.

    This is my point - the high turnover of the engineers assigned to a customer contract means they have the poorest of documentation that they themselves have created as their only knowledge of context of product use. It's not the technology so much, as how it's used in the environment. That, is of course when things are working. Like outsourcing, it's only great when it actually works. And when it doesnt then you're given the scenario that has RBS/Natwest without transaction services - the absolute core of their multi-million pound establishment, in ruins.

    I'm thinking about creating a meme on this, with that guy from Anchorman:

    Outsourcing:

    <insert jpg>

    60% of the time, it works everytime

    Why AC? Moral of the story I guess. The company in question expected a launch at 300p, launched at 220p and dive bombed to 70p... all a result of mis-management by those same people who thought outsourcing 25 people to India the great company I worked half my life for and had existed since the early 60s is no more.

    1. Anonymous Coward
      Anonymous Coward

      Re: Outsourcing...

      Once Upon A Time, in a land Far Far Away, there was a publicly-listed networking device company that closed a sh*t-ton of offices globally including one or two newly-built and taken over from competitors. They outsourced quite a bit of development to India and the stock price promptly nosedived, the only thing that saved them from circling the drain quite so closely was canning the India office immediately and taking themselves OFF the stock market.

      Thus ends the Parable of The Stupid Fat Hippy Buddhist Boomer and his New Office.

      This parable may or may not be related to the Parable of The Brown Envelopes and the Parable of The Developer That Wasn't and The Stockholder Who Was. All rights reserved. The names of people and places may have been changed to protect the innocent, if there were any.

  62. JimmyPage Silver badge
    Pirate

    Closing your bank account ....

    may I respectfully suggest a better way to vent your ire, is not to close your bank account, but to simply open another, and reduce the balance in your NatWest account to (say) £1.01. Make sure you are signed up for monthly statements. By post.

    I did this 16 years ago when Barclays ****ed me off. Every month I get a statement. Last year they almost twigged that the account may be dormant - I got a warning letter. So I shifted £5.00 in, and then a day later £5.00 out.

    Just make sure there's no overdraft availale on the account.

  63. The_Equaliser

    Governonce

    If your outsourcing governance is solid which for a mutlinational bank it would have to be then your outsourcers would not be in a position to make any errors

    OR........

    You could hawk it out to the cheapest bidder and use the money you "saved!?!?!?!" to bonus up your senior management

    I wonder which of the two approaches RBS is using ?

    Also I wonder what will become of the onshorebased management team for the outsourced departments. (New yachts for everyone from what I can gather).

  64. Anonymous Coward
    Anonymous Coward

    Hey, give me a toy inflatable globe of the world and a marker pen, and I can show you know what the "defective part" is. I don't need an ouija board either ;)

This topic is closed for new posts.

Other stories you might like