back to article RBS collapse details revealed: Arrow points to defective part

The tech problems at the RBS banking group that left millions of people unable to access money for four days last week were caused by a failure in a piece of batch scheduling software, sources have told The Register. And at least some of the support staff for that software have been outsourced to India - as recently as …

COMMENTS

This topic is closed for new posts.

Page:

    1. teebie

      Re: UK based

      It is a suspiciously specific piece of PR

      1. Anonymous Coward
        Anonymous Coward

        Re: UK based

        Yes it's spin designed to give the impression that outsourcing had noting to to with the problem.

    2. GitMeMyShootinIrons

      Re: UK based

      When a company makes a comment that specifically refers to "UK-based piece of software", I start smelling a rat. Why would the company's PR-drones be so specific, unless they were paranoid about media comments regarding the risks of off-shoring?

      Also, it's a suitably vague PR-drone written comment - the software may well be based in the UK, but it could be written by an Indian, in Somalia, using a Brazilian keyboard and uploaded using a proxy in Thailand, after being QA'd by a kid in Poland. Where it's based is somewhat meaningless in this context.

      1. James Micallef Silver badge
        Facepalm

        Re: UK based

        Also, what the hell does "UK-based piece of software" mean? This is software we're talking about, it isn't "based" anywhere. If he meant "it's running on a physical server in the UK", even that means nothing, because it still could have been coded anywhere in the world, and it presumably still could be remotely executed by RBS staff at a remote location.

        So it's nothing but PR-bullshitting meant for the consumption of people who don't have a clue about software.

    3. HappyInTheValley
      Angel

      Re: UK based

      I saw this posted by http://twitter.com/#!/YamalDodgyData over at The Telegraph, it certainly sounds plausible!

      "RBS PR has specifically referred to a "UK-based piece of software" as the fault.

      Why would the PR-drones at RBS be so specific, unless they were paranoid about media comments regarding the risks of cheap third world off-shoring ?

      Perhaps a few questions you should ask RBS are:

      Was it an Infosys employee on a Tier 2 visa at the mainframe site in Edinburgh who bungled the customer accounts batch job, or

      Was it an Infosys employee working remotely from Chennai that bungled the batch job in Edinburgh ?

      Does Mr Hester regret making the last British mainframe specialist at RBS redundant three weeks ago ?

      and does he think the savings from sacking 1,800 British IT professionals and third world off shoring will pay off ?"

      1. Anonymous Coward
        Anonymous Coward

        @HappyInTheValley

        It's RBS Gurgoan, so a mix of cheap permies, contractors and some 3rd parties. RBS gave Infosys the boot ~3 years ago for being sh*t.

        Also Hester just told the head of IT to reduce costs, it is they who decided to go off-shoring despite two previous failed attempts.

        I knew many of the mainframe batch guys and they were a good bunch of long termers who were all showing a lack of morale when I left, but there again senior IT management had removed much of the desire to do a decent job and company loyalty with their announcements and actions.

        Oh and as to sources areas to lay blame on for this fiasco. What's the bets RBS Incident management did their usual block on any repair/back out actions for amny hours while they searched their own navels and those of senior management (oh the joys of working support there).

    4. Callum
      FAIL

      Re: UK based

      I saw Stephen Hester say it was "UK based" on the news last night - my immediate reaction was that they are already trying to dodge the resulting FSA fine. Especially since the largest fine ever levied was to Zurich regarding an out of control outsourcer to their outsourcer.

  1. Anonymous Coward
    Anonymous Coward

    Not convinced

    Think there must be more to the story than this.

    Whilst I've been personally responsible for scheduling issues that clobbered the Month End of a bank - I'm struggling to understand how that on its own could have caused a weeks worth of problems.

    Think there is more to come on this.

    Anon obviously.

    1. sugerbear

      Re: Not convinced

      It is easy. If on the first night of your batch run you miss a few input file and it isn't picked up until say the Tuesday afternoon (and let us say for example your support staff who specialise in CA7 had left for the day), you would be into Wednesday before you could be anywhere near identifying what the problem is and what caused it and what was missed the previous night. Then you have to fix the problem and work out how you are going to get the missing transaction into your batch file (they would then most likely reject because the batch header didn't have the correct day on it .. and so on..)

      Of course all the time its getting escalated and people want answers, so that delays the fix because they dont want the fix screwing up something else.

    2. Anonymous Coward
      Anonymous Coward

      Re: Not convinced

      One possible reason is the lack of knowledge of the RBS configuration as opposed to the product.

      When the first batch failure occured, the "old" staff who knew RBS would probably have found the error quickly and could back-out the issue to the known good position.

      The "new" staff however, who probably know the product, but not the RBS configuration, didn't find the issue quickly enough, further batches had completed over the top, and it was no longer possible to return to the known good position. You're then into a fix-forward scenario to apply the missing batches to an updated position which is then out of transactional order.

      Think of it like an old fashioned engine - any old mechanic can probably fix it eventually, but old Bob who's looked after it for 30 years knows just where to hit it with a hammer when it stutters.

      The lesson for the accounts therefore is to consider a persons salary not just as a book cost this year, but also on the basis of the investment you've already made in that resource. You can get a YTS mechanic on minimum wage, but would you really trust them to service your Aston Martin.

    3. Anonymous Coward
      Anonymous Coward

      Re: Not convinced

      "Think there must be more to the story than this."

      I think so as well, as sugarbear points out it should be a simple enough task to add the missing transactions to the next days run, I'm quite positive that there must also be some sort maintainance application what can be used to apply ad-hoc transaction to accounts and that the experienced maintainence programmers would know what to do to recalculate interest, refund fees etc.

      wait a minute, they got rid of the experienced people didn't they

  2. Anonymous Coward
    Anonymous Coward

    £10k costcutting job

    costs RBS, sorry - the UK public how much money?

  3. Nash_Equilibrium
    IT Angle

    I have not used CA-7 or anything like it, god I would hate to be left alone with a mainframe.

    But after a bot of googling, I assume you don't want people like this either

    http://ibmmainframeforum.com/viewforum.php?f=38

    "cann any one please explain the commands like PF(page forward),PB (page backward) and SR(save Replace)

    Also TSO CA7cLIST"

  4. Anonymous Coward
    Anonymous Coward

    Freedom of Information

    Since "we" own RBS, what about a Freedom of Information request to find out what really happened :-)

    1. sofaofthedamned
      FAIL

      Blimey...

      ... I found the post you're on about - go Google for the guys name on linkedin, He's apparently been a 'mainframe programmer' for 15 months.

      Then go to his companies home page, and they're providing mainframe guys for HSBC!

      Frankly, i'm going to keep my cash in my house from now on :)

      1. irish donkey
        Happy

        Re: Blimey...

        Outstanding analysis.

        LMFAO 15 mths dribble

    2. nichomach
      Big Brother

      Re: Freedom of Information

      Upvoted the idea, but I can see the phrase "commercially sensitive" popping up like a flipping meerkat on speed in the rejection letter...

      1. Stuart Halliday
        WTF?

        Re: Freedom of Information

        Fair enough.

        So ask your local MP why a UK Government owned Bank is using non-UK staff?

        1. Anonymous Coward
          Anonymous Coward

          Re: Freedom of Information

          >So ask your local MP why a UK Government owned Bank is using non-UK staff?

          My MP asked a question in Parliament about fines over dog shit once, but that's her sole contribution to governance so far. May as well ask the cat.

          I've seen loads of horrible tales of woe...but has anyone else noticed that international payments which normally sits in the system for a couple of days - notably Google Adsense - have arrived on the day of payment this month? I've never understood why this doesn't happen all the time - it does the other way.

  5. Anonymous Coward
    Anonymous Coward

    UK-based, sort of....

    Ha Ha The country continues to reap the results of citizen goodwin's ego purchase of abn-amro.

    Yes CA-7 sits on the cluster of mainframes in Gogarburn, the idiots fucking it up are in India.

    No, I take that back, the idiots who fucked it up were Gogarburn based as well, fred 'the pleb' goodwin, mckillop et al. 8K a year to manage CA7 Pay peanuts etc. No wonder hester thinks he earned his £1M bonus.

    1. Anonymous Coward
      Anonymous Coward

      Re: UK-based, sort of....

      There are no mainframes at Gogarburn.

      While Goodwin's ego over ABN-AMRO may have been the major cause of the financial crash, it was Hester et al (appointed by the LABOUR government) who aggresively outsourced the IT services to India and Accenture.

      If Goodwin had backed off from ABN-AMRO, Barclays would have one the deal, which was a share swap. Come the crash, Alistair Darling would have approached RBS to take over Barclays, much as they did Lloyds for HBOS.

      1. Anonymous Coward
        Anonymous Coward

        Re: UK-based, sort of....

        AC @ 12:46 - not necessarily. I worked for both during that time.

        RBS was outsourcing to India way, way before the ABN farce. I know as I ran teams there (and here in the UK, Chicago, Singapore, etc). Some of those in India were good, some poor, some very bad. Sometimes the cultural issues made it worse.

        As for the RBS vs. Barclays issue. When RBS "purchased" ABN they really didn't look very closely. Having seen both sides, I think I can say that Barclays would've either backed off or made a better job of the purchase.

    2. Anonymous Coward
      Anonymous Coward

      Re: UK-based, sort of....

      @AC12:31 - Sorry, nothing to do with Goodwin's purchase of ABN, RBS used z and CA-7 for decades before ABN was purchased. Also, Gogaburn is the de-facto group head office (technically RBS St Andrew's Square is head office), they don't run the datacentres from a group head office.

      Now take your name calling of an easy media hate figure and work out something better to do with your day.

      1. Anonymous Coward
        Anonymous Coward

        Re: UK-based, sort of....

        Fred was "asset-sweating" local staff long before this kicked off and long before Gogarburn and the India-outsoucing mania. So, it is his fault, actually, for permitting this kind of behaviour to start under his watch.

        The opposition across the road were much less shitty. You know, the ones without the word "Royal".

        Cheers Fred, was that a fried potato of some sort on your shoulder BTW?

        Here's a clue, buying things then breaking them by trashing inside processes to make this years bottom line look good isn't actually "business".

  6. Anonymous Coward
    Anonymous Coward

    Oh dear

    Hope TATA didn't get the contract.

    From my experience TATA systems admins, sorry got to laugh at that, wouldn't know how to recognize a computer even if they'd read "Peter And Jane Meet A Computer" or "Identifying a Computer for Dummies"

  7. Anonymous Coward
    Anonymous Coward

    Nice job advert

    Using a google mail & the indian version of gumgree for job adverts? Yes I'm sure that is very legitimate.

  8. Anonymous Coward
    Anonymous Coward

    Shame on El Reg

    It looks like El Reg has recruited editors from the Daily Mail for this article and the comments also reflect the type of readership of that publication. Shame.

    1. Anonymous Coward
      Anonymous Coward

      Re: Shame on El Reg

      Well, if by Mail-readership, you mean racist, sexist, climate-change-denying, Euro-sceptic, hanging's-too-good-for'em, illogical, knee-jerk-reacting, warmongering carnivores, then - welcome to El Reg!

      1. Anonymous Coward
        Anonymous Coward

        @FatsBrannigan

        Who the fuck are YOU calling illogical?

    2. Yamal Dodgy Data
      Thumb Down

      Re: Shame on El Reg

      The most catastrophic IT failure in banking history, on a supposedly failsafe system with NO explanation or answers coming from RBS.

      And AC at 12.48GMT decides no one should know why... because YOU think the answer is rawwwwwcist.

      you wanker

    3. Blitterbug
      FAIL

      Re: Shame on El Reg

      You utter twat. So having a pop at the banking establishment makes us Daily Fail readers now? Do you even know what a right-wing rag is? Have a guess who Mail readers are more likely to support; fat work-shy tax dodgers like ex-Sir Fred & Mr Pester, or hard working IT grunts like us?

      You muppet.

  9. Anonymous Coward
    Anonymous Coward

    ouch

    If proven (can be, will be?), the cost of this minor (cough) glitch might offset all those fabulous offshoring / outsourcing savings they have been creaming their pants about over the last... what, 10 years?

    now, how to recover this cost, so that the punters don't notice they foot the bill again....mmmm... how about bringing back the Fred?

    1. Anonymous Coward
      Anonymous Coward

      Re: ouch

      Or maybe they could hire a "manager". You know, there are plenty of nice, honest, qualified, young folks in UK and EU. Some even have experience. Oh and they might be qualified in a thing called "banking", or even better, also in "business". Did I mention that you can get qualifications in "banking" these day at all? Apparently they don't do courses in "funny handshakes", though.

  10. Anonymous Coward
    Anonymous Coward

    Well I'm closing my accounts at RBS.

    Yep! Bye Bye RBS.

    Thank god I had an account an another Bank that didn't need bailing out originally.

    1. Anonymous Coward
      Anonymous Coward

      Re: Well I'm closing my accounts at RBS.

      Go for Barclay's. Every other bank calls it's cashiers 'sales' staff and gives them sales-based performance targets; which involves flogging credit cards to grannies and worse.

      1. JimmyPage Silver badge

        Re: Well I'm closing my accounts at RBS.

        Go for Nationwide:

        1) It can't play casino games with your money

        2) It doesn't worry about shareholders

        3) It hasn't had a penny of government cash

        Is it just me who thinks if we had lost fewer building societies, we might not be quite as deep in this mess as we are ?

        1. Anonymous Coward
          Anonymous Coward

          Re: Well I'm closing my accounts at RBS.

          But you've still got the same sort of city fat cats on the board of Nationwide, with the CEO helping himself to over £2m last year. For an organisation that isn't particularly complex, supposedly works for its members, and allegedly doesn't engage in risky trading activities, I can't see much justification for that (other than of the "because I can" sort).

        2. Tharglet

          Re: Well I'm closing my accounts at RBS.

          I have a bank account with Nationwide. Had a problem with my account and their customer service left a lot to be desired. Not entirely trusting of their backend.

      2. Blitterbug
        Happy

        Re: Go for Barclay's

        Nah, try Nationwide - ethical banking (if there is such a thing) and all that.

    2. Anonymous Coward
      Anonymous Coward

      Re: Well I'm closing my accounts at RBS.

      COOP banking, not the best customer service, but has all the services and as ethical as banking can get without going for a credit union (which often just bank at a high street branch anyway).

  11. Anonymous Coward
    Anonymous Coward

    @12:48 How is this racist ? (which I infer from your "Daily Mail" editor line).

    At no point has anyone said the chaps now looking after the NW systems are incompetent because they are Indian, which would be discriminating due to race. The point being made is that NW have outsourced to inexperienced staff to save costs, and that inexperience has possibly caused the problem and contributed to the length of time taken to fix. The same thing would happen if they outsourced to less experienced staff anywhere in the world. By screaming "Racist" over this shows what little grasp you have of the word, devalues any genuine attempts to eradicate racism and is simply an annoying and uneccessary input to this story .

  12. Fred M

    Been there, done it

    Having seen first-hand what happened to bits of RBS as they were outsourced to India, this comes as no surprise at all. Incompetent doesn't even come close.

  13. heyrick Silver badge
    WTF?

    Unwanted redirect?

    Reading this on my Android phone via Orange France (WiFi). Why did it, the first two times, point me at http://www.windowsphone.com/en-GB/apps/7f8072c8-57a1-4dc7-af48-5776e16c28e0 ???

  14. Anonymous Coward
    Anonymous Coward

    Dealing with offshore (Indian/Pakistani) devs is an art from in itself

    Which can end up costing more than the saving. I managed 3 teams over 2 years, and by the end of it could pretty much know how to word a spec so they would deliver something that vaguely resembled it. However, this meant you had to spec so tightly you may as well have written the code yourself.

    Also, they have a worrying tendency to say "yes" without understanding the issue.

    1. Anonymous Coward
      Anonymous Coward

      Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

      "Also, they have a worrying tendency to say "yes" without understanding the issue."

      It's a cultural thing, as anyone familiar with Norwich Union's cross-culture training CBTs now knows.

      In my experience, Indian staff are not comfortable disagreeing with superiors, so say "yes" when they mean "not a snowball's chance in hell", out of politeness. I got caught out a few times (prior to my NU cultural training, of course!) before I realised the issue. And knowing is half the battle!

      Also beware "Insh'Allah" in the Middle East. In the UK you can't get away with saying "with a bit of luck" after giving formal deadlines in senior management meetings, but in the ME, you can add a cast iron excuse, and never be taken to task for it!

      1. Anonymous Coward
        Anonymous Coward

        Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

        Perhaps there is a similar English phrase to inshAllah with a meaning closer to "providing all goes well" than "with luck"

        1. Anonymous Coward
          Anonymous Coward

          Re: Perhaps there is a similar English phrase to inshAllah

          I can personally guarantee it

        2. Anonymous Coward
          Anonymous Coward

          Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

          "Perhaps there is a similar English phrase to inshAllah with a meaning closer to "providing all goes well" than "with luck" "

          You'd still be taken to task for it not all going well, though. Whereas in the ME, several times I walked out of crash meetings scot-free, having said "well, I did say that it would only work God-willing" and everyone pretty much nodding and saying "Fair enough!".

          1. Keep Refrigerated
            Alert

            Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

            Well the ME is not India, but I can attest through a number of adventure-laden taxi rides that Indians have a strong aversion to saying "No" (seen as a negative) to anyone they are providing a service for - regardless of truth or consequence!

      2. keithpeter
        Windows

        Re: Dealing with offshore (Indian/Pakistani) devs is an art from in itself

        Would that be the Hofstede thing?

        http://www.kwintessential.co.uk/intercultural/power-distance-index.html

Page:

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019