back to article RBS collapse details revealed: Arrow points to defective part

The tech problems at the RBS banking group that left millions of people unable to access money for four days last week were caused by a failure in a piece of batch scheduling software, sources have told The Register. And at least some of the support staff for that software have been outsourced to India - as recently as …

COMMENTS

This topic is closed for new posts.

Page:

    1. Steve I

      Re: Hell yeah!

      "This seems to be a ISV (independent software vendor) Product (CA7) upgrade issue, ... Any Tom or any Ram might run to this issue if your systems are way too 'obsolete' and refuse to pay for updated s/w."

      This seems to be a failed s/w upgrade - which happens all the time - followed by a botched backout to a known good restore point.

    2. Anonymous Coward
      Anonymous Coward

      Re: Hell yeah!

      You'ŗe looking a bit foolish now, mate. Just sayin'·

  1. Scotty

    the fault lays at the door of the guy who was hiring over there .............called Krash :D

  2. The Alpha Klutz

    uk based stalling tactics

    one of the big buckets of money fell over and got spilled and now they crawl on the floor putting the coins back in the right accounts

  3. Anonymous Coward
    Megaphone

    Let's have the "Royal" back!

    Given the systematic incompetence that RBS have demonstrated, their cavalier disregard for either private or state shareholder, their snivelling unfelt apologies for pissing off millions of customers, their well dodgy corporate banking arm, isn't it about time that the government did what they did for Fred?

    Sir Fred is now plain Fred. Let's follow the pattern, and whip the "Royal" out of Royal Bank of Scotland. Obviously HBOS own the name "Bank of Scotland", but RBS could perhaps investigate other possible names. A quick check on Companies House shows that "Bank of Thieving Bungling Twats" doesn't yet appear to be taken.

  4. Anonymous Coward
    Anonymous Coward

    Lets's have the "Royal" Back

    "A quick check on Companies House shows that "Bank of Thieving Bungling Twats" doesn't yet appear to be taken."

    Sounds like a sound move to me - keeps them out of trouble with Trading Standards as well!

    AC

  5. Anonymous Coward
    Anonymous Coward

    Seen it before at a bank outsourced to the subcontintent

    batch failed... I'll run it again

    and again

    and again

    until it was truly stuffed.

  6. Steven Burn

    Hmmm ...

    Anyone else find it highly suspect that the advert for the job, shows a Gmail address? Perhaps because I've been investigating Indian based scammers for years, I've be come overly skeptical, but no bank or financial company, should be dealing with ANY company that uses a Gmail address as a contact spot, especially for something as important as this.

    1. Anonymous Coward
      Anonymous Coward

      Re: Hmmm ...

      Maybe their real email systems were down, because they had been outsourced ....... ;)

  7. Anonymous Coward
    Anonymous Coward

    Deja Vu

    Makes you wonder how many ex employees of other large companies are saying "that could be XXXX". I know I am.

    Skilled UK people bailing at the sight of a reasonable redundancy - check

    Skilled Indians leaving almost before they had finished company specific training - check

    Morale diving in the UK as what was once a relatively smooth efficient machine starts making grinding noises - check

    Management proclaiming how successful it all is while everyone else wonders what they are smoking - check

    Cultural differences causing problems as the two ways of working clash - check

    The cost savings justifications being eroded over time as the cost of the Indian work force rises - check

    The cost savings justifications being destroyed as one or more easily avoidable blunders eat up cash like crazy - check

    My own poison was Tesco and while I have many fond memories of the company I still shake my head at how gung-ho they were about the off-shoring process. After the accidental deletion of the entire UK user tree in AD followed by the not so accidental loss of the relevant log files and the discovery that the backup procedures hadn't been followed that THEN they might actually have learned something but no..... that was the signal for me that I'd be better off elsewhere.

    1. Anonymous Coward
      Anonymous Coward

      Re: Deja Vu

      Mumbai lads took over English Support, then, not 6 months later, we get asked to fix the backlog .... 1 month of backlog because the Mumbai call centre boss grabbed the cash and left ... Mumbai office closed "unexpectedly", took 1 month to figure out it had happened... LOL.

  8. Anonymous Coward
    Anonymous Coward

    Offshored email

    The reason for the gmail address is that the email systems in India are unreliable and, therefore, offshored to Google... which is , of course, very reliable....

  9. aussieroof
    FAIL

    CA7 not at fault

    CA7 has been around for many years - i was using it in the 90's. It works and performs very well, provided you know the tool and you know the environment. it is uniquely mainframe.

    Not only did they give it to people who dont know how to use it nor do they know what their environment is, but they scheduled this upgrade on a tuesday. Had they done this on a saturday after midnight they would have at least had the sunday to back it out.

    This is incompetence at its grandest.

    It also proves... you get what you pay for.

  10. Anonymous Coward
    Anonymous Coward

    I used to work for them too....

    I was at Direct Line, Croydon when RBS shifted it all off up to Gogarburn, c2003/4

    Relocate to Edinburgh? No thanks, next stop probably Mumbai, could see it coming even back then.

    Initial redundancy offer was the state minimum - from a company (DL) making over £1M a day in profit!

    Whereas RBS staff T+Cs = far more lucrative.

    Make us RBS staff first then? NO.

    And they wondered why we had no enthusiasm for knowledge transfer.

    No disrespect to the Edinburgh lads - they were a decent bunch and our situation wasn't of their making.

    And re RBS hardware investment - Can we have STK tape robot/silos please? NO. But do have this unreliable hunk of junk instead. Whatever it was, nasty thing. Probably cheap though.

  11. Anonymous Coward
    Anonymous Coward

    CA-7 ?

    CA-7..?

    It was Control-M when I worked for one of the RBS companies.

  12. Alien Doctor 1.1

    Did anyone else...

    read the contact name on the job advert as "krash me live jobs"?

  13. Anonymous Coward
    Anonymous Coward

    Too many Ops staff

    TSB

    Mid 1990's, some mid-management fool asked why we have 24 shift ops staff.

    Well, dimwit, it's 4 shifts of six staff.

    But why four shifts?

    Well, dimwit, would you want to work more than an average of 8hrs a day, covering 24x365?

    24hrs = three 8 hour shifts, plus 8 hours off, which = 4 shifts.

    1. Anonymous Coward
      Anonymous Coward

      Re: Too many Ops staff

      >> Well, dimwit, would you want to work more than an average of 8hrs a day, covering 24x365?

      >>

      >> 24hrs = three 8 hour shifts, plus 8 hours off, which = 4 shifts.

      What pattern did you work 3 on 1 off? I work 12 hour shifts, so we have 2 x 12 hour shifts plus 2 x 12 off - still works out as 4 shifts. We work for days (or nights) on and 4 off. Means longer days, but fewer trips to work (so less wasted on travel), also we avoid rush hour traffic.

      BTW, I'm not with NatWest or RBS, however my wife's employer is - and she was due to have been paid her monthly salary last Friday.

      1. Richard 41

        Re: Too many Ops staff

        What pattern did you work 3 on 1 off?

        Eight week pattern, meaning, by average, only one weekend a month.

        WK1

        Sat + Sun 20:00 - 08:00

        Mon-Fri 00:00 - 08:00

        WK2

        Sat+Sun OFF

        Mon-Fri 16:00 - 20:00

        WK3

        Sat+Sun OFF

        Mon + Tue OFF

        Wed-Fri 08:00 - 1600

        WK4

        Sat+Sun 08:00 - 20:00

        Mon+Tue 08:00 - 16:00

        Wed-Fri OFF

        WK5

        Sat+Sun OFF

        Mon-Fri 00:00 - 08:00

        WK6

        Sat+Sun OFF

        Mon-Fri 16:00 - 20:00

        WK7

        Sat+Sun OFF

        Mon + Tue OFF

        Wed-Fri 08:00 - 1600

        WK8

        Sat+Sun OFF

        Mon+Tue 08:00 - 16:00

        Wed-Fri OFF

        ....and repeat, for very nearly a decade until Lloyds took it all away, redundancy #1

  14. Revelationman

    Why outsource in the first place saving money give me a break

    How many on you on this discussion love calling a overseas call centre love giving your details or knowing in this case your sensitive banking information is somewhere thousands of miles away.

    The Lord of the Banks say we are saving money, yes you are but now this will cost you millions of pounds to fix the potential loss of customers could damage the bank for years. The common sense approach is to bring back all IT services back to the UK were they belong, bring back the jobs that were taken,

  15. FreeTard
    FAIL

    Still not fixed

    I'm with ulsterbank (natwest/RBS) and my wife's pay has not arrived at all.

    My pay is supposed to arrive on the 28th but I'm not convinced it will, as none of te other transactions are working. I have four accounts with ulsterbank and none of them seem to be working as before this fiasco.

    I bet the feckers charge me for late fees on my ulsterbank credit card, let them try.... I've never failed to pay it in full before!

  16. Anonymous Coward
    Anonymous Coward

    jObScHeDuLinG

    You know, it is better to use one brand of software in scheduling across the landscape, especially if you can have one central point of control. I guess the sales guyz of the other schedulers will be knocking on RBS's door soon. I want run my jobs, total control, no maintenance costs, no downtime ... one of them must have the solution ...

  17. Anonymous Coward
    Anonymous Coward

    https://en.wikipedia.org/wiki/CA7

    responsible for (among other things) the production of hot air and spit.

  18. Nick G
    Flame

    And another thing...

    Whilst it's all very fascinating hearing about peoples issues with outsourcing (& I have my own experiences...) nobody seems to have touched on the other major show of incompetence, namely - who, in the name of all that's holy, schedules an upgrade to a major finance system in the middle of the week?

    I'd love to hear the reasoning behind that!

  19. Anonymous Coward
    Anonymous Coward

    This sounds more like the real story...

    I didn't believe they could be so dumb as to do an upgrade midweek. These comments tally with what their spokeperson said on Moneybox last Saturday: basically it was the backout on Tuesday that caused the problems. These posts spell out why they needed to do the backout, following an upgrade at the weekend that caused problems on Monday.

    The following is taken from a number of posts on the westhamonline dot net forum by a poster called CTM who is currently working on the Natwest/RBS fiasco and who was due to be made redundant 2 weeks ago. The most worrying comment is that 'The backout caused the "queue" of jobs for that batch to be unexpectedly deleted. '

    "Ok, since I work on the mainframe at RBSG, I'll let you know some facts - I'm currently on my 5th 12 hour shift in 4 days (and am quite busy so won't be back here for a while)

    It is a mainframe problem and not a virus It was not deliberate by the bank

    It was a third party product that was upgrading that caused a big whole in the batch schedule

    RBS wasn't actually affected directly as the schedule is on a different LPAR - however focus was on NWB & UB and normal everyday errors on RBS side were not getting dealt with

    COBOL and Assembler on the Mainframe, like most big banks. But it's not the application code that is the problem as I stated below. CA7 was updated to a new level last weekend. It was causing problems with the schedules on Monday so was backed out on Tuesday. The backout caused the "queue" of jobs for that batch to be unexpectedly deleted.

    Not sure why DR wasn't invoked to be honest, but it wouldn't have made much difference as we'd still have been hours behind on our batches.

    All transactions made on Tuesday and since WILL be processed eventually.

    As for the account thats being sold to Santander - you can just phone the number on your letter and you won't be moved, so stop fucking whining like you've been made to do it. We could have done without the work and expense that was foisted on us! Anyway, there's a pot of gold at the end of the RAINBOW. SWT

    Also, in my previous posts, the "not sure" was my own, not the banks - I'm sure that the discussions were had about DR and there must have been a good reason why it wasn't invoked. In truth DR is great in certain circumstances if it works, but in this one it wouldn't have. Obviously i can't go into too much detail here, but although the outsourcing to India did not cause the intial problem, the recovery has prompted questions to be asked as to who thought dumping over 50% of the UK support staff was a good idea - indeed people made redundant less tha a month ago have now been offered short term contracts to come back and help. Unfortunately, the balance sheet and pressure from the government to become independent again hasn't helped in the outsourcing/offshoring decision and I speak as a someone who was due to have been made redundant two weeks ago."

    1. Chilky

      Re: This sounds more like the real story...

      Of everything that I have read so far, this seems to be by far the most plausible explanation. Combination of upgrade not working and then the decision to roll back causing a serious error.

      From (admittedly back in the 90's) experience, you can recover if you lose the CA7 data. But only if you have a bunch of guys in the computer room who have worked on those batch schedules for years and know them off by heart. You can't simply re-create without them. However, they are generally the first in the chopping line as on a resourcing spreadsheet you simply see very expensive resources doing something that the management consultant doesn't understand.

      Again from experience of working in IT in retail banks, the business / management neither understand nor respect IT. As such they are dealt a crap hand every budgetary cycle and simply have to make do. Until the business understand that they are dead in the water without IT and start giving it the money it requires then this event is going to be repeated over and over again.

  20. Anonymous Coward
    Anonymous Coward

    Outsourcing...

    A few years ago I was made redundant when my job role was offshored to Chennai, India. It was a nice investment management company with circa 500 people and visions of an IPO. I had been at this company for 14 years. The knowledge transfer guys were pleasent and nice guys with a little, but not too much knowledge on the subject (the subject being, for me Information Security). Their production of Run Books for their offshore team, however, wasnt.

    This is my point - the high turnover of the engineers assigned to a customer contract means they have the poorest of documentation that they themselves have created as their only knowledge of context of product use. It's not the technology so much, as how it's used in the environment. That, is of course when things are working. Like outsourcing, it's only great when it actually works. And when it doesnt then you're given the scenario that has RBS/Natwest without transaction services - the absolute core of their multi-million pound establishment, in ruins.

    I'm thinking about creating a meme on this, with that guy from Anchorman:

    Outsourcing:

    <insert jpg>

    60% of the time, it works everytime

    Why AC? Moral of the story I guess. The company in question expected a launch at 300p, launched at 220p and dive bombed to 70p... all a result of mis-management by those same people who thought outsourcing 25 people to India the great company I worked half my life for and had existed since the early 60s is no more.

    1. Anonymous Coward
      Anonymous Coward

      Re: Outsourcing...

      Once Upon A Time, in a land Far Far Away, there was a publicly-listed networking device company that closed a sh*t-ton of offices globally including one or two newly-built and taken over from competitors. They outsourced quite a bit of development to India and the stock price promptly nosedived, the only thing that saved them from circling the drain quite so closely was canning the India office immediately and taking themselves OFF the stock market.

      Thus ends the Parable of The Stupid Fat Hippy Buddhist Boomer and his New Office.

      This parable may or may not be related to the Parable of The Brown Envelopes and the Parable of The Developer That Wasn't and The Stockholder Who Was. All rights reserved. The names of people and places may have been changed to protect the innocent, if there were any.

  21. This post has been deleted by a moderator

  22. JimmyPage Silver badge
    Pirate

    Closing your bank account ....

    may I respectfully suggest a better way to vent your ire, is not to close your bank account, but to simply open another, and reduce the balance in your NatWest account to (say) £1.01. Make sure you are signed up for monthly statements. By post.

    I did this 16 years ago when Barclays ****ed me off. Every month I get a statement. Last year they almost twigged that the account may be dormant - I got a warning letter. So I shifted £5.00 in, and then a day later £5.00 out.

    Just make sure there's no overdraft availale on the account.

  23. The_Equaliser

    Governonce

    If your outsourcing governance is solid which for a mutlinational bank it would have to be then your outsourcers would not be in a position to make any errors

    OR........

    You could hawk it out to the cheapest bidder and use the money you "saved!?!?!?!" to bonus up your senior management

    I wonder which of the two approaches RBS is using ?

    Also I wonder what will become of the onshorebased management team for the outsourced departments. (New yachts for everyone from what I can gather).

  24. Anonymous Coward
    Anonymous Coward

    Hey, give me a toy inflatable globe of the world and a marker pen, and I can show you know what the "defective part" is. I don't need an ouija board either ;)

Page:

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019