back to article IT MELTDOWN ruins Cyber Monday for RBS, Natwest customers

RBS, Natwest and Ulster Bank customers were hit by an "IT meltdown" on Cyber Monday that stopped card payments, borked ATMs and closed down online banking, leaving them with no way to pay for anything. Problems started around 6.30pm yesterday as folks trying to do their Christmas shopping online and those looking for a few …

COMMENTS

This topic is closed for new posts.

Page:

And Kudos to the scammers who quickly turned around a new spam email which asked me to download a utility to verify my account:

"Dear Card User,

We urgently wish to inform you that your

RBS card could not be verify due to some

systematic error. therefore we urge you to

download the security file and follow the

instructions.

Thank you for choosing RBS

Royal Bank of Scotland

"

3
0
Anonymous Coward

Resolved? I think not.

I transferred a large amount of money to my RBS account yesterday. It's now disappeared. What a fucking joke of a bank.

4
0
Silver badge

Re: Resolved? I think not.

"I transferred a large amount of money to my RBS account yesterday. It's now disappeared"

Then I suggest you (and all other RBS & Natwest customers) explore the new fast account switching service, ideally to a new banking market entrant or a mutual. Admittedly that doesn't avoid future IT related problems with your new provider, but it does stop rewarding the persistent failure of RBS (or endorsing their unethical behaviours by their "global restructuring group", pushing businesses over the brink to take control of the assets).

I left RBS in 2007 purely over the unethical behaviours that wiped out my then employer, andI've had no problems at all from Nationwide since then.

26
0

Re: Resolved? I think not.

+1! I also jumped from the useless Natwest to Nationwide nearly two years ago and have never looked back. And I get free European travel cover.

6
0
Anonymous Coward

Re: Resolved? I think not.

I left a "big four" bank in about 1995 to join a more ethical alternative.... the Co-Operative Bank. Look how that turned out! I am a little bit cynical that there is actually much difference. If your bank (or your utility company, etc) hasn't suffered an IT disaster, financial disaster, ethical disaster etc. yet, then maybe it is now overdue and will be the next to get egg on its face, be caught with its pants down, or run out of money.

3
0
Silver badge

Re: Resolved? I think not.

well, coop was new. why not go with something a little more established? like the commenter above pointed out.

0
2
Silver badge

Re: Resolved? I think not.

" I am a little bit cynical that there is actually much difference."

I think that's a very fair comment. But is that reason enough to stick with poor service or unethical behaviour? If you don't switch then the incompetent or dishonest not only continue with their existing behaviours, but you continue to reward them for it. Better to take a chance, in my view.

If you look at the state of the energy market, you might easily conclude that the market failings are due to the fact that most customers can't be bothered to switch "because they are all the same", and because of the linked issue of British Gas' dominant market share.

7
1
Anonymous Coward

Re: Resolved? I think not.

and on top of those, I also get a shitload of adverts of all kinds on my Nationwide accounts pages. Fortunately, careful (key word here :) pruning, i.e. blocking individual frames with adblock plus has made my visits to their spamsite what it should be - nothing spectacular, just... normal.

1
0
Anonymous Coward

Re: Resolved? I think not.

@Ledswinger "switch...ideally to a new banking market entrant ..."

Having worked for several old and new, I'd have to say DO NOT TOUCH a new entrant.

If you think RBS et al can screw it up, wait until someone who has NO EXPERIENCE has their first major outage. And they will all have a major outage, some sooner than others. Older mutuals at least have some track record in running an IT business. Supermakets?, not so much.

2
0
Silver badge

Re: Resolved? I think not.

"Having worked for several old and new, I'd have to say DO NOT TOUCH a new entrant."

Each to their own, But I work in a very large customer facing business. We have a strong track record, documented processes, oodles of resource, all of which make our service worse than new entrants and systems less reliable because the new entrants aren't weighed down by vast crappy legacy systems, and hidebound by years of "doing it this way", or a business model built on millking retained customers.

And in the case of RBS, have you noticed how the IT failures often seem to be ancient legacy code or processes that they no longer know how to operate or support? I have no illusions that new entrants, mutuals, or any other player could have their own IT disaster, but I wouldn't personally equate "new" with "worse".

3
0

Re: Resolved? I think not.

@NomNomNom (12:15)

The Co-Op bank was formed in 1872, and became a 'proper' clearing bank in 1975 - so it's not a new entity.

4
0
Anonymous Coward

Re: Resolved? I think not.

@Leadswinger - Of course the code is old - RBS' first computer was a LEO2, now they won't be running that code, but they will be running the code from their first IBM mainframe, why wouldn't they? They know the code inside out, the problem with the last outage was that the CA7 scheduling staff were let go, not anything to do with not understanding the underlying code.

If they rewrite the code in, presumably some new flashy language, there will be problems while it beds in.

3
1

Re: Resolved? I think not.

After reading this, I thought I ought to check my own RBS account and sure enough, the money that I transferred in yesterday is no longer showing.

As a result I phoned RBS, who were very apologetic but couldn't help in any way.

0
0

Re: Resolved? I think not.

RBS ... Is that not the bank that is deliberately bankrupting its own customers and buying their assets on the cheap from the receivers?

Oh Yes, it is:

http://www.nakedcapitalism.com/2013/12/ian-fraser-the-financial-terrorism-of-royal-bank-of-scotland.html

Good luck getting your money back, they probably snorted all of what the hookers left of it already

1
0
Anonymous Coward

Re: Resolved? I think not.

"RBS ... Is that not the bank that is deliberately bankrupting its own customers and buying their assets on the cheap from the receivers?"

That is under investigation and very much not proven. In the interests of balance, even Robert Peston said that this is way more complicated than that. For a start the companies that are alleged to have been deliberately failed by MS were already on the rocks and relying upon RBS' support at a time when the bank were under vast amounts of pressure to cut off credit from zombe companies.

1
1
Anonymous Coward

Re: Resolved? I think not.

That is under investigation and very much not proven.

Read Private Eye - they've been running articles on a number of cases where banks and the big accountancy firms have been putting companies into receivership then selling their assets when it's clear that the companies were viable. In most cases they were only experiencing temporary cash flow problems, often caused by the banks withdrawing long standing lines of credit (funnily enough, that was the original purpose of a bank before they got into exotic investments).

10
0
Gold badge
Thumb Up

@Ledswinger

If you don't switch then the incompetent or dishonest not only continue with their existing behaviours, but you continue to reward them for it. Better to take a chance, in my view."

Damm right. thumbs up for saying it.

I saw a statistic that only 35% of all utility company users switch.

And I think the proportion of people switching bank accounts is much lower.

AIUI that's a churn rate mobile phone companies dream of having.

Any talk about "customer satisfaction" or "Customer service" is basically b**lcks.

That will only change when customer realize that customer "loyalty" basically means the right of the bank to screw you sideways

1
1
EJ

Re: Resolved? I think not.

All the smart money has been moved to Bitcoins.

0
1
Anonymous Coward

Re: Resolved? I think not.

Why does Cameron think the CoOp deserve an inquiry into their leadership when the City in general are doing far worse? Wrong kind of bankers at the CoOp?

That aside, look at any survey of banking sector customer satisfaction over the last ten years, maybe more.

Top two, almost without fail, First Direct (HSBC owned, but mostly not HSBC operated where it matters) and CoOp/Smile.

There *is* a reason FD are always up there (I'm a longstanding customer, my experience matches the surveys). I'm an occasional CoOp customer too, and whilst I've experienced far worse , in my experience the CoOp simply don't come close to FD. CoOp are, however, streets ahead of e.g. Sandtander (always close to the bottom in the surveys).

As suggested above, pick a bank with a record of customer satisfaction (no guarantee for the future, but...).

Then try the fast switch process. What can possibly go wrong ?

FD are currently offering £100 when you switch to them, and if you choose to leave within 12 months, you get another £100. Small print applies, see

http://discover.firstdirect.com

http://www.youtube.com/watch?v=ZXHckAFMzaw

1
2
Silver badge

Re: @Ledswinger

Switching...? Bollocks. So who do you switch to - not necessarily in banking but in another industry, where you can count the choices available on one hand - after every single one screwed you over in various ways, quite possibly the same ones, so you don't assume but *know* that they're all *exactly* the same...? How about when you didn't personally tried them all but between you and your mates, there isn't a single choice without its own collection of first-hand horror stories...?

I don't know if there ever was a time when customer choice actually did have any sway over business practices, but in today's global economy and uniformly lowest-possible-cost maximum-screw-factor service there's exactly f***all anyone and everyone gets to punish / reward / change. There's nowhere to go but from the frying pan into the fire, and the players, safe and secure in that knowledge, ROFL at any customer indignation...

0
0
Anonymous Coward

Re: Resolved? I think not.

""RBS ... Is that not the bank that is deliberately bankrupting its own customers and buying their assets on the cheap from the receivers?"

"That is under investigation and very much not proven. "

Not proven to you perhaps. But I worked for a listed tech company that was put through the mangler by RBS GRG a few years ago, before the credit crunch. It was in difficulties and needed refinancing certainly, but RBS chose to squeeze the trigger, and take control. In rough terms the company had £150m of debt, and an equity value of perhaps £20-70m. By forcing it into administration RBS were able to sell it in a suspect pre-arranged transaction for over £200m. Even over above outrageous "professional advisor" fees from a couple of the big four accunting (sic) firms that I recall totalled about £13m, and £2m of "fees" to a board of RBS placemen for a few months work, RBS made around £40m of profit at the expense of the shareholders and unsecured creditors who got nothing. Other delicious ironies included the appointment of a certain dodgy US investment bank to "sell" the company out of administration, the same bank that had previously been the bookrunner for a bond sale that raised £40m or so from bondholders who saw their cash evaporated.

All of this has been raised with the chairman of RBS himself ("no, no, wasn't us, you don't understand, maybe sailing close to the wind, that's how capitalism works etc etc"). It has been reported to the City of London Police, to the FSA/FCA (who did nothing), and over two years ago was raised with those useless cretins Cable, Osborne and also the Labour party's various spokesmen. Cable says he hadn't heard anything other than rumours before the Tomlinson report, but Cable is a liar - I have copies of the emails sent to him in my inbox, because I know the complainant.

Also, you may wish to pay attention to the ad hominem defence that RBS and the government are running. Note the freshly started attempts to discredit Lawrence Tomlinson (eg I understand that there will be an FT article attacking him today - pink paper, not going to offend the banks, is it?). And Cable (fat, useless, old, and evidently working for the 1%) is also distancing himself from Tomlinson.

You can choose to believe that RBS didn't do this. But the f***ers have been caught with their hand in the till so many times I wonder why anybody in their right mind would not start from a presumption of guilt.

4
0
Gold badge
Unhappy

@DropBear

"I don't know if there ever was a time when customer choice actually did have any sway over business practices, but in today's global economy and uniformly lowest-possible-cost maximum-screw-factor service there's exactly f***all anyone and everyone gets to punish / reward / change. There's nowhere to go but from the frying pan into the fire, and the players, safe and secure in that knowledge, ROFL at any customer indignation.."

And as long as people believe they can make no difference they never will The 35% of utility customers who do switch have made a difference. Look at UK gas & electricity suppliers versus water companies who litteraly don't give a s**t.

In the UK there is a magazine called "Money Facts." It lists all bank accounts, credit card, mortgage, loan and other financial providers in the UK, and their interest rates and qualifying requirements.

Yes a lot of them are very similar, but some are not, and the differences can be staggering.

The system only has a chance of working if customers reward the good and punish the bad.

If you live in the UK that means you.

2
0
Gold badge
Unhappy

@Tim J

"The Co-Op bank was formed in 1872, and became a 'proper' clearing bank in 1975 - so it's not a new entity."

Except as fo 2013 it's now 70% owned by a bunch of US Hedge Funds and assorted "High net work" individuals.

Not really a "co-operative" any more, is it?

1
0
Gold badge
Unhappy

@fajensen

"RBS ... Is that not the bank that is deliberately bankrupting its own customers and buying their assets on the cheap from the receivers?

Oh Yes, it is:"

This is more ironic than you probably realize.

Someone did a study on bank ordered bankruptcy in the UK and found a)Banks ordered solvency reports from accountants b)Most reports called for receivership. c)Most receiverships were handled by the accountants who wrote the original report. d) However one bank tendered the receivership work if the accountant said the company was insolvent. That bank had a much lower level of companies in receivership.

That bank was the RBS, pre Fred the shred.

0
0

Agreed

Just one person's experience. But 25 years with 1 st. Direct. . Complex personal finances. Their standard non-premium telephone service - 100%.

1
0
Bronze badge
Coat

Re: @DropBear

The "Bank of Dave" documentary was very interesting as it pointed out that "banks" are actual legal entities in the UK.

Banking is not rocket science...

P.

1
1

Re: Resolved? I think not.

I wonder if RBS's issues with IT are down to the multitude of other banks it gobbled up, presumably each with their own special flavour of software?

1
0

Re: Resolved? I think not.

I've just moved away from First Direct due to their ludicrous new online banking login system and newly-introduced "foreign transaction fee" on Euro-based Visa purchases. Plus the fact that the First Directory extras are largely useless due to terms buried in the policy small print. Nationwide FlexPlus here I come.

0
0
Anonymous Coward

Re: Resolved? I think not.

"I've just moved away from First Direct"

Fair enough, best of luck. At one point I had accounts with Nationwide and with FD. Following an unresolved multi-thousand-pound foul-up where Nationwide failed to pass on the proceeds of an investment in a timely manner, I am an ex-Nationwide customer.

"[FD's] ludicrous new online banking login system"

I still see my chosen username (in full), enter a few characters of their choice (1st, 3rd, 5th,etc)from one password, and another security question. Keylogging not impossible but not immediately useful either. What ludicrousness should I look forward to?

"newly-introduced "foreign transaction fee" on Euro-based Visa purchases."

I didn't like it when they dropped Mastercard (some places I travelled to accepted Mastercard everywhere but not always Visa). So I got a Mastercard from somewhere else.

"First Directory extras are largely useless due to terms buried in the policy small print. "

May well be true, but amazingly enough I'm happy to pay a few quid a month for UK-based banking I can trust (I've dealt with Santander so I know the impact bad banking can have on me).

Mind you I don't buy my broadband from race-for-the-gutter suppliers either (hello TalkTalk). Maybe I'm in a minority.

0
0
Gold badge
Unhappy

@Mike Richards

"I wonder if RBS's issues with IT are down to the multitude of other banks it gobbled up, presumably each with their own special flavour of software?"

Given bank mergers have been (partly) cancelled because they couldn't get the IT systems to mesh (and give that "synergy" management con-sultants like to talk about) I'd say that could be a factor.

And those integration issues can take years to resolve....

0
0
Silver badge

Again?

Once? Ok, everyone has the odd hiccup here and there.

Twice? Really? Well hopefully they've learned their lessons now...

Again? This is starting to look less like random glitches and more like rank incompetence.

13
0
Anonymous Coward

Re: Again?

"This is starting to look less like random glitches and more like rank incompetence."

Yes, rank incompetence. Let's take 7,500 man years of site knowledge and walk it out the door, since technical skills are the same the world over and you don't really need site knowledge.

(for anyone who doesn't know, RBS moved ~750 jobs offshore, with those in the UK who were made redundant having on average 10 years experience INSIDE RBS). Bean counters only see bottom line costs - well they'll certainly be seeing how much the offshoring is costing in the delays to remedy the faults, the customer compensation and the FCA fines).

18
0
Silver badge
Unhappy

Re: Again?

I bet the RBS shareholders are seriously worried about the security of their investment. Oh, hang on....

3
0

Re: Again?

Get with the times: RBS is a "Systemically Important" Bank, this means that it has large, unbreakable, cash-umbilical right to the treasury and is pretty much immune from prosecution too. The taxpayers supplies RBS with capital, it does not need customers at all, in fact customers is the main driver in operating costs! It simply makes sense to drive customers away as much as possible!!

Systemically-Important banks exists only for the sake of their own importance - exactly as it says on the tin!

1
0
FAIL

Re: Again?

sell the assets to other banks, and revoke their license, as they are clearly not a fit and proper organisation.

0
1
Bronze badge

I'd like to know how this was even possible?

Surely there is some sort of backup system in place right?

Reading twitter last night - it would appear that even the call centres were offline - the phones rang - but never got answered.

They have promised to compensate people that were "left out of pocket" but that doesn't really make me feel better about them - most people were not left out of pocket - because they couldn't actually spend anything in the first place! I do wonder though how people coped at things like self-service petrol stations and the like - are we to be expected to check online-banking and Twitter now before we attempt to use our cards to make sure everything is working? Checking Natwest's online 'Service Status' page last night was pointless because it claimed everything was running normally - leading me (and thousands of people on Twitter) to believe something must have gone wrong with just my account.

Anyway - I discovered a lovely site last night which instead of just pinging their website, appears to scrape Twitter (and other networks?) to give you the real story - http://downdetector.co.uk/problems/natwest quickly confirmed that it wasn't just me having problems.

3
1
Anonymous Coward

Backup systems

There are backup systems, of course. However, failing over to a backup system has its own costs and risks, particularly when dealing with the systems which hold things like account information. So it can be perfectly reasonable to decide to try and fix the problem on the current live system rather than fail over to the backup / DR system.

1
0
Anonymous Coward

Of course there are backup systems (by which I presume you mean DR systems) but you don't always want to move to a DR system. RBS is an extremely complex computing environment, probably more complex than most people commenting here can begin to comprehend. There are a lot of reasons that you wouldn't automatically fail over to a DR site, many companies elect to manually fail over so that they know exactly what is happening.

Here is one scenario where a DR system would potentially take hours to bring up:

1) Production system fails, let's say by a corrupt filesystem

2) due to the synchronous replication of the disks the filesystem corruption is instantly replicated to the DR site

3) The system needs to be taken offline, services/daemons stopped, failed filesystems unmounted etc.

4) The failed disks are recovered from start of day snapshots

5) The logs are replayed up to the point of the corruption

6) The system is checked to make sure that everything is working

7) It's put back online again.

That sort of thing takes quite a while.

12
1
Silver badge

"RBS is an extremely complex computing environment, probably more complex than most people commenting here can begin to comprehend."

And evidently more complex than many of the people working there can begin to comprehend, to judge by their history of IT out(r)ages.

11
1
Anonymous Coward

By "history of" you actually mean two. One of which was really bad, the other was a few hours of cards not working, hardly the end of the world, although within RBS it will be an arse-kickings all round type of problem. Cards, ATMs, Mortgages, New account opening and Online (bascially anything customer facing) are considered cardinal sins to take out, even temporarily.

No one person understands everything at RBS, this was made worse by the redundancies, I know that I took 7 man years of knowledge with me, but that does not mean that they're an incompetent shower as you seem to be suggesting.

2
2
Silver badge

I always have a good chuckle when someone (usually someone who has no experience of other environments) tries to tell me how unbelievably complex wholesale banking systems are. Fundamentally, they're just subtracting X from account A then adding X to account B and writing a journal record - that's not particularly complex, even for a financial services business. What people generally mean is that these systems are very large, which is true - tens of millions of accounts and tens of millions of transactions a day - but that's a problem of capacity management and scaling, not coding.

Of course, the banks (like many long-established environments) can make very simple systems into very complex ones by adding layers of new architecture to mask the limitations of old systems (usually old systems that people are afraid to touch because the last person who understood them left 10 years ago). That's understandable, but not something to be particularly proud of.

6
9
Anonymous Coward

"they're just subtracting X from account A then adding X to account B and writing a journal record"

No, no they really aren't. That you snigger about the fact that you think the people working in financial service IT make it sound more complex than it is, then suggest that it's that simple suggests that you have literally no knowledge of the inner workings of a large financial institution.

10
4
Anonymous Coward

Or simply: They did not have any money in their accounts so they have to wait until salaries and pensions are deposited? In Denmark we experience a "bank IT problem day" just about every third month *always* on payday!

1
5
Silver badge
Stop

@Chris Miller

You've described a small part of the functionality of a general ledger. That is connected to MANY other systems (for example links to SWIFTNET, FPS, credit management, reporting, direct debits etc.) ONE of those links is to to ATM network. The failure could have been due to anything from network, through hardware, the interaction of existing bugs or release management (though one would hope that they weren't releasing new software on what was expected to be one of the busiest days of the year).

6
0
Silver badge

@Steve Todd

Nothing that you or your AC colleague(s?) have said begins to justify the statement that RBS has an: "extremely complex computing environment, probably more complex than most people commenting here can begin to comprehend". The fact that it has links to external systems distinguishes it from every other computer environment - how, exactly?

I guess it's the BSD syndrome so prevalent in banks that causes those working there to insist that their systems are complex beyond imagination, but that doesn't necessarily make it so. FWIW I had 25 years working in financial services, many of them managing equally (if not more) complex systems. And those systems in turn were far less complex than (say) a complete SAP implementation for a multinational conglomerate.

4
5
Anonymous Coward

Re: @Steve Todd

"And those systems in turn were far less complex than (say) a complete SAP implementation for a multinational conglomerate"

Hmm 'a complete SAP implementation' being one such system in a multitude of others for a large international bank. A nice pretty vendor supplied and run to vendor when things go wrong system, and pretty insignificant one at that in relation to the problem at hand.

Comparing SAP to core banking systems - built on layer upon layer of generational in-house development, consolidation, cost cutting, acquisitions, middle-ware layers, protocol wrap ups, across multiple platforms which you really don't find in too many places these days - Mainframe, AS400, VMS, Tandem (and of course integrating with your common old garden variety) UNIX, Linux, Windows and oh 30 + years - is in the nicest possible way, a bit silly.

5
2
Silver badge

Re: @Steve Todd

Oh dear, we've got a proper banker here. Completely clueless about the rest of the world, but never wrong ('cos I'm a banker). Have it your way. The rest of us are just far too thick ever to be able to work with your hugely complex systems (for which you're incapable of presenting any evidence). You do have the biggest dick. Congratulations.

And they wonder why their systems crash and burn so regularly ...

4
7
Silver badge
FAIL

Re: @Steve Todd

As it happens I'm an ex-banker. I spent 10 years working for the cash management division of one if the largest investment banks in the world. For 15 years before that I was a contractor, working for everything between a flight booking company through a water utility to the research division of a large pharmaceutical company. I think I'm pretty well qualified to say what is or isn't a complicated environment.

The high level system diagram of the bank's IT environment ran to a page of A2 IIRC, and that was with a small font. They had a whole system responsible just for cataloging the systems that they had, the technologies that they used, the systems that they connected to and their level of compliance with IT rules.

6
1
Anonymous Coward

Re: @Steve Todd

I expect your mind is made up but, as someone who has worked for retail banks and also in other large organisations: yes, they are complicated, very complicated. Remember this is not only a bank, its a bank which has eaten and partly, but only partly, digested the IT systems of a number of other banks, leaving the kind of semi-maintainable mess you would expect. I don't think they *need* to be anything like as complicated as they are, but I also think organisations like RBS, in particular, find it very hard to simplify things while keeping it all working.

3
0
Bronze badge

Re: @Steve Todd

Regardless of the argument about whether banking IT systems are complicated or not (I would prefer to call them "multiply-interfaced"), is it not the case that we have quite a number of banks other than RBS/Nat West/Ulster Bank whose systems seem to be much more reliable? Cue the people badly affected by the current débacle moving across...

Over the past dozen years, at least, RBS management has done a very thorough job of "wrong-sizing" by getting rid of the people who knew how their IT systems worked.. This policy works fine - until such point that something nasty goes wrong. At present nobody even seems to know exactly what has gone wrong, and the response was presumably simply to "turn everything off and on again"...

2
0

Page:

This topic is closed for new posts.

Forums