Feeds

back to article IT MELTDOWN ruins Cyber Monday for RBS, Natwest customers

RBS, Natwest and Ulster Bank customers were hit by an "IT meltdown" on Cyber Monday that stopped card payments, borked ATMs and closed down online banking, leaving them with no way to pay for anything. Problems started around 6.30pm yesterday as folks trying to do their Christmas shopping online and those looking for a few …

COMMENTS

This topic is closed for new posts.

Demand

To be fair, they're nowhere as intellectually challenging as pensions systems, but they are exceedingly high volume time critical.

If something goes wrong in a schedule they've become so overloaded demandwise that the impact of failures is way beyond what the original designers conceived.

0
0
Silver badge

Re: @Steve Todd

"I don't think they *need* to be anything like as complicated as they are, but I also think organisations like RBS, in particular, find it very hard to simplify things while keeping it all working."

So we're agreed it *is* the fault of RBS management past and present. In fact they've admitted as much in the press. Whilst we and they talk of "IT" failures, this is actually a failure of their core business, which is keeping transactions flowing and accounts balances correct through IT. Instead, the dogf*ckers who manage RBS have considered IT as an evil necessity, a support service, to be done as cheaply and nastily as possible. And meanwhile they have persuaded themselves that casino banking and reckless lending are the core purpose of the bank. In my view even regular "high street" lending isn't really the core business of the bank, it is simply a by product of holding account balances of savers and depositers, and sitting on funds whilst they clear.

Somewhile back one of the Reg articles referred to banking's core business as being an IT operation with a financial services business attached. That's self evidently been proven by RBS who chose to believe the opposite, and in which form it doesn't work very well. Will RBS admit their incompetence, bring IT back in house, onshore, properly designed and resilient? Somehow I think not, they'll just do the usual, in the form of unbelievably expensive sticking plasters that won't cure the underlying failings.

1
0
Anonymous Coward

The company I work for had a very large money transfer in yesterday. It's disappeared. Our finance guy is probably bald by now.

1
0
Bronze badge

or..

or lying back on the beach earning 20%...

3
0

Did they just roll back to the previous day's data?

It's shocking that people have seen money go missing. I mean, being unable to use your credit card or withdraw cash is obviously bad, but you can get around that by keeping credit cards from more than one provider. Seeing your wages vanish is another thing entirely.

4
0
Anonymous Coward

Re: Did they just roll back to the previous day's data?

Agreed on multiple bank accounts. This has saved me on a number of occasions.

2
0
Anonymous Coward

Re: Did they just roll back to the previous day's data?

The money won't be "missing" if they have rolled back, they will be scheduling an end of day batch to catch up, not all of these systems are "online" many are batch driven.

1
0
Big Brother

Re: Agreed on multiple bank accounts

forget multiple bank accounts, it's starting to look like stuffing money under the matress is the most secure option

1
0

... because you can run a bank on a shoestring budget with all your development out of India.

15
1
Silver badge

It might almost make you think that getting rid of all your most experienced (and therefore most expensive) IT staff is a bad idea!

5
0
Silver badge
Meh

It might almost make you think that getting rid of all your most experienced (and therefore most expensive) IT staff is a bad idea!

Nah!! - Been-counter general, RBS

0
0
Bronze badge

STOP complaining and be thankful

Just think how much worse it would have been if the management had not been so extremely incentivised to reward their brilliant work in serving their customers ... or even their stockholders!

5
0

are there any UK techs left who might want to mention what happened? obviously, anonymously. Having had previous at RBS, i just can't figure out what batch or online process could cause such a widespread problem and why the business continuity processes didn't kick in. Sounds horrific.

7
0

It does make you wonder what sort of hardware our banking network is running on.

I've read some horror stories that suggest some of it is built on very very old hardware indeed.

1
1
Eek

Re: It does make you wonder what sort of hardware our banking network is running on.

Suggests..... That was just careful phrasing.

I think just about the only bank I would trust nowadays is HSBC.... I've heard enough scare stories about the others that I wouldn't go there. That is not to say HSBC are any better it may be that they are competent or it could be that they don't employ people who frequent the same network of contacts I have...

0
0
Silver badge

Re: It does make you wonder what sort of hardware our banking network is running on.

Of course, it'll be a smogasbord of legacy code, built up through a dozen deadmarch projects spanning decades, held together with hope and fear. Can't go under it, can't go over it, can't rewrite it, have to bodge through it. Management and marketing will demand ever ambitious new features be bolted on top to match what they see on their IPads. Why can't our system intercommunicate using Twitter? Come on you lazy ass developers, I want a system upgrade pushed through tonight.

5
0
Silver badge
Boffin

Re: It does make you wonder what sort of hardware our banking network is running on.

There's nothing wrong with old hardware, or old software, as long as you have the processes and skills in place to maintain it and fix it.

Upgrades are often the cause of outages, so if it ain't broken, why rush to fix it (subject to statement 1)..

2
0
Anonymous Coward

Re: It does make you wonder what sort of hardware our banking network is running on.

Very new hardware, quite old software (in terms of custom code.)

2
0

Re: It does make you wonder what sort of hardware our banking network is running on.

I've not seen any indication of hardware failure in these incidents. It's all process or software related.

When I opened a bank account with RBS in my local branch in 2003 I found their office computers were running OS/2 (the banking app that the guy was using to open the account crashed and I saw the desktop). I suspect that's because the backend was an IBM mainframe and they were using one of the proprietary IBM communication protocols. The desktops (and ATMs, which also ran OS/2 for the same reason) have all been upgraded. You can see the result of the ATM upgrades on flickr and other photo sharing sites - Windows error boxes all over the place.

The backends? To be honest I'd probably trust a 20+ year old IBM mainframe that's under a proper support contract than I would a lot of the newer gear and newer OSs.

4
1
Anonymous Coward

Re: It does make you wonder what sort of hardware our banking network is running on.

You wouldn't believe what's being run on "very very old hardware" in this country! In fact, you'd be scared if you did know!!

2
0
Anonymous Coward

Re: It does make you wonder what sort of hardware our banking network is running on.

You wouldn't believe what's being run on "very very old hardware" in this country!

Bits of the London Underground still run on DEC Vax hardware - the real thing, not emulated - they've even been known to buy spare parts from eBay.

2
0

Re: It does make you wonder what sort of hardware our banking network is running on.

And bits of network Rail rely on Victorian cast iron. One bright young man has even proposed computerised signalling, but with the safety interlocking handled by said Victorian iron - cheaper and maybe safer than trying to do it in software.

2
0
Silver badge

OH NO I CANT SPEND MY LIFE POINTS ON INTERNET THINGS

3
3
Silver badge

I can understand fresh transactions being declined if there were IT problems. But how could already committed transactions suddenly disappear?

Enquiring minds wish to know.

0
0
Silver badge

maybe they rolled back to try and revert whatever changes were made and lost some recent transactions in the process, like nuclear option.

1
0
Anonymous Coward

Sounds very much like they intentionally denied everyone access to their accounts because of screw up with a batch process.

0
0
Silver badge

But how could already committed transactions suddenly disappear?

Easy. Your database goes tits up. You have a backup, you have a transaction file.

You stop the system, fix the problem, you revert to backup, and then tonight you will roll the transaction log forward and reinstate all the transactions you just erased. Probably duplicating the error that was originally made that cased the thing to crash...

4
0
Silver badge

I've never had any trouble...

... with Coutts Private Office.

But I accept that it may not be suitable for everyone.... :)

2
0
Anonymous Coward

Re: I've never had any trouble...

... who are RBS

0
0
Anonymous Coward

Re: I've never had any trouble...

I've never had any trouble ... with Coutts Private Office.

A whistle blower recently exposed large money laundering at Coutts, along the same lines as HSBC.

0
1
Anonymous Coward

Re: I've never had any trouble...

One would hope they launder the money, oiks have touched it

2
0
Anonymous Coward

while I understand the sentiment...

I really would, if, say, had I been waiting in the supermarket check out queue for 10 min, scanned the items, swept the card and then - bang: fuck off... I would still wish to point out, that it wasn't a calamity, a disaster or planet-wide tsunami that wasted 4 mln lives. Nothing anything of that type, just a 24 hr or so RBS failure (or fuckup, depending on how much you dislike RBS) - and the world moves on. So please, people, stop being hysterical, it's only Tuesday.

1
6
Silver badge

Re: while I understand the sentiment...

" So please, people, stop being hysterical, it's only Tuesday."

How about "no"?

It's two thousand and f***ing thirteen. Expecting one of the world's largest banks, backed by the entire largesse of the state, located in one of the most technically advanced nations on the planet to offer a reliable and continuous service is unreasonable is it?

In this rotten, festering POS bank there are about one hundred people paid over a million quid a year. And every year they are on the hook for something new - manipulating LIBOR, wilfully bankrupting business customers, mis-selling PPI for donkeys years, mis-selling interest rate swaps to SME's, misleading investors during the cash call before seeking a state rescue. And then there's the reason WHY the f***ers are owned by the state to be taken into account.

Face it: RBS & Natwest are useless. Has any other bank had repeated large scale screwups of customer service? Has any other bank been as routinely associated with incompetence, deviousness, dishonesty and fraud?

13
1
Anonymous Coward

Re: while I understand the sentiment...

Inside knowledge - one UK retailer attempted ~300,000 transactions during a three hour affected period, and ~120,000 were declined by RBS group.

Standing at a supermarket till is one thing - you can walk away from the goods - what if you've just filled your car with petrol... "Hello Mr Police Officer, I have no idea why my bank card has been rejected, and no I don't carry a second payment method"

The world has come to expect banks to work always. Perhaps they all need a bit more down time. (I remember the days they were open Monday to Friday 09:30-12:00 and 13:30-15:30). If you didn't get to the bank before half three then it was next day before you ate.

5
1
Anonymous Coward

Re: while I understand the sentiment...

Did RBS kill your puppy?

You're going totally off the deep end.

2
9

Re: while I understand the sentiment...

really no there are worst things but what has happened to some of us wellmlots of us has massive ongoing issues so if you want to kindly f off yourself and get back in the queue and scan yourself through the check out get in the bag and tie it up as tight as you can. idiot. how about making a online application for a loan for whatever reason they have nine applications on my account within minutes of each other which causes massive issues forget that the reason I required the loan and went for the 24 hr funds transfer in order to make a purchase that I will potentially now loose, but it ok I am still alive, but NatWest have carried out nine credit checks because that doesn't have any major implications.

really it has caused massive issues for hundreds and thousands. and the question is why? and not thank goodness it not a massive hurricane, or tsunami. but just ask yourself who will foot the bill for this little tiny little hiccup according to you.

1
2
Anonymous Coward

I know I shouldnt laugh, but twitter was far more entertaining than anything on TV last night because of this.

1
0
Anonymous Coward

Here we go again

Yet again the 'two pay days away from bankruptcy" brigade are out in force they can't use their electronic money because they have no cash. Sort it out! You've had since 2008....

0
2

Who's stolen our flag?

Britain's (quite clearly NOT Great Britain's any more) flagship economic sector is supposed to be banking.

But if we are so good at fucking this sector up, what hope is there? Anything else we are good at?

Looking forward to the rest of the 21st century -- we're all doomed. :-(

0
1
Coat

Re: Who's stolen our flag?

"Great Britain" means "the larger Britain", "Lesser Britain" being roughly the same place as Normandy these days. Nothing to do with greatness, sadly.

1
0

Its only money

The tax payer will...oh, yeah, haha! I see what you did there...

so I should send a note to the bank saying, "due to an unintentional IT related incident, I will not be able to pay my mortgage/loan this month, but am working to resolve the issue"

I am sure they will be absolutely peachy with that.

when you put more focus on profits than infrastructure of any kind, bad things eventually happen.

I wonder if anyone made a living crap ton of money from this "mistake", there usually is someone somewhere who does

3
0
Silver badge
Mushroom

Re: Its only money

"so I should send a note to the bank saying, "due to an unintentional IT related incident, I will not be able to pay my mortgage/loan this month, but am working to resolve the issue"

I am sure they will be absolutely peachy with that."

Err, yes, they will be absolutely peachy. Because you've already agree that if you pay them late you'll also pay them compensation (interest and late fees). And RBS have already confirmed they will do this in reverse for anyone who was affected - what part of COMPENSATION WILL BE PAID did you not understand.

1
2
Silver badge

COMPENSATION WILL BE PAID

So all Wonga/RBS customers should forward their correspondence to the boss of RBS to explain why they missed the payment on their loans.

2
0
Silver badge
Thumb Down

Re: Its only money

what part of COMPENSATION WILL BE PAID did you not understand."

They have not offered compensation. They have said they will refund people who are "out of pocket" due to the problems. I doubt they will be paying compensation for the embarrassment of having a transaction refused, especially if you just filled up £60-70 worth of petrol and are standing at the till.

2
1
Anonymous Coward

Re: Its only money

"I doubt they will be paying compensation for the embarrassment of having a transaction refused, especially if you just filled up £60-70 worth of petrol and are standing at the till."

Sorry but there is a reason cards are still embossed with raised characters. Can you guess why? Thought not.

0
0

downdetector is down - oh the humanity!

0
0

Failed sla ...

A 3 hr non scheduled outage , takes rbs into failed SLA country , which will result in a full explanation to the FSA.

The previous breach would have resulted in a similar report. (What a mess that was)

This time the same excuse or failure will not be thought of as a 'moving forward' situation.

Mindful of the steps detailed earlier , whilst generally they are correct , the issue here may focus on time to determine issue then fix , of a sufficiently detailed nature.

Outsource or no outsource , I have no doubt the root cause is known to RBS already.

4
0
Silver badge

Re: Failed sla ...

"which will result in a full explanation to the FSA."

Following the musical chairs the took the FSA away and made the FCA, I'd wager there's little continuity in enforcement.

0
0
Silver badge
Joke

Thank goodness Alex Salmond has is going to rid us of the Royal Bank of Scotland after Scottish independence.

What? He doesn't want it?

3
0
Facepalm

Completely unrelated statements...

1) RBS cut hundreds of experienced IT staff and outsourced to India in late 2010 in a cost saving exercise.

2) Online job advert in India, February 2012

"Urgent Requirement by RBS."

Looking for candidates having 4-7 years of experience in Batch Administration using CA7 tool,"

(CA-7 is a batch processing system, which deals with the movement of money in and out of accounts).

3) In June 2012 RBS managed to have thousands of transactions to the sum of millions of £££s not happen, this was apparently due to an issue with a CA-7 tool update.

4) RBS has had IT f̶u̶c̶k̶ ̶u̶p̶s̶ "issues" every 10 months since late 2010 that has so far cost them millions of £££s

According to RBS these are completely unrelated.

10
0
This topic is closed for new posts.