Brief Natwest's online, mobile and telephone banking services are having an unscheduled rest today, Saturday, November 5. At time of writing customers who try to log-in to their accounts are greeted with the following message: We're sorry our Digital, Mobile and Telephone Banking services are currently unavailable. We're …
Not a bug but a feature
When I was on their banking website yesterday, there was a prominent notice during the login sequence announcing this morning's maintenance shutdown.
That's right, but it was only 1am-5am...
Strangely enough, the banks have been forcing cut after cut in the rates of their IT operations contractors recently. What a coincidence!
Mind you we can be assured that the casino gamblers will not see any cuts in their £million pound bonuses in a couple of months.
Yup, I work with several finance companies, and all of them are shedding IT staff left right and centre. Unsurprising that they've got the people who cost the less, who are overworked developing stuff at the moment - and then we get fall over after fallover.
You can expect Lloyds \ HBOS to be next, if it's a similar deployed package that many of the banks that have been working on to cause things to collapse soon.
I've seen a couple of people comment on Facebook that they have been unable to take out or deposit money into their accounts. I do not know if this includes using a debit card for payment though but it should be taken into account when shopping, just in case.
This problem did not commence at 7 am as you stated, scheduled maintenance in the form of a large software update commenced at 01:30 this morning and at circa later the message reminding their customers of this was replaced by the current message. I have been informed by Natwest that all accounts are frozen as they were yesterday and that the problem is affecting their systems globally.
This suggests that the problem lies deeper than the web Customer Self Management and Customer Self Management and Customer Relationship Management systems that the automated and advisor assisted telephone banking services use respectively.
Testing shall commence at 12:00 and while unable to confirm a time for the services to be restored their is a rough estimate "of some time this afternoon".
For those of you like myself who through a quirk get paid today with direct debits also being taken out over the weekend it shall be interesting to see how this one pans out.
Source: My insomnia.
Should read "circa 05:45" not "circa later".
Correction to correction
Read the article - there is nothing to correct:
"According to Reg readers, the service was unavailable at 7am."
It merely states that there are reports the service was not available at 7am. It doesn't state the outage started at 7am.
Later, some people might report they found it unavailable even earlier than 7am (as seems to have happened in the comments section). This does not make the earlier statement incorrect.
Also, there is a difference between schedules maintenance - which according to several sources, was supposed to have ended sometime in the early morning - and what seems to have turned into an unscheduled event - to end later in the day (or even later than that).
Just because the initial outage was scheduled doesn't make the whole of it so. As soon as it outruns it's designated time slot, it automatically becomes unscheduled downtime.
A technical argument for managing customer expectation is always due to fail particularly in banking, where people are actually unable to travel home because there debit & credit cards are not getting authorised.
While I value you technical opinion the gross majority of RBS/Natwest's customers see the customer experience, not an ITIL breakdown of pain points in an SLA.
RBS is also down
The Royal Bank of Scotland is also down at the moment with the same message. Natwest is now just a part of RBS so it makes sense that they are all down together.
Frustrating stuff, but as long as my money is still there after I am ok about it. just a few years ago we didn't even have ibanking etc. so whatever.
Of course it is...
They're on the same physical back end hardware.
Is there anyone who doesn't think there is more to this...
i'm not buying that it's coincidence.
there is something afoot. something mysterious.
Yes, I was wondering that. If it happens to another bank tomorrow, my suspicions will deepen!
yes, two in the same week does seem like enemy action
Re: Enemy action
Foreign government, motive, oh hai, just noticed your door was open and was curious.
Al Quada, motive, plunge the infidels into chaos.
Anonymous, motive, vengeance for wiki leaks / 4tehlolz
Organised crime, motive, cash money.
Bond super villain, motive, world domination.
Government, motive, big brother upgrades.
Software developer, motive, none, unintentional, banks systems unable to cope with ligitimate user requests from new smartphone app.
Banks offline, SSL cert authorities pawned, RSA encryption seeds stolen, Military defence contractors burgled, nuclear reactors sabotaged, something new and exciting every day.
Script kiddies, organised crime and governmental hacks all blurred together, exploiting networks and systems that were designed for utility rather than security.
This will end badly.
In-branch ATMs seem to be working around 10:30am this morning. But external ATMs such as Link machines were refusing to process transactions. Haven't tried paying by debit card as yet, but grabbed a wad of cash from the ATM just in case.
We would like to apologise for any inconvenience caused.
Any? It's self-evident that it's causing inconvenience. The policy where I worked was to apologise for THE inconvenience caused.
Go back and read it again:
Apologising for _any_ inconvenience means that they're apologising for anything that may be the fallout of what's gone wrong, not apologising on the off-chance that you're a bit hassled.
That will be why they are queueing out the door of the local branch then. Couldn't happen to a nicer bank.
It's quite refreshing that there hasn't been the usual calls of "can't they afford to put in redundant systems" in both these comments and the HSBC ones.
Is probably in place, I know where I work we have rolling business continuity testing. However getting it to fallover seamlessly in an actual emergency situation is another story altogether. Often it works fine others it can take a few hours. You can't predict every part that will fail, just have to practice with the scenarios with think might and deal with others on the day.
There is probably no redundancy per-se at this point in time - the systems being upgraded will be separated from the DR site, with the DR site as failback, there will also be snapshots of disks at the production site. Having a failback site doesn't mean that it'll be quick though, it's highly likely that many systems are having changes applied to them, so failing back manually is a complex procedure.
Having been present on several recovery management calls at RBS, I'd say that they will almost certainly be backing out. Fixing forward is not something that is done at RBS, unless there is no alternative. If they go off SOE (Schedule of Events) the decision is more often than not to follow the backout process, if at all possible, as fixing forward brings unknowns into a meticulously planned change.
Anyway, as mentioned, I previously worked for RBS, until my job went to India. We were having problems with the quality of staff in India then and from what I've heard from people still there, it's not got any better. I wonder if this is the new management reaping their just rewards?
Will please some one confirm if it actually does work if you pay by debit card regardless of balance seeing i had a payment due in for today? i have been to my local cash point 4 time this morning! as i have 3 crazy kids who want to go shopping!
hiya if you hand funds from yestoday it will work if they were due to clear today then i am afarid not as my dd card did not work in asda and i should have my wages in there today :-( rang up and thy told me no funds have cleared yet till things are up nd running again
All in the timing?
A lot of stuff going on at the same time - BBC main page, HSBC, NatWest/HSBC.
Good for conspiracy theories.
My conspiracy theory is that all the banks are scheduling their upgrades to take place at the same time so if it all goes nipples to the sky no one bank stands out as incompetent.
On the other hand this may be a global conspiracy which will make the occupation of St. Pauls seem like a minor event.
All seems up & working.
Just used (NW) debit card at a local co-op & all went thru fine
yeah but if my balance was 0.00 due to myself transferring what was left to my savings, and i receive regular payments into my current account on Saturdays, can i use the debit card still regardless of my current balance shown as of yesterday?
my balance was 0.00 as of last night and a payment was due in this morning, as natwest haven't updated. if i go to the local store to pay by debit card, will it go though? or will it say insufficient funds as reflected before the outage?
off down to shop soon, will try it out many thanks buddy
i am a customer with NatWest and I am due to be paid today i am taking my children on holiday today buy until they fix this I am unable to go anywhere my children one of whom has learning difficulties will be left disappointed if they do not sort this out what where they thinking doing any changes to the system on a day like bonfire night when people need their money all money hits NatWest accounts at 3.30 am so why didn't they carry out the maintenance after 4am .
Still down - and there was me going to transfer some money from one state organisation (Natwest) to another state organisation (HMRC) ... presumably I can reclaim the interest of the corp tax I couldn't pay today.
Natwest/RBS going tits up
The fault is online banking, telephone banking, cash machines and paying for goods via debit card and credit cards. I suppose to pay my rent today and haven't been able to get to my bank account since 1.30am. I can't even buy any food at my supermarket and might have to borrow some money of my mother in law to get some food to feed my family.
The person who is in charge of this so called technical department should be sacked and also who authorised this update on a Saturday morning should be sacked. I thought updates like should have been done on a Sunday morning so minimise problems like this.
Your rent is due on a non-banking day?
Have you called RBS and asked if they can authorise an emergency payment - they can even do them out of ATMs with no card, so you don't even need to go to a branch while it's open.
When would you rather they do essential upgrades? During the week? Maybe they need to carry out two updates to systems over the weekend, it's not unheard of. Possibly they needed to reserve Sunday for contingency, in case of any problems? There are also constraints with backups and the amount of time they take to run. RBS have world class uptime and their changes very rarely go wrong in such a way which would be visible to the customer. They don't do changes without meticulous planning, so I'd imagine that a Friday/Saturday change was essential in the larger scheme of things.
Wonder if they will compensate for the inconvenience caused. They are quick to charge you if you stray over your O/D limit.
RBS Mortgage system also unwell
When I phoned the RBS mortgage centre this morning, they were unable to help due to the mortgage system being down. Its all going wrong!
It's probably middleware that is down, rather than back end systems.
are continuing at the branches. All systems are down so you can't deposit or withdraw at a branch either
Credit where it is due.
A show of hands for the call centre staff who have been fielding irate customers all day, with Natwest management not saying anything apart from the ambiguous statement on the site disussed in this thread.
An update is due at 18:00. I wonder if there is a connection with the 3D secure compromise they had a week or so ago. Natwest meets Sony, now that would be fun.
It's still down at 5.30pm
must be costing them alot of money on the phone to their IT staff in India.
HSBC Common packages
What do RBS and HSBC have in common as regards Hardware and Databases?
Once is happenstance, twice is coincidence
Three times is enemy action.
You know I have to wonder about the decisions made by the UK banks in regard to security. My bank, HSBC, has just changed its security system to use a small PIN generator device from China. Now its not too hard to put in a few little extra bits in order to enable some sort of 'nullify and reset' signal, or even, given that Chinese companies (and government) have the algorithms to hand, a brute force type of attack. Given also that from my conversations and letters to him, the security head of HSBC, is either a Oscar worthy actor or as dumb as a stump, whats the odds that they've checked the devices properly?
From my point of view; given the chance to bet £10 on a bank paying out money to have the things taken apart and checked from top to tail, or Commander Sir Samuel Vimes Vs a Quasi-Demonic entity of extreme power; I have to say I would be voting for the guy in the chain mail...
Now, how long do you think it would take for our nice tea drinking, cricket loving civilization to turn into a cutscene from Halo:Reach if all the banks that used a similar system that was obtained from China (and whats the betting thats alot of them) were all hit at once. People cant get money for food, bills or petrol/Diesel - They cant get paid, and so on. I know there are other systems in place - but the thing there is that the banks have automated so much, that what was the entirety of a bank 20 years ago is a shadow of its former self (the physical staff etc) and it will *not* be able to cope for an extended period alone. Thats not even to mention the additional havoc that could be caused by pulling a stunt like that in the middle of Great Depression II...
I know it sounds paranoid, but the thing about being paranoid is its like being a pessimist. You are never ever disappointed.
if the Chinese could pull something like that off for an extended period then you wouldnt even need a nasty war like was going on in Libya. You just wait a little while and then saunter down and pick up the pieces after the countries you want are practically in the middle of their own little civil wars.
Even the most elementary tactics tells one that you don't build an army with only one type of unit, you just don't do it. Similarly, you don't put all your eggs in one basket when it comes to bank security - and I wonder how many people, when they look at the their security pin device see 'Made in China' on the back. There have already been cyberattacks traced to China mentioned in the media, given that we probably only hear about 1/10th of the total that happen, Im not happy about the odds that the 'system failures' we hear about are as innocent as they seem...
Time to find me an old sock to put under the mattress methinks.
heads up - alliance leicester and possibly Santander next!!!
Saw this when logged in:
"The service will be undergoing essential maintenance on Sunday 6th November between 3:45am - 7:00am. Customers may experience some disruption. We apologise for any inconvenience this may cause. "
What are the banks up to I wonder....
@ Militant Rodent. Co-Op Debit Working?
Stocked up on fresh fillets of barbecued sauce Cat then?
Fed up of BBQ Cat...
So I looked in the Ghanaian isle and picked up a nice Chihuahua & Gari dish for my sunday lunch... yum !!
Online banking back up (for now...)
Have finally been able to login. Seems to be working, but for how long?...
Also, I wonder whether us Nat West customers will get a proper apology and explanation?
To err is human
To really fuck up requires a machine.
I for one, will be asking the Royal Bank of Scotland for compensation for telephone calls, time wasted, embarrassment & the sum paiid on to a credit card that I am trying to pay off!