...another one ?
HSBC had a global card outage on the 31st of May, this year... I know, I was stranded in Wembley !
Barclays Bank is suffering an IT outage that has seen cash points refusing to pay out and online banking going offline, with some readers reporting that even fleshy-staffed branches are having problems. The problems started over lunch time, and it seems that anyone trying to get money from a Barclays account should try and …
failure to communicate with a datacentre, doesn't necessarily have to be an issue with the bank's IT systems.
What they need is some sort of system that lets them continue working locally, even if the communication links are down, then updates the system when they come back up, like we have :)
Your news yesterday ran with job cuts in IT departments at Barclays, so presumably some BOFH indulging in some rm -f style payback when given his papers, or a lack of skilled staff left with the ability to fix the problem because they have all been given the bullet...
I've just had to withdraw cash over the counter, which brought on a gentle tinge of nostalgia.
Paris, because she always has the personal touch
They probably turned off the workstation belonging to the ex-employee, that had the command line COBOL application running on it, that was responsible for managing the account synchronisation across the network. Now they're desperately running around switching them all back on again, in a bid to find which machine it was, since it turns out the only documentation for the job was in a README file sitting in the same directory, alongside the COBOL program, on the machine in question. Hopefully, all being well, it won't turn out to have been one of the old SPARC Ultras, that they sent off in a skip, this morning.
This is standard management practice. When you have finished destroying the business sack the IT department and then blame all your failures on the resulting mess. We should be flattered that in spite of all the abuse management do actually realise that we are the ones holding the business together.
Try getting cash back instead.... I have a feeling it defaults to the VISA part, or whatever, on a debit card if the bank is unavailable. A few weeks ago I was getting the can't connect to your bank message at all the cash machines I tried from various banks but buying a pack of fags and getting cashback in tesco worked fine.
I had the same problem on Saturday (13th June) I tried 3 different cashpoints at my local branch before giving up and going inside. 1st Machine gave a this card is damaged error, the other two let me check my balance but threw a wobbler when I tried to make a withdrawal (unable to contact....).
Hope it stays fixed this time.
My step daughter tried drawing out cash to go shopping for herdaughter's prom dress. Screen went blank just before dispensing cash. Result: no cash but debited from account.
Took her ages to get through, but they restored the balance straight away; so must have been a few with the same thing.
Not particularly robust software if a disk array going down causes that.
"presumably some BOFH indulging in some rm -f style payback when given his papers, or a lack of skilled staff left with the ability to fix the problem because they have all been given the bullet..."
I work for barclays let me assure you itll be the latter! I see people given a manual task with excel sit there and manually calculate one row at a time for thousands of rows when there is a very very simple beginner style formula that will do the lot in seconds, it's actually terrifying how little 96% of anyone there is pretty clueless when it comes to using a PC
You jest, but exactly this happened with their NTP time service a few years ago. Someone had an OpenBSD (I think) system (called Dougal if I remember correctly), sitting under a desk that was turned off, and their entire UNIX application farm lost their reliable time source (it may have been wider than this; I was only interested in the Unix servers). It was not spotted until NTP on the DNS servers (which were also distributing time as a peer NTP tier) started complaining about being unable to sync with their higher-tier time source after about 10 days.
Fortunately. the system was still present, and could be turned on again while they worked out how to remove it properly. Mind you, I was the local 'expert' on their NTP and DNS servers at the time, and I was a contractor. Says volumes about their own expertise, as I only have a passing knowledge of NTP. Think they use InfoBlox now, so I won't be doing that again.
I feel reasonably safe to talk about this now, as all of the people who may recognise me have since left Barclays when the Poole support centre was shut down. Still, black helicopters...
Not really one disk array. Actually a two hands-full of large HDS SAN servers, but it is true that several critical application servers will be on the same SAN server. HDS always told Barclays that their SAN servers had multiple-redundancy, so even when a High Availability solution was deployed on a single site for server resilience, both servers tended to use the same SAN server for storage. And everything else (FC cards, switches, links etc.) was duplicated for each UNIX system. Of course, this was pre-SVN. Don't know whether they have actually implemented this since I left.
Barclays used to have split-site with cross-site disk mirroring for their most critical systems, but when this became too costly, they started reducing the places where this was deployed. Apparently, (high bandwidth) X (long distances ) = lotsa money. Whoda thunk this strategy could cause an outage!