Posted Tuesday 16th June 2009 15:43 GMT
HSBC had a global card outage on the 31st of May, this year... I know, I was stranded in Wembley !
Posted Tuesday 16th June 2009 15:43 GMT
Barclays lay off more contractors
Barclays IT systems have a strop
Posted Tuesday 16th June 2009 15:43 GMT
Well, I guess all that head-chopping in the IT department are starting to pay-off...
Posted Tuesday 16th June 2009 15:43 GMT
failure to communicate with a datacentre, doesn't necessarily have to be an issue with the bank's IT systems.
What they need is some sort of system that lets them continue working locally, even if the communication links are down, then updates the system when they come back up, like we have :)
Posted Tuesday 16th June 2009 15:43 GMT
Your news yesterday ran with job cuts in IT departments at Barclays, so presumably some BOFH indulging in some rm -f style payback when given his papers, or a lack of skilled staff left with the ability to fix the problem because they have all been given the bullet...
I've just had to withdraw cash over the counter, which brought on a gentle tinge of nostalgia.
Paris, because she always has the personal touch
Posted Tuesday 16th June 2009 15:43 GMT
But will you still be able to go to Sainsbury's and buy a pint, sorry, half-litre of milk with your debit card and get cashback??
Posted Tuesday 16th June 2009 15:43 GMT
...Barclays haven't offshored* large parts of their IT department - oh, hang on a sec.
*they're not outsourced, they're offshored and then those staff are, quite aptly, referred to as captives!
Posted Tuesday 16th June 2009 15:43 GMT
They probably turned off the workstation belonging to the ex-employee, that had the command line COBOL application running on it, that was responsible for managing the account synchronisation across the network. Now they're desperately running around switching them all back on again, in a bid to find which machine it was, since it turns out the only documentation for the job was in a README file sitting in the same directory, alongside the COBOL program, on the machine in question. Hopefully, all being well, it won't turn out to have been one of the old SPARC Ultras, that they sent off in a skip, this morning.
Posted Tuesday 16th June 2009 15:43 GMT
Or is it any ATM when used by a Barclays customer?
Either way, rather shonky.
/me puts bet on it being a fat-finger moment.
Posted Tuesday 16th June 2009 15:46 GMT
Didn't Barclays fire a lot of their I.T. Staff last week?
VPN Accounts still enabled?
I.T. pic as "Where is their I.T." !!??
Posted Tuesday 16th June 2009 15:46 GMT
With all this recent cutbacks on IT staff, its nice to see the reality hit home.
Perhaps the systems gained AI and went and took a payout-cut before they were asked when there contracts came up for renewal.
Posted Tuesday 16th June 2009 15:48 GMT
Barclays cuts more IT jobs (7 January 2009)
Barclays to lay off more tech staff (11 May 2009)
Job cuts start to bite at Barclays (11 June 2009)
... Major system issues (16 June 2009)
Posted Tuesday 16th June 2009 15:48 GMT
My girlfriend is a cashier at Barclays and even the branches have collapsed. The best thing to do is to stay home and write an angry letter explaining they probably shouldn't have fired those 400 IT staff a few months ago...
Posted Tuesday 16th June 2009 15:55 GMT
http://www.dailymail.co.uk/news/article-1193440/Barclays-cash-machines-online-banking-hit-hardware-failure.html
Staff still not back though...
Posted Tuesday 16th June 2009 15:57 GMT
I heard it was part of their drive towards PCI compliance. All transactions have to be hand written and delivered to their data center by secure carrier pigeon to be manually keyed in within their secure network
Posted Tuesday 16th June 2009 16:11 GMT
This is standard management practice. When you have finished destroying the business sack the IT department and then blame all your failures on the resulting mess. We should be flattered that in spite of all the abuse management do actually realise that we are the ones holding the business together.
Posted Tuesday 16th June 2009 18:54 GMT
Try getting cash back instead.... I have a feeling it defaults to the VISA part, or whatever, on a debit card if the bank is unavailable. A few weeks ago I was getting the can't connect to your bank message at all the cash machines I tried from various banks but buying a pack of fags and getting cashback in tesco worked fine.
Posted Tuesday 16th June 2009 18:54 GMT
Loose a load of IT staff, and then have the system break... Who would have thought it!
I bet they laid off too many Indians and kept too many chiefs!
Oh well, at least they'll be able to have a nice meeting about it.
Posted Tuesday 16th June 2009 18:54 GMT
http://news.bbc.co.uk/1/hi/business/8103271.stm - apparently they had a faulty disk array... guess it might be unsurprising but one disk array for the entire network? talk about a failover failure....
Posted Tuesday 16th June 2009 18:54 GMT
my cash card withdrawal won't get back to my bank?
Posted Tuesday 16th June 2009 20:50 GMT
many of their offshore staff were involved in fixing this issue....or was it left to the heroic efforts of the few remaining onshore staff.....
Posted Tuesday 16th June 2009 20:50 GMT
17:45 - couldn't get money from the machines at tesco. At least it's the bank, not me!
Posted Tuesday 16th June 2009 22:03 GMT
I had the same problem on Saturday (13th June) I tried 3 different cashpoints at my local branch before giving up and going inside. 1st Machine gave a this card is damaged error, the other two let me check my balance but threw a wobbler when I tried to make a withdrawal (unable to contact....).
Hope it stays fixed this time.
Posted Wednesday 17th June 2009 09:57 GMT
they would learn.. remember what happened when they under-invested in IT in Canada? One bank nearly down the toilet (cant remember it's name for the life of me)
Posted Wednesday 17th June 2009 09:57 GMT
Whihc bank is that? I'd love to have a crack at an old fraud trick where you could suck the machine dry of cash in about 10 minutes
Pirates - hand over your jewelry too..
Posted Wednesday 17th June 2009 09:57 GMT
My step daughter tried drawing out cash to go shopping for herdaughter's prom dress. Screen went blank just before dispensing cash. Result: no cash but debited from account.
Took her ages to get through, but they restored the balance straight away; so must have been a few with the same thing.
Not particularly robust software if a disk array going down causes that.
Posted Wednesday 17th June 2009 12:17 GMT
"presumably some BOFH indulging in some rm -f style payback when given his papers, or a lack of skilled staff left with the ability to fix the problem because they have all been given the bullet..."
I work for barclays let me assure you itll be the latter! I see people given a manual task with excel sit there and manually calculate one row at a time for thousands of rows when there is a very very simple beginner style formula that will do the lot in seconds, it's actually terrifying how little 96% of anyone there is pretty clueless when it comes to using a PC
Posted Wednesday 17th June 2009 13:27 GMT
You jest, but exactly this happened with their NTP time service a few years ago. Someone had an OpenBSD (I think) system (called Dougal if I remember correctly), sitting under a desk that was turned off, and their entire UNIX application farm lost their reliable time source (it may have been wider than this; I was only interested in the Unix servers). It was not spotted until NTP on the DNS servers (which were also distributing time as a peer NTP tier) started complaining about being unable to sync with their higher-tier time source after about 10 days.
Fortunately. the system was still present, and could be turned on again while they worked out how to remove it properly. Mind you, I was the local 'expert' on their NTP and DNS servers at the time, and I was a contractor. Says volumes about their own expertise, as I only have a passing knowledge of NTP. Think they use InfoBlox now, so I won't be doing that again.
I feel reasonably safe to talk about this now, as all of the people who may recognise me have since left Barclays when the Poole support centre was shut down. Still, black helicopters...
Posted Wednesday 17th June 2009 15:27 GMT
Not really one disk array. Actually a two hands-full of large HDS SAN servers, but it is true that several critical application servers will be on the same SAN server. HDS always told Barclays that their SAN servers had multiple-redundancy, so even when a High Availability solution was deployed on a single site for server resilience, both servers tended to use the same SAN server for storage. And everything else (FC cards, switches, links etc.) was duplicated for each UNIX system. Of course, this was pre-SVN. Don't know whether they have actually implemented this since I left.
Barclays used to have split-site with cross-site disk mirroring for their most critical systems, but when this became too costly, they started reducing the places where this was deployed. Apparently, (high bandwidth) X (long distances ) = lotsa money. Whoda thunk this strategy could cause an outage!
Posted Tuesday 23rd June 2009 09:16 GMT
Barclaycard on-line systems down again Tuesday 23rd June!
Can't help thinking that having systems staff in India with servers in UK is not a good idea.
Back to the article
