Feeds

back to article Lloyds Group probes server crash behind ATM, cash card outage

Lloyds Banking Group is investigating the cause of a Hewlett-Packard server failure it blames for taking down thousands of its ATMs and crippling cash cards at the weekend. The crash saw thousands of customers unable to withdraw money from their accounts or make payments using debit cards on Sunday afternoon. The outage left …

COMMENTS

This topic is closed for new posts.

Page:

Anonymous Coward

I don't know anything about bank setups but wouldn't they use failovers?

If not, why not?

2
0
Silver badge

Never mind Banking, clearly you know nothing at all about IT.

Yes, they have and use failovers. But it depends on the nature of the fault before you can invoke a failover.

Hardware generally yes, no problem, automatic failover can be almost instant, even geographically dispersed.

Software, middleware, transaction processing - maybe not - it just depends what has failed and if its possible to move the multiple transaction threads or if you need to stall the processing and move it in a controlled manual manner.

3
5
Bronze badge

Failovers dont always work

You can have the odd issue where the machine being mirrored is responding to heart beats but not actually working anymore, hardware problems can be a nightmare, if the storage system for instants starts responding sluggishly it may not flag a failure anywhere even though it can no longer complete all requests thrown at it before they timeout.

1
0
Anonymous Coward

No only do they use failovers, but it's likely that the machine which failed was a tandem, they're more reliable than mainframe, but the only thing you can be sure of with a system is that the system will fail.

1
0

This post has been deleted by a moderator

Re: Failovers dont always work

@Cpt Scarlet

Thanks for the explanation

0
0
Silver badge

Tandem

"it's likely that the machine which failed was a tandem" - if by that you mean HP NonStop (formerly Tandem), they're effectively immune to single points of hardware failure (if correctly configured - which I imagine/hope a bank's systems would be). Dual hardware failures can be the result of (a) extreme bad luck - second component fails before the first failed component can be repaired; or (b) human error - engineer sent to replace failed component pulls the good one by mistake (should never happen, but I've seen it done).

If a dual failure does occur, it can cause significant problems as the software tends to be designed on the assumption that it can never happen.

2
0
Bronze badge

Re: Tandem

"On the assumption it can never happen."

Yes, that's wise...

PS, I want to see some sci-fi/fiction where the characters have infinite "back up plans" and redundant equipment. Like Mission Impossible, cross Star Trek and some McGiver for... well, backup.

0
0
Silver badge

@ TechnicalBen

You can code elaborate software recovery routines that will be useful once every 50 years (and potentially reduce the stability of the software), or accept a very rare outage as a result of dual hardware failure. Your choice.

3
0

Re: Tandem

When I wrote some code for "Non-stop" Tandems (TAL, 20 years ago or so, as part of an industrial placement), you had to write the code to synchronise across two CPUs yourself. I think the only thing running like that on those production systems was the command shell. I'm sure it sounded great to whoever got the original sales pitch, but the reality was a long way away.

0
0
Silver badge

CEO's tweet implied hardware

The CEO's tweet clearly implied that one specific HP server had failed. As software is not specific to any single computer, that clearly implies it was a hardware fault. If not, HP has grounds for complaint - if not legal redress.

I found it extremely odd that he saw fit to name a supplier in that way.

0
0
Bronze badge

Re: CEO's tweet implied hardware

Yes, it's odd. Maybe they've decided to burn the bridges.

0
0
Thumb Down

Cash

Time to start carrying cash again, or having bank accounts with two separate banks.

2
1
Anonymous Coward

Re. multiple bank accounts

Anyone with any nouse should already have accounts with several banking providers, to take advantage of the various offers freebies and incentives they all dish out. Me, I've got 3 bank accounts, and 5 credit cards with 3 different providers.

0
0
TRT
Silver badge

"@dangerfield_gem Gemma, no truth in this. The issue was caused by a HP Server failing here in the UK. PDP"

Is he blaming DEC as well?

9
0
Bronze badge

Compaq bought DEC. And HP bought Compaq. So, yes, it's DEC's fault. Nothing whatsoever to do with Lloyd's management, cost savings, hacked together back office systems etc.

7
1
Bronze badge

Note that he says that "the HP server failed in the UK" when the assertation was that had they have had a competent and experianced IT support team in the UK they wouldn't have the issue.

Is it just because i'm a hardened cynic that I notice Sir Humprhy-esque evasion in answers like this and assume that he means that the lack of experianced staff (and BCM/DR tests) was the issue that caused this problem...?

1
0
TRT
Silver badge

I was going to post something about him not getting any brown sauce on his sausage sandwich this morning, but then I remembered what the HP on HP sauce stands for and it suddenly made sense. failure is the usual mode for HP and IT.

0
0
Anonymous Coward

No announcement of the problem

Servers going down is one thing, (there but for the grace of $deity...), but why were there no annoucements of the problem? Nothing on their web site, and their official Twitter feed stopped just at the same time as the incident started and didn't resume until this morning.

At least the all other unfortunates posting to Twitter and Facebook reassured me that it wasn't just my account that might have been rifled of its balance, but it was a very unpleasant half hour between having my card declined and finding out what had happened.

2
2
Silver badge
Mushroom

Re: No announcement of the problem

Maybe they don't have 24/7/365 twitter writers, maybe it was their day off and they were using Facebook instead, or maybe Google+, or even a blog, heck maybe they shared some photo's in Pinterest whilst syncing it with their Flickr accounts ...who knows.

Still glad after 30 minutes your entire would stopped collapsing and you could return to you instant gratification.

6
4

PDP?

I notice that Mr Pester ends his tweet with PDP. Are these his initials or the technology the system runs on ?.

1
0

Re: PDP?

They're his initials (middle name David) and he signs off all his tweets that way.

0
0
Anonymous Coward

Single point of Failure

Memo to bosses: Time to pay more peanuts to your monkeys.

1
1
Anonymous Coward

HP Servers?

Surely a bank would know that it's cheaper to buy your kit outright than doing a Hire-Purchase agreement???

Thanks, my coat's the rental one over there ->

5
0
Bronze badge

Re: HP Servers?

While it may be cheaper, it's the banks we are talking about here.

They would (read as "did") sell the office, building, chairs and hardware to have a nice large sum of cash to declare to the share holders, along with the reduced costs of having to replace and repair those things.

Never mind in the long run the price rises in rental could eat up all that temporary cash. If it looks good on paper, and for a short enough time to get the market buying your shares, well, you can always parachute out before the reality hits the ground.

8
0
Anonymous Coward

Re: HP Servers?

> Thanks, my coat's the rental one over there ->

Is it a Rumbelows' shop coat?

2
0
Bronze badge
FAIL

Re: HP Servers?

... well, you can always parachute out before the reality hits the ground. FAN!!!

FTFY

0
0
Silver badge
Coat

Re: HP Servers?

No it was the one from 'Grace Bros'

0
0
Silver badge

Re: HP Servers?

They could be doing an HP agreement with Blackhorse Finance to shift profits elsewhere in the group.

0
0
Anonymous Coward

ATM Network?

It wasn't the ATM Network that was down, was it?

I thought all Lloyds Banking Group debit card transactions were not working - i.e. in shops too and regardless of whether it was a LBG owned ATM

0
0
TRT
Silver badge

Re: ATM Network?

It was the TSB. Or Titsup Savings Bank.

2
1
Anonymous Coward

Re: ATM Network?

That's not correct, the outage affected all LBG banks (including TSB). It was just that TSB's CEO Paul Pester chose to comment.

1
1

Dear Banks

Sort your fucking IT acts out. It's the 21st Century FFS.

(I also see too much "clearly you know nothing at all about IT" and comments of that nature.

So. Fuck. What.)

I think that customers have a right to demand far more robustness from Bank (and other vital institutions) IT systems than there currently is. Stop paying your already grossly overpaid executives obscene amounts of money, bolster your infrastructure (including security) and treat your customers with more respect.

10
2
Silver badge

Re: Dear Banks

I think that customers have a right to demand far more robustness from Bank (and other vital institutions) IT systems than there currently is.

I don't entirely agree - a large proportion of personal customers in the UK don't actually pay for a current account (okay, charges for overdrafts etc) but basic banking is free. Once this was funded by the fact that they didn't pay interest on current accounts, but with interest rates at zero they don't have any income to pay for the services. It's a bit like Google - you can't really complain or have expectations about something you're getting for nowt.

But yes, on the whole bankers are overpaid parasitic scum who should be condemned to twenty years chained to an oar in the bottom level of the slave galleys.

2
1
Silver badge

Re: Dear Banks@ Pen-y-gors

"I don't entirely agree - a large proportion of personal customers in the UK don't actually pay for a current account (okay, charges for overdrafts etc) but basic banking is free. Once this was funded by the fact that they didn't pay interest on current accounts, but with interest rates at zero they don't have any income to pay for the services."

Let's stop for a moment and consider why interest rates are so low. Would it be the fault of the self same banks bleating they've no money, whose reckless lending meant the state had to bail the thieving scum out in the first place, accompanied by QE and near zero interest rates?

If they have to provide "free" current accounts at a loss for a few years, simply because their collective greed, incompetence and dishonesty wrecked the whole system, then I say tough luck. If they don't want to do retail banking, then let them close or sell their high street branches.

9
0

Re: Dear Banks@ Pen-y-gors

"why interest rates are so low"

It's also worth noting that the interest rates paid to the account holders are low; but the banks themselves use that money to fund loans. Often at 10 times what they pay to the account holders.

Some of that money also goes into the stock market in various forms. Value of investments can go down as well as up, but even so, many of them are making 20 or 30 times what they pay the account holder. (I saw one indication where a bank was making more per day than they were paying out per year)

As for the "free" banking accounts; they reserve the right to make charges for certain things. Some of these will only be levied if you make a mistake, but even so, just one of these charges could mean that your "free" account is costing you more than you are earning in interest on the money in your account.

For what it's worth, my bank keep pestering me to move to their "premier" account. I'd end up paying more per month than my savings would earn in interest over 5 years.

3
0
Silver badge

Re: "don't actually pay for a current account"

Really? I (and all the other customers) give the bank all my money, and let them do what they like with it, so long as they hold enough in reserve that I can take it back out again. Interest rates might well be in the dirt, but I'm sure that they've found some way of turning a profit from my money.

4
0
Bronze badge

Re: Dear Banks@ Pen-y-gors

Completely false.

they have your money, and use it as collateral to get 5x 10x that amount in the interbak-funds market and lend it at market price + premium.

So yes, just by having money in your account, you are paying them.

Plus, if you get your salary by bank transfer, they get paid, plus credit/debit card transactions, plus you getting money out of a different network ATM, plus non free money transfers (YMMV), plus the extraordinary expensive currency rates they apply...

So yes, you ARE paying them.

7
0
Silver badge

@ Pen-y-gors

"Once this was funded by the fact that they didn't pay interest on current accounts, but with interest rates at zero they don't have any income to pay for the services."

Please think this through. Interest rates are at zero for money that WE lend to BANKS. Have you checked out mortgage or other loan interest rates for money BANKS lend to US lately? Not to mention the far higher rates they can get for lending to actual productive ventures such as industry and commerce.

Even at a very low rate of interest, banks could make plenty of money to run their businesses out of the interest they can get on the vast amounts of money deposited with them. Don't forget, that's a substantial fraction of all the money in the UK.

2
0
Anonymous Coward

Credit cards ?

I always carry my main (Nationwide) debit card. However I also have a Barclaycard (which is almost exclusive reserved for expenses-type purchases). In the event my debit card fails, I can always cover it with the Barclaycard.

And then get medieval on the Nationwides ass ....

0
0
Anonymous Coward

Re: Credit cards ?

Same here, I may always carry a Debit Card for withdrawing cash, but I do all my purchases by Credit Card if possible, and I carry multiple cards with me from different banks... so I have multiple ways to pay if I have to, and of course I carry cash just incase the whole thing goes tits up or I want to purchase from small vendor.

3
0

Re: Credit cards ?

Ironically, Nationwide is one of the few with a core banking platform designed and built this decade. Almost all the rest are derived from the last century. This doesn't make them immune from failure of course, nor from the challenges of managing all the interconnected systems, but at least they don't have to treat the whole lot as the IT banking equivalent of Buckaroo.

0
0
Anonymous Coward

End of the world

Some ATMs not working for a few hours on a rainy sunday !>= End of world

4
2

Re: End of the world

Hopefully it was for the PHB bean counter who recommended the cuts and has been asked to clear his/her desk.

we can but hope.

can we expect it to finally come out that the support is done in 'Bongo Bongo land'?

5
1
Anonymous Coward

Re: End of the world

Some ATMs not working for a few hours on a rainy sunday !>= End of world

It's not the ATMs which are the problem, it's the epos systems.

People standing in front of a cashpoint being told to fuck off, might not be pleased, but they're not embarrassed enough to go shouting about it in public. Stand them in front of a checkout girl, and refuse them access to their money, and they're going to shout very loudly, and very publicly about how you're an incompetent wanker, who has ruined their life.

10
0
Anonymous Coward

Re: End of the world

"It's not the ATMs which are the problem, it's the epos systems."

Exactly. Standing at the counter of a petrol station with your car full of petrol and finding that your bank is refusing to let you have any of your money to pay for it is a very unpleasant experience.

Gas station attendants basically treat you like the lowest form of life on earth in this situation and make you fill out some very evil sounding documents before they'll allow you to leave - even offering to leave the vehicle and its keys with them while you go source the required dosh isn't always accepted(*).

(* Happened to me once a long time ago, when I found after filling my bike with gas that my wallet wasn't in my pocket. Only lived just down the road, but they wouldn't let me walk home to get it)

4
0
Anonymous Coward

Re: End of the world

Oddly enough as an ex- retail manager and having worked in IT at financials this sounds exactly like what happens at both ends of the outage.

0
0
Anonymous Coward

Re: End of the world

even offering to leave the vehicle and its keys with them while you go source the required dosh isn't always accepted(*).

Car could be stolen even with keys. Same goes for any iPhone or whatever you offer. Most places in the UK though will just check your drivers license to see who you are and copy the info down and copy your reg plate and let you go.

Also you made the mistake of riding a motorbike which automatically means your invisible to 95% of road users, a legitimate target to another 2% and only seen as a nice guy to the remaining 3%. I've given people pillion rides back to there bike when they have turned up on foot (we took fuel +£5 deposit on petrol can) while on shift so not all people are asses.

4
0
Anonymous Coward

Re: End of the world

Not sure the support is done in Bongo Bongo Land, but HBoS outsourced much of their support to HCL in India in about 2007. Many of the in house staff were made redundant a few years later when the LTSB thing happened.

A/C cos I still have friends there.

0
0

Re: End of the world and backups

No, the bean counter won't cop it - the man responsible for cuts and outsourcing at RBS is still hanging on, despite all his colleagues doing the decent thing and falling on their swords. All the BC has to do is ignore all decency, carry on like nothing's wrong and plan what he's going to spend his next bonus cheque on.

Regarding the fact that Lloyds had no backup, well they do have - every other bank in the country - you can use most other cash machines for free, pay with cash, credit cards etc., so not sure why everyone is whittling on about it, unless they are hoping for a handout as well.

1
0

Page:

This topic is closed for new posts.