Angry retailers have launched a tirade against HSBC after its e-payments system fell over on Friday and stayed down for the whole weekend. Retailers were unable to sell anything online for over 48 hours, and have started calling for compensation. HSBC's e-payments system has proved particularly attractive for retailers because …
Eggs and baskets
Surely, nobody would have all their eggs in the same basket when it comes to something as critical as their online payment system? And surely, they would have performed non-functional testing including to make sure that their system was able to cope with one of the payment suppliers being offline so that they could re-direct their customers to another payment supplier?
100% uptime doesn't exist and it is as true of HSBC as anybody else so if a critical part of your business (i.e. getting customers to pay you) relies on a single third party system, you're asking for trouble.
Trying to order for 3 days now
Been trying to order from TescodDirect.com since Saturday (although managed to get a payment through on Dabs on Saturday using the same system). I can't register a new card or use an exisiting registered one. Wife just spoke to HSBC who said it was a problem at Mastercard End and it was meant to be up this morning but they think it will be another 24 hours at least.
What is embarassing is that someone as big as tesco cannot process maestro/mastercard transactions. Still at least the web is safer without the ability to use payment cards.
The problem in January wasn't just the outage, it's that HSBC lost a days transaction data in the process, buggering up everyone's dispatch databases in the process, and then expected us to call each customer asking for them to place the order again.
I could also even forgive HSBC for having outages, 100% doesn't exist, but it's the extremely poor level of service you get from them - firstly outright denials that there's a problem (often insisting it's you websites), then several hours later an admission that "something is being looked at", until after many frustrated phone calls to a very apathetic "support" team in bangalore or wherever, you get told that yes, the service is down - but no, we can't tell you why or when it will be back up. They really don't seem to care.
I don't expect an apology from HSBC, I'll probably receive a letter stating they had "intermittent issues" and that, oh yes, my charges are going up. I haven't had any contact from my HSBC account manager about this. It's beyond shoddy, and you get the feeling they would be much happier if they didn't have customers who keep annoying them by wanting a service to be available.
Fortunately, we also allow customers to pay cash on delivery, so we were still able to take orders, but Sunday's takings were half what they should be, we must have lost a good £10k, and I can't guarantee that those customers would try again today.
So you are suggesting that all the small and medium sized companies using HSBC for the payment gateways should be investing in an infrastructure to provide near 100% uptime since their quite large multinational bank shouldn't really have to? Anyway a whole weekend outage already puts them in the region of around 99.5% already for this year, not great!
Bit more to it than that
I couldn't get any money out of a cash point machine on Friday or Saturday and had to drive to the next town to get to a HSBC one.
So, whatever they were doing was rather more widespread than just the payments system.
Actually it's a cost and simplicity issue. You have to pay each merchant for the transaction type, so that adds to your costs. You pay commission per sale which is standard, but most need you to pay a fee as well (especially if sales are below a certain amount) which many retailers would rather not have to do.
More so, if you got to many e-tailers (take amazon as an example) although they take several payment types you only see one system. Having looked at such implementations for some clients, you tend to find that most need you to use their system or at least their system wrapped in your page layout rather than you having control over each. Thus adding multiple payment systems, many of which may overlap, would simply confuse and irritate customers.
If you went to an online retailer and wanted to pay by card, if you were offered 3 ways to pay by visa you'd be a bit miffed as to why they couldn't just sort themselves out. I accept your alternative would allow for the retailer to auto-switch to another merchant but that comes back to cost, maintaining the other pages that rarely get used, and the confusion for regular customers depending on the sort of company you are.
As a professional merchant providing a service, HSBC have a duty to inform their customers of any issues, especially at peak sales times such as weekends and evenings, and this isn't just an exclusive communication to their large branded clients. I certainly would think twice before suggesting a client use them for their systems in future, such events can only be bad for business.
You'll also find that where a company looks like they're processing their own card details, 60% of the time they're simply connecting to an external api for a more technical merchant, so still aren't processing their own payments.
Prince Albert yesterday... but not Today
Evan Davis, the BBC's economics editor and now one of the presenters of the Today Programme on Radio 4, is widely rumoured to have a Prince Albert...
He studiously refuses to comment about said rumours, seemingly taking great pleasure in maintaining the air of a man of mystery. Whilst Mr Davis may well have once had more than his RDA's worth of ironwork about his person, I bet he doesn't take pleasure from any such metal adornment these days - it wouldn't quite be in keeping for one of the Beeb's top news hounds to be setting off metal detectors left right and centre whilst out and about reporting on the latest sharp movements in gilt edge stock or whatever.
Presumably, when the light bulb blows above the MD's desk he will congratulate himself on his forethought at having arranged a replacement.
More Info from HSBC
"Approached by The Reg, HSBC would not confirm or deny the outage."
Approached by an angry customer (myself), the ePayments team said this:
"We are extremely sorry to advise that the Secure E Payments application
encountered a database problem with some of the larger database tables.
The space used to control the database keys became fragmented resulting in
performance degradation and the database becoming locked.
In addition, this same degradation affected the performance of all the
database utilities we use to manage the database problems with our Secure
We are continuing to do everything possible to resolve these issues.
Robust testing of the system is being undertaken to ensure that the system
is stable and is able to fully support your business."
One payment gateway...
Er excuse me - do you have 2 bank accounts because you think your bank might accidentally lose all data about payments in and out of your account for 3 days in January? Or because you expect your debit card would not work for 3 days from 3rd-6th Jan 2008 and then from 28th-31st March 2008?
I don't think so.
What I expect is a PCI-DSS validated Merchant system run by one of the worlds biggest banks to have 0.01% down-time. That is what I was told when they sold this system to me.
What I also expect is instant notification of any down-time - we have still had no communication from ePayments!
What I expect is my emails to be answered. My calls to be picked up.
What I also expect is that my customers credit card data is kept secure and doesn't get 'lost' (or stolen??)
What I don't expect is to be told that I am the only company complaining about it.
What I don't expect is their IP to be completely uncontactable.
What I don't expect is a message 3 days later on their website saying" Please accept our apologies for the recent disruption." ie no explanation or reporting.
What I don't expect is to have to fight for months for the bank to even accept there was a problem (in Jan) and that I was not the only customer that had a problem and that in fact they had been lying to me.
What I don't expect is a media black-out.
And what I don't expect is to have to take them to court becuase they refuse to accept liability.
Jonathan "Hampshire-based catering equipment company"
I am totally with you Jonathan
Its bad enough dealing with your business being paused for 3 whole days - the stress levels are bad enough if my ecommerce service is out for 1 hour can you imagine 3 days of it and not knowing why?
I have had no response from epayments, however have recieved an email from HSBC customer service and a call from the regional commercial director who was vert appologetic and insisted to ensure that i send in a claim for compensation and cc him in.
Lets see if they can keep to the promise of responding to this in 5 days.
I totally understand the outRage at HSBC's latest blunder, having been affected myself (in january) and receiving no help or apology, I switched by recommendation of a friend to Barclaycard e-pdq, the set up was easy, I have an account manager who is a real person at the end of the phone! I know systems have errors as nothing is 100% but so far so good.
Jo, Do you work for Barclays? lol!
HSBC down again 8/4/2008
Can you believe that the server is down again?!?!?!?!
They have a lot to answer for, its like they are trying to sabotage us! which is rather appropriate under the circumstances.
oh my jesus-ing h-ing christ
it's down again!
ok, I'm taking bets it down until tomorrow morning, what's your wager?
I agree, well I hope that it will be back tomorrow am, they are useless, they seem to work like my staff, 5pm and wooooosh, they're off home. I bet no one looks at this until tomrrow am, Unbelievable for one of the biggest banks on the planet - they can't get people to work overtime and fix this use less system for good!
Still down as at 7:40am. The API went down at 5:05pm yesterday with the CPI following less than 10 minutes later.
As of 11pm last night (when the close), there was no ETA although they were "confident" it would be running by 8am this morning with engineers "working on it" still.
Unfortunately the problems at HSBC are far deeper than their technical in abilities to keep a key service running. As a loyal customer for over 10 years, we are moving all our banking services away from HSBC following a catalogue of issues over the last 12 months. This includes card processing, factoring, insurance, direct debit processing and general banking.
For those of you looking for a new gateway and wary of moving to the likes of Protx, Secpay and Worldpay - take a look at Netbanx who cost a little bit more (and have minimum T/O requirements).
Best of luck everyone! Don't forget to get those compensation claims in early.
Paris, because even her panties stay up more than HSBC's servers.