HSBC e-payments system goes titsup

argh

The problem in January wasn't just the outage, it's that HSBC lost a days transaction data in the process, buggering up everyone's dispatch databases in the process, and then expected us to call each customer asking for them to place the order again.

I could also even forgive HSBC for having outages, 100% doesn't exist, but it's the extremely poor level of service you get from them - firstly outright denials that there's a problem (often insisting it's you websites), then several hours later an admission that "something is being looked at", until after many frustrated phone calls to a very apathetic "support" team in bangalore or wherever, you get told that yes, the service is down - but no, we can't tell you why or when it will be back up. They really don't seem to care.

I don't expect an apology from HSBC, I'll probably receive a letter stating they had "intermittent issues" and that, oh yes, my charges are going up. I haven't had any contact from my HSBC account manager about this. It's beyond shoddy, and you get the feeling they would be much happier if they didn't have customers who keep annoying them by wanting a service to be available.

Fortunately, we also allow customers to pay cash on delivery, so we were still able to take orders, but Sunday's takings were half what they should be, we must have lost a good £10k, and I can't guarantee that those customers would try again today.

@Bruno Girin

Actually it's a cost and simplicity issue. You have to pay each merchant for the transaction type, so that adds to your costs. You pay commission per sale which is standard, but most need you to pay a fee as well (especially if sales are below a certain amount) which many retailers would rather not have to do.

More so, if you got to many e-tailers (take amazon as an example) although they take several payment types you only see one system. Having looked at such implementations for some clients, you tend to find that most need you to use their system or at least their system wrapped in your page layout rather than you having control over each. Thus adding multiple payment systems, many of which may overlap, would simply confuse and irritate customers.

If you went to an online retailer and wanted to pay by card, if you were offered 3 ways to pay by visa you'd be a bit miffed as to why they couldn't just sort themselves out. I accept your alternative would allow for the retailer to auto-switch to another merchant but that comes back to cost, maintaining the other pages that rarely get used, and the confusion for regular customers depending on the sort of company you are.

As a professional merchant providing a service, HSBC have a duty to inform their customers of any issues, especially at peak sales times such as weekends and evenings, and this isn't just an exclusive communication to their large branded clients. I certainly would think twice before suggesting a client use them for their systems in future, such events can only be bad for business.

You'll also find that where a company looks like they're processing their own card details, 60% of the time they're simply connecting to an external api for a more technical merchant, so still aren't processing their own payments.

Prince Albert yesterday... but not Today

Evan Davis, the BBC's economics editor and now one of the presenters of the Today Programme on Radio 4, is widely rumoured to have a Prince Albert...

http://www.guardian.co.uk/media/2005/nov/06/broadcasting.observerreview

He studiously refuses to comment about said rumours, seemingly taking great pleasure in maintaining the air of a man of mystery. Whilst Mr Davis may well have once had more than his RDA's worth of ironwork about his person, I bet he doesn't take pleasure from any such metal adornment these days - it wouldn't quite be in keeping for one of the Beeb's top news hounds to be setting off metal detectors left right and centre whilst out and about reporting on the latest sharp movements in gilt edge stock or whatever.

Light bulbs

Presumably, when the light bulb blows above the MD's desk he will congratulate himself on his forethought at having arranged a replacement.

More Info from HSBC

"Approached by The Reg, HSBC would not confirm or deny the outage."

Approached by an angry customer (myself), the ePayments team said this:

"We are extremely sorry to advise that the Secure E Payments application

encountered a database problem with some of the larger database tables.

The space used to control the database keys became fragmented resulting in

performance degradation and the database becoming locked.

In addition, this same degradation affected the performance of all the

database utilities we use to manage the database problems with our Secure

ePayments system.

We are continuing to do everything possible to resolve these issues.

Robust testing of the system is being undertaken to ensure that the system

is stable and is able to fully support your business."

One payment gateway...

Er excuse me - do you have 2 bank accounts because you think your bank might accidentally lose all data about payments in and out of your account for 3 days in January? Or because you expect your debit card would not work for 3 days from 3rd-6th Jan 2008 and then from 28th-31st March 2008?

I don't think so.

What I expect is a PCI-DSS validated Merchant system run by one of the worlds biggest banks to have 0.01% down-time. That is what I was told when they sold this system to me.

What I also expect is instant notification of any down-time - we have still had no communication from ePayments!

What I expect is my emails to be answered. My calls to be picked up.

What I also expect is that my customers credit card data is kept secure and doesn't get 'lost' (or stolen??)

What I don't expect is to be told that I am the only company complaining about it.

What I don't expect is their IP to be completely uncontactable.

What I don't expect is a message 3 days later on their website saying" Please accept our apologies for the recent disruption." ie no explanation or reporting.

What I don't expect is to have to fight for months for the bank to even accept there was a problem (in Jan) and that I was not the only customer that had a problem and that in fact they had been lying to me.

What I don't expect is a media black-out.

And what I don't expect is to have to take them to court becuase they refuse to accept liability.

Jonathan "Hampshire-based catering equipment company"

HSBC

I am totally with you Jonathan

Its bad enough dealing with your business being paused for 3 whole days - the stress levels are bad enough if my ecommerce service is out for 1 hour can you imagine 3 days of it and not knowing why?

I have had no response from epayments, however have recieved an email from HSBC customer service and a call from the regional commercial director who was vert appologetic and insisted to ensure that i send in a claim for compensation and cc him in.

Lets see if they can keep to the promise of responding to this in 5 days.

Carl

HSBC

Jonathan,

I totally understand the outRage at HSBC's latest blunder, having been affected myself (in january) and receiving no help or apology, I switched by recommendation of a friend to Barclaycard e-pdq, the set up was easy, I have an account manager who is a real person at the end of the phone! I know systems have errors as nothing is 100% but so far so good.

Jo

HSBC

Jo, Do you work for Barclays? lol!