Merchants are furious at HSBC UK after the bank's e-payments system fell over for an extended period for the second time in less than a fortnight. Retailers using the system have not been able to process transactions since around 5pm (BST) on Monday. Problems with processing credit card payments continued into Tuesday afternoon …
there'll be a IT Director job advertised pretty soon.
Remember HSBC, fire them, don't let them resign and don't give them any golden parachutes. It's long passed time these jokers were punished for thier failures just like the rest of us.
while its not acceptable for HSBC to be down like this and not keep clients updated, there is also a finger to be pointed at clients who rely on this exclusively and dont have any alternative.
If something is _SO_ crucial to your business, why no backup option?
I can never understand this.
Anything to do with computers can and will go wrong eventually. Fact of life.
If you could potentially lose thousands at any outage, then its your responsibility to have backup plans. No excuses. Even if its just paypal. anything.
Paris knows alot about titsup, i'll bet
The best complaint number for HSBC is the Head Office complaints on 0800 881155. You normally get a manager phoning you back with 24 hours.
Fixing the problem
Nothing gets a business moving faster than customers closing accounts and joining competitors.
Banks may have fancy web sites, but their IT systems are still ancient.
Their IT departments are run like civil service departments - lots of meetings , lots of changing priorities, and very little done.
The real work gets done by external contractors and a few skilful souls in the bank who have to battle against blind incompetence and 'job for life' managers who prefer trying their luck with the call-centre staff, the silly old fools..
that would be the case if their "account closing" application were to be running smoothly. ;o)
- "Sorry Dear Customer but there is a processing fee to close a business account and since we cannot process payments ..."
HSBC e-payments system is horrific to integrate with, to add another service and be able to switch between the 2 adds yet another level of complexity. Sticking a paypal button might seem like a good idea but
a) you'd lose cusotmer confidence more than just saying "we're having a problem"
b) you can't do transactions such as pre-Auth and Post-Auth, which we use so that the card is charged correctly post-delivery.
Currently, we're developing a buffering system that hopefully(!) will allow us to buffer transactions bound for pre-auth when HSBC invariably go down again.
Although, by that time, we're likely to have moved.
The way it is sometimes
I have to agree with Seanie that people should have some form of business continuity plan should a service or services like these fail. Even taking manual payments with a PDQ keeps ya going!
It would not be hard to integrate a second gateway into most if not all shops/stores/whatever and fallover to that should or when HSBC breaks. You get paid into the same account it is not a major issue then.
Look at the mess with Protx a while back when people were screaming the same thing that they were loosing £10,000s as it was offline - surely if your putting that much into a gateway you 'should' have a backup.
And surely as a "business" you don't have one copy of your accounts ... payroll ... customers list ... Do you????
Mind you lost faith with integrating HSBCs payment thing ages ago. Far faster ones exsist nowadays thank god.
sure this is again an good example of bad change control... (and roll back program).
I couldn't agree more.
It's called business continuity planning, and most companies never bother to think about it or realise that it's something, just like all the other costs of doing business, you need to pay for. The problem is that doing business via the web has been tarted up as some kind of panacea of fault tolerancy and commercial utopia, when in fact it's exactly not. Businesses are brainwashed into thinking that they should pay little or nothing for a 24x7x365 presence to run their web shop fronts.
It never fails to amaze me the irate calls we get claiming losses of 10's of thousands of pounds of sales if one of our shared (tenner a month) boxes hits some problem for a few minutes. Yet when we offer them a dedicated box with failover which probably costs them less a month than the leasing payments on their Jags and Astons sitting in their company parking spaces they suddenly lose interest.
If your business is that bloody important and you do it on the web make sure you have two of everything (at least) - and that includes an alternate payment gateway (which may cost you a bit more per transaction, but at least you can still take a payment/authorise in real time).
Paris, because I bet she's got two of everything.
My HSBC Fix
My clients who use hsbc have switched to nochex today and have started trading today.
Tongue in cheek?
"The real work gets done by external contractors and a few skilful souls in the bank who have to battle against blind incompetence and 'job for life' managers who prefer trying their luck with the call-centre staff, the silly old fools.."
Is this supposed to be tongue in cheek?! Has this idiot ever worked for a Bank? I love the attitude of people who feel this could never happen to them. Has it ever occurred to anyone that it could be an external vendor at fault?! It's all too easy to criticise without the necessary facts to hand...
This is what happens when you try to build banking systems with Visual Basic and graduate programmers. Bring back the AS/400's!
It's not like I can call every customer and take details over the phone
-->It never fails to amaze me the irate calls we get claiming losses of 10's of thousands of pounds of sales if one of our shared (tenner a month) boxes hits some problem for a few minutes.
but when your annual HSBC processing bill comes to just under £20k - you can see why you could get a bit irate
integrating with another system to provide failover would cause more problems that it solves.
For example with HSBC you get a response code - 22 I think it is, to say that the customer's supplied postcode doesn't match their card accounts. If you then integrate with RBOS or whoever, who return code NoMATCHPOST instead - you have to have logic for every type of transaction and every possible outcome, twice, all the time relating it to which processor you're using at the time, just becuase the major international bank you've been using can't get it's arse in gear.
If your payments system is suitably abstracted, handling different responses from different gateways should be a simple case a of writing a translation layer.
The additional time and cost spent implementing a second payment gateway far outweighs the risk of not being able to process transactions.
If your business depends heavily on taking online payments you'd be crazy to rely on a single provider, no matter how good their track record. It's the same rule that applies to hosting, backup and pretty much anything else that can fail.
thanks for the Tip (hsbc fix)
I changed to Nochex thanks for the Tip (hsbc fix)
It's easy to close an HSBC account; you go into a branch, and say "I'd like all my money please. Bankers Draft. If you can't do that because of a computer error, cash will be fine. I assume you can still count without using a calculator?"
You then promptly take your money down to another bank, and deposit in a new account. Setting up the services you need may take longer, but there are alternatives in the mean time. Assume HSBC have outsourced the fault finding to India, and they're using a remote desktop session over a 28.8k modem, and a local engineer whos qualifications top out at "have upgraded the ROM in a CPU."
The important thing is your money is not with HSBC, and that's what they don't want to happen.
"HSBC's e-payments system is attractive for retailers because of the relative speed in which the money can be transferred into online bank accounts."
Down to greed, then..how fast is it now?
Service will not be resumed until 9am tomorrow at the earliest
Nice new voice message on the phone number stating that service will not be resumed until 9am at the earliest tomorrow.
All businesses need contingency
We have had a backup system in place on our API based HSBC system that checks to see if the HSBC server is there using CURL. If you are a developer then look into this. You can then allow your system to automatically decide if the payment is taken there and then or it stores it for later, so that it can be manually processed when the system comes back up.
Yes we have lost thousands in the past from HSBC but sh1t happens people and it will continue to happen all over the world with many things. Take some time off tomorrow and plan your contingency, do not waste time on complaining!
The transaction buffer we've spent today coding is now up and running. HSBC are going to pay!(for the development of the buffer on a time and materials basis)
Latest news from HSBC is that they are turning the service off (when was it on?) until 9am tomorrow.
Thanks to Nochex
thanks to the people at nochex.com they have saved my bacon and what a bonus they dont even charge a monthly fee :-) + they give me an account manager that helped me switch
I have been in contact with a few "insiders" who have been able to confirm the cause of the problem.
After the recent loss of CDs they decided to upgrade their systems to use DVDs, they used up their stocks of CDRs and were due to go live with their new improved DVD based communications system for handling transactions - however they realised a bit late that the server only had a CD burner, not a DVD burner and naturally they had no spare CDRs to revert to the old system.
The delay in getting back online isn't being helped by the fact that for some reason their supplier is not taking any orders due to some sort of payment processing outage.
@ all new nochex clients
You still dont get it do you?
Lets say that next week NoChex decide to do an upgrade and it all goes wrong.
What then? It _could_ happen.
You have just seen how easy it was to switch. Why didnt everyone who is complaining setup a nochex account months ago and have, at the very LEAST, 2 sets of html/php/asp pages, so you can put your nochex in when you want, and your hsbc (or other) when you want. Its that easy guys!!.
I just hope you all kept your HSBC account active and a copy of the integration.
That means you NOW have a backup for the future. Pat yourself on the back.
hope NoChex doesnt buckle under the new load !! ;-p
Hong-kong/Shanghai Banking Concern is not the combination of words I first think of when the acronym HSBC is mentioned.
I shall leave it to the collective imaginations of the assembled to come up with more...accurate...interpretations of their initials
They have mismanaged just about *every* aspect of my company's account since it's inception in December 2005.
So this comes as no surprise whatsoever.
Cunch of useless bucking funts
If you're doing 10k of business a day, a 20k bill from HSBC is a drop in the ocean....and it's called 'cost of sale' which so many retailers fail to understand....but that's another rant for another time.
In answer to "integrating with another system to provide failover would cause more problems that it solves.", as Whelk correctly states, you wrap your payment processes up in an API that abstracts away the specifics of the payment service.
re: Tongue in cheek?
Yes it was slightly tongue in cheek, no I don't know the specific facts of the case, and yes I have worked for a bank - for a very short period of time (NR as it happens - ho ho ho).
But you are quite right - I should know better. On the other hand, I think there was more than a grain of truth in what I said. Apologies to all hard working sould who work in banking IT depts. Blame the managers is always my watchword - they get paid to do the planning after all...
Relax, guys !!
They've just been hit by one of the biggest losses in the sub-prime crisis. Those poor things are still reeling from the loss of their massive bonuses and their annual junket^H^H^H^H^H^H board meeting in their Bahamas Headquarters !!
Please take pity on those poor, deprived persons struggling to make do with their last Rolls Royce and multi-million quid country home !!