back to article Routine WHAT NOW? Bank of England’s CHAPS payment system goes TITSUP

The CHAPS system used to handle most bank-to-bank transfers in the UK has collapsed, with all mortgage payments currently on pause. We were told that the problem was due to an update over the weekend, when the details of new CHAPS users were being added and old ones removed. This is meant to be a routine operation, but a …

  1. Anonymous Coward
    Anonymous Coward

    my money would be on:

    :Goldman Sachs :~{

    1. Ted Treen
      Childcatcher

      Re: my money would be on:

      It wouldn't be on them for long...

      They could smell a halfpenny at 25 miles against a force 12 gale.

      And they'd be there in a flash to slurp it up...

  2. Tom 38 Silver badge

    Apparently they added a new institution over the weekend, which is what caused this kerfuffle.

    1/5 - it bust the bounds of a fixed sized array

    2/1 - unicode decode error

    7/1 - File Not Found

    1. Proud Father

      "Apparently they added a new institution over the weekend,....."

      Institute of Management Consultants?

    2. We're with Steve

      9/1 - New name not added to the legacy "Tea Rota Application Package" system (running on legacy Tandem) that no one thought was used anymore but some Charlie was using to do a lookup for short company name on as it was easier than filling in a change request to use "Core Names Thingy 2.1" and senior management had promised some Treasury wonk it would be live by the end of the week (we will fix it after go live, right Steve?).

      1. Danny 14 Silver badge

        found it

        #*****************************************************************************************

        # don't forget to comment out the following before the next update!

        # Not fixed CHAPS import fully yet.

        #*****************************************************************************************

    3. Anonymous Coward
      Anonymous Coward

      Williams & Glyn?

      perhaps

      1. Phil Endecott Silver badge

        Re: Williams & Glyn?

        > Williams & Glyn?

        Yes, some youngster decided to type in "Williams & Glyn", to be safe...

    4. Will Godfrey Silver badge
      Thumb Up

      Like it

      @ Ted Treen

      Have one of these as well as an upvote :)

    5. Mr Handle

      They added a user account.

      Username: JohnSmith

      Surname: ;Drop Database Table Banks;

      1. Gazareth

        Re: They added a user account.

        Little Bobby Tables has grown up & opened his first bank account!

        OXR: http://xkcd.com/327/

  3. Risky

    This will be seriously messing stuff up internationally. CLS Bank will be dusting off their emergency procedures.

    1. jonathanb Silver badge

      People generally use SWIFT for international transfers. As far as I'm aware, it wasn't affected.

  4. g e

    Or GCHQ spliced the wrong fibre (or botched the right one) for the Yanks

    See title

  5. fixit_f

    Not a huge issue

    Sounds like all payments will be made today and they're extending the late clearing cutoff for delayed payments

    http://www.chapsco.co.uk/media/press_releases/-/page/3143/

    Just means that estate agents and conveyancers might have to work a bit later so those house keys can still get handed over!

  6. Anonymous Coward
    Anonymous Coward

    Coincidence?

    Barclays Bank had an online banking login notice on Sunday evening saying they were doing maintenance. What should have been an instant bill payment elicited a warning that it was accepted - but that the actual transfer would be delayed "For security checks" until Tuesday.

    Coincidence?? or just a false sensitivity to a small transfer on Saturday to prove the destination account - followed by a larger transfer on Sunday?

    1. Danny 14 Silver badge

      Re: Coincidence?

      bill payments are normally BACS rather than CHAPS.

      1. Anonymous Coward
        Anonymous Coward

        Re: Coincidence?

        Bill Payments is innocent...er...sorry, got the wrong Bill there:(

  7. Ken 16 Silver badge
    Coat

    Steady on, CHAPS!

    El Reg miss worst possible title for this article, must try harder.

    1. Nigel 11

      Re: Steady on, CHAPS!

      I don't know ... an American might say that CHAPS in its current state was "about as much use as the tits on a boar hog".

      1. GBE

        Re: Steady on, CHAPS!

        <pedant>

        quote: an American might say that CHAPS in its current state was "about as much use as the tits on a boar hog".

        I've heard that many times, but never with the word "hog" on the end

        In common parlance, "boar" means adult male hog, so the "hog" is redundant and makes the listener think you might think there's such a thing as a "boar sheep" or "boar cow".

        In less common usage, "boar" means adult male <certain other mammals>, in which case tits are just as useless.

        </pedant>

  8. Peter Depledge
    Mushroom

    Title is now optional

    Ah. Good ole Bobby Tables.

    http://xkcd.com/327/

  9. JaitcH
    FAIL

    Is this NEWS?

    What UK government computer system has ever worked right first time ... or the fiftieth software upgrade?

    Memories of NHS.

    1. Anonymous Coward
      Anonymous Coward

      Re: Is this NEWS?

      It isn't a government system. It's run by a private company owned by the banks.

    2. jonathanb Silver badge

      Re: Is this NEWS?

      Congestion Charging, or at least there were no major problems.

      1. Anonymous Coward
        Anonymous Coward

        Re: Is this NEWS?

        but the IT Architects on that one were simply brilliant.

        (and worked like one handed paper hangers to paste over the cracks)

        Pint owed if they read this.

  10. Marketing Hack Silver badge
    Joke

    Everyone knows....

    A few CHAPS that are just no good!!

    /rimshot

    Thank you! I'm here all week!! (No, you don't have to be happy about that)

  11. Harry the Bastard

    message just received in arecibo...

    all your cash are belong to us

    you are on the way to destruction

  12. LOL123

    RTGS - Real Time Gross Settlement

    Real Time not Retail time..

    House buying is treacherous enough as it is!

  13. kain preacher Silver badge

    This really CHAPS my hide.

    1. Marketing Hack Silver badge
      Happy

      @ kain preacher

      You obviously didn't read the story, or you would know that this is all about hiding your CHAPS!

  14. kellerr13

    Accident?

    If any of you really believe this is an accident, then you are the fool in the army of fools.

    1. Anonymous Coward
      Anonymous Coward

      Re: Accident?

      Of course it was an accident, but that doesn't mean we should throw away the valuable information gathered by the "financial disaster preparedness monitoring software", installed the day before the accident.

  15. Anonymous Coward
    Anonymous Coward

    Was it still on NonStop? Something stopped.

    Like many similar systems, BoE RTGS was on NonStop (Tandem/CPQ/HP) (until at least 2012). A Tandem usergroup whitepaper says so, as does a public profile on a well known Facebook-for-businesspeople of a NonStop person with RTGS experience who left BoE a year or two ago.

    Who can tell me what the system now runs on, in which country it is hosted, and who wrote and approved the DR plan?

    1. Anonymous Coward
      Anonymous Coward

      Re: Was it still on NonStop? Something stopped.

      Almost certainly still on Non-Stop/Himalaya.

      HP has a successor in the works, but still in alpha at the moment. Looks interesting, will say no more.

      Posting anon because I wasn't supposed to mention that...

  16. Alan Denman

    Legacy system ?

    It almost sounds like the IT bod Chap who actually knew what he was doing left years ago.

    I hazard a guess it should have been a 2 minute job.

    1. Gordon 11

      Re: Legacy system ?

      It almost sounds like the IT bod Chap who actually knew what he was doing left years ago.
      left was off-shored a few years ago.

      1. Anonymous Coward
        Anonymous Coward

        Re: Legacy system ?

        "It almost sounds like the IT bod Chap who actually knew what he was doing left years ago.

        left was off-shored a few years ago."

        That's why I'm hoping someone can tell us where the system now is.

        Courtesy of public data on Facebook-for-professionals (no account needed), I apparently know the name of the man in charge of the infrastructure till a couple of years ago, and I also apparently know where he now works (still working with NonStop - it's a relatively small world).

        Among the many things I don't know is why he left, or what the sucession plan was.

        Some of these details may emerge in the inquiry.

  17. Anonymous Coward
    Anonymous Coward

    CHAPS

    the only time I've ever had anything to do with CHAPS was a few years ago after I put a few grand inheritance into my Barclays account after my old man had died. Put it in to the current account and then wanted to move it on to a saving's account in another bank. Anyway paid the cash in then couldn't move it out as Barclays had put a limit on the amount of money that you could move out of the account and it was quite a small limit, grrrrrrrrrrrrrrrrr Trip in to branch to be told the only way to move it was via CHAPS which of course they wanted something like £30! ROBBING TW&TS

  18. Nifty

    The transaction size...

    needed for an average UK house purchase just caused a numerical overflow

  19. Smartini
    Joke

    Failover?

    What, didn't the failover to PayPal kick in?

  20. Anonymous Coward
    Anonymous Coward

    Why no love for NonStop.. :)

    this or part of this is still on NonStop (good job IMHO) but not a Non-Stop/Himalaya those are a little old now and no longer supported by HP.

    CHAPS runs on a NonStop system which IIRC is smaller than the test systems used by say VocaLink or the Banks. This is ok though CHAPS only does like 200K transactions per day VocaLink do 12-14m (real time) transactions per day.

    Nothing at all wrong with NonStop.. it has lowest TCO for this kind of environment. This outage wasn't an infrastructure failure it was a failure to implement change.

    You can be Active Active on the most reliable hardware available on the planet (they are) but if you F**K up an application or configuration change it all goes down.

    Should have been trivial to roll back though.. looks like it wasn't

    Common causes of system failure #1 Operator, #2 Software Change

    Thats like 90% of failures taken care of right there.. then comes power failure, network failure and infrastructure failures.

    Seems odd however that a systems that only has to be available BoE office hours went down.. FPS is 24/7/365 and still manages 100% availabilty (core systems) given this bonus they should have been able to veryify the change prior to the opening bell.

    Cheers (Anonymous Availability Nerd..)

    1. Anonymous Coward
      Anonymous Coward

      Re: Why no love for NonStop.. :)

      "CHAPS runs on a NonStop system"

      That's as maybe, but isn't RTGS (the system that failed) distinct from CHAPS?

      Other than that, yes, this all sounds a bit unfortunate; properly designed and managed NonStop setups (especially office-hours-only ones) shouldn't be falling over like this due to (apparently) lack of proper testing.

      Bedside reading for availability fans (this month including a 6page summary of, and a link to a free copy of, NonStop for Dummies):

      http://www.availabilitydigest.com/

      1. NonStopBoy

        Re: Why no love for NonStop.. :)

        Good point I was unclear..

        I am pretty sure RTGS is also run on NonStop.. I have had discussions with programmers of this systems at NonStop events in recent times.

        Good link to the Availability Digest, Dr Bill is a good friend of mine and his site is always very informative.. I especially Like the Active Active 60's style.. write up he did for NASA historical work. Brilliant.

  21. Terry Cloth
    Paris Hilton

    Any update?

    I haven't seen any stories about the collapse of the UK housing market, so I presume they got it fixed, but I don't see the promised Reg followup. If I'm just blind, I'd appreciate somebody pointing it out to me. Thanks.

    (Paris for the big question mark.)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019