back to article UK.gov demands urgent answers as TSB IT meltdown continues

The government is demanding urgent answers over a botched systems upgrade at TSB that has locked out up to 1.9 million customers. The IT meltdown happened after a planned upgrade between 4pm on Friday 20 April and 6pm on Sunday 22 April as TSB migrated from former parent Lloyds Banking Group's systems to shiny new ones. TSB …

  1. Anonymous Coward
    Anonymous Coward

    Whose idea was this?

    No doubt the deadline was forced onto TSB by the government who now deny responsibility.

    1. tiggity Silver badge

      Re: Whose idea was this?

      Maybe they needed changes in place for GDPR (only a month to go) compliance?

      1. Korev Silver badge

        Re: Whose idea was this?

        Or they want to do the change shortly after the end of the UK tax year.

        Maybe all three reasons?

    2. Austin Montego

      Re: Whose idea was this?

      It's nothing to do with the government. It's so that TSB's parent company can stop paying Lloyds Banking Group millions of pounds each year for access to their IT systems.

      The switch is to Banco Sabadell's systems (i.e. those of TSB's parent).

      1. Anonymous Coward
        Anonymous Coward

        Re: Whose idea was this?

        Lloyds are transitioning their systems into IBM cloud. Maybe they gave TSB a very hard deadline to get out?

        1. Anonymous Coward
          Anonymous Coward

          Re: Whose idea was this?

          IBM cloud. Oh no.

        2. This post has been deleted by its author

      2. Anonymous Coward
        Anonymous Coward

        Re: Whose idea was this?

        "Maybe they needed changes in place for GDPR (only a month to go) compliance?!"

        "Maybe they needed changes in place for GDPR (only a month to go) compliance, so that the fines will not be quite so bad so they balls it up and expose customer details to random other customers."

        TFTFY

      3. Mickybhoy55

        Re: Whose idea was this?

        And these systems are now run from Barcelona! It should have been a straight forward migration. Obviously wasn’t, why didn’t TSB tell customers about Data Centre move? Frightened of backlash? I think so!

        1. Anonymous Coward
          Anonymous Coward

          Re: Whose idea was this?

          Banks are always so transparent about where their data centers are and what they are doing with them, I find.

    3. Anonymous Coward
      Anonymous Coward

      Re: No doubt the deadline was forced onto TSB by the government

      No need for a conspiracy when incompetence will do.

      1. Loyal Commenter Silver badge

        Re: No doubt the deadline was forced onto TSB by the government

        No need for a conspiracy when incompetence will do.

        Yup, sounds like there was a bit of a hole somewhere in their change management, unit testing, integration testing, UAT, scale testing, and dry-run that they definitely must have done because of due diligence.

        What do you mean, they just pressed the 'Go' button and crossed their fingers?

    4. Anonymous Coward
      Anonymous Coward

      Re: Whose idea was this?

      Maybe someone unplugged the wrong server.

  2. silks

    Backout?

    Wonder if TSB are unable to backout their change in order to revert to the old Lloyds platform, or maybe the latest maintenance outage is to do just that?

  3. }{amis}{ Silver badge
    Joke

    TSB

    Totally Shagged Back-end.....

    1. malle-herbert Silver badge
      Joke

      Re: TSB

      Total Server Bullocking...

    2. DaveTheRave

      Re: TSB

      Totally Shit Bank....

    3. David Webb

      Re: TSB

      Time (to) Switch Banks

      - happy member of Starling Bank

  4. Doctor Syntax Silver badge

    "We are currently experiencing large volumes of customers accessing our mobile app and internet banking which is leading to some intermittent issues with people accessing our services."

    But isn't that what your mobile app and internet banking is for? Or are you still expecting customers to call into the branches that you've shut?

    1. Anonymous Coward
      Anonymous Coward

      Ah, you’ve clearly not visited a branch then - their front end systems aren’t working either, but they’re doing a good job of hiding that. Basically deferring keying of transactions by the looks of it. It may be better today.

  5. JakeMS

    Who Me?

    Can we expect a Who Me? article about this? Hopefully the people at fault read the reg and can contribute a good story!

  6. Anonymous Coward
    Anonymous Coward

    Damage Control

    Time for TSB to issue the usual statement "Lessons have been learnt, we have fired the poor peon in India who was entirely and solely responsible for this cockup, all affected customers will receive a free year of credit check courtesy of TSB Credit Checks (after this period, you will be charged £40 a month)... "

    1. Aitor 1 Silver badge

      Re: Damage Control

      In this case the poor peon in Lima, Peru.

      1. Anonymous Coward
        Anonymous Coward

        Re: Damage Control

        They outsourced it to Paddington Bear's Aunt Lucy?!

  7. LucreLout Silver badge

    I wonder....

    ... how will they detect all the fraud they're exposing themselves too? We've already seen in the media stories of people able to access £XX,XXX in other peoples accounts. What are the odds some dishonest scrote hasn't moved some of that "free money" to their own accounts? What are the odds other scrotes won't view their legitimate out going transactions and respond "Wasn't me guv".

    And that's to say nothing of those simply writing down names, account numbers, balances, standing orders, and direct debit info for whatever accounts they see, knowing they can phone up later and hack into the account then.

    Buggering up your outsourced IT is par for the course - you choose to do that when you choose to offshore - but displaying other peoples private information wen you do so is inexcusable.

    1. Killing Time

      Re: I wonder....

      'What are the odds some dishonest scrote hasn't moved some of that "free money" to their own accounts?

      Pretty slim probably as, if true, this access clusterf@@k means they have no guarantee the funds will end up where directed...

      1. LucreLout Silver badge

        Re: I wonder....

        Pretty slim probably as, if true, this access clusterf@@k means they have no guarantee the funds will end up where directed...

        They'd probably short cut that and transfer the money to an online account with another provider or bitcoin account. Relying on your screwup to be so bad that it saves you from fraud because literally nobody can do anything seems counterproductive and unlikely to engender goodwill amongst customers.

    2. TkH11

      Re: I wonder....

      Easy to detect fraud. They record details of all transactions and the time and date of when it was performed.

      All they have to do is sit back and wait for people to complain money has been stolen from their account. Then when the complaints roll in, investigate those complaints relating to transactions which occurred during the time window of the change.

      1. BongoJoe

        Re: I wonder....

        Assuming that they can tell that Account A's funds were transferred by someone logging in on Account B.

        If they can do that, then why were Account A's details shown in real time?

      2. Jtom Bronze badge

        Re: I wonder....

        Except the money has been wired out of the country, cash withdrawn, and the account closed. Fraud is easy to detect. Getting the money back, isn't.

    3. Joe 99

      Re: I wonder....

      I suspect Russia and North Korea will be taking a great deal of interest in the TSB infrastructure and other UK banks that have outsourced their IT/Ownership/Responsibility/Security (tick all that apply) based on this evidence.

  8. Allonymous Coward
    FAIL

    TITSUP

    TSB IT Service Unavailability Problem

    TSB IT Serviceable Usury Postponed

    TSB IT Silly Useless Plonkers

    TSB IT Switch to Unrelated Provider

    1. ArrZarr Silver badge

      Re: TITSUP

      Total Inability To Send Users Pennies

  9. Alister Silver badge

    [Nicky Morgan] said the reports of unauthorised transactions, access to other customers' accounts, and failures of in-branch services "have all the hallmarks of an IT meltdown".

    Is "IT meltdown" an officially designated term now? How is it quantified?

    1. Anonymous Coward
      Anonymous Coward

      needs to become an official el-reg measure

      1 TITSUP

      2 FOOBAR

      3 BORKED

      4 MELTDOWN

      for starters...

      1. davidp231

        Re: needs to become an official el-reg measure

        Where would SNAFUBAR fit?

        1. TkH11

          Re: needs to become an official el-reg measure

          You're combining two acronyns: SNAFU, FUBAR.

    2. Korev Silver badge
      Black Helicopters

      Is "IT meltdown" an officially designated term now? How is it quantified?

      Meltdown now has quite a specific meaning for IT security; hopefully politicians and the "mainstream" media* will stop using it in this sense.

      *By mainstream media I mean the non-technical press, not that lot -->

    3. Doctor Syntax Silver badge

      "How is it quantified?"

      By being big enough for politicians to notice something's wrong.

      1. AndrueC Silver badge
        Joke

        By being big enough for politicians to notice something's wrong

        ..so bigger than Brexit then?

  10. Brewster's Angle Grinder Silver badge

    Select Committee !== Govrernment

    C'mon, Kat, select committee's ain't part of the government and describing them so is sloppy; it's like calling Ofwat "the water companies", or misrepresenting a gamekeeper as a poacher.

    I seriously wanted to make a joke about how the government, given their track record, can't afford to lecture anyone else on IT. But I can't , in good faith, make that joke on the basis of this report. Except elliptically.

    1. The First Dave

      Re: Select Committee !== Govrernment

      Such committee's are usually _selected_ almost entirely from current and previous government members. The clue is in the name.

  11. Tom 7 Silver badge

    Excellent idea

    i'm sure the Government getting involved is going to reduce the pressure on the poor bastards trying to fix this so they can see the problem more clearly and fix it.

    1. Allonymous Coward
      Pint

      Re: Excellent idea

      True. As a TSB customer and an IT professional it's easy to mock. But we've all been there, know someone who has, or had to clean up the mess someone else has made. So the techies scrambling to fix this have my sympathies. The architects, BAs, PMs and C-level execs who ultimately should've made sure this didn't happen, not so much.

      Icon, for what I hope someone buys the developers, DBAs and ops people when it's all over.

      1. Anonymous Coward
        Anonymous Coward

        Re: Excellent idea

        Maybe the devs, dba’s and op’s people ignored the architects and did their own thing as they thought they knew better...

        1. tfb Silver badge

          Re: Excellent idea

          Maybe the devs, dba’s and op’s people ignored the architects and did their own thing as they thought they knew better...

          I can see you haven't worked in banking IT.

          1. Anonymous Coward
            Anonymous Coward

            Re: Excellent idea

            I can see you haven’t, i have and do...

          2. Anonymous Coward
            Anonymous Coward

            Re: Excellent idea

            A little bird tells me that the majority of the problems do indeed lie with Sabadell's IT team generally ignoring the requests and requirements given to them by TSB.

        2. Anonymous Coward
          Mushroom

          Re: Excellent idea

          More likely they followed the architects design, despite the techies telling them it was an incomplete, half-arsed, ill conceived bag of shit.

          Of course that's never happened here.

        3. Loyal Commenter Silver badge

          Re: Excellent idea

          Maybe the devs, dba’s and op’s people [sic] ignored the architects and did their own thing as they thought they knew better...

          Much more likely, the devs DBAs and ops people did know better, since their job roles will involve a degree of expertise in their chosen fields, but their concerns went unheard because their middle-managers didn't want to jeopardise a comfortable life by passing problems upwards.

          The 'architects' in this case were probably a bunch of sales types who got a Groupon to do a cheap PRINCE2 course and now believe that they know everything there is to know because their pinstripe suits are snappier, and are almost certainly the root cause of the problem. They probably think testing starts and finishes with making sure the software doesn't emit errors when you run it up.

          1. Anonymous Coward
            Anonymous Coward

            Re: Excellent idea

            Maybe the architects took a TOGAF course instead of a PRINCE2 course, and just maybe they are technical experts in their field, just like all the other architects ive met working in Technical Architecture for banks...

  12. Gordon Pryra

    Ironic Headline?

    "UK.gov demands urgent answers as [insert name here] IT meltdown continues"

    Normally it would be the other way around.

  13. Anonymous Coward
    Anonymous Coward

    Statements just announced

    Banco Sabadell - 'We have successfully migrated the TSB customer data'

    TSB - 'We are experiencing "intermittent" issues'

    Customer 'I can not access my accounts, funds or get any information'

    Financial Conduct Authority and Information Commissioners Office - AN ABYSS OF SILENCE

    A truely Fu**ed up situation with government bodies doing the same as the TSB systems , sweet FA.

    1. Anonymous Coward
      Anonymous Coward

      Re: Statements just announced

      "Financial Conduct Authority and Information Commissioners Office - AN ABYSS OF SILENCE"

      From the article:

      An ICO spokeswoman said: "We are aware of a potential data breach in relation to the TSB and are making enquiries."

      Also a few points to consider:

      1. Just because you haven't heard anything doesn't mean that they aren't doing anything.

      2. Investigations take time and they are likely waiting for TSB to fix the problems before looking into what has happened.

      3. The FCA is not a government body.

  14. Anonymous Coward
    Anonymous Coward

    Do the TSB not have anywhere left where you can just go and draw your money out?

    If you have dealings with a someone who is not giving you the service you require then you complain and if they ignore you then you stop doing business with them.

    Clearly there is little point complaining as they are not listening so just move to another bank that does listen, where is the problem? you stoopid or sum fing

    If they refuse to give you your money then, you have something to complain about and doing it via the courts it is less easy to ignore.

    1. Doctor Syntax Silver badge

      Re: Do the TSB not have anywhere left where you can just go and draw your money out?

      "Clearly there is little point complaining as they are not listening so just move to another bank that does listen, where is the problem?"

      Finding that other bank that does listen. Most of us are running out of choices.

      1. Anonymous Coward
        Anonymous Coward

        Re: Do the TSB not have anywhere left where you can just go and draw your money out?

        "Finding that other bank that does listen" then use cash instead and be in complete control of your finances.

        There used to be life before banks and it is only recently that having a bank account was compulsary, given what has happened since banks were guaranteed access to the everyones wallet then perhaps it is time to review our options.

        Clearly the customer is not being put first here and for every banking fkup the same management never learn from their errors, simply because we also come last with the Government too

        1. Peter2 Silver badge

          Re: Do the TSB not have anywhere left where you can just go and draw your money out?

          The option that everybody really ought to exercise is to have two accounts. Preferably one that you use daily, and another (with a completely seperate supplier) which holds savings.

          When a problem hits on your main account, switch to using your savings account until they fix it.

          1. TkH11

            Re: Do the TSB not have anywhere left where you can just go and draw your money out?

            Different accounts in different banks and make sure those banks really are using separate IT systems,

          2. jabuzz

            Re: Do the TSB not have anywhere left where you can just go and draw your money out?

            I would recommend making sure you have at least one Visa card and one Mastercard, and some actual cash hidden somewhere safe in the house. I also recommend a third card that you don't actually carry around for when you loose/have stolen your wallet/purse.

  15. Milton Silver badge

    Ah the sweet smell of irony

    There's something almost endearingly charming about listening to yet another political mouth-on-a-stick criticising someone—anyone—else for incompetence. It's even more darkly hilarious coming from a government that f***s up every single IT project it ever touches, costing taxpayers tens of millions.

    I'm sure TSB have been playing to the Tory handbook of cost-cutting, cheap half-measures, fat bonuses for the board, screwing the customers at every opportunity and all the other shyte that major corporations do to score short-term returns (yum, especially those nice bonuses: grab it and move to the next fiasco, quick!), while storing up disastrous weaknesses and catastrophic problems for the future.

    Anyway, not to worry. None of the senior management who lovingly laid the groundwork for this clusterf**k will be punished; no lessons will be learned; no meaningful regulation will be imposed; lobbyists' enticements and inducements will flow unabated; politicians will continue to dodge responsibility; customers will suffer; and the torrent of lies, evasions and excuses will be epic.

    Er, yes, and everything else it touches, really ... doesn't seem to matter whether it's health, education or defence, you can always trust them to bring their own unique combination of stupidity, arrogance and dishonesty to the table and leave a steaming mess behind.

    1. RancidRodent

      Re: Ah the sweet smell of irony

      "The Tory handbook"? Remember Tax Credits? £5bn stolen wasn't it? Notice the vast majority of IT outsourcing (including government IT contracts) to India occurred under 13 years of Labour - the same people who took the monitoring of UK bank liquidity off the Bank Of England and gave it to a toothless quango - the rest is history. Don't get me wrong - I can't stand the current shower in government but it was under Labour's watch the UK IT industry was decimated and the UK banking industry brought to its knees - the former by easy access given to our markets (visa rubber stamping), the latter partly caused by the one-eyed idiot's banking act(s).

      1. Anonymous Coward
        Anonymous Coward

        Re: Ah the sweet smell of irony

        I think that was the same lot that removed the controls on the bookies and neutered the licensing laws.

    2. anothercynic Silver badge

      Re: Ah the sweet smell of irony

      Whaa whaa Tory whaa whaa! Stop this godforsaken 'let's blame the Tories for everything' game. I am not a Tory, but I find this kind of cowardly buckpassing completely pathetic. The Tories are as bad as Labour, or the Lib Dems, or any other party that's been in government (coalition or otherwise) since the 60s.

      Labour cocked up with Windrush (not requesting the imaging of the landing cards of Windrush arrivals, instead ordering the paper copies irretrievably destroyed in 2009) which has now led to the Tories carrying the can for it (thanks to their 'hostile environment' policy at the Home Office), Labour cocked up with the Passport Agency, the Child Support Agency, the NHS and its PFI clusterf***s (which is driving NHS Trusts into failure), LibDems cocked up with the tuition fees and probably the all-digital Cabinet Office strategy (i.e. GDS), but hey *WHAA WHAA TORIES WHAA WHAA*.

      Do f*** off.

      1. Dan 55 Silver badge

        Re: Ah the sweet smell of irony

        The actual destruction of the landing cards was carried out in 2010. I think the Tories had time to decide whether or not to a) scan before destroying or b) not destroy the cards. They were also advised that it would hit the Windrush generation, but they pressed ahead anyway and removed legal protection for them in the 2014 Immigration Act for good measure.

        So they may be as bad as each other, but in this case one is worse than the rest.

        1. Peter2 Silver badge

          Re: Ah the sweet smell of irony

          Somehow, I suspect that nobody went and asked the cabinet minister if a decision made by the previous minister to destroy some old documents from the 1960's ought to be reviewed.

          And the constant "it's Labour's/Tories fault11!!11!" poisionous party posturing hides more fundemental issues.

          For instance, let's see what the Labour cabinet minster had to say about his department in 2006.

          "Our system is not fit for purpose. It is inadequate in terms of its scope, it is inadequate in terms of its information technology, leadership, management systems and processes," he told MPs"

          This was IIRC greeted by laughter in the commons instead of serious determination to expunge, eliminate, eradicate and exterminate the problems and reform the civil service department in question to the point it could efficiently and effectively acomplish it's purpose for existance. One wonders if more recent minsters agree with his assessement of the department, and if so what is being done about it.

      2. Loyal Commenter Silver badge

        Re: Ah the sweet smell of irony

        Labour cocked up with Windrush (not requesting the imaging of the landing cards of Windrush arrivals, instead ordering the paper copies irretrievably destroyed in 2009)

        Whilst there is certainly a lot of buck-passing going on, I think the fact that the decision may have been made in 2009 is not as relevant as the fact that it was made by a civil servant, with no involvement of the Home Secretary, whomever it was at the time.

        Theresa May, however, is on the record, when she was Home Secretary, as promising the unreachable immigration targets, and preaching the 'hostile environment'. The last Labour lot may have been bad (I'm no fan of Blairite Red Tories), especially with their data-collection fetishism and creeping authoritarianism, but May et al are arguably a lot worse, especially with May's attitude towards human rights (which can be summed up as "pesky humans, why should they have rights?").

        Also, the "was it Labour / was it the Tories" argument neatly sidesteps the fact that you can hardly blame the current Labour leadership of sharing an ideology with Blair and Brown, which was a lot further to the authoritarian right than the party currently lies.

        1. anothercynic Silver badge

          Re: Ah the sweet smell of irony

          The fact remains that Labour made the decision, Labour also made other decisions. Just like the Tories made some decisions in the past that the current Tories should/cannot be lambasted for.

          Both major parties are as full of visionaries and sh**bags as each other. That's the point here. Any party follower pointing fingers at the other party should first acknowledge that their party are just as big a bunch of cockup merchants as the ones they are pointing at.

      3. TkH11

        Re: Ah the sweet smell of irony

        Agreed. Politicians don't specify or build the systems.

        Blame the civil service for constantly changing their minds about what they want.

        Blame the companies building the systems for poor project management, and poor technical staff.

  16. smudge Silver badge
    Holmes

    Morgan added: "This is yet another addition to the litany of failures of banking IT systems. Potentially millions of customers could be affected by uncertainty and disruption.

    "It simply isn't good enough to expose customers to IT failures, including delays in paying bills and an inability to access their own money.

    "Warm words and platitudes will not suffice. TSB customers deserve to know what has happened, when normal services will resume, and how they can expect to be compensated.

    Not defending TSB, of course, but for a politician and former Government minister to say all this, without a hint of embarrassment or self-awareness, just beggars belief. Does she know nothing of the Government's record in IT?

    Does the term "Universal Credit" mean anything to you, Nicky? Because you have just described it in a nutshell.

    1. Anonymous Coward
      Anonymous Coward

      The forces recruiting system is still a mess and that is four months later.

    2. TkH11

      Hold on. There may be irony involved because of the government's appaling record of IT failure, but that does not preclude the government from criticising companies. These companies need telling off and holding to account, and the government is the right party to do that.

      It is just a shame that the people they employ on House of Commons Select Committee hearings know nothing about IT.

  17. Anonymous Coward
    Anonymous Coward

    Banco Sabadell regard it as a success, so there you go:

    Banco Sabadell: Banco Sabadell successfully completes TSB technology migration - 23/04/2018

    As for techy stuff, TSB's system is an in-house system called Proteo4UK, based on Sabadell's own Proteo which itself was based on something Accenture knocked up (Alnova). Sources:

    Sabadell looks to TSB in UK growth strategy - non-paywalled version via Google News.

    TSB unveils new banking tech platform, Proteo4UK

    Banco Sabadell and Accenture Team to Transform the Bank's Technology Infrastructure - January 19, 2000

    In other words, abandon hope all TSB customers who bank here. They can't roll back because TSB are off Lloyds, they just have to grin and bear it while it gets fixed which could take a while. Or switch to another bank, probably involving a visit to a TSB branch, a banker's draft, and changing direct debits with every company manually.

    1. yoganmahew

      Thanks for the links!

      From one of them:

      Carlos Abarca, CIO of TSB, says ... “Proteo4UK was built in close co-operation with world-class companies, and has very few legacy systems,” he states. “It is a brand new core banking system for us.”

      This would be that meaning of "legacy" that roughly translates to "working".

      Unlike others, I'm having a hard time being sympathetic to the peons working on the emperors new clothes. Too much bad in IT is already waved off as somebody else's fault. This TSB IT thing is rotten from the head of IT down.

      1. anothercynic Silver badge

        You mean the kind of 'working legacy' systems that rely on ancient batch systems to do stuff overnight instead of instantly? Oh, right, *those* kind of legacy systems...

        Sometimes an all-new digital system is very useful. It gets rid of a lot of cruft that builds up over decades. A banking group by the name of ABSA in South Africa successfully merged *four* banks together onto a new platform in the nineties... granted, back then we didn't have Internet banking, but damn, the novelty of being able to pay some money to someone else and have it instantly appear was great, especially when the grand dames of British banking still believed that things *had to* take at least 3 days to happen (unless you withdrew it from one bank over the counter, walked down the road with a massive wodge of cash in your pocket and looking over your shoulder for fear of being robbed, and then paid same wodge of cash into the other bank and hoped it would be cleared instantly).

        So yes, please, don't give me that crap of 'legacy = working'. Legacy might = working, but possibly working by shoe string, bubble gum, some staples and a paperclip.

        1. Sgt_Oddball Silver badge

          You forgot the half a tennis ball that no one else notices until it's not there anymore... then all hell breaks lose on a really tricky edge case...

        2. Anonymous Coward
          Anonymous Coward

          I do so love the smell of noobs in the morning! :)

          Batch processing may be a bit out of fashion these days, but it's still often the best way to check up on the "instant" stuff to see if that's actually working correctly. It can also still be very appropriate for quite a few other things, such as managing loads where resources are tight.

          For example, I used to run batch exception audits on a system where the real-time transaction flow was incessant and on busy days was quite extreme (the system might have real problems keeping up with the load). The trick was to run that audit twice in a row in order to see what showed up there. Something that turned up as out-of-balance on the first run but not the second was just an inflight, incomplete transaction which could be safely ignored. But anything that showed up on both runs was almost always a genuine problem which needed to be investigated and corrected.

          The weird part about this was that I often had to explain to my younger colleagues what I was doing here and why I was doing it. I was usually the best in the group at tracking down problems quickly and finding and correcting any systemic issues behind them, but I often left other folks scratching their heads after I tried to explain to them that this batch audit was just part of my "secret sauce".

          As for load issues, one way to often correct these was to split the transactions into those pieces which pretty much needed to happen immediately, versus any pieces that needed to happen fairly quickly but not right away (high-priority batch), versus any pieces that could generally take their time (low-priority batch). Once again I often had to explain to my younger colleagues the wisdom of doing this, since their knee-jerk reaction to any overload situation was usually "Just upgrade the infrastructure!", something which would take both time and money. These days this might not involve much more than a button-click or whatever, but at the time such an endeavor could turn into a big freaking deal.

          BTW, as for your money transfer example I think you may be confusing "instant" processing with "optimistic" processing, which are two very different things. And yes, "legacy = working" is frequently an accurate assessment of the situation, so I probably wouldn't go around dissing that too much if I were you.

  18. cantankerous swineherd Silver badge

    branches. cheques. cash.

    use them or lose them.

  19. Anonymous Coward
    Anonymous Coward

    Buck passed to...

    ...overseas IT outsourcing in 3..2..1..

    1. TkH11

      Re: Buck passed to...

      I am not sure in this case if that would s the case, but if it was, they would never admit it.

    2. TkH11

      Re: Buck passed to...

      When my employer offshored system support to the Asian subcontinent to save money, we saw one key effect - the duration of operational outages increased dramatically.

      One incident took several months to investigate;I got involved. Gave them a few pointers, made them think, and within 24 hours they had found the fault.

      There are huge cultural problems with using low cost workers from the Asian subcontinent, and you shouldn't use them for any kind of support or development. I s'pose I shouldn't complain too much: their incompetence kept me in work.

  20. Anonymous South African Coward Silver badge

    TSB having an ELE?

  21. Anonymous South African Coward Silver badge

    TITSUP

    TitSup Bank?

  22. adam payne Silver badge

    There were also reports that TSB customers could access the accounts of other TSB users, including their account numbers, sort codes and transaction histories, and possibly having the ability to perform actions on these accounts, including transferring money.

    A month before GDPR, lucky you TSB.

    A TSB spokesperson told us yesterday: "We are currently experiencing large volumes of customers accessing our mobile app and internet banking which is leading to some intermittent issues with people accessing our services."

    Well after you close so many branches you would I hope have been expecting that.

  23. noboard
    Coat

    I just hope...

    that the top brass at TSB have learned from this unfortunate situation and will take steps to make sure it doesn't happen again.

    1. Anonymous Coward
      Anonymous Coward

      Re: I just hope...

      Hardy har har.

    2. TkH11

      Re: I just hope...

      Top brass hasn't a clue what is going on. They rarely are IT professionals.

  24. Anonymous Coward
    Anonymous Coward

    Good luck to TSB

    If you could just lose details of my mortgage I will not complain.

    1. JimBlueMK

      Re: Good luck to TSB

      You would when you remember that the Bank will have its name as an interested party on your the deeds at Land Registry which would make selling the house or leaving it to the kids a legal nightmare to sort out what with all the missed payments and accruing interest.

    2. Sasswell

      Re: Good luck to TSB

      >If you could just lose details of my mortgage I will not complain.

      I wonder if Elliot Alderson works for TSB.

  25. thegrouch

    If a substantial number of people go to withdraw their balances from TSB, is there a risk of a run on the bank? I don't know how much needs to be removed for the bank to physically run out of money. Expect the BoE to keeping an eye on this.

    1. TkH11

      Are you really advocating bringing down the bank? Shame on you....

      1. Richard 12 Silver badge

        I wonder how long

        Before they are dead.

        They've almost certainly lost more than 10% of their customers by now.

        If they lose 20% or so, they are calling on the FSCS and winding up.

  26. RancidRodent

    Damn those legacy - er, um, new fangled systems!

    The sound of silence is deafening over the fact that this is a modern state-of-the-art banking system, dragged off the boring, reliable, Lloyds mainframe and put on edgy x86 cloudy stuff - and - oh dear - it's all gone wrong. If this was a mainframe system, there would be finger-pointing at the platform from every commentator - as it is - nada - nothing to see here, move along...

  27. Anonymous Coward
    Anonymous Coward

    Use of Live Test Data

    Did it work along these lines....

    1. Information Commissioner: "Live data cannot be used for testing"

    2, Information Commissioner: "You should have tested the systems properly".

    Rinse and repeat.

    1. Doctor Syntax Silver badge

      Re: Use of Live Test Data

      "You should have tested the systems properly"

      I suppose it could be tested by moving accounts over in tranches. Whose accounts should go in the first tranche? How about senior managements'? Second tranche? Senior management of any big consultants and audit firms involved? What - none of them bank with TSB?

      1. John G Imrie Silver badge
        Joke

        Re: Use of Live Test Data

        What - none of them bank with TSB?

        Well some of them did, but they closed their accounts last week, allegedly.

    2. Killfalcon Silver badge

      Re: Use of Live Test Data

      You can get exceptions to that policy. You just need to ensure Test is secure as hell and your developers (with access to whatever you're calling the 'live data test' environment) are audited at least as well as your production staff. It's a *risk* you can *control*, though it is genuinely best to not have most dev work done on live data, and use your experiences with the 'live test' to craft better test cases for the lower environments most dev work is done against.

      1. TkH11

        Re: Use of Live Test Data

        The problem is always going to be that when you construct test data how realistic is it? I did some work on a system a few weeks ago and I could not obtain a data model of my source database and only in time I discovered problems in the live dataset, which I needed to cater for. Had I constructed test data I would have built it to what I expected the data model to be and my software would have failed.

        An employee working with sensitive live data simply has to sign an NDA, now that doesn't guarantee they won't steal the information, so you have to also consider who the people are that are working on that data, and which country they are in. And worst case, you can pseudoanonymise it by tokenisation.

    3. TkH11

      Re: Use of Live Test Data

      What is the problem with using live/real data in a test system? As long as it is protected in all the usual ways. And as long as you ensure the test system is kept separate and isolated from the production system so you don't inadvertently update the production system with test transactions from the testing?

  28. open_paul

    They obviously had to move off Lloyds systems which makes sense but oh boy - what went wrong between the testing of this and the live implementation?

    Surely you would do a dummy run using your staff that have accounts first or would phase this in over a couple of weeks.

    I'm a TSB customer and was locked out yesterday with payments not happening as they should. Not ideal.

  29. MartyOhr
    FAIL

    The bank that likes to say si

    How hard can it be?

    Big bang IT changes are always fraught, but an online banking website and app is not really that hard surely? In my simple head it's just a long database of transactions against account numbers with a relatively simple front end.

    Of course if I'd been doing it I would have moved customers in batches. Or I'd have offered reduced functionality over the weekend rather than shutting up shop. At least then there would be a small element of goodwill from customers rather than pent up demand desperate to get on the new platform.

    The official reason for the move is to stop paying Lloyds for use of its IT systems by the Spanish parent company of TSB. It seems ludicrous in this day and age of 'anything/everything as a service' to expose yourself to so much risk by taking stuff back in house. But I've spoken to other retail finance companies who are in the process of 'de-clouding'. Is this a new trend?

  30. ghp

    It happens to the best ...

    It was 18 days ago today

    https://www.theregister.co.uk/2018/04/06/belgian_bank_argenta_outage_botched_it_infrastructure_upgrade/

  31. ChoopaChoopa

    https://www.tsb.co.uk/news-releases/news-releases/tsb-unveils-new-banking-platform/

    They didn't even put in the effort to spell effort right... also I think they meant 2,500 man hours not man years...

    You trust them to bank with?

    1. Anonymous Coward
      Anonymous Coward

      No, I think man years is actually correct. Assuming a team of around 1000 people across TSB, Sabadell, and third party contractors (a reasonable estimate for manpower) and the two and a half year timescale - this has been in the pipeline since at least 2015, 2500 man years is correct.

      2500 man hours is nothing. It's the equivalent of 250 people doing one day's work.

  32. Mickybhoy55

    IT Shambles.

    This was meant to be a relatively straight forward migration from a Lloyds data centre to Banco Sabadell one, run from Barcelona. Despite years of planning it has gone badly wrong. TSB never told customers that data was moving, afraid of an exodus backlash I think...

    1. Mike Henderson

      Re: IT Shambles.

      Post-Brexit, will an English-registered retail bank actually be able to outsource its entire operations to what will suddenly be a completely foreign jurisdiction? So its actual banking operations* will no longer be subject to English law? Really?

      The Reserve Bank of NZ stopped the Australian-owned main trading banks in NZ (i.e. 90% of the market) from doing that, insisting they do their processing in NZ, not Australia.

      Just wonderin'

      * the bank branches, ATMs etc are just interfaces between the customer and the bank. All the real banking operations are IT systems nowadays

  33. TkH11

    The government can demand what it likes, companies like this hide behind obfuscation. They rarely disclose what actually happened.

    Look at the big BA scandal recently, they bluffed their way through with an incomplete explanation claiming they had a power surge when too many of their systems were turned back on at the same time, but they never disclosed what caused the original power failure and why their battery and generator systems did not kick in.

  34. Roger Ramjet

    Rest assured - we're in safe hands....

    https://pbs.twimg.com/media/Da_CYjVXUAEVsOz.jpg:large

  35. John70

    TSB Online banking is back up but it's still flaky as shit.

    Most of the time it's "We're limiting access to Internet Banking" and when you get that window of opportunity to login, it's slow as fuck. Get a blank screen but "viewing source" reveals there is HTML there and if you refresh the page you get logged out and the "We're limiting access to Internet Banking" message again.

  36. Anonymous Coward
    Anonymous Coward

    Some commentators clearly in the know on here. Work for TSB or LBG I wonder?

    When TSB was spun out of LTSB, they had a 10 year contract to use the Lloyds systems with a 5 year break clause in that contract.

    In order to save money, as others have pointed out, the owner of TSB (Banco Sabadell) activated that clause and for the last three years, have been building to this point.

    Unfortunately, SABIS - that's the IT arm of Banco Sabadell - haven't quite succeeded in switching TSB's customers across. This should have all been completed at least 6 months ago and properly tested but because of SABIS's wish to do it as cheaply as possible, instead of spending a bit more money, SABIS have cut a few corners in both Madrid and Barcelona and ended up with the mess we saw now.

    The blame lies firmly with SABIS, not TSB.

    1. Tom 7 Silver badge

      Re: The blame lies firmly with SABIS, not TSB.

      I'd imagine it would lie with the Director whose bonus required the job to be completed soon.

      1. Anonymous Coward
        Anonymous Coward

        Re: The blame lies firmly with SABIS, not TSB.

        That'll be three directors (at SABIS) then

  37. John 61
    FAIL

    Online banking is totally secure...

    is what I'm told when I visit my local open plan bank (which is bad enough). I refuse to use online under any circumstances.

  38. uccsoundman

    Heard inside the develpment office

    We're Agile; we don't need to test.

  39. GSTZ

    Shiny new systems ...

    From the artcle: "TSB migrated from former parent Lloyds Banking Group's systems to shiny new ones" ...

    Moving from centralized and highly deterministic systems to "shiny new" systems that (from the outside) may even look centralized too, but are in fact a highly complex conglomerate of many thousands of "PC's" all doing more or less their own thing but also being dependent on the outcome of many other "PC's" to complete their tasks isn't easy. It doesn't help much that these "PC's" are no longer small separated physical machines like in the early days of distributed computing, rather myriads of virtual machines running on some kind of x86 infrastructure coming with bombastic marketing wording but behaving like a bunch of PC's anyway. Predictability suffers, such systems are certainly "good enough" to handle enormous workloads for less critical applications like Facebook and Twitter but might be less than ideal for really critical stuff like banking operations.

    This is not crying for the good old past based on legacy systems, as it has already been pointed out that old systems eventually become a real pain when too much new functionality gets added. At some point, it is better to start with a clean slate - but also on a highly deterministic system providing better reliability, predictability and security than the "good enough" gear that has become the de facto default for each and every new application these days. Unfortunately, most of the younger IT folks do not even realize that alternatives do exist.

    The prevailing hardware stuff comes relatively cheap, but the business results tend to be mixed as reliability, efficiency and security have "room for improvement" and the cost to run and support those very complex systems becomes too high. Many user organisations now do escape to the public cloud, even the military are now considering such moves. However, it is unclear how cloud providers having less knowledge of the business requirements and less incentive to provide superior service levels for critical applications will be able to serve their customers better.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019