back to article TSB outage, day 5: What do you mean you can't log in? Our systems are up and running. Up and running, we say!

Embattled bank TSB has claimed its IT systems are now "up and running" – despite many users still reporting they can’t get into their accounts. The bank’s online services have been down for almost a week, after TSB's planned migration off former parent Lloyds’ tech infrastructure – some five years after its split from the …

  1. OnlyMortal

    I'd switch bank if I was one of the effected. It's very easy to do nowadays.

    1. vtcodger Silver badge

      In the short term, switching may not be so easy if you can't access your assets in the old bank. In the longer term, switching might be an excellent idea.

    2. wolfetone Silver badge

      "I'd switch bank if I was one of the effected. It's very easy to do nowadays."

      I wouldn't. I'd open another bank account with a different bank not connected to the bank you already have.

      Why? Well at least you have access to some money if your bank has gone up shit creek with their 2,500 man years of work as a paddle.

      I was considering dropping the Co-Op for TSB and have accounts with both of them, but as you can guess right now I'm feeling fairly safe(ish) that while I can't access my TSB account I still have access to the Co-Op one.

    3. Inventor of the Marmite Laser Silver badge

      I, too, would Effect a transfer if I was one of the Affected

      1. Dr Who

        Yes, but did he make the error pacifically to piss people like you and me off?

        1. Allonymous Coward

          Definately.

          1. TRT Silver badge

            That's the most annoyingist thing...

            1. Deej

              OMG, you grammer Nazi's

              1. DontFeedTheTrolls

                Sounds like the grammer Nazi's need some comforting....

                Their, they're, there

                1. Anonymous Coward
                  Anonymous Coward

                  Re: Sounds like the grammer Nazi's need some comforting....

                  Here hear heir

                2. Jtom

                  Re: Sounds like the grammer Nazi's need some comforting....

                  It's important to distiguish the difference between knowing your shit, and knowing you're shit.

              2. Anonymous Coward
                Anonymous Coward

                OMG, you grammer Nazi's

                Nope its: OMG, you grammAr Nazis (no possessive apostrophe)

                1. Anonymous Coward
                  Anonymous Coward

                  Re: OMG, you grammer Nazi's

                  It's (contraction of it is), not its (possessive).

    4. justAnITGuy

      Try doing that when the donor bank's systems are down. The gaining bank won't be able to perform the switch.

  2. Anonymous Coward
    Anonymous Coward

    All code is written by offshore idiots to the lowest price

    This shitty code is in your medical devices, cars, industrial systems, phones, apps and most devices in your homes. It's present on every website you visit.

    Insecure by negligence and stupidity, it's everywhere in your life.

    But hey - psychopaths are running the companies that make this stuff & they don't give a shit. They are cutting cost to get paid. You are not the 1%, so fuck you.

    1. Anonymous Coward
      Anonymous Coward

      Re: All code is written by offshore idiots to the lowest price

      Accenture in Sant Cugat (outside Barcelona) are Sabadell's outsourcer of choice.

      1. colinb

        Re: All code is written by offshore idiots to the lowest price

        Accenture rarely get their hands dirty with code, that's Avanade's (who they own) job and whose devs will mostly reside in India.

        Firewalls of blame are very important to the PHB's

        1. Anonymous Coward
          Anonymous Coward

          Re: All code is written by offshore idiots to the lowest price

          Sorry most of the coding has been done in Spain by Accenture and other Spanish companies.

          1. katrinab Silver badge

            Re: All code is written by offshore idiots to the lowest price

            Accenture isn't Spanish. It is an American company that is registered in Ireland for non-taxpaying purposes.

          2. colinb

            Re: All code is written by offshore idiots to the lowest price

            That's for clearing that up, in that case Accenture Spain is different to Ireland in that respect.

            Spain is offshore to the UK in this case.

            Worked out well for them, hasn't it. Normally their failures are kept very private. They should have paid more attention to how public this would be.

            1. Anonymous Coward
              Anonymous Coward

              Re: All code is written by offshore idiots to the lowest price

              The full list of consultancies involved are: Accenture, Everis, Indra, GFT, IBM, HP and BT. Everis and Indra are Spanish, the others have branches in Spain.

              Link

              1. Anonymous Coward
                Anonymous Coward

                Re: All code is written by offshore idiots to the lowest price

                It’s quite extraordinary that neither Capita or Lockheed got their blundering tentacles into a fuck up of this magnitude.

                1. Experience ? whats that

                  Re: All code is written by offshore idiots to the lowest price

                  I understand that allegedly HP / DXC and Accenture did ,so all's good from a continued feck up / continuity point of view

              2. Anonymous Coward
                Anonymous Coward

                Re: All code is written by offshore idiots to the lowest price

                Don't forget KPMG were involved with the testing.

      2. Anonymous Coward
        Anonymous Coward

        Re: All code is written by offshore idiots to the lowest price

        Steer clear of Accenture and anyone who ever worked there.

    2. Anonymous Coward
      Anonymous Coward

      Re: What about auto-updates?

      There are idiots, offshore and onshore, in this and every other field. Bootnotes have served up many a tale.

      The corollary to your statement then is that if you aren't the highest paid in your industry you are an "idiot" to someone else. There is some truth to this - we all don't know something..

      The race to the bottom on price is where the problem is at. You get what you pay for.

    3. Anonymous Coward
      Anonymous Coward

      Re: All code is written by offshore idiots to the lowest price

      "You are not the 1%" .....

      Oi..... some of us AC's might be, in the 1% that is. Plenty more will be psychopaths.

      1. Yet Another Anonymous coward Silver badge

        Re: All code is written by offshore idiots to the lowest price

        All code is written by offshore idiots to the lowest price

        That's their mistake - offshore the customers, it's much cheaper and easier

  3. Flywheel
    Joke

    So it's the bl00dy accountants!

    Bean creation not allowed .. do not request a bean from the bean factory

    They've obviously blown their budget.

    Let's see the b*ggers wriggle out of *that* one! Ha!

    1. Anonymous Coward
      Anonymous Coward

      Re: So it's the bl00dy accountants!

      > Bean creation not allowed .. do not request a bean from the bean factory

      (Cash) cow traded for 3 beans and Jack to show for them. :-)

  4. Roger Ramjet

    Experts at the helm....

    https://pbs.twimg.com/media/Da_CYjVXUAEVsOz.jpg:large

  5. Anonymous Coward
    Anonymous Coward

    AIOOB and BeanFactory Exceptions being surfaced all the way to the front end HTML in a retail bank's customer facing web products, after it's been claimed that everything is fixed.

    This may qualify as one of the more comprehensive failures of software engineering, testing and delivery that I've seen.

    And yet that failure pales in comparison to the organisational failures and managerial incompetents that caused it happen.

    1. Anonymous Coward
      Anonymous Coward

      It's 2018 and people are still using arrays...

      1. HieronymusBloggs

        "It's 2018 and people are still using arrays..."

        Which array-free OS/compiler/VM/interpreter are you using?

        1. Anonymous Coward
          Anonymous Coward

          Must must be someone from academia - they're in a parallel universe.

      2. Anonymous Coward
        Anonymous Coward

        "It's 2018 and people are still using arrays..."

        ... because in the hands of competent analysts and programmers they remain one of the best ways of representing a myriad kinds of data, and because, properly used, they enable the functionality of some of the most advanced high level languages available.

    2. Anonymous Coward
      Anonymous Coward

      If it's surfacing this sort of crap to the user it's a certain bet that the security design is broken too. The best security defence they have left is that their site is too broken for people to log on.

    3. rmacd

      The masochist in me wants to see some of this code. Especially where AIOOBs aren't being caught. The other part of me wants to see it just to give me an ego boost.

      There is something intensely gratifying about seeing CIO's fucked over by outsourcing.

      No doubt though, the blame will be pinned on the (non-technical) PM than the shitty devs who don't know their arses from their elbows.

      1. Anonymous Coward
        Anonymous Coward

        "Especially where AIOOBs aren't being caught"

        The code is clearly Spring in nature, which means the entire application is almost certainly wrapped in a global exception handler; Spring often uses Exceptions as a standard control flow mechanism. This means someone has decided it's a good idea to pass through AIOOBs to the presentation layer (probably through a default catch & rethrow "handler"), and the presentation layer has decided it's a good idea to display internal Exceptions verbatim.

        This is first year software engineering student shit that doesn't even pass static analysis, never mind professional code review. Stinks of a rush job.

  6. Anonymous Coward
    Anonymous Coward

    TSB stands for The Shit Bank?

    1. Ye Gads

      Oh come on, we can do better than that!

      TSB:

      Test Software Before!

      Test Software Better!

      Tanked So Badly

      Trashed Someone's Bonus?

      So many possibilities

      1. Ye Gads

        Re: Oh come on, we can do better than that!

        Terrible Software Blamed

        Techies Shaft Board

        Testing Saves Bonuses

        Terribly Stupid Board

        Try Software Backup?

        Tools Sometimes Break

        Totally Shafted Business

      2. Ye Gads

        Re: Oh come on, we can do better than that!

        Try Some-other Bank...

        (yes, that's cheating, but good advice)

        1. John H Woods Silver badge

          Re: Oh come on, we can do better than that!

          Try Switching Bank, surely

  7. Voidstorm
    WTF?

    "Load Balancer Errors" is the clue

    Having been a client-server Java developer for many years, this is the sort of thing you get in a postQA environment when you rollout a live solution and the stress-test hasn't been done well, and the stress-testing environment doesn't replicate the live one sufficiently effectively.

    The internal resource shortages (threadpool, database connections, config problems etc are good examples) don't truly manifest themselves until a *genuine load* hits the *live techstack*; even with the best will in the world.

    The fact that all three front end layers (Web, App and In-branch clients) are evidencing "Javabean" errors speaks to problems at the server layer of the architecture; hence the restrictions on number of logged in clients => less chance of server resource exhaustion.

    Clearly there are issues with the not-so-adequate scale of the backend infrastructure. From experience, these are the hardest to assess from the point of view of a development team; even if the solution is properly written and tested, and passes QA, the live environment can hilight resource problems where the infrastructure isn't well provisioned.

    And thats assuming a perfectly solid Dev/QA process, into the bargain

    1. Anonymous Coward
      Anonymous Coward

      Re: "Load Balancer Errors" is the clue

      It seems they threw in Netflix's open source load balancer and hoped it worked. You can tell because the error gets thrown to the mobile app.

      Twitter

      They're boasting about throwing together a mobile app for TSB in "SIX MONTHS" (17:40). Use cogwheel icon to switch subtitles to Spanish (Auto generated) then again to turn on auto translate.

      Innovación en AWS - Docker en AWS

      1. Anonymous Coward
        Anonymous Coward

        Re: "Load Balancer Errors" is the clue

        Regardless of all the stuff about load testing etc (which would certainly be a good idea, but probably doesn't catch all possible errors).

        *** You don't migrate 1.9 million users in one big bang, and with no rollback plan!!! ***

        You migrate 1,000 of them. Then next week 10,000 of them. Then maybe 100,000. Then if you're being cautious, 100,000 every week after that for the next 18 weeks. And if you're messing with people's money and their livelihoods, you should be cautious.

        At each stage you check for system errors and user complaints, fix the problems if you can, and if necessary migrate them back again until you solve the problems.

        It's not rocket science. It's certainly harder work to plan and implement (directing inbound requests to the correct system where each account could be on one of two different systems), but it's the right way to do the job.

        You will almost certainly discover a bunch of stuff that you didn't know about your user base, such as legacy products and services which nobody even realised were there. You might be left with a small runt userbase on the old system while you come up with a solution for how to move them. This is better than breaking them.

        1. Anonymous Coward
          Anonymous Coward

          Re: "Load Balancer Errors" is the clue

          You migrate 1,000 of them. Then next week 10,000 of them. Then maybe 100,000. Then if you're being cautious, 100,000 every week after that for the next 18 weeks. And if you're messing with people's money and their livelihoods, you should be cautious.

          Trying to run old and new systems in parallel just brings a whole host of unnecessary headaches. Trying to reconcile transactions in two disparate databases which may not have the same structure or schemas is destined for failure.

          1. Anonymous Coward
            Anonymous Coward

            Re: "Load Balancer Errors" is the clue

            Been there, done that and happens to be a specialty here as I migrate systems from mainframe to (network of) server PC's and, sometimes, even back to mainframe, too. It's when you can't do this that should start Big Ben loud alarms inside your head.

            Extra-points for turning your validation feature into an additional biz continuity feature.

          2. Anonymous Coward
            Anonymous Coward

            Re: "Load Balancer Errors" is the clue

            Especially when different products may have been hosted by different banks - although having to log in to different systems and having working products may have been preferable to no access to money.

        2. eamonn_gaffey

          Re: "Load Balancer Errors" is the clue

          As a veteran of many such migrations of bigger proportions, I can attest this this is the only sensible approach. To go 'big bang' on something like this is just too risky. I'm guessing the reason TSB did it was because some megalomaniac in charge decided to go for the macho high risk approach, and fucked it up royally.

          Interesting to hear Paul Pester,TSB CEO, on BBC Radio 4 this morning. Sounded very upbeat, almost smug (if you can beleive it), now that he has a global IBM team on board !. I'm sure that will be productive for TSB's customers (or rather, IBM in terms of fees generated).

          .

          1. thegrouch

            Re: "Load Balancer Errors" is the clue

            The Big Bang approach was driven by timescales and the desire to secure a bonus by the senior management. Everyone on the ground knew it wasn't fit for purpose but it went ahead anyway. Pester and Helen Rose should step down ASAP.

          2. RegMigrant

            Re: "Load Balancer Errors" is the clue

            I heard the same report and couldn't believe how confident he sounded 'now that IBM are on-board' - he still missing the point that no-one get's rich from out-sourcing by putting their best (and therefore expensive) people on an account. Now IBM will charge a small fortune for fixing the issues which his own team probably highlighted as risks but were ignored because the expensive consultants told the board it would all work out - and yes there's a level of bitterness in there. It always reminds me of the Monty Python surgeon telling a pregnant mother to be quiet because she's not qualified to have a baby.

        3. justAnITGuy

          Re: "Load Balancer Errors" is the clue

          Who, precisely, are the four idiots who Downvoted this comment? Please Ghod let them not work in IT.

        4. Experience ? whats that

          Re: "Load Balancer Errors" is the clue

          Acknowledging that I'm an old Global Telco networks design and test chap who is obviously way out of date and sees agile when used in large apps/ infra projects as a bit "overly risky" . Not being nasty to the customers who are having difficulty this sort of complete and utter incompetence isn't actually unusual in the dark side of IT where I've crossed over to. PMs think that Risk based testing doesn't actually ever have a bad outcome or think doomsday will come true... and if it does we can make loads of dosh with a patch and release programme using the all new singing and dancing dev ops / CI Super Dooper automated high quality approach. Well I hope that ... the Cinderella that approved the risk register that agreed to : . no comprehensive live like representative test environment .. ok: or lets not do rollback and fail over - ok ; or lets go for Big Bang migration - ok ... is now looking for a new job. Hopefully its not the Test manager who was most likely bull dozered by the wish to meet a date.

  8. Anonymous Coward
    Anonymous Coward

    BeanCreationNotAllowedException

    Off to investigate Spring vulnerabilities in the hope of a payday

    1. Bill M

      Re: BeanCreationNotAllowedException

      The code is in the Spring Cloud GitHub repo

  9. colinb

    Experts

    Who needs them, right?

  10. Anonymous Coward
    Anonymous Coward

    That's the problem with mahooosive waterfall releases...

    You can't break small and fix quick. I can only imagine the CAB meetings to pile up the requested changes and upgrades to go alongside each other. Not knowing what broke what (I've been there in the past scrambling to find out the multiple root causes) and using more firefighting time to plaster up the cracks until the breaks again.

    Traditional banks just won't change :(.

    1. Anonymous Coward
      Anonymous Coward

      Re: That's the problem with mahooosive waterfall releases...

      Wassup with the thumbs down?

      1. Killfalcon Silver badge

        Re: That's the problem with mahooosive waterfall releases...

        Downvotes? Probably that full stop attached to the end of a smiley. End your sentence, then emote.

        (seriously though, I dunno)

  11. Oh Homer
    Joke

    Now I know what TSB stands for...

    (Our) Technology Sucks Balls.

  12. Alan J. Wylie

    loading failed for script

    https://pbs.twimg.com/media/DbiPvH7WsAAj3Iu.jpg

    "Loading failed for the <script> with the source "https://dpm.demdex.net/"...

    "Loading failed for the <script> with the source "https://visitor-service.tealiumiq.com/"...

    What on earth are these doing on a supposedly secure page?

    1. Dan 55 Silver badge
      Coat

      Re: loading failed for script

      You can tell it's secure, it's got https.

      1. Bill M

        Re: loading failed for script

        It's secure https won't last for long

        (index):1 The SSL certificate used to load resources from https://www.tsb.co.uk will be distrusted in M70. Once distrusted, users will be prevented from loading these resources. See https://g.co/chrome/symantecpkicerts for more information.

    2. IneptAdept

      Re: loading failed for script

      Tealium is the bane of my life seriously......

      Fucking tracking bullshit

    3. DontFeedTheTrolls
      Devil

      Re: loading failed for script

      It's what happens when the Sale and Marketing people get priority over testing

  13. Ben1892

    Seriously, what clowns wrote this and why aren't any of the exceptions being caught and handled properly?

    1. Korev Silver badge
      Coat

      Hopefully the "exceptions" who wrote it will be punished suitably...

  14. swisstoni

    Think I'll be moving accounts

    Blimey you've only got to open the console in Chrome to see that these people really shouldn't be trusted with your money.

  15. Anonymous Coward
    Anonymous Coward

    TITSUP

    Totally Inept TSB Screws Up Perfectly

  16. AbelSoul

    Still humped

    Been trying to get in all afternoon.

    Finally got in a couple of minutes ago and second stage of 2 factor authentication crashed.

    Tried again and finally got to see a list of my accounts, tried to access one and was greeted with a "Thank you! You have successfully logged out of your account." message.

    Omnishambles springs to mind.

    1. Pascal Monett Silver badge

      Re: Still humped

      From here it seems to me that they should have added another 2,500 man-years on the project.

      Just my 2 cents.

      1. Tom 7

        Re: Still humped 2 cents

        that's more that I got.

      2. Anonymous Coward
        Anonymous Coward

        Re: Still humped

        "From here it seems to me that they should have added another 2,500 man-years on the project."

        Not necessarily. Twice as much incompetence will not necessarily fix the problem.

        The issue is more quality than quantity, at every level - design, project management, QA, programming, executive decision, security, operations, and probably a few things I missed.

  17. adamat49
    FAIL

    TSB Business or Not

    Well just checked TSB Business online is still off line. I have worked for many of the main banks (software bod) and I have to say it looks like they had no rollback position to go too.

    1. Anonymous Coward
      Anonymous Coward

      Re: TSB Business or Not

      Against this backdrop https://www.theregister.co.uk/2017/06/06/lloyds_confirms_ibm_cloudy_outsourcing/

  18. Daggerchild Silver badge
    Paris Hilton

    But it was tested! Look, the box is ticked, see!

    I am currently looking at another department making a staging environment for the stuff run by our department.

    This staging environment looks, and acts, nothing like the live one, and is made from different tech. This has been explained, at length, to no effect, since the parts involved have the same *names*, that's good enough for them.

    So, I entirely understand how TSB came about. I'm betting they have the loadbalancer in DeathBeam mode (concentrate all firepower on first backend to stand up, it goes into swap death, drops, restarts, slowly, meanwhile you wait and fry the next to stand up, rinse and repeat). On paper they genuinely have enough backends for the load...

  19. Anonymous Coward
    Anonymous Coward

    Just emptied my account.

    I used a cheque, I will take bets on whether it clears.

    1. katrinab Silver badge

      Re: Just emptied my account.

      I'm going to bet that it will clear.

      They have to respond by a deadline if they want to bounce it, and they won't. So it will be paid into your account, and not reversible because it isn't a fraudulent transaction.

      What happens at the TSB end is another matter.

      1. Ken Hagan Gold badge

        Re: Just emptied my account.

        "They have to respond by a deadline if they want to bounce it, and they won't. So it will be paid into your account, and not reversible because it isn't a fraudulent transaction."

        So any non-fraudulent (ie, you have the balance in your account) movement of funds out of TSB towards some other bank will clear in the usual time, but TSB might end up with an even bigger mess on their hands as a result.

        Hmm ... That sounds like something that a lot of TSB customers ought to know.

        1. katrinab Silver badge

          Re: Just emptied my account.

          Even if you don't have the funds, but it is a "normal" level of transaction, ie don't write a cheque for £1m if your balance is 10p, then it will likely clear, but of course they are entitled to charge the usual unauthorised overdraft fees and ask you to pay it back. They have 6 years to do that, or 5 if your account is with a Scottish branch, sort codes beginning 87, which is around 1/2 of all customers.

          1. Anonymous Coward
            Anonymous Coward

            Re: Just emptied my account.

            What if I empty someone else's account?

            1. Doctor Syntax Silver badge

              Re: Just emptied my account.

              "What if I empty someone else's account?"

              You'll need more in yours to pay for your defence.

              1. Anonymous Coward
                Anonymous Coward

                Re: Just emptied my account.

                Provided TBS has the ability to sort out who has done what.

  20. Allonymous Coward
    Flame

    It's getting to the point where I hope this kills the company

    Or at least severely maims them. Every so often the beancounters need a sharp lesson in what happens when you treat IT as an overhead and farm it out to low-cost body shops, rather than treating it as a core part of delivering your business.

    1. Anonymous Coward
      Anonymous Coward

      Re: It's getting to the point where I hope this kills the company

      It’d be a delight if Accenture (and their shitty little accent that fucks up linespacing) went with it, but I suspect it’s unkillable.

  21. Anonymous Coward
    Anonymous Coward

    Fungible software engineers...

    I had heard this so many times I had given up even discussing it any more. Then one fine day someone tried it at my current place of employment. The business hated it, the staff hated it and questions were asked. Ultimately the exec responsible for bringing it in was removed. Management still discuss this time and it's viewed as a big mistake.

    Just because someone writes code for one application solution doesn't immediately make them fully skilled to work elsewhere even if it's the same language. It takes time to learn the existing code base, application domain, business policies and procedures. Unfortunately the C-suite types aren't aware of any of this.

    The roles I find immediately fungible are the C-suite roles. There are laws to followed and regulators to obey. These are going to be largely the same for each organisation, can't these roles be outsourced?

    Anon, for obvious reasons.

    1. colinb

      Re: Fungible software engineers...

      This still surprises me. I've seen Oracle people put in to do a large SQL Server site, it didn't go well.

      Its the same as saying look you fly an Airbus, just jump in that Boeing and do these runs for us please.

      They are both planes, has a couple of engines, wings, steering wheel of some sort. I mean what's the problem.

      The problem is its quite likely the plane would crash with all souls lost.

      The regulators stop this with ratings, if you have not passed training in the type you cannot fly it. You can't even fly an A330 if you are rated on a A320.

      That's why we have regulators and not PHB making these decisions.

    2. Doctor Syntax Silver badge

      Re: Fungible software engineers...

      Visited Leeds today. Getting off the train there was a poster: "Write code/solve problems/save lives No experience needed". My emphasis. What really worries me - this is the firm that does my doctors' practice S/W.

    3. CrazyOldCatMan Silver badge

      Re: Fungible software engineers...

      doesn't immediately make them fully skilled to work elsewhere even if it's the same language

      The same holds for support skills - about 30% of your effectiveness is *site* knowledge (ie - yes, you needs to do this in order to get that to work - even if it's not the $VENDOR-approved way of doing it because that's the way it works in *our* environment).

      Which is why the outsourcing model of 'assigning a support person from a pool, none of which know anything about your environment' is generally a disaster. Unfortunately, all the beancounters see is 'we are getting 5 people's experience for the cost of one!'.

  22. low_resolution_foxxes

    I hope this is due to a 1-off account creation process the first time someone logs in? Falling over due to overloaded servers? Still, it looks like a self-made DDoS attack !

  23. Anonymous Coward
    Anonymous Coward

    The Post-COBOL Apocalypse Has Arrived!

    COBOL has been abandoned, so now the Post-COBOL Apocalypse has arrived. But that's okay ... not for the bank or its customers, just for the IT staff. Job Security forever!!

    1. colinb

      Re: The Post-COBOL Apocalypse Has Arrived!

      Actually given the information on other threads it seems like Proteo4UK is based on the Parents Proteo which is based on the Alnova banking systems which was a port of the Cobol version but retaining the Cobol syntax, so Cobol compiling to .NET using MicroFocus tools.

      Any pro would have parsed the cobol and extracted the business logic, created a banking DSL and throwing away the god awful verbose Cobol syntax.

      11 millions lines of code but easily 8 millions lines of framework crud.

      That's actually probably the most stable part, the issue here seem to be the middleware and frontend services. All the new code basically.

      1. Phil Endecott

        Re: The Post-COBOL Apocalypse Has Arrived!

        > a port of the Cobol version but retaining the Cobol syntax, so

        > Cobol compiling to .NET using MicroFocus tools.

        FUCKING HELL

        1. Steve Channell
          WTF?

          Re: The Post-COBOL Apocalypse Has Arrived!

          So, the Java server falls over with EJB errors caused by Spring dynamic instantiation, and dumps unhandled error messages in the browser because of dodgy JavaScript code.. but lets all pile in and kick the COBOL code that works, and stops the money vanishing into suspense accounts.

          MicroFocus COBOL has been porting mainframe systems for forty years, and running on .NET Common Language Runtime for twenty years.

      2. katrinab Silver badge
        Mushroom

        Re: The Post-COBOL Apocalypse Has Arrived!

        "Cobol compiling to .NET using MicroFocus tools"

        F*********************************************************ck

  24. DaveB

    I didm't win the lottery

    Well I didn't win the lottery, but it sounds that opening a TSB account may be the next best thing.

  25. d3vy

    Well, Business banking is still very much down.

  26. Anonymous Coward
    Anonymous Coward

    Blimey! This outage is the most outrageous since...

    ooh... October 2016, KCL, perhaps?

    1. Anonymous Coward
      Anonymous Coward

      Re: Blimey! This outage is the most outrageous since...

      Nope, I reckon this is more outrageous, at least with KCL the initial cause was a hardware failure, whereas here, it's a 100% homegrown clusterfuck due to crap software, insufficient resources and bad planning.

      1. d3vy

        Re: Blimey! This outage is the most outrageous since...

        Dont forget communication... Ive yet to receive any kind of acknowledgement from TSB that there is a problem or any communication of timescales etc.

        No Text, No Email nothing.

        I knew it was still fucked because my card didnt work on sunday night Ive got status updates from facebook and the news.

        Now I imagine some of TSBs older customers might not be on facebook monitoring the TSB page for news... They might be in for a shock when they try to buy their shopping.

    2. Ivan Headache

      Re: Blimey! This outage is the most outrageous since...

      I read that as the KFC shortage of 2018.

      Must be something I ate.

      Anyway it's now 2018 (on the clock - not the year) and I still can't get in - BUT - the status page now shows Mobile being Up and Telephone being Down.

    3. Anonymous Coward
      Anonymous Coward

      Re: Blimey! This outage is the most outrageous since...

      I bet it has a lower impact than the KFC cock up ;-)

      1. TRT Silver badge

        Re: Blimey! This outage is the most outrageous since...

        The

        Secret

        Blend

        of 11 herbs and spices is what makes

        Kentucky Fried

        Chicken so finger

        Lickin'

        good.

  27. Anonymous South African Coward Bronze badge

    Eiah wena.

    Definitely a job for Sisyphus then.

  28. Uberior

    I'm wondering if the details of people who have posted on social media are being harvested right now by crooks ready to make the call:-

    "Hi, It's Sonia from TSB. I'm really sorry about the trouble you've been having with your account. Can I take you through security and we'll start to get things sorted out..."

    Most of the cheques written in the UK are still using the heritage system of shifting bundles of paper around the country. I hate to think about the risks facing TSB right now if they haven't got a clue about the balance of their customer's accounts when the cheques hit. If someone timed it right by banking a large cheque on Thursday to hit on Monday in hope of system issues. The money will have cleared and swiftly moved on by now.

    Tomorrow and Friday will be fun. It'll be payday for:-

    Anyone who gets paid weekly.

    Anyone who gets paid on the last Thursday/Friday of the Month

    Anyone who gets paid on the 26, 27, 28 , 29th

    It's going to be bloody.

    1. katrinab Silver badge

      Indeed. Payday is Friday for me. Fortunately it is going into a different Spanish bank. If I wanted to change it, I would have had to tell payroll before about last Friday / Monday.

  29. Doctor Syntax Silver badge

    Pester said in September last year that "over 2,500 man years of effort by TSB, Sabadell and our technology partners" had gone into TSB's "new banking platform for the digital age".

    What counts is what comes out, not what goes in. He's emphasising the wrong thing.

  30. Fcoz

    Switch Bank NOW

    This Bank has been shot and is now wounded. It is therefore vulnerable. Hackers are looking closely because this Bank definitely has serious internal IT vulnerabilities which will be exploited. Bye Bye TSB.

  31. Anonymous Coward
    Anonymous Coward

    Former HBOS employee here, one that left prior to the crash and the redundancies that followed. We spent a decade modernising HBOS sales-side infrastructure; taking client side stuff off Z/OS into onto Windows Client/Server arrangements. Quite successfully if I do say so myself. The backend remained on Z/OS. A year after I leave, Lloyds get their grubby mitts on it and roll the whole shebang onto an old-school TSB-derived mainframe system. Leftovers from TSB first time around, not the "new" bank. No doubt this is the same system that was foisted onto "new" TSB. Owing to separation requirements, they were no doubt given marching orders to migrate away. Or advice that mainframe is old hat and hard to maintain. Maybe both. There is absolutely no illusion to anyone concerned that this was a very, very wet run of the migration that HBOS (under Lloyds ownership) will now have to repeat to exit mainframe legacy. Not to mention Lloyds itself (which I understand is also on an old TSB-derived mainframe!) Not saying any other banks are any better, but let's be honest, the proof is in the pudding. None of what is left of what used to be respectable organisations have a scooby do. Damn shame. Blame squarely pointed at the Exec and laissez-faire global non-regulation.

    1. Uberior

      At the front end, I recall switching from the legacy NCS to Core Banking System. Whilst CBS wasn't pretty to look at it, I don't recall it ever going "down" during the time it was in use and was amazingly quick to use.

      Then we switched from CBS to Lloyds CBS - awful. Glad I left.

      (When I started work, we were still keying on a 3604 terminal and had microfiche in the sub-offices.)

  32. Anonymous Coward
    Anonymous Coward

    test1.int.uk.tsb

    I wonder if the fact that the login page seems to attempt to want to load JavaScript from test1.int.uk.tsb might have anything to do with it?

    Aren’t you supposed to configure your build system so that all references to the test environment are replaced with the correct values for the live environment when you actually go live?

    And why do so many internet banking systems rely so heavily on just so much JavaScript? Surely that’s the first thing that a would-be hacker would try to compromise to see what sort of weaknesses they could exploit?

    (At least this shows that they do have a test system, anyway!)

  33. Anonymous Coward
    Anonymous Coward

    I'm a Tech Recruiter and not an Engineer so what follows may be bunkum. I suspect the major root cause of this lies with what I believe to be a migration to an IBM BlueMix and Microservices environment. More specifically, they used guys with J2EE & WebSphere, etc. skills who grew up building solutions in monolithic environments. In their rush to go Cloudy these J2EE guys don't yet have the expertise to build and test Microservices properly. Put another way, they're looking at their Cloudy environment through old, monolithic prisms and have created a massive TITSUP in the process.

    For an effort of this magnitude, using BlueMix & Spring Boot Microservices, you need top, top, Engineers. i. e. FAANG quality guys who ain't cheap and don't reside offshore. It's all well and good jumping on the latest tech trends but if you ain't got the staff to do the work....well,you get shit like this.

    1. Anonymous Coward
      Anonymous Coward

      "For an effort of this magnitude, using BlueMix & Spring Boot Microservices"

      And there's your problem right there.

      The whole 'microservices' concept is another of the miracle cures that pop up in IT every few years, and often do way more damage than they cure.

      "Microservices" is ill-conceived and will probably die a well deserved death relatively soon, if we are lucky.

  34. Anonymous Coward
    Anonymous Coward

    TSB's New Platform

    TSB's new stack - that is, Sabadell's - comprises the following. This is from an employee's LinkedIn profile. Again, I bet their engineers are traditional J2EE/JEE guys who have no clue about Microservices development & testing in Cloudy environments. It's a very different animal to old-school monolithic software development.

    Component of the BancSabadell Architecture team in the TSB project (TSB Bank in UK acquired by BancSabadell group) for the definition and implementation of a new banking platform based on the latest technologies and methodologies and oriented to a hybrid infrastructure between on-premises and public cloud

    Technologies:

    -PaaS (TIBCO SilverFabric)

    -Micro services (Spring Cloud Netflix)

    -SOA (TIBCO AMX Service Grid, TIBCO BusinessWorks, TIBCO API Exchange Gateway)

    -Single Page Application (AngularJS)

    -Asynchronous Messaging (TIBCO EMS)

    -APM (Application Performance Monitoring)

    -Distributed Search & Analytics (ElasticSearch)

    -Containerization (Docker)

  35. hypnos

    Have witnessed several bank migrations done by "offshore idiots". . .

    . . . in Banks that could be named "provincial" compared to the UK/Western Europe behemoths. Never seen such shambles. Now that I think of it, there was this project by Accenture in Greece in the late ninetys. They just could not finish the implementation of a new system called Altamira. Eventually the customer bank threw them out and took over code and all. Migrated successfully in a few months. System running happily until present with in-house staff and monthly development cycles.

  36. Anonymous Coward
    Anonymous Coward

    TSB, legacy IT

    Back in 2004 I remember visiting the head of an IT department that had responsibilities for TSB's IT. There was an enormous system map diagram on a huge whiteboard in the guy's office, it wasn't pretty with lots of old systems cobbled together that somehow just 'worked', many undocumented and originally written in house. The challenge back then was that the IT team had an average age of 55 and final salary pensions that were going to kick in at 60, it was a ticking time bomb some 14 years ago...

    1. colinb

      Re: TSB, legacy IT

      They have just replaced that enormous system map with a new one, full of Service Buses, SOA, MicroServices and Containers (Docker). This will be even less documented as event type systems are very hard to reason about.

      So its new, you can add new services quicker but is it better?

      1. You have probably created more links, not less. With products you have to keep upgrading introducing who knows what issues in future.

      2. The old system had known failures and all the main wrinkles stomped on over time, decades probably.

      3. The new system has a whole new batch of failure modes, both in the scale, links and the logic some of which have yet to be found.

      4. End to end testing now involves a mass of products that probably means a single developer will never see it in action, just a small piece.

      The people there on the old system probably sat quietly, keeping the systems humming, to a foolish person it looks like they are doing nothing since the systems always work.

      My take away, if you are going to replace decades of work, even with a huge head start of a running banking system, 24 months is not enough time, you need multiple end-to-end test environments with realistic load testing and you need real experts in the tech you will be using.

      1. Anonymous Coward
        Anonymous Coward

        Re: TSB, legacy IT

        "My take away, if you are going to replace decades of work, even with a huge head start of a running banking system, 24 months is not enough time, you need multiple end-to-end test environments with realistic load testing and you need real experts in the tech you will be using."

        And you need to choose the right tech, which is not always the newest tech.

        Anyone remember the "second system effect"?

  37. GSTZ

    Shiny new systems ...

    From yesterday's article: "TSB migrated from former parent Lloyds Banking Group's systems to shiny new ones" ...

    Moving from centralized and highly deterministic systems to "shiny new" systems that (from the outside) may even look centralized too, but are in fact a highly complex conglomerate of many thousands of "PC's" all doing more or less their own thing but also being dependent on the outcome of many other "PC's" to complete their tasks isn't easy. It doesn't help much that these "PC's" are no longer small separated physical machines like in the early days of distributed computing, rather myriads of virtual machines running on some kind of x86 infrastructure coming with bombastic marketing wording but behaving like a bunch of PC's anyway. Predictability suffers, such systems are certainly "good enough" to handle enormous workloads for less critical applications like Facebook and Twitter but might be less than ideal for really critical stuff like banking operations.

    This is not crying for the good old past based on legacy systems, as it has already been pointed out that old systems eventually become a real pain when too much new functionality gets added. At some point, it is better to start with a clean slate - but also on a highly deterministic system providing better reliability, predictability and security than the "good enough" gear that has become the de facto default for each and every new application these days. Unfortunately, most of the younger IT folks do not even realize that alternatives do exist.

    The prevailing hardware stuff comes relatively cheap, but the business results tend to be mixed as reliability, efficiency and security have "room for improvement" and the cost to run and support those very complex systems becomes too high. Many user organisations now do escape to the public cloud, even the military are now considering such moves. However, it is unclear how cloud providers having less knowledge of the business requirements and less incentive to provide superior service levels for critical applications will be able to serve their customers better.

    1. Anonymous Coward
      Anonymous Coward

      Re: Shiny new systems ...

      "Predictability suffers, such systems are certainly "good enough" to handle enormous workloads for less critical applications like Facebook and Twitter but might be less than ideal for really critical stuff like banking operations."

      YES!

    2. Anonymous Coward
      Anonymous Coward

      Re: Shiny new systems ...

      "However, it is unclear how cloud providers having less knowledge of the business requirements and less incentive to provide superior service levels for critical applications will be able to serve their customers better."

      And it appears very doubtful that cloud solutions can be made secure, partly for technical reasons, and partly for legal reasons (recent US laws giving them - thousands of US organizations - access to anything anywhere that a US company can somehow get, as one example of many).

  38. GSTZ

    Wrong architecture ...

    From today's article: "Shujun Li, professor of cybersecurity research at the University of Kent, said the main issue was not the initial failure – modern IT systems are too complicated and dynamic to be totally bug-free, he said – but because of the bank’s poor risk management."

    Many people would agree to the above opinion, but only few would be willing to draw the resulting further conclusions: Modern IT systems (ie., the currently prevailing "good enough" stuff) are far too complex to provide deterministic behaviour and predictable results automatically by themselves. Hence, a lot of additional and pretty difficult work needs to be done to raise service levels beyond the threshold of "good enough". Which means that the number and severity of complaints need to be reduced far enough that an overall impression of somewhat acceptable system behavior can be achieved - which however isn't exactly the level of reliability one should expect from critical IT systems, and it is pretty costly too. How about another IT architecture that delivers more predictable results ? That's not rocket science, it has been done in the past and it is done in other areas like industrial IT and OT.

    Maybe sometimes not having access to your account or somebody else having access to your account isn't seen as a major problem in today's banking industry. In industrial IT, the equivalent would be frequent production outages and small explosions all over the plant every once in a while ... (;-))

  39. Jove Bronze badge

    The Ministry

    Would that headline be, in part, a homage to "The Men from the Ministry"?

  40. dijital

    Untrusted Cert on the Complaint Form...

    As this guy's just noticed, they're using an old Symantec-issued cert that is no longer trusted by Chrome:

    https://twitter.com/mrw34/status/989477977892229120

  41. Potemkine! Silver badge

    This story shows how we became dependent of online apps. When the next major solar storm will hit Earth and disrupt everything, it will be chaos.

  42. Miss Config

    "Embattled bank TSB"

    As a Guardian reader I presume The Reg will write another article about TSB if and when the banks becomes Beleagured

    1. Tom 7

      Re: "Embattled bank TSB"

      I think it will pass straight to bankrupt - if I can get in to my account I'm shifting the whole damn lot the minute it comes back.

      1. theExecutive

        Re: "Embattled bank TSB"

        To another Indian Tech, best of luck!

  43. theExecutive

    Bitcoin

    And Segregated Witness, now looking a bit better than a Bank

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon