Whose idea was this?
No doubt the deadline was forced onto TSB by the government who now deny responsibility.
The government is demanding urgent answers over a botched systems upgrade at TSB that has locked out up to 1.9 million customers. The IT meltdown happened after a planned upgrade between 4pm on Friday 20 April and 6pm on Sunday 22 April as TSB migrated from former parent Lloyds Banking Group's systems to shiny new ones. TSB …
"Maybe they needed changes in place for GDPR (only a month to go) compliance?!"
"Maybe they needed changes in place for GDPR (only a month to go) compliance, so that the fines will not be quite so bad so they balls it up and expose customer details to random other customers."
No need for a conspiracy when incompetence will do.
Yup, sounds like there was a bit of a hole somewhere in their change management, unit testing, integration testing, UAT, scale testing, and dry-run that they definitely must have done because of due diligence.
What do you mean, they just pressed the 'Go' button and crossed their fingers?
"We are currently experiencing large volumes of customers accessing our mobile app and internet banking which is leading to some intermittent issues with people accessing our services."
But isn't that what your mobile app and internet banking is for? Or are you still expecting customers to call into the branches that you've shut?
Time for TSB to issue the usual statement "Lessons have been learnt, we have fired the poor peon in India who was entirely and solely responsible for this cockup, all affected customers will receive a free year of credit check courtesy of TSB Credit Checks (after this period, you will be charged £40 a month)... "
... how will they detect all the fraud they're exposing themselves too? We've already seen in the media stories of people able to access £XX,XXX in other peoples accounts. What are the odds some dishonest scrote hasn't moved some of that "free money" to their own accounts? What are the odds other scrotes won't view their legitimate out going transactions and respond "Wasn't me guv".
And that's to say nothing of those simply writing down names, account numbers, balances, standing orders, and direct debit info for whatever accounts they see, knowing they can phone up later and hack into the account then.
Buggering up your outsourced IT is par for the course - you choose to do that when you choose to offshore - but displaying other peoples private information wen you do so is inexcusable.
Pretty slim probably as, if true, this access clusterf@@k means they have no guarantee the funds will end up where directed...
They'd probably short cut that and transfer the money to an online account with another provider or bitcoin account. Relying on your screwup to be so bad that it saves you from fraud because literally nobody can do anything seems counterproductive and unlikely to engender goodwill amongst customers.
Easy to detect fraud. They record details of all transactions and the time and date of when it was performed.
All they have to do is sit back and wait for people to complain money has been stolen from their account. Then when the complaints roll in, investigate those complaints relating to transactions which occurred during the time window of the change.
C'mon, Kat, select committee's ain't part of the government and describing them so is sloppy; it's like calling Ofwat "the water companies", or misrepresenting a gamekeeper as a poacher.
I seriously wanted to make a joke about how the government, given their track record, can't afford to lecture anyone else on IT. But I can't , in good faith, make that joke on the basis of this report. Except elliptically.
True. As a TSB customer and an IT professional it's easy to mock. But we've all been there, know someone who has, or had to clean up the mess someone else has made. So the techies scrambling to fix this have my sympathies. The architects, BAs, PMs and C-level execs who ultimately should've made sure this didn't happen, not so much.
Icon, for what I hope someone buys the developers, DBAs and ops people when it's all over.
Maybe the devs, dba’s and op’s people [sic] ignored the architects and did their own thing as they thought they knew better...
Much more likely, the devs DBAs and ops people did know better, since their job roles will involve a degree of expertise in their chosen fields, but their concerns went unheard because their middle-managers didn't want to jeopardise a comfortable life by passing problems upwards.
The 'architects' in this case were probably a bunch of sales types who got a Groupon to do a cheap PRINCE2 course and now believe that they know everything there is to know because their pinstripe suits are snappier, and are almost certainly the root cause of the problem. They probably think testing starts and finishes with making sure the software doesn't emit errors when you run it up.
Banco Sabadell - 'We have successfully migrated the TSB customer data'
TSB - 'We are experiencing "intermittent" issues'
Customer 'I can not access my accounts, funds or get any information'
Financial Conduct Authority and Information Commissioners Office - AN ABYSS OF SILENCE
A truely Fu**ed up situation with government bodies doing the same as the TSB systems , sweet FA.
"Financial Conduct Authority and Information Commissioners Office - AN ABYSS OF SILENCE"
From the article:
An ICO spokeswoman said: "We are aware of a potential data breach in relation to the TSB and are making enquiries."
Also a few points to consider:
1. Just because you haven't heard anything doesn't mean that they aren't doing anything.
2. Investigations take time and they are likely waiting for TSB to fix the problems before looking into what has happened.
3. The FCA is not a government body.
If you have dealings with a someone who is not giving you the service you require then you complain and if they ignore you then you stop doing business with them.
Clearly there is little point complaining as they are not listening so just move to another bank that does listen, where is the problem? you stoopid or sum fing
If they refuse to give you your money then, you have something to complain about and doing it via the courts it is less easy to ignore.
"Clearly there is little point complaining as they are not listening so just move to another bank that does listen, where is the problem?"
Finding that other bank that does listen. Most of us are running out of choices.
"Finding that other bank that does listen" then use cash instead and be in complete control of your finances.
There used to be life before banks and it is only recently that having a bank account was compulsary, given what has happened since banks were guaranteed access to the everyones wallet then perhaps it is time to review our options.
Clearly the customer is not being put first here and for every banking fkup the same management never learn from their errors, simply because we also come last with the Government too
The option that everybody really ought to exercise is to have two accounts. Preferably one that you use daily, and another (with a completely seperate supplier) which holds savings.
When a problem hits on your main account, switch to using your savings account until they fix it.
I would recommend making sure you have at least one Visa card and one Mastercard, and some actual cash hidden somewhere safe in the house. I also recommend a third card that you don't actually carry around for when you loose/have stolen your wallet/purse.
There's something almost endearingly charming about listening to yet another political mouth-on-a-stick criticising someone—anyone—else for incompetence. It's even more darkly hilarious coming from a government that f***s up every single IT project*¹ it ever touches, costing taxpayers tens of millions.
I'm sure TSB have been playing to the Tory handbook of cost-cutting, cheap half-measures, fat bonuses for the board, screwing the customers at every opportunity and all the other shyte that major corporations do to score short-term returns (yum, especially those nice bonuses: grab it and move to the next fiasco, quick!), while storing up disastrous weaknesses and catastrophic problems for the future.
Anyway, not to worry. None of the senior management who lovingly laid the groundwork for this clusterf**k will be punished; no lessons will be learned; no meaningful regulation will be imposed; lobbyists' enticements and inducements will flow unabated; politicians will continue to dodge responsibility; customers will suffer; and the torrent of lies, evasions and excuses will be epic.
*¹ Er, yes, and everything else it touches, really ... doesn't seem to matter whether it's health, education or defence, you can always trust them to bring their own unique combination of stupidity, arrogance and dishonesty to the table and leave a steaming mess behind.
"The Tory handbook"? Remember Tax Credits? £5bn stolen wasn't it? Notice the vast majority of IT outsourcing (including government IT contracts) to India occurred under 13 years of Labour - the same people who took the monitoring of UK bank liquidity off the Bank Of England and gave it to a toothless quango - the rest is history. Don't get me wrong - I can't stand the current shower in government but it was under Labour's watch the UK IT industry was decimated and the UK banking industry brought to its knees - the former by easy access given to our markets (visa rubber stamping), the latter partly caused by the one-eyed idiot's banking act(s).
Whaa whaa Tory whaa whaa! Stop this godforsaken 'let's blame the Tories for everything' game. I am not a Tory, but I find this kind of cowardly buckpassing completely pathetic. The Tories are as bad as Labour, or the Lib Dems, or any other party that's been in government (coalition or otherwise) since the 60s.
Labour cocked up with Windrush (not requesting the imaging of the landing cards of Windrush arrivals, instead ordering the paper copies irretrievably destroyed in 2009) which has now led to the Tories carrying the can for it (thanks to their 'hostile environment' policy at the Home Office), Labour cocked up with the Passport Agency, the Child Support Agency, the NHS and its PFI clusterf***s (which is driving NHS Trusts into failure), LibDems cocked up with the tuition fees and probably the all-digital Cabinet Office strategy (i.e. GDS), but hey *WHAA WHAA TORIES WHAA WHAA*.
Do f*** off.
The actual destruction of the landing cards was carried out in 2010. I think the Tories had time to decide whether or not to a) scan before destroying or b) not destroy the cards. They were also advised that it would hit the Windrush generation, but they pressed ahead anyway and removed legal protection for them in the 2014 Immigration Act for good measure.
So they may be as bad as each other, but in this case one is worse than the rest.
Somehow, I suspect that nobody went and asked the cabinet minister if a decision made by the previous minister to destroy some old documents from the 1960's ought to be reviewed.
And the constant "it's Labour's/Tories fault11!!11!" poisionous party posturing hides more fundemental issues.
For instance, let's see what the Labour cabinet minster had to say about his department in 2006.
"Our system is not fit for purpose. It is inadequate in terms of its scope, it is inadequate in terms of its information technology, leadership, management systems and processes," he told MPs"
This was IIRC greeted by laughter in the commons instead of serious determination to expunge, eliminate, eradicate and exterminate the problems and reform the civil service department in question to the point it could efficiently and effectively acomplish it's purpose for existance. One wonders if more recent minsters agree with his assessement of the department, and if so what is being done about it.
Labour cocked up with Windrush (not requesting the imaging of the landing cards of Windrush arrivals, instead ordering the paper copies irretrievably destroyed in 2009)
Whilst there is certainly a lot of buck-passing going on, I think the fact that the decision may have been made in 2009 is not as relevant as the fact that it was made by a civil servant, with no involvement of the Home Secretary, whomever it was at the time.
Theresa May, however, is on the record, when she was Home Secretary, as promising the unreachable immigration targets, and preaching the 'hostile environment'. The last Labour lot may have been bad (I'm no fan of Blairite Red Tories), especially with their data-collection fetishism and creeping authoritarianism, but May et al are arguably a lot worse, especially with May's attitude towards human rights (which can be summed up as "pesky humans, why should they have rights?").
Also, the "was it Labour / was it the Tories" argument neatly sidesteps the fact that you can hardly blame the current Labour leadership of sharing an ideology with Blair and Brown, which was a lot further to the authoritarian right than the party currently lies.
The fact remains that Labour made the decision, Labour also made other decisions. Just like the Tories made some decisions in the past that the current Tories should/cannot be lambasted for.
Both major parties are as full of visionaries and sh**bags as each other. That's the point here. Any party follower pointing fingers at the other party should first acknowledge that their party are just as big a bunch of cockup merchants as the ones they are pointing at.
Morgan added: "This is yet another addition to the litany of failures of banking IT systems. Potentially millions of customers could be affected by uncertainty and disruption.
"It simply isn't good enough to expose customers to IT failures, including delays in paying bills and an inability to access their own money.
"Warm words and platitudes will not suffice. TSB customers deserve to know what has happened, when normal services will resume, and how they can expect to be compensated.
Not defending TSB, of course, but for a politician and former Government minister to say all this, without a hint of embarrassment or self-awareness, just beggars belief. Does she know nothing of the Government's record in IT?
Does the term "Universal Credit" mean anything to you, Nicky? Because you have just described it in a nutshell.
Hold on. There may be irony involved because of the government's appaling record of IT failure, but that does not preclude the government from criticising companies. These companies need telling off and holding to account, and the government is the right party to do that.
It is just a shame that the people they employ on House of Commons Select Committee hearings know nothing about IT.
Banco Sabadell regard it as a success, so there you go:
Banco Sabadell: Banco Sabadell successfully completes TSB technology migration - 23/04/2018
As for techy stuff, TSB's system is an in-house system called Proteo4UK, based on Sabadell's own Proteo which itself was based on something Accenture knocked up (Alnova). Sources:
In other words, abandon hope all TSB customers who bank here. They can't roll back because TSB are off Lloyds, they just have to grin and bear it while it gets fixed which could take a while. Or switch to another bank, probably involving a visit to a TSB branch, a banker's draft, and changing direct debits with every company manually.
Thanks for the links!
From one of them:
Carlos Abarca, CIO of TSB, says ... “Proteo4UK was built in close co-operation with world-class companies, and has very few legacy systems,” he states. “It is a brand new core banking system for us.”
This would be that meaning of "legacy" that roughly translates to "working".
Unlike others, I'm having a hard time being sympathetic to the peons working on the emperors new clothes. Too much bad in IT is already waved off as somebody else's fault. This TSB IT thing is rotten from the head of IT down.
You mean the kind of 'working legacy' systems that rely on ancient batch systems to do stuff overnight instead of instantly? Oh, right, *those* kind of legacy systems...
Sometimes an all-new digital system is very useful. It gets rid of a lot of cruft that builds up over decades. A banking group by the name of ABSA in South Africa successfully merged *four* banks together onto a new platform in the nineties... granted, back then we didn't have Internet banking, but damn, the novelty of being able to pay some money to someone else and have it instantly appear was great, especially when the grand dames of British banking still believed that things *had to* take at least 3 days to happen (unless you withdrew it from one bank over the counter, walked down the road with a massive wodge of cash in your pocket and looking over your shoulder for fear of being robbed, and then paid same wodge of cash into the other bank and hoped it would be cleared instantly).
So yes, please, don't give me that crap of 'legacy = working'. Legacy might = working, but possibly working by shoe string, bubble gum, some staples and a paperclip.
I do so love the smell of noobs in the morning! :)
Batch processing may be a bit out of fashion these days, but it's still often the best way to check up on the "instant" stuff to see if that's actually working correctly. It can also still be very appropriate for quite a few other things, such as managing loads where resources are tight.
For example, I used to run batch exception audits on a system where the real-time transaction flow was incessant and on busy days was quite extreme (the system might have real problems keeping up with the load). The trick was to run that audit twice in a row in order to see what showed up there. Something that turned up as out-of-balance on the first run but not the second was just an inflight, incomplete transaction which could be safely ignored. But anything that showed up on both runs was almost always a genuine problem which needed to be investigated and corrected.
The weird part about this was that I often had to explain to my younger colleagues what I was doing here and why I was doing it. I was usually the best in the group at tracking down problems quickly and finding and correcting any systemic issues behind them, but I often left other folks scratching their heads after I tried to explain to them that this batch audit was just part of my "secret sauce".
As for load issues, one way to often correct these was to split the transactions into those pieces which pretty much needed to happen immediately, versus any pieces that needed to happen fairly quickly but not right away (high-priority batch), versus any pieces that could generally take their time (low-priority batch). Once again I often had to explain to my younger colleagues the wisdom of doing this, since their knee-jerk reaction to any overload situation was usually "Just upgrade the infrastructure!", something which would take both time and money. These days this might not involve much more than a button-click or whatever, but at the time such an endeavor could turn into a big freaking deal.
BTW, as for your money transfer example I think you may be confusing "instant" processing with "optimistic" processing, which are two very different things. And yes, "legacy = working" is frequently an accurate assessment of the situation, so I probably wouldn't go around dissing that too much if I were you.
When my employer offshored system support to the Asian subcontinent to save money, we saw one key effect - the duration of operational outages increased dramatically.
One incident took several months to investigate;I got involved. Gave them a few pointers, made them think, and within 24 hours they had found the fault.
There are huge cultural problems with using low cost workers from the Asian subcontinent, and you shouldn't use them for any kind of support or development. I s'pose I shouldn't complain too much: their incompetence kept me in work.
There were also reports that TSB customers could access the accounts of other TSB users, including their account numbers, sort codes and transaction histories, and possibly having the ability to perform actions on these accounts, including transferring money.
A month before GDPR, lucky you TSB.
A TSB spokesperson told us yesterday: "We are currently experiencing large volumes of customers accessing our mobile app and internet banking which is leading to some intermittent issues with people accessing our services."
Well after you close so many branches you would I hope have been expecting that.
The sound of silence is deafening over the fact that this is a modern state-of-the-art banking system, dragged off the boring, reliable, Lloyds mainframe and put on edgy x86 cloudy stuff - and - oh dear - it's all gone wrong. If this was a mainframe system, there would be finger-pointing at the platform from every commentator - as it is - nada - nothing to see here, move along...
"You should have tested the systems properly"
I suppose it could be tested by moving accounts over in tranches. Whose accounts should go in the first tranche? How about senior managements'? Second tranche? Senior management of any big consultants and audit firms involved? What - none of them bank with TSB?
You can get exceptions to that policy. You just need to ensure Test is secure as hell and your developers (with access to whatever you're calling the 'live data test' environment) are audited at least as well as your production staff. It's a *risk* you can *control*, though it is genuinely best to not have most dev work done on live data, and use your experiences with the 'live test' to craft better test cases for the lower environments most dev work is done against.
The problem is always going to be that when you construct test data how realistic is it? I did some work on a system a few weeks ago and I could not obtain a data model of my source database and only in time I discovered problems in the live dataset, which I needed to cater for. Had I constructed test data I would have built it to what I expected the data model to be and my software would have failed.
An employee working with sensitive live data simply has to sign an NDA, now that doesn't guarantee they won't steal the information, so you have to also consider who the people are that are working on that data, and which country they are in. And worst case, you can pseudoanonymise it by tokenisation.
What is the problem with using live/real data in a test system? As long as it is protected in all the usual ways. And as long as you ensure the test system is kept separate and isolated from the production system so you don't inadvertently update the production system with test transactions from the testing?
They obviously had to move off Lloyds systems which makes sense but oh boy - what went wrong between the testing of this and the live implementation?
Surely you would do a dummy run using your staff that have accounts first or would phase this in over a couple of weeks.
I'm a TSB customer and was locked out yesterday with payments not happening as they should. Not ideal.
How hard can it be?
Big bang IT changes are always fraught, but an online banking website and app is not really that hard surely? In my simple head it's just a long database of transactions against account numbers with a relatively simple front end.
Of course if I'd been doing it I would have moved customers in batches. Or I'd have offered reduced functionality over the weekend rather than shutting up shop. At least then there would be a small element of goodwill from customers rather than pent up demand desperate to get on the new platform.
The official reason for the move is to stop paying Lloyds for use of its IT systems by the Spanish parent company of TSB. It seems ludicrous in this day and age of 'anything/everything as a service' to expose yourself to so much risk by taking stuff back in house. But I've spoken to other retail finance companies who are in the process of 'de-clouding'. Is this a new trend?
No, I think man years is actually correct. Assuming a team of around 1000 people across TSB, Sabadell, and third party contractors (a reasonable estimate for manpower) and the two and a half year timescale - this has been in the pipeline since at least 2015, 2500 man years is correct.
2500 man hours is nothing. It's the equivalent of 250 people doing one day's work.
Post-Brexit, will an English-registered retail bank actually be able to outsource its entire operations to what will suddenly be a completely foreign jurisdiction? So its actual banking operations* will no longer be subject to English law? Really?
The Reserve Bank of NZ stopped the Australian-owned main trading banks in NZ (i.e. 90% of the market) from doing that, insisting they do their processing in NZ, not Australia.
* the bank branches, ATMs etc are just interfaces between the customer and the bank. All the real banking operations are IT systems nowadays
The government can demand what it likes, companies like this hide behind obfuscation. They rarely disclose what actually happened.
Look at the big BA scandal recently, they bluffed their way through with an incomplete explanation claiming they had a power surge when too many of their systems were turned back on at the same time, but they never disclosed what caused the original power failure and why their battery and generator systems did not kick in.
TSB Online banking is back up but it's still flaky as shit.
Most of the time it's "We're limiting access to Internet Banking" and when you get that window of opportunity to login, it's slow as fuck. Get a blank screen but "viewing source" reveals there is HTML there and if you refresh the page you get logged out and the "We're limiting access to Internet Banking" message again.
Some commentators clearly in the know on here. Work for TSB or LBG I wonder?
When TSB was spun out of LTSB, they had a 10 year contract to use the Lloyds systems with a 5 year break clause in that contract.
In order to save money, as others have pointed out, the owner of TSB (Banco Sabadell) activated that clause and for the last three years, have been building to this point.
Unfortunately, SABIS - that's the IT arm of Banco Sabadell - haven't quite succeeded in switching TSB's customers across. This should have all been completed at least 6 months ago and properly tested but because of SABIS's wish to do it as cheaply as possible, instead of spending a bit more money, SABIS have cut a few corners in both Madrid and Barcelona and ended up with the mess we saw now.
The blame lies firmly with SABIS, not TSB.
From the artcle: "TSB migrated from former parent Lloyds Banking Group's systems to shiny new ones" ...
Moving from centralized and highly deterministic systems to "shiny new" systems that (from the outside) may even look centralized too, but are in fact a highly complex conglomerate of many thousands of "PC's" all doing more or less their own thing but also being dependent on the outcome of many other "PC's" to complete their tasks isn't easy. It doesn't help much that these "PC's" are no longer small separated physical machines like in the early days of distributed computing, rather myriads of virtual machines running on some kind of x86 infrastructure coming with bombastic marketing wording but behaving like a bunch of PC's anyway. Predictability suffers, such systems are certainly "good enough" to handle enormous workloads for less critical applications like Facebook and Twitter but might be less than ideal for really critical stuff like banking operations.
This is not crying for the good old past based on legacy systems, as it has already been pointed out that old systems eventually become a real pain when too much new functionality gets added. At some point, it is better to start with a clean slate - but also on a highly deterministic system providing better reliability, predictability and security than the "good enough" gear that has become the de facto default for each and every new application these days. Unfortunately, most of the younger IT folks do not even realize that alternatives do exist.
The prevailing hardware stuff comes relatively cheap, but the business results tend to be mixed as reliability, efficiency and security have "room for improvement" and the cost to run and support those very complex systems becomes too high. Many user organisations now do escape to the public cloud, even the military are now considering such moves. However, it is unclear how cloud providers having less knowledge of the business requirements and less incentive to provide superior service levels for critical applications will be able to serve their customers better.
Biting the hand that feeds IT © 1998–2019