'Big Bang': Great for creating the universe, but not as an approach to IT migration, TSB told TSB's disastrous migration of its core banking data and payment records of five million customers has today been slammed by an independent report from London City law firm Slaughter and May. The botched move last year left 1.9 million customers unable to view their accounts, some of whom had money disappear or were able to …
"With 2000 testing defects I think the migration plan was little more than the executive summary 'Copy the DB from A to B'"
You neglected the caveat of "do you think it will make any difference if we start writing data rather than just using read-only copies in all our tests? I know all our tests failed but it can't make anything worse can it?"
Even one 'showstopper' is too many to even think of going live when the application is so central to your business.
The real problem with testing large systems like retail banking packages is that they are far more complex to test than the typical bank management ever realise if they haven't been through it previously. Whether the system is running on a mainframe, server farm or in the cloud doesn't really affect the size of this task. Any such system has to handle transactions input via:
* ATMs, web browsers, phone apps, FastPay, SWIFT, CHAPS, Paypal and probably some others I've missed.
It needs to handle various account types:
* Current accounts, savings accounts, mortgages and loans working in multiple currencies
..and must handle complex staff activity using PCs and other terminals with big displays as well as printing fancy statements, etc.
Testing something like this is expensive and time consuming because testing MUST exhaustively cover everything from normal daily operation through account creation, deletion and migration in and out. On top of this there is special processing at month end and year end (both company and tax years). Additionally backups and restores MUST be tested and so must input overloads and auditing.
All this testing needs to be scripted and repeatable simply because its unreasonable, and expensive, to expect testers with phones/laptops/ etc. to do this sort of testing repeatedly without making mistakes.
Sabis already had Proteo, an apparently configurable retail banking package, and so you'd expect a set of development/configuration/testing/regression tools as part of the standard package. However, judging from the way things panned out it either lacked these tools or it wasn't configurable enough to meet UK standards without modification, and maybe some PHB thought tweaking the development etc tools to match UK requirements was unimportant.
The other problem is that TSB didn't go live by stages, but all at once. Has TSB decided to go live on a subset of customers, it would have had the possibility of seeing the storm that was brewing and scale back and correct the issues before the whole thing blew up in their faces.
In any case, they certainly got a big bang, just not the one they were hoping for.
Oh, and a note to all high-level managers : let this be a lesson to you. Never believe someone who tells you that there will be no migration issues to worry about and we can do it all in one go.
It never happens like that.
The problem they had was that when customers had multiple accounts with the bank of different types. You would have diminished the customer experience. And migrating some customers entirely demands complete functionality on all products.
From what I hear Lloyds were also part of the problem. TSB were tied to them, paying an arm and a leg for a static service and any phased approach (e.g. Migrate credit cards first but still show all the products on the Internet Banking) would have needed unbelievably expensive services from Lloyds. So TSB were screwed. Can't get Lloyds to help you (because Lloyds have no interest in making it a success), so you have to make sure it is going to work in one go.
I think the failure was when TSB was bought from Lloyds. At that point, the few people around the world with experience of massive separation of banks (and it will only be a few experts) should have crawled over all the commitments and costs Sabadell were getting Lloyds to commit to. Lloyds gave Sabadell £400m to subsidise the separation (I think that's the figure). That is far too small and actually, much more valuable than money would have been a committed capacity of Lloyds IT support staff during the separation, without the horrible management layers to get anything done between the companies.
Major problems were that all the IT knowledge was in Lloyds, TSB were just users. So when Sabadell asked TSB 'now tell us what you need your systems to do.?' they just stared at the techies and said 'we don't know, we just use the systems, we don't know what they actually do or what they do underneath' and Lloyds wouldn't tell them either. So Sabadell had to reinvent everything just to stand still.
In an ideal world you would parallel run and prove the systems 'green field' with new customers, then migrate portfolios of existing customers onto systems that have already been proven with new customers at low volumes.
But all of these approaches demand proper support from Lloyds, who had no interest in making it a success.
When was the last time a bank split in two?
Citizens Financial Group was split off from Royal Bank of Scotland, but that was their US division and probably pretty separate anyway, but one single integrated division of a company being split in two; that has never happened. So nobody has experience of it.
The other thing; Sabadel didn't buy TSB off Lloyds. If they had, they would have negotiated much better terms for the IT stuff. TSB was spun off as a separate company and sold on the stock exchange. Then Sabadel took it over a couple of years later.
Correct, although TSB and Lloyds continued to use the same systems (well, TSB a copy of the Lloyds systems) to ease the spinoff process. TSB/Sabadell had a choice: Either continue to pay some other bank (Lloyds) for access to the system (which didn't get the improvements that Lloyds' own systems were getting), or build their own. I don't blame Sabadell for choosing the latter... Why contribute to someone else's profits when you know you should have that knowledge in-house.
It's unfortunate that this migration was a massive clusterf***, but ultimately I think TSB and Sabadell will be better off for it.
How many times do we read about insufficient testing leading to a FUBAR.
After it was all fixed they were offering a nice 5% rate on the first £1,500 in the current account to try and get some (any?) customers back (it's 3% now, still better than most) Now I still wouldn't trust them with pay & mortgages (give it a few years) but it's been worth parking some cash there and using some shuffling between accounts to meet the t&c.
"How many times do we read about insufficient testing leading to a FUBAR."
I'm not sure the issue was insufficient testing - I mean the testing that was carried out was certainly insufficient but the tests were also significantly scaled back until the system would pass. If you are going to move the goal posts to ensure your tests are passed regardless of the outcome AND you don't intend to address any of the findings, I'm not sure any testing will produce a worthwhile outcome.
The report also appears to show a lot of people who knew the project was in trouble and tried to get assurances from other parties that "all necessary checks had been carried out" to cover their asses rather than trying to address the issues.
The conclusion should probably have said "the board gave the project two extra years to complete the migration and then said it won't get any more ready than it is, fingers crossed"
Age-old problem, someone simply would not put their signature to the cost of buying an automated testing system. In a day and age of Automated DevOps, Code Reviews > Unit and Security Tests > Performance and Functionality tests and (if passed) automated deployment, why is a bank showing 2000 unresolved test issues?
I do hope that IBM threw the book at them.
While an automated testing system may have helped with the software defects, I'm not sure it would have addressed the other shortcomings of the project:
- the timescale was too short (also a potential reason for not getting around to using automated test tools)
- the initial capacity requirements were understated. While there were mitigating factors (it was based on the previous gen platform and time was short) it was never addressed later in the project.
- the new architecture was untested. Not only were defects an issue but the whole environment was new and poorly understood.
- the tests that were carried out were lowered due to concerns around the impact on the new system as the ATM network was already using the new platform. Instead of testing at the expected system capacity, they tested to 50% capacity which was
- they knew there were shortcomings in the Customer Services platform that meant that it was not possible for all customer service staff to use the system and that an issue with on data centre would reduce this further. Again testing was not carried out in a way that represented a normal day due to known issues with the application stability meaning tests lasted for 2 hours rather than 8+ hours..
And IBM threw a very large bill at them....I guess that is sort of similar.
Skimming through the report the only real surprise must be that they got through it at all.
TSB/Sabis appears to have known prior to the migration that their testing was woefully inadequate and that they didn't have the necessary capacity in a number of key areas and would likely fail to the extent observed if they reached their predicted load. Rather than addressing the issues, they reduced the size of the tests...by around 50% when actual load turned out to be ~300% in the case of the mobile platform but there appears to have been no area where they had adequate capacity at day one irrespective of application issues.
And this wasn't just known by technical staff or project management teams - it went to the CIO levels of TSB/Sabis and possibly beyond (still reading...)
I've been through some shoddy project go-lives where we knew there were major issues and areas that we would likely have to fire fight for weeks or months, but this seems particularly poor.
And I assume this is the carefully polished, gold plated turd that TSB were happy to release to the public rather than the in-depth, detailed report that will help the organisation grow and avoid a repeat of this issue.
I look forward to the FCA/PRA investigations for an even more damning insight into what happened.
I suspect that anyone who has ever cut over a system or been remotely involved with a cut over of a system would have been aware that testing is absolutely vital. Not only that but testing the assumptions on which forecasts were built is essential. The stimulus of a new system upon customer demand for services is very often woefully forecast or understood. The only good thing in this case, was that those customers who set out to see what breaks, were spoilt for choice; they did not know that the system was already broken and untested in so many areas that they were bound to have some fun.
So the fraudsters had a party time.
Many years ago I tested what would happen when the demand rush really started. I simulated the effect and listened to fuses blow. When working with a supplier I questioned some 'example data', believing it was for demonstration purposes. Sadly, it was real planning data from an unnamed customer. I predicted the go live failure. Their customer had forecast less than 10% of the actual user impact; ouch.
It’s actually the opposite problem. Sabadell have successfully converted their core system twice, and, successfully absorbed five or six smaller banks. I think the problem was hubris and overconfidence.
P.S. is it just me but shouldn’t we worry that 10-15% of retail banking is owned by Spanish companies. I worry as a Spanish resident that the local economy will get sucked into the inevitable collapse the UK property market.
Don't forget that all previous integration was with v3 of their code. They'd decided to let tsb, really lloyds, pay for a system upgrade. New Middleware, Web interface and mobile apps, and they also used the project to get rid of their legacy VB code.
Firstly, WTF does a law firm know about IT systems, Banking platforms, Migrations methodologies, strategies and testing procedures
Secondly, as already mentioned, the number of defects is irrelevant, it is the number of showstoppers. I don't give a damn if there are 5,000 in the queue if they are all category 5 such as "Snr Mgr can't change colour of menu bar". We've all had ones like these. The question that needs to be answered is "How many showstoppers did the decisions makers know about & downgrade so they could meet the go live date and so as to not incur financial penalties?"
Thirdly with statements like this "we specifically asked for a fully independent and thorough inquiry. Although the report doesn't paint the full picture of migration"
If the report doesn't paint a full picture of the migration then "IT"S NOT A FUCKING THOROUGH INQUIRY"
I'm surprised you overpaid wankers can tie your showlaces in the morning.
Finally, If the IT contractors had multiple successes in data migrations before, the argument can simply be made the only variable were the the overpaid tosspots , aka PHB, at TSB.
A law firm knows a great deal about assessing evidence and presenting it clearly.
I've only got part way through it but one thing's quite clear. The project was presented to TSB upper management as a migration onto a tweaked version of an existing product. In this situation the migration of the data was the riskiest part of it That's what they were actually told. The truth was that it was largely new software developed for the job and that the Spanish parent was hoping to get a new system for themselves off the back of it. It doesn't need an in-depth experience of development to see that this was a recipe for disaster and that the "riskiest" part of it went smoothly, balancing to the penny as they emphasised. You don't need to read a line of code to see all that.
As far as I've got with it I'm left wondering why the board showed such lack of inquisitiveness. Instead of being fed multi-hundred page information packs they might have learned a lot more by paying visits to the offices where the work was being done and asking people what they were actually doing.
Funny you should say that. A few times I've looked at the members of a company's board (usually looking for a name to complain to) it's appeared to me that they were, by and large, vaguely generic. Even when they'd been promoted from within the company I got the impression that for the most part they could have been senior managers in any company doing anything.
What I didn't get an impression of was a committee of the knowledgeable about that business. The main qualification for being on the board seemed to be "the right sort of chap" preferably with an MBA.
I dunno. Just an impression.
According to the report they were presented with "packs" of hundreds of pages. Much easier to hide stuff in that. If you hide something by omitting it from a bullet list summary you're apt to find that when the shit hits the fan it then sticks to you because you can't deny hiding it. Put in on pages 834 to 842 out of 967 and nobody can accuse you. Of course you can do even better by putting on a single loose sheet that then gets accidentally trapped in the 967 page file. The Sir Humphrey tactic.
Thanks TRT, but it's not arrogance. It's common sense.
If you are sick, see a doctor.
You want to build a bridge, get a engineer
You want to implement a banking system, system network get an IT professional in that area
If you want to absolve or distance yourself from any legal matter, you get a lawyer
Sorry. I may have misunderstood you. The company that was providing TSB with IT facilities and transitioning the customers onto a new platform was a company of lawyers, then? I thought you were referring to Slaughter & May, the legal company that prepared the report. The same Slaughter & May that provided legal advice to Carillion, as well as an £8m bill, just prior to the Carillion collapse.
To suggest that a company of lawyers cannot appropriate the technical expertise required to conduct an independent investigation into an incident and convert facts and expert opinions from technical specialists into an informed opinion on the legal, commercial and moral consequences of a particular incident would bring into question the entire industry of independent investigation.
I think I'll have to go and read the article again. Or one of us will at least.
"An ambitious timetable..."
I've met that phrase a few times. Locally and small scale.
But I'd say the meaning is the same. It means allowing less time than the experts say is the minimum.
But with added pressure to meet it.
Which is never a sensible way to make sure a job is done thoroughly or safely.
I got sick and tired of telling managers and directors that the timescale isn't long enough and there will be problems.
Usual response is, "we've agreed it with our clients" or "our 3rd party support says it's ok".
Then when the go live is a complete cluster fuck, it's everyone else's fault.
Oh they really don't like the "told you so" statement.
Then when the go live is a complete cluster fuck, it's everyone else's fault.
You mean, you don't keep copies of the emails where you've told them the problems and they've ordered you to do it anyway?
And what about the copies of the emails where you reply to that saying "I'll go ahead if you agree to take responsibility", and if their response isn't "I'll take responsibility", your email telling them (perhaps politely) to fuck off?
I have several electronic copies, stored in corporate 'immutable' record keeping systems, and in personal locations, and several hard copies, of all such communications.
Item 4 from the summary on the TSB site:
"This final migration of customer data was completed with every customer account transferred to the penny. However, despite this, the main migration was followed by extensive service disruption and instability for TSB customers."
Translation: the operation was a success but the patient died.
Having just wasted most of the evening skimming through all 262 pages....
They had two data centers in an “active/active” configuration, which neither Sabadel (the Spanish parent bank) nor IBM UK had done before.
ATMs had moved over to the this new infrastructure early, so during the load testing they didn’t want to push it until it fell over because that would break the live ATMs. So for load testing they used a configuration where the tests ran against one data center while the other served the ATMs.
And the transactions they used for these tests were read-only, because it was live data.
Their target for this testing was only 100% of normal peak load, i.e. no margin for growth or miscalculation.
So in real use the performance was inadequate.
Code quality was poor. Not much detail. about that, but it sounds like functional testing was poor and anything that wasn’t tested had no chance.
Documentation was poor esp. for architecture/infrastructure. There was a lack of configuration management e.g. duplicated IP addresses.
Branches used remote desktops (Citrix) and had a memory leak which caused them to crash frequently. They had not been tested for a whole 8-hour working day.
The real underlying cause seems to be that TSB (uk) was not treating Sabis as an external provider, as they should have been. The TSB CIO had been working at Sabis previously, and after moving to TSB he knew more about this new system than anyone left at Sabis. So he was conflicted between being customer and supplier. Maybe the TSB board should have been more circumspect and asked harder questions but not really, there is only so much you can expect them to do; in particular they were not told how different this was from what Sabadell had done when acquiring smaller Spanish banks.
Maybe.
I suspect it is more a function of TSB never having had do do any complex IT work for themselves since it was all done by Lloyds at arms length and even then basically no projects had been done by TSB since it was carved out of Lloyds.
So their management had no idea how to manage technology in what is essentially an IT company. But they didn't recognise their own lack of knowledge in what is probably the most complex It project ever done at a UK bank. Sabadell should have called it out and TSB should have recognised their own lack of skill. Both to blame.
>They had two data centers in an “active/active” configuration, which neither Sabadel (the Spanish parent bank) nor IBM UK had done before.
Don't know about Sabadel. but IBM UK had certainly done active/active DC's in the UK banking sector, suspect those with experience have either retired or weren't involved in TSB.
"Code quality was poor."
Another factor was thet there shouldn't even have been that much new code according to what the board were told. They thought they were being moved to an existing platform with a few tweaks to meet local requirements. Instead they were getting a largely rewritten system.
In my opinion you are going very easy on the TSB CIO. The report contains multiple examples of him wilfully ignoring or misrepresenting evidence that the project was not ready, including ordering staff to falsify RAG project scores on more than one occasion. I imagine he will cop a lot of heat off the back of this report.
Everybody talks about how crap MS is for not testing their Windows builds correctly - and they're right. Sadly, this is just the way software is done these days, apparently by everyone, if El Reg's stories are anything to go by. Wanna program? Just half-ass code it, fart in the general direction of testing it, then push it out the door and try to fix it when it breaks and the users scream.
There's an old saying "Why is there never time to do things right, but always time to do things over?" Not to mention damage to one's reputation and possible legal problems, but nobody seems to care much these days about that. Instead, we always keep hearing the same, old crap: "We take quality/security/privacy/whatever VERY SERIOUSLY" = "We don't give a toss because we don't think it will impact bottom line, or we'll be gone by the time it does."
</rant>
I have been involved in two core banking system replacements, both were successful and both were basically Big Bang conversions.
The phased approach is superficially attractive, but falls down for various reasons.
The main problem is that a large percentage of transactions are between a banks own customers. Having to deal with transfers between a customer in the old system and a customer in the new system, requires a whole new set of business rules, special coding and nearly doubles the use cases to be tested. The only real advantage is that phased conversions are usually quietly dropped half way through as the workload increases and the budgets get blown leaving the customers blissfully unaware of the chaos in the IT department.
When the major 3rd party is the supplier of your core business application and also part of the same group of companies following the purchase of Sabadell, I would be cautious about this being caused by not bringing things in-house.
Ambitious time frames, poor capacity predictions not accounting for new platform requirements or changing business requirements, poor understanding of the magnitude of the change inherent in a new architecture/platform, software bugs in an untested platform (I'm including this through lack of in-depth information - I would expect this in any implementation...), deliberately scaling back testing to achieve a pass and poor governance both at project level (they weren't able to hit their project deadlines but scaled back/incomplete testing results to achieve a pass, going live without adequate capacity for customer services) and corporate level (the boards desire to avoid additional payments to Lloyds seemed to be the overwhelming driver for the project deadlines).
In summary, they attempted a 5 year project (my estimate based on it taking an additional year to get close to fully operational and didn't carry out suitable testing) in 3 years, delivered it in 3.5 years and cost the bank customers/growth for at least an additional year, cost an additional £200m plus any future fines/regulatory costs and reputational damage that will last years.
The total number of defects, and the number of defects reported to the Board, are irrelevant.
OK there were 2,000 defects, of which the Board were told of 800. What were the ratings of these defects? If 1200 were Sev 5 and 800 were Sev 3, then that's probably sensible.
However, if >0 were Sev 1, then somebody's arse must be on the line here for having signed off the permission to proceed.
If the 800 were Sev 1&2, then they were fucking crazy to go ahead.
Apologies if this is in the report somewhere - haven't had time for a trawl yet.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
