back to article TSB outage, day 5: What do you mean you can't log in? Our systems are up and running. Up and running, we say!

Embattled bank TSB has claimed its IT systems are now "up and running" – despite many users still reporting they can’t get into their accounts. The bank’s online services have been down for almost a week, after TSB's planned migration off former parent Lloyds’ tech infrastructure – some five years after its split from the …

I'd switch bank if I was one of the effected. It's very easy to do nowadays.

25
3
Silver badge

In the short term, switching may not be so easy if you can't access your assets in the old bank. In the longer term, switching might be an excellent idea.

33
0
Silver badge

"I'd switch bank if I was one of the effected. It's very easy to do nowadays."

I wouldn't. I'd open another bank account with a different bank not connected to the bank you already have.

Why? Well at least you have access to some money if your bank has gone up shit creek with their 2,500 man years of work as a paddle.

I was considering dropping the Co-Op for TSB and have accounts with both of them, but as you can guess right now I'm feeling fairly safe(ish) that while I can't access my TSB account I still have access to the Co-Op one.

22
0
Silver badge

I, too, would Effect a transfer if I was one of the Affected

39
0

Yes, but did he make the error pacifically to piss people like you and me off?

26
0

Definately.

20
0
TRT
Silver badge

That's the most annoyingist thing...

15
0

OMG, you grammer Nazi's

21
2

Sounds like the grammer Nazi's need some comforting....

Their, they're, there

12
1
Anonymous Coward

Re: Sounds like the grammer Nazi's need some comforting....

Here hear heir

3
0
Anonymous Coward

OMG, you grammer Nazi's

Nope its: OMG, you grammAr Nazis (no possessive apostrophe)

2
6
Bronze badge

Re: Sounds like the grammer Nazi's need some comforting....

It's important to distiguish the difference between knowing your shit, and knowing you're shit.

8
0
Anonymous Coward

Re: OMG, you grammer Nazi's

It's (contraction of it is), not its (possessive).

1
0

Try doing that when the donor bank's systems are down. The gaining bank won't be able to perform the switch.

0
0
Anonymous Coward

All code is written by offshore idiots to the lowest price

This shitty code is in your medical devices, cars, industrial systems, phones, apps and most devices in your homes. It's present on every website you visit.

Insecure by negligence and stupidity, it's everywhere in your life.

But hey - psychopaths are running the companies that make this stuff & they don't give a shit. They are cutting cost to get paid. You are not the 1%, so fuck you.

74
0
Anonymous Coward

Re: All code is written by offshore idiots to the lowest price

Accenture in Sant Cugat (outside Barcelona) are Sabadell's outsourcer of choice.

11
0

Re: All code is written by offshore idiots to the lowest price

Accenture rarely get their hands dirty with code, that's Avanade's (who they own) job and whose devs will mostly reside in India.

Firewalls of blame are very important to the PHB's

38
0
Anonymous Coward

Re: All code is written by offshore idiots to the lowest price

Steer clear of Accenture and anyone who ever worked there.

17
0
Anonymous Coward

Re: All code is written by offshore idiots to the lowest price

Sorry most of the coding has been done in Spain by Accenture and other Spanish companies.

0
4
Silver badge

Re: All code is written by offshore idiots to the lowest price

Accenture isn't Spanish. It is an American company that is registered in Ireland for non-taxpaying purposes.

22
0

Re: All code is written by offshore idiots to the lowest price

That's for clearing that up, in that case Accenture Spain is different to Ireland in that respect.

Spain is offshore to the UK in this case.

Worked out well for them, hasn't it. Normally their failures are kept very private. They should have paid more attention to how public this would be.

6
1
Anonymous Coward

Re: What about auto-updates?

There are idiots, offshore and onshore, in this and every other field. Bootnotes have served up many a tale.

The corollary to your statement then is that if you aren't the highest paid in your industry you are an "idiot" to someone else. There is some truth to this - we all don't know something..

The race to the bottom on price is where the problem is at. You get what you pay for.

20
1
Anonymous Coward

Re: All code is written by offshore idiots to the lowest price

"You are not the 1%" .....

Oi..... some of us AC's might be, in the 1% that is. Plenty more will be psychopaths.

4
0
Anonymous Coward

Re: All code is written by offshore idiots to the lowest price

The full list of consultancies involved are: Accenture, Everis, Indra, GFT, IBM, HP and BT. Everis and Indra are Spanish, the others have branches in Spain.

Link

4
0
Silver badge

Re: All code is written by offshore idiots to the lowest price

All code is written by offshore idiots to the lowest price

That's their mistake - offshore the customers, it's much cheaper and easier

10
0
Silver badge

Re: All code is written by offshore idiots to the lowest price

It’s quite extraordinary that neither Capita or Lockheed got their blundering tentacles into a fuck up of this magnitude.

16
0
Anonymous Coward

Re: All code is written by offshore idiots to the lowest price

Don't forget KPMG were involved with the testing.

4
0

Re: All code is written by offshore idiots to the lowest price

I understand that allegedly HP / DXC and Accenture did ,so all's good from a continued feck up / continuity point of view

0
0
Silver badge
Joke

So it's the bl00dy accountants!

Bean creation not allowed .. do not request a bean from the bean factory

They've obviously blown their budget.

Let's see the b*ggers wriggle out of *that* one! Ha!

27
0
Silver badge

Re: So it's the bl00dy accountants!

> Bean creation not allowed .. do not request a bean from the bean factory

(Cash) cow traded for 3 beans and Jack to show for them. :-)

22
0

Experts at the helm....

https://pbs.twimg.com/media/Da_CYjVXUAEVsOz.jpg:large

3
0
Anonymous Coward

AIOOB and BeanFactory Exceptions being surfaced all the way to the front end HTML in a retail bank's customer facing web products, after it's been claimed that everything is fixed.

This may qualify as one of the more comprehensive failures of software engineering, testing and delivery that I've seen.

And yet that failure pales in comparison to the organisational failures and managerial incompetents that caused it happen.

48
0
Silver badge

It's 2018 and people are still using arrays...

1
29
Silver badge

"It's 2018 and people are still using arrays..."

Which array-free OS/compiler/VM/interpreter are you using?

26
0
Anonymous Coward

Must must be someone from academia - they're in a parallel universe.

6
3
Anonymous Coward

If it's surfacing this sort of crap to the user it's a certain bet that the security design is broken too. The best security defence they have left is that their site is too broken for people to log on.

20
0
Anonymous Coward

"It's 2018 and people are still using arrays..."

... because in the hands of competent analysts and programmers they remain one of the best ways of representing a myriad kinds of data, and because, properly used, they enable the functionality of some of the most advanced high level languages available.

1
1

The masochist in me wants to see some of this code. Especially where AIOOBs aren't being caught. The other part of me wants to see it just to give me an ego boost.

There is something intensely gratifying about seeing CIO's fucked over by outsourcing.

No doubt though, the blame will be pinned on the (non-technical) PM than the shitty devs who don't know their arses from their elbows.

0
0
Anonymous Coward

"Especially where AIOOBs aren't being caught"

The code is clearly Spring in nature, which means the entire application is almost certainly wrapped in a global exception handler; Spring often uses Exceptions as a standard control flow mechanism. This means someone has decided it's a good idea to pass through AIOOBs to the presentation layer (probably through a default catch & rethrow "handler"), and the presentation layer has decided it's a good idea to display internal Exceptions verbatim.

This is first year software engineering student shit that doesn't even pass static analysis, never mind professional code review. Stinks of a rush job.

2
0
Anonymous Coward

TSB stands for The Shit Bank?

10
0

Oh come on, we can do better than that!

TSB:

Test Software Before!

Test Software Better!

Tanked So Badly

Trashed Someone's Bonus?

So many possibilities

15
0

Re: Oh come on, we can do better than that!

Terrible Software Blamed

Techies Shaft Board

Testing Saves Bonuses

Terribly Stupid Board

Try Software Backup?

Tools Sometimes Break

Totally Shafted Business

9
0

Re: Oh come on, we can do better than that!

Try Some-other Bank...

(yes, that's cheating, but good advice)

11
0
Silver badge

Re: Oh come on, we can do better than that!

Try Switching Bank, surely

20
0
WTF?

"Load Balancer Errors" is the clue

Having been a client-server Java developer for many years, this is the sort of thing you get in a postQA environment when you rollout a live solution and the stress-test hasn't been done well, and the stress-testing environment doesn't replicate the live one sufficiently effectively.

The internal resource shortages (threadpool, database connections, config problems etc are good examples) don't truly manifest themselves until a *genuine load* hits the *live techstack*; even with the best will in the world.

The fact that all three front end layers (Web, App and In-branch clients) are evidencing "Javabean" errors speaks to problems at the server layer of the architecture; hence the restrictions on number of logged in clients => less chance of server resource exhaustion.

Clearly there are issues with the not-so-adequate scale of the backend infrastructure. From experience, these are the hardest to assess from the point of view of a development team; even if the solution is properly written and tested, and passes QA, the live environment can hilight resource problems where the infrastructure isn't well provisioned.

And thats assuming a perfectly solid Dev/QA process, into the bargain

33
0
Anonymous Coward

Re: "Load Balancer Errors" is the clue

It seems they threw in Netflix's open source load balancer and hoped it worked. You can tell because the error gets thrown to the mobile app.

Twitter

They're boasting about throwing together a mobile app for TSB in "SIX MONTHS" (17:40). Use cogwheel icon to switch subtitles to Spanish (Auto generated) then again to turn on auto translate.

Innovación en AWS - Docker en AWS

9
0
Anonymous Coward

Re: "Load Balancer Errors" is the clue

Regardless of all the stuff about load testing etc (which would certainly be a good idea, but probably doesn't catch all possible errors).

*** You don't migrate 1.9 million users in one big bang, and with no rollback plan!!! ***

You migrate 1,000 of them. Then next week 10,000 of them. Then maybe 100,000. Then if you're being cautious, 100,000 every week after that for the next 18 weeks. And if you're messing with people's money and their livelihoods, you should be cautious.

At each stage you check for system errors and user complaints, fix the problems if you can, and if necessary migrate them back again until you solve the problems.

It's not rocket science. It's certainly harder work to plan and implement (directing inbound requests to the correct system where each account could be on one of two different systems), but it's the right way to do the job.

You will almost certainly discover a bunch of stuff that you didn't know about your user base, such as legacy products and services which nobody even realised were there. You might be left with a small runt userbase on the old system while you come up with a solution for how to move them. This is better than breaking them.

39
4
Anonymous Coward

Re: "Load Balancer Errors" is the clue

You migrate 1,000 of them. Then next week 10,000 of them. Then maybe 100,000. Then if you're being cautious, 100,000 every week after that for the next 18 weeks. And if you're messing with people's money and their livelihoods, you should be cautious.

Trying to run old and new systems in parallel just brings a whole host of unnecessary headaches. Trying to reconcile transactions in two disparate databases which may not have the same structure or schemas is destined for failure.

7
10
Silver badge

Re: "Load Balancer Errors" is the clue

Been there, done that and happens to be a specialty here as I migrate systems from mainframe to (network of) server PC's and, sometimes, even back to mainframe, too. It's when you can't do this that should start Big Ben loud alarms inside your head.

Extra-points for turning your validation feature into an additional biz continuity feature.

10
0

Re: "Load Balancer Errors" is the clue

As a veteran of many such migrations of bigger proportions, I can attest this this is the only sensible approach. To go 'big bang' on something like this is just too risky. I'm guessing the reason TSB did it was because some megalomaniac in charge decided to go for the macho high risk approach, and fucked it up royally.

Interesting to hear Paul Pester,TSB CEO, on BBC Radio 4 this morning. Sounded very upbeat, almost smug (if you can beleive it), now that he has a global IBM team on board !. I'm sure that will be productive for TSB's customers (or rather, IBM in terms of fees generated).

.

12
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018