Deja vu
I could have sworn I read about some remarkably similar "research" by the Institute for Studies several weeks ago - oh look, I did.
Old, complex code could cause another UK banking TITSUP – study
Another major banking outage similar to the RBS disaster back in 2012 is likely to happen again in the UK, given the amount of legacy code in the sector, according to research. The average mission critical banking application has around 600,000 lines of code, according to a study by software firm CAST which analysed systems …
-
-
Tuesday 3rd May 2016 09:40 GMT Warm Braw
Re: Deja vu
Interestingly, the CAST Press Release that led to the earlier El Reg article was entitled:
Revealed: Why UK Banks Crash All The Time [dateline March 23, 2016, London]
It was followed by another press release citing the same report [dateline March 24, 2016, New York] and entitled:
Study Reveals U.S. IT Systems Unfit for Digital Future
So it could all perhaps be summed up as:
Vendor alleges crisis in $MARKET can be solved by giving them $CURRENCY
-
-
Tuesday 3rd May 2016 08:54 GMT Anonymous Coward
Statistics ...
For some reason I remember the story of an army officer (pre 1918, possibly) who did some number crunching and "predicted" that so many men would end up being castrated by being kicked by horses in the coming year.
It wasn't magic - just a realisation that the event happened with a frequency greater than 1/year.
It *may* have been the Belgian army.
-
-
-
Tuesday 3rd May 2016 10:23 GMT Anonymous Coward
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
I am struggling to understand why Old Software is Bad Software. Software doesn't perish or rust like hardware, nor do the component parts wear out with use.
What this actually means that it has 20 years of being used in a real environment, 20 years of bugs being identified and fixed. Not like the new, shiny, barely tested stuff.
-
Tuesday 3rd May 2016 10:27 GMT Electron Shepherd
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
True, but in a lot of industries, especially finance, the regulatory environment changes, and that makes the software out of date. Effectively, the software gains bugs by not changing, and so failing to keep up with the world in which it operates.
-
Tuesday 3rd May 2016 11:10 GMT Skoorb
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
Yes, Things Change.
A brilliant example of this is RBS/NatWest Telephone, Online and Mobile banking.
A telephone banking system was developed to hook into the main banking system. This telephone banking system is separate to the main branch system, logging all transactions against a four digit call number, before passing them over to the main banking system.
Becauase parts of the main branch banking system dates back to the 60s, it was not possible to actually integrate the telephone banking system fully; it has to sit separately and just pass transaction messages back and forth as if it were a special type of branch terminal.
The online banking system was developed to hang off the end of the telephone banking system, so all transactions are passed to the telephone banking system and treated as coming from a telephone "call", before being transferred to the main branch system with a call number.
The mobile banking system hangs off the end of both the online and telephone banking systems, but cannot generate proper call numbers, so simply logs everything against telephone call 0, then leaves the telephone banking system to pass the transaction over to the branch system.
So, a right mess there then, with lots of potential to go wrong. No decent software engineer would even consider designing a system that way if implemented today - the potential for failures and security flaws is just too high.
You will also notice how everything is predicated on having a telephone banking login, and how your telephone banking details also double as your online banking log in details, with no easy way to separate the two.
-
Tuesday 3rd May 2016 14:15 GMT Doctor Syntax
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
"that makes the software out of date"
No, it makes it require maintenance. Development is the process by which software is launched into maintenance. It usually spends most of its life there so it's no excuse for assigning the least competent staff to the job. Neither is it an excuse for relying on maintenance to do all the bug-fixing that should have been done during development (did someone say continuous release?) so documentation and testing are equally important in both phases.
-
-
Tuesday 3rd May 2016 11:33 GMT glen waverley
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
Smooth Newt "Software doesn't perish or rust like hardware, nor do the component parts wear out with use."
I used to work in a place where we used to get a phenomenon called "bit rot". Stuff would stop working for no apparent reason.
-
Tuesday 3rd May 2016 12:44 GMT Anonymous Coward
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
The Tao of programming states:
5.1
A well-used door needs no oil on its hinges.
A swift-flowing stream does not grow stagnant.
A deer blends perfectly into the forest colors.
Software rots if not used.
These are great mysteries.
-
Tuesday 3rd May 2016 17:45 GMT Anonymous Coward
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
I used to work in a place where we used to get a phenomenon called "bit rot". Stuff would stop working for no apparent reason.
Bit rot is the phenomenon in which code which is written with certain assumptions about the environment in which it is used, breaks, because the environment changes in subtle ways that cause those assumptions to no longer hold true.
e.g. you might have a LSB (systemv) init script that takes arguments after the "start"/"stop" argument to allow you to start/stop subservices. Then systemd comes along and wraps your LSB init script up in a wrapper that ignores and strips these arguments: causing bit rot that breaks the feature.
I've spent the last few days battling bit rot in the Xaraya CMS. Ideally, I'd rip out Xaraya and replace it with something else, but that would require locating the original design docs (most of which were in the head of a now deceased colleague).
-
-
Tuesday 3rd May 2016 17:51 GMT Daniel von Asmuth
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
Indeed. What would happen if the banks start replacing old complex code with fashionable ultra-complex code? The one thing you can be sure of is that the performance will decline (espciallty with Java code).
-
-
Tuesday 3rd May 2016 10:52 GMT PNGuinn
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
"As any fule kno: software documentation wasn't invented until 2003 so the code will be undocumented."
That's news to me.
I naively thought that that patent was years away yet.....
All together now:
"Does - your - sourcecode - loose it's comments on the bedpost overnightttt?"
-
-
Tuesday 3rd May 2016 21:23 GMT a_yank_lurker
Re: "Even if something has been written in Java in 90s that is still 20 years ago."
If they it was Cobol or something as ancient they I would say they definitely have something. Java is still active but I could see some PHB refusing the money to update some ~2000 Java code which is using some deprecated features that runs erratically at best on Java 8.
-
-
-
-
Tuesday 3rd May 2016 15:43 GMT JeffyPoooh
LoC: "...just remove the line breaks..."
Back in the Tandy Colour Computer days (circa 1981), there were 'One Line (of BASIC code) Contests'. About 240-some characters to work with.
Many examples were outstanding and/or hilarious. Tiny video games. Utilities. An Adventure game engine (additional DATA statements holding the content).
-
-
-
Tuesday 3rd May 2016 11:17 GMT Skoorb
With most banking systems that would be an IBM Mainframe or two running z/OS and z/TPF etc OSs, with something like Computer Associates' (remember them? ) CA-7 actually triggering all the batch jobs.
Mainframes and their OSs are a little bit different from pretty much any other system you will ever touch.
-
-
-
Tuesday 3rd May 2016 09:23 GMT Warm Braw
if its there then it has purpose
In the case of very-long-lived commercial applications, it's often the case that "if it's there, it may have had a purpose at some indeterminate point in the lifetime of the application". Exactly what purpose that was, and whether the code path might be required for some random future transaction is not always something that is clear.
Given banks' reliance on contractors and their enthusiasm for merging, splitting and outsourcing, it's quite surprising that any of it works at all.
-
Tuesday 3rd May 2016 10:02 GMT Doctor Syntax
"Exactly what purpose that was, and whether the code path might be required for some random future transaction is not always something that is clear."
It should be if the code was properly documented and the best place to do that is in the code itself, not in some separate document that's either been lost or never updated since day one.
-
-
-
-
Wednesday 4th May 2016 10:16 GMT cloth
Re: "I am a coder..."
I did once argue that in order to be a good coder you had to have a good command of grammatical structures and ability to communicate well. I don't know many story telling 'coders' or 'programmers' - but I do know an awful lot of introverted techies who are always looking for something more interesting than 'making sure that thing runs well and is maintainable'.....Sigh.
-
-
-
Tuesday 3rd May 2016 09:24 GMT David Roberts
Distant memories
Once in my youth I was a COBOL programmer and I still remember the heady excitement of a week writing all the active code.
Followed by a month or so writing all the exception routines which made up over 90% of the average programme and were probably never used in production.
So sheer volume is no real measure of how good or reliable the code is. Although the more lines of code there are the higher the odds of an undetected flaw.
-
Tuesday 3rd May 2016 10:05 GMT Doctor Syntax
Re: Distant memories
"all the exception routines which made up over 90% of the average programme and were probably never used in production."
Like all safety devices, you hope they're never needed but when they are they're really needed.
"So sheer volume is no real measure of how good or reliable the code is. Although the more lines of code there are the higher the odds of an undetected flaw."
You're arguing against yourself here. All those exception handlers are there so that you can detect flaws. It's lot's of lines that don't do checking that are the problem.
-
-
-
-
Tuesday 3rd May 2016 09:51 GMT kmac499
Re: "...more comprehensive error checking and handling?"
Yeah banks love audits, Never happier than when they found that 10p that fell down the back of the cash drawer in branch.
But when it comes to auditing code this usually means a tick box exercise. Did we have a spec (yes)
Did we do some testing (yes) Did we sit around risk assesing the changes (Yes). Ok go live
What I have neve seen is some form of single code vault akin to the banks money vault, where software is received, stamped, filed and held. This should be the one and only place that code can be retrieved from and put into production or formally signed out for future mods.
-
-
Tuesday 3rd May 2016 09:40 GMT JeffyPoooh
"...a 10-15 per cent charge on that project..."
If that's true, then the amazing conclusion is that some software projects are actually getting finished on budget, thus protecting this "10-15 per cent" reserve. Traditionally, the "last 10%" (suc) of the code requires the other 200% of the budget.
This improvement is amazing news.
Humanity, perhaps via the UN, should celebrate. Maybe a cake.
-
Tuesday 3rd May 2016 10:10 GMT Mage
Outsourcing
Losing core competence due to outsourcing, so called "cloud" or simply off shore developers is stupidity.
These business need to be just as expert at software development as at trading and marketing or else ultimately they are not actually really banks or other financial institutions, but just resellers.
-
Tuesday 3rd May 2016 11:29 GMT jake
"Lev Lesokhin, CAST's senior vice president of strategy and analytics, said: "In consumer banks, there are core components been there for a long time. Even if something has been written in Java in 90s that is still 20 years ago."
Earth to Lev Lesokhin, Java was never secure. Ever. Really. And never used in truly secure systems.
Thankfully, the core systems in the world's banking systems are Fortran & COBOL, neither of which the skiddies are capable of breaking into.
Well, to be truthful, COBOL & Fortran aren't exactly secure as a stand-alone product, but the skiddies have absolutely zero clue as to how they work ...
-
Tuesday 3rd May 2016 13:51 GMT Caff
RBS
Bit misleading to mention the RBS outage expect possibly to highlight scale.
The outage would have happened regardless of the age or complexity of the code. It was an operational issue caused by poor lines of reporting and badly designed escalation proceedures in the outsourcing contract.
-
Tuesday 3rd May 2016 14:29 GMT Anonymous Coward
The real problem is Britain invaded India...
... and made people there speak English also. In turn it made for English-speaking countries easier to offshore software development to India today.
That's the real reason why English banks code is crappier than continental ones, where, for pure language reasons, offshoring to India or the like is less appealing.
There are good chances code has been developed locally, also software developers salaries may be lower in many European countries, making offshoring somewhat less advantageous.
-
Tuesday 3rd May 2016 18:18 GMT JulieM
Old code that doesn't get run
There's probably a whole bunch of code in there that doesn't do anything, in the form of functions that are declared but never called. Especially if anyone wrote routines to deal with Imperial (240 pence) pounds -- which they might well have done, just in case someone was ever crazy enough to try to revert back from the decimal (100 pence) pound that was already well-established before computerisation.
-
Wednesday 4th May 2016 14:57 GMT Zippy_UK
DON'T WORRY - RBS HAS IT COVERED
"Because of this problem, RBS executive management has acted quickly and efficiently to solve this problem by 1. Offshoring to cheap, inexperienced Indian IT operatives with a quality industry experience of less than 2 years 2. Cut back on all infrastructure spending and maintenance to bare minimum to re-invest in critical areas 3. Invested a whooping 50% of the money saved in point 2 in quality providers such as TCS and Gap Gemini so we can get the future delivered in the future where its needed. 4. Invested key priority areas such as restarting dividends, so share buy back programs can be re-started. 5. Paid extra money (including share options schemes, see point 4) to invest in the talent at the top (of our organisation)
to ensure it is retained"
DON'T WORRY CONSUMER, YOU MONEY IS SAFE WITH US.
Biting the hand that feeds IT
