Bank fail: Ready or not, here's our new software Every now and again The Register runs articles from sysadmins around the world about the horrors of working in IT. From time to time, however, it is probably worth reading something from the user's point of view. This is one such story. Today at the grocery store, I saw an advertisement for a smallish regional bank. I used to …
...to do proper testing anymore.
It's boring, tedious, costs money etc. The guys that built it, lost interest in the project a week after it started and this is 11 months later.
But it does need to be done if you want to stay in business or maintain any sense of credibility.
to do proper testing anymore.
There just isn't time.
Gone are the 1980s/90s days of carefully managed projects with an end release date, and carefully reviewed life cycles. When software was treated like a car.
Nowdays software is more akin to the power grid - never finished, and always under maintenance somewhere.
As usual we need to wait for the managers who came up through the 80s and 90s to either get this, or die.
I think part of the problem is Google. By releasing services to the general public (even if they are invite only) they label as beta, they've made it acceptable to release products without proper testing.
no, you can go much further back to mickey^H^H^Hrosoft for that... everyone wanted to be a beta tester so they let them and moved all testing into the public sphere...
"to do proper testing anymore.
There just isn't time."
Yes there is - but management don't like time spent on non productive - as they see it - activities. In fact they even renamed development to account for limited or no testing - Agile. Code branches constantly being updated , little to no testing of the aggregated changes before its put live , all to please a "customer" , often internal , who doesn't even know what they want anyway but just come up with some handwaving ideas that some idiot IT manager things Must Be Implemented NOW! regardless of any long term consequences.
Bullshit. That may be true if you're producing something unimportant like Facebook, but for things like banking there is absolutely enough time if enough personnel and money are thrown at the problem. Computer platform changes have long lead times, so that can be planned. Regulatory changes? Usually signposted well in advance. New browsers? Released in beta months before official release. Mobile access? Browsers are a known quantity, as are the mobile platforms.
There is time, it's just that people don't want the loss of revenue by releasing their software later.
It may be that some older software suffers by not being developed with a view to unit testing, but that seems a little better than more modern software which is constantly a least a bit shit, looks like it has never been properly unit tested, and has definitely never had more than a cursory user test.
I think that one of the problems, oddly enough, is the move away from the waterfall model, with it's stages of system test -> integration test -> user acceptance test.
There seems to be an idea developing that TDD and automated unit testing results in full-working software.
"What do you mean, it has bugs? I checked it in, and our CI server said it passed all the tests. That means it works"
There is a bit of "Quis custodiet ipos custodies?" here - how do you know the tests are correct and provide full coverage? To bring the Latin up to date, "Who tests the tests?"
I've seen plenty of code that has lots of unit tests, and the code passes all of them, but they either end up testing that basically the compiler works (e.g. making sure that getter/setter pair get and set as expected), or the actual functionality is mocked out, just to get the test to pass.
This isn't to say that unit tests are a bad idea (they aren't) or that the waterfall model is the best way to write software (it isn't), but in the rush to improve the way software is developed, some of the key tasks have been left behind.
As a software test engineer, can I just say that you all are correct and you need to be heard.
Good testers are like rocking horse shit, I've been through a lot of interviews to find people who can actually test. There're companies who think that they don't need to do integration or regression testing and that if anything goes wrong, they'll just patch it in the next sprint. These companies seem to think that unit testing and TDD can find every bug. They are wrong. They're great tools, but they're an adjunct to proper testing, not a replacement for it. A tester is not just the user's proxy in the development flow, they're the adversary's proxy too. If we don't try and attack the product and use it, you're letting the public have a go.
I also agree about black box testing, I try as much as possible to not look at our source repos, because when I know how it's expected to behave, that biases me to test it in that way. I like to test the processes of a user or attacker, not the expected code paths. That said there is definitely a place for whitebox testing where you do proper static analysis and identify as many code paths as are practical to test.
Testing does not, and cannot, guarantee quality or security, but it's a hell of a lot better than the alternative.
</rant>
I sort of went off on one there, sorry.
These companies seem to think that unit testing and TDD can find every bug. They are wrong. They're great tools, but they're an adjunct to proper testing, not a replacement for it.
Unit testing / CI for me really just tests that the programmer hasn't mullered the system somewhere they weren't expecting. One of the primary benefits I see of TDD is that it forces forethought about class design, and usually leads to some level of adherence to SOLID, Design Patterns, and other industry standard concepts. In essence then, it ameliorates the cowboys, it does not certify the code.
UT, however, are not the only thing you can automate. A proper enterprise level public facing application should have a test suite that simulates users clicking every button; entering range tested, type variable inputs into every field; generally doing what proper fleshy testers do, just automated as part of the build.
Most of the RBS style banking problems aren't developer related though, they occur because some low cost chair warmer in India didn't have the skills, experience, or training to understand that dumping a queue, or skipping a batch step is not the answer they're looking for. Stopping the off-shoring is the answer here, rather than Dev Ops minutiae.
@Thecowking - The more eyes a project passes through the more likely the final output will be correct. There are some many ways for a project to get screwed up from the initial specification, through programming, unit testing, review, and final testing that skipping any step is flatly idiotic.
"TDD can be great but as always with testing it should be quality over quantity.
30% coverage with good quality tests beats 90% coverage with bullshit tests."
For me coverage is a key measure of the quality of the testing... A figure as low as 30% suggests that they aren't even close to finishing writing and testing the code yet. If they really have delivered and tested all the functionality then 70% of their code is bloat. Neither outcome would be acceptable in my view.
In my experience the fundamental problem remains managers & bean-counters thinking they can reduce development times with an acceptable fall in quality by eliminating testing. They don't learn because they never try doing things properly so they just don't know any better.
People forget that while Automated Unit Testing is a good tool for testing, it's nothing more than that. It's a tool. It's not a substitute for human testing. It may be able to test for thousands of potential bugs, but it will not find any bugs it's not been set up to test for.
IMO, the best test of any system is to let a few users loose on it, as well as a few testers with a devious mind.
One of my colleagues tested a website I developed for internal use and tried all sorts of tricks I'd never even thought of to break the error checking I'd built in. He did succeed and I fixed the errors. Funnily enough, I didn't even have to try to break the site he'd set up. Both sites were linked. Mine was a public facing one to do with booking equipment and his was an internal site dedicated to equipment inventory management. I entered a few codes for equipment that didn't exist, and the site let me manage them.. When I highlighted this, he told me that as the site was for staff only, he hadn't implemented any error checking (obviously he thought staff don't make mistakes).
> I think that one of the problems, oddly enough, is the move away from the waterfall model, with it's stages of system test -> integration test -> user acceptance test.
For what it's worth, if anyone can bear to listen to an old fart... years ago when I developed 4GL systems we would code and unit test our own work then *demonstrate* it[1] to a peer, who then did the testing on it.
This had two key benefits: firstly it was amazing how often something mysteriously failed when it came to the demonstration [2]; and secondly if the test needed some fiddly data to be set up (think in terms of an bank account that needs a small overdraft; then 2 failed direct debits; then a small payment received; in that order before you can test that the right customer letter is sent out) you could show the tester what was required. This saved getting numerous 'failures' because the tester didn't understand the prerequisites rather than any actual error in the code.
If the demonstration worked with no obvious errors (just a simple path through - it's not the full test remember) then the peer would take over to properly test and code review. (The code review was to ensure coding standards were being followed.) If it passed that, only then would it get marked as complete.
I don't see any reason why agile teams can't adopt a similar approach. Everyone does some coding. Everyone does some testing. You all get to see each other's work which is good for learning - both from the good examples and from the bad.
(NB This was not the only testing, of course: there were still formal test scripts in order to show coverage etc. )
[1] 'It' in this case is the equivalent of a user story.
[2] Which puts the emphasis on simple, reliable code that works first time. The risk of embarrassment when demonstrating is a powerful incentive to be thorough.
All those bloody automated test cases are all well and good but they don't and can't replicate the quirks of the average user who will try every know keypress in 5seconds in order to speed up the system.
I'm just spent two days getting to the bottom of a nasty UI interaction problem. Do this, this, this, then that and opps!
All has to be done within a minute or things going on in the background make it all work. Timing Sucks.
Can anyone write an automated test for this? Somehow I doubt it.
Systems are getting more and more complicated. Proper (not the automated type) Testing takes lots of time and even more money.
We used to put people in front of a system and tell them 'Go Break it'.
Lots of very obsure bugs were discovered that way. Why don't we do it now?
Answers on a pinhead please?
The guys that built it are the last ones that should be testing. They (I...) know how it works, what you need is testers who haven't seen the code and have the devious sort of mind that tries all sorts of "unusual" inputs the original coders (and spec?) never considered.
One group that may be able to assist in the testing process but are frequently ignored are the first line support staff. I used to be included in this when I was supporting an in house application and would try everything I'd had reported from users on support calls. The purpose of the testing wasn't to prove that the application was bug free (the approach most seem to have nowadays) but to find ones that could only be exposed by using the application incorrectly. Use the whole ASCII character set in free text fields to find out which ones broke what and how, break the normal workflow to see if it can recover when the normal, logical order isn't followed. The likelihood of getting it fixed prior to release was minimal but at least we knew and could provide an answer.
In short, if everything goes pear shaped because someone entered a postcode backwards while typing left handed on the second Thursday of the month at least we knew in advance.
Proper testing does need to be done. Proper testing also costs money without any direct benefit to the profitability of the company. As such, the beancounters may see it as a cost with no profit, and some may choose to reduce or scrap it.
They forget to include the costs if the system is rolled out company wide without adequate testing and goes even partly tits up. They forget to include the costs to the company reputation, lost customers, refunds and other gifts given to keep the customers you've just failed happy.
They also forget to include the costs of any resulting legal action.
Don't get me wrong, I've been involved in the testing of systems as both a user and sys admin, and I know that even with the most comprehensive testing, there can still be problems but when I see a failed upgrade taking out a bank's core systems for days at a time, I have to wonder how much they have tested both the upgrade and procedures used to do it.
Proper testing significantly de-risks delivery and the money spent on testing should be proportional to the potential impact if the product goes TITSUP when it goes out the door - no need to always sweat the small stuff, but for something as important to a core business function like this..
If the PM or project team can't get that across to the beancounters, or the beancounters cock a deaf 'un, then there's going to be a problem!
I have to call every time I want to get into an online account with a bank in the backwoods of America, they can't\won't even provide me with an IBAN Number for my account?!
Ah, but that's just to stop you from handing back your passport. If you can't get to your US based funds, it makes it harder to make the sensible choice and run while you still can..
I love the classic American approach to the passport idea.
You experience a sharp ramp up in the volume of passports handed back. What do you do? Fix the problem? No, no - it's a market opportunity! Put the price up!
Sometimes I wonder if we shouldn't help them fence off the place. All it lacks is an "asylum" sign.
If the software wasn’t ready – and it clearly wasn’t – it wouldn’t have been expensive to inform customers that the release date was being delayed. I doubt anyone would have cared. But by pushing it out the door unfinished they created such a fiasco...
I've had that argument so many times, sometimes with managers who listren, sometimes with ones who wont.
Late delivery of a project annoys the hell out of a customer, but if it works they soon forget. Deliver something on time but broken and they'll be reminded every day they have to use the stuff.
@Doctor Syntax
The distinction is getting less clear. The bank is, to a large extent, its software.
It is, but unfortunately that isn't how banks see themselves, as software companies. IT is still just a cost to be managed as low as possible. Quality costs, it always has and it always will, but the drive for the past 8 years has been to ever cheaper just barely good enough software and staff, with the inevitable public overshoots that engenders.
> Any glitch, bug or issue will reduce my confidence in the bank as a whole rather than just the software
I recall and old fart I used to work with and some of his tales. His earlier life was spent on the nightshift for a large and well known confectionary manufacturers - the one that built a whole town in it's name - processing batch jobs on a single tasking mainframe.
One day he went into his bank to query something on his statement. After a short while he got what was then a common answer - "the computer made a mistake". Being a bit of a pedant, and mischievous as well, he replied along the lines of "well you'd better fetch the manager". When asked why, he said that he needed to get all his money out while they still had any - before the crash that comes with a run on the bank.
The manager was quickly called, and the old fart then explained that if the computer made mistakes, it couldn't be trusted - and that meant that no figures it produced could be trusted, and anyone who had any sense would get their money out of a bank that couldn't say with any certainty how much money anyone had once it became public knowledge.
The manager sussed that old fart knew about computers, and after a short conversation turned to the cashier and told her "never use that excuse again".
Late delivery of a project annoys the hell out of a customer, but if it works they soon forget. Deliver something on time but broken and they'll be reminded every day they have to use the stuff.
Late delivery can incur penalty charges. Defective delivery never really does. It's a lesson they learned from government projects - even the quality is similar.
@AC - "Late delivery can incur penalty charges. Defective delivery never really does. It's a lesson they learned from government projects - even the quality is similar."
This.
Just remember people, everytime you join in the criticsm of a massively complex government IT project for lateness or cost overrun, you're fuelling the drive to deliver on time and cost, rather than on quality. Naturally the testing regime will suffer to satisfy this.
Small scale but still damn annoying, Lloyds are so desperate to show me marketting material on their online banking site they'll happily completely freeze for 30s+, repeatedly, waiting for the content. Content that's blocked by multiple browser plugins as unsafe for various reasons including coming from a different domain.
Other sites carry on working while partially blocked like this. Did Lloyds not bother testing their shitty implementation or is it deliberate? They seem to think I'm using an advertising site with added account management functions.
Abusive idiots.
Yes, I'm doing a spot of teeth grinding due to that too. I notice that this is very much a character of sites that have been "tabletised" where I have to wait for the content to "unfreeze". Presumably while I'm being data-mined (tinfoil alert) or served crappy ads of no interest.
I highly recommend using stylish (https://addons.mozilla.org/en-gb/firefox/addon/stylish/) for firefox and reformatting the terrible new design. There's only a few bits of CSS you need to tweak to get a pretty acceptable desktop experience. Of course this isn't exactly a mainstream option!
@AC - "Lloyds online banking...completely freeze for 30s+, repeatedly, waiting for the content. Content that's blocked by multiple browser plugins as unsafe for various reasons including coming from a different domain."
I had the same problems, so I'm going to assume you are using NoScript (which seems pretty sensible with a banking site) and have just found out how to solve this, without turning off NoScript.
You have to specifically whitelist ONLY secure.lloydsbank.co.uk without allowing all of lloydsbank.co.uk, this isn't the norm for NoScript. The site works fine now that I have done this.
If you told Lloyds that their site didn't work properly they would simply tell you to turn NoScript off, which isn't particularly responsible behaviour IMHO. Considering that cyber is one of the big 4 threats to our nation, I can't help thinking that some good old fashioned regulation of what security compromises banking sites can and cannot allow is in order.
Late/delayed delivery probably means some managers won't be getting a bonus... so better for them it's deployed whatever state it's in, even if it causes havoc on other departments.
Have seen it in the past where "no new bugs" are recorded against the pre-gold version, which basically means we'll stop adding them to the bug tracker until it's signed off and my bonus is guaranteed.
I've sat a number of times, over the years, in my strange combined role ( it could probably only happen in education) as a senior frontline education manager and with added responsibility for local IT management and firstline IT support and training.
The Powers That Be will come along with a new piece of software. It sounds good, but I ask the questions.
Who designed it? - Usually an IT team somewhere far from the world of education. Often apparently using a different subset of English to the one my team uses.
Who did they test it on? - Usually a hand picked collection of administrators and suits.
How are we going to be trained to use it? - Usually me getting sent for two hours to a deserted office somewhere, then expected to "cascade" something I barely know how to use myself to staff who don't even know why we have been given this when the existing system worked really well. With no time allocated to do the "training" and little or no backup when it all turns out to be a dog's breakfast.
Typical Edukashun set-up.
I have come across similar problems twice; once at school and once a few years later.
1/ The head math teacher was tasked with teaching IT (this is in the days of the first 32K RM machines); with no training and no budget he had to raise the money for the machine (~£3,000), figure out how to use it, and then teach us.
Shortly after we took our exams he told us the truth - we now had better qualifications in IT than he did, and that he was usually only a week ahead of us on the coursework.
2/ Was actually my mum, as head of a Chamber of Commerce training centre, she was given the task of teaching office skills to teenage students.
The problems being
a/ She had no clue herself (and neither did any of her staff).
b/ They had no computers or software to train on.
The Friday before the first course started, a box of Amstrad CPC machines turned up, she brought one home and I spent Saturday figuring it out, spent SUNDAY teaching her what I had figured out, and at 8AM Monday morning she started teaching the staff, ready for the students arriving at 10AM
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
