RBS must realise it's just an IT biz with a banking licence Banks need to start thinking of themselves as IT companies, said David Chan of City University London. "A senior banking technologist has said to me: 'A retail bank is nothing but an IT company with a banking licence'," Chan told The Reg. "While this may seem extreme, when one looks at the economics of any retail bank, it is …
Lots of talk about change management preventing the problems, which is always a good approach, but RBS is suppose to have a BC/DR plan.
In this day and age it should be unacceptable for a major bank to lose mission critical capabilities for more than 24 hours.
So where was the BC/DR plan that says this part of the system has been "nuked", (in this case by operator error rather than the data centre burning down), this is how we restore it?
Even if they have one, what are the odds of it being tested regularly?
Most businesses cost-save, so having a duplicate system to fully try stuff out on (without breaking the operational system) is seen a a luxury, until there is a centrifugal excrement incident to be managed...
[quote]
Even if they have one, what are the odds of it being tested regularly?
[/quote]
Every system goes to DR at least once a year (well, that's the intention), there's an entire team for it. This issue was because the old version couldn't be restored with backup data from the new version, the old data couldn't be used because it was two days out of date.
Imagine having an iPhone with iOS 5, you have a backup and upgrade to iOS 6, you use your phone, have emails, messages, games etc. after a few days you find out there are serious issues and need to go back to iOS 5, fine you have a backup, but you'll lose any changes, not a big deal, little annoying, but hey - now imagine the same thing with a major banking system, you have a "point of no return" where you've been running fine for two days, data is changed significantly so you can only "fix forward", but it turns out that that ceases to be an option, you're in a no man's land, what if you were told that the backup from the new system could be loaded into the old system? DR and BC plans are all well and good, and even testing them is practical, but there are limits, can you test a gun without putting a live round in it? can you test a payment system without actually making a real payment? and if you can, is one payment enough? how many days before a new system is tested?
The question that interests me, and I suspect many high-ups at RBS and other banks, is how much damage will this ultimately do to their bottom line? I.e. how many customers will move their accounts etc.
For years, telecoms companies obsessed about getting the reliability of their systems up e.g. 99.9999% uptime on your landline. Then along came mobiles and VoIP, which were popular because they were more ubiquitous and cheaper respectively, despite being much less reliable. Companies like AT&T were looking the wrong way and were too big to change when they realised what was going on; they were optimising the wrong thing.
So, do bank customers really care enough about problems like this to actually close their accounts? If most of them don't care and stay with RBS, then RBS got it right by making their IT less reliable. What do customers care about enough to change banks? I suspect that no-one really knows....
"how many customers will move their accounts?"
Speaking from experience, one. When I needed to release some equity from my home to re-organise my finances (like a small scale version of a bank bailout), RBS didn't want to know, even though it would have reduced my total outgoings. So I'm moving from RBS to the Co-op as I'm tired of the mis-managment, rewards for failure, privatisation of profit and nationalisation of risk.
It just happens to be a convenient time for me to change banks, not everyone is so lucky.
O rly?
This is the definition of a diktat, or as Jacques Chirac once said about a young upstart of a home secretary (Nick Sarkozy) : "I decide, He Executes". I dictate what I want, just make it happen.
With the size of these "too big to fail" companies, this is what delegation is. You need underlings to sign off because there are not enough hours in the day to read 2000 emails, sign 30 contracts, manage unions, budgets, meetings etc etc etc ad nausam.... the problem is also that you are so far up the chain and removed from day to day operations, what ever those operations are (IT, HR, FM, banking operations that can be subdivided into zeus knows how many subdivisions (high street, business, corporate, stocks, investment etc...), then you end up managing your company though the balance sheet. Screw the effects, my accounts say that salaries, IT and real estate rental for the high street banks are my 3 biggest net costs, let's axe 20% across the board... but the net operational effects are totally different from front office and back office operations. This has not always been taken into account...
There are two sides to this particular coin.
The business managers who say "I don't do IT", and think that is a good thing. Rather than a reason to be fired the same way that would happen instantly if they told the CFO " I don't do budgets"
The other side of the coin is the number of IT departments and managers that I have met that are incapable of explaining the implications of the various IT options to a business manager in business terms, so that he/she can take an informed business decision. (this includes one knot where IT where saying iminent collapse of business critical system, but what the business manager was hearing was IT want to spend money on new toys that don't add business functionality. Once explained, the budget for the replacement kit was authorised PDQ)
The edge of this coin is made up of the issue of credibility in the organisation, and IT managers tend to be pretty bad at "selling" IT to the organisation as a proffesional advisor that finds ways to make things happen, with many companies viewing their IT department as an obstacle rather than an enabler and innovator.
It depends on your perspective.
If you are the person planning the change in a hurry, or the management committing the resource to do the change, then an over-rigorous, time consuming process is the last thing you want to add to your work or costs, so you do your best to short-circuit the process to make the change happen faster and cost less.
If you are the risk manager, who is on the line if changes cause problems, then you want as much process around you to protect your butt (and to a lesser extend, the organisation they are working for), and then a bit more. You feel most secure if there is no change at all (that's counter-productive).
If you are a diligent IT professional, then you want the *right* amount of change management to make sure that the change has been carefully considered, and has a good backout plan, but not so much as to make planning the change more difficult than it has to be.
It is this balance that is missing. You see it swinging from cost-reduction to risk management according to the current trends in risk and management style and the most recent disaster. And always nowadays, the people who understand it least are the ones dictating the processes.
If you are in a large organisation involved in change management, take a change and estimate how much the change costs in people and financial terms. Look at the time necessary to cross the 't's and dot the 'i's. Count the number of people involved. Look at the number of people who have to read and understand the change. Add up the people-hours spent sitting in the change board meetings.
You can often find in places like a bank that a change to switch servers from one DNS or time server to another (often simple but with a potential high risk and impact if it goes wrong), which may actually only take minutes to do, ends up costing you dozens of man-hours (or even man-days), involving people on quite high salaries, and many days or weeks to drive the process. All of these things cost money, one way or another.
I can give you one example - a process so convoluted that 80% of the change raisers time is spent on navigating the multiple levels of sign-off - leaving just 20% to focus on the impact of the change.
Most of the time, if the people are good enough this is sufficient - but sometimes it isn't.
Great article btw Anna, hits all the right spots without being rabid :)
I agree about the great article btw Anna, Sir Runcible Spoon, but it missed out on all the juicy hot spots which are highlighted here ....... "The American people have suffered for decades from the declining purchasing power of the dollar. The Federal Reserve has abused its position as the monopolist issuer of currency to enrich Wall Street and impoverish Main Street," stated Chairman Paul. "The Fed can effectively create money out of thin air with impunity, while creators of gold and silver currencies face jail time. This is a travesty. The only way to stabilize the economy is to return to monetary freedom by legalizing Constitutional money. Until the American people are free to choose the money they want to use, and not what the government forces them to use, the economy never will be truly stable and any recovery will be illusory." ..... with the Bank of England also falling into that sticky sweet honey trap of creating wealth from nothing and trying to deny it is something which is roundly abused in closed circles for fabulous riches. And we haven't even mentioned the euro trying to get in on the act and create another layer of extremely well paid fools who would think they can use money as power to try and control the masses with its rank withholding rather than stock supply ...... and another group of self-serving, wannabe emperors who would excel at fiddling to the detriment of everything else.
Apparently tomorrow there is a pow-wow about it ...... http://www.thedailybell.com/4140/Paul-Subcommittee-to-Examine-Sound-Money-and-Parallel-Currencies
And as Libyans and Iraqis have found to their cost and at the cost of increasing monumental US debt which is never intended to be repaid, but rather more crazily expects to be able to expand exponentially with no ceiling, is military invasion and a war of terror and death and resultant destructive chaos the other preferred option to trump any nonsense of jail time for creators of gold and silver currencies and/or any other default method of payment outside of existing increasingly worthless paper notes/dodgy government bonds/disgraceful IOUs. And that is an unsustainable and self-defeating socio-political strategy which creates a world of smarter enemies to attack and remove all means of wealth generation and global support.
Mr Mars, as time goes by we are seeing and hearing more about the strings that are controlling our destinies. Trouble is what we as individuals can do about it - and if someone does have a good idea about what to do how are we to trust again? … Anonymous Coward Posted Wednesday 1st August 2012 19:53 GMT
That is quite the paralysing Catch 22 for controlled humans, AC, which fortunately does not affect that and/or those which lead them/feed and seed them new processed information, which would be novel advanced intelligence already comprehensively beta-tested in a radically stabilised alternative situation ……. completely changed environment which would require and provide completely different leadership arrangements.
In such completely changed environments, are new ideas easily exercised autonomously by revisionary leaderships as they encounter no prevaricating opposition and significant encouragement from earlier catastrophically failed administrations which have themselves benefitted immeasurably from revised and revolutionary human programming and ITs Reprogramming of Powerful Command and Awesome Remote Control of Virtual Reality Systems for the new completely changed environments.
And with regard to ….. "Trouble is what we as individuals can do about it - and if someone does have a good idea about what to do how are we to trust again?" …. you as individuals need do absolutely nothing about it, which is more than just fortunate should you not be intellectually equipped to do anything about it, for IT is doing everything for you. And trust will return, if ever it is again needed, whenever intelligence levels are raised and one starts to see the benefits of changes that radical solutions you may not have been immediately aware of, are shared/revealed/exposed/leaked. ……. and one further realises that that is no chance coincidence either, but all part of a pre-ordained and simply complex master plan, with SMARTR Leading Parties/IntelAIgently Designed Entities simply following complex instructions received from communicating nodes in …… well, let us just call them Surreal InterNetworking Systems for Virtual Space Places in Live Operational Virtual Environments and be done with it, and get on with it and get it on with IT.
Or you could just remain much as you are, paralysed in ineffective inaction in systems which are collapsing.
Methinks a quantum leap into radical change is a much better option though. What say you?
That's not hard to see, for managing at a distance is harder than being there in-person. And that distant relationship also tends toward "work to the letter, not the spirit", even without factoring in cultural differences. Not to mention that arguing back and not losing the contract come renewal time is that much harder at a distance, too.
Now please follow my reasoning here: If you're good at managing, you know how to attract and retain good people. You can make do with a small crew of highly skilled people, and don't need to resort to off-shoring to try and "cut costs" despite all the extra back-and-forth due to, well, the myriad differences you get from hiring people a continent or two away. Thus, while you could argue that there's nothing "intrinsically" wrong with outsourcing apart from the black box-thinking that also manged to exacerbate the problems, the reasons for trying it on stem from failure to get it right locally. IE, bad management.
Then again, we'd already established there were signs on the walls aplenty, and we have a nice reminder that rules alone do not an organisation make. So what do they do when the rules fail? Add more rules, of course. Might as well put the whole thing in the bin and start over. With qualified management, this time.
Too bad that banking itself is regulated to the gills, so you won't see an actual lean&mean IT-shop-offering-banking anytime soon. There is no such thing as a sanely run bank for exactly that reason. And yes, I'll take a no-red-tape-attached banking licence suitable for an entirely 'net-run banking IT shop for a hundred quid, Adair.
Since more and more IT bizzes are run like banks now anyhow. That's why there has been so little innovation in the last years.
What they need to realize is that IT is important. That spending money on it, is a strategic investment into the future. They need to understand that it's not just about keeping their current systems running, but also about inventing the future of automated banking. They need to understand that if they don't so that, they will go broke eventually, loosing the market to new upstarts like PayPal. PayPal is not successful because it's good, but because the other banks are so bad at doing their business.
However in a world where banks are "to big to fail", that doesn't matter. It doesn't matter what you do because in the end the taxpayer is going to pay your bonus anyhow.
This post has been deleted by its author
This article pretty much sums it up. There are however 2 other issues that were touched on but which I have drawn my own conclusions over the years.
1 - Pretty much all organisation consider IT as only a cost centre (as opposed to a profit centre). They therefore continually press for reduction in operating costs. The lack of real intelligence is startling as there seems to be little appreciation of the fact that without IT the organisation will cease to be. In the RBS case it would be very interesting to see the figures provided for the cost savings of the IT restructure and how that stacks up against the financial impact of this one incident (I'm sure there are others that have costs the organisation but have not been so widely publicised)
2 - The change management regeime at RBS has (for many, many years) been driven with the wrong philosphy. It is run like a police state. They use it to create fear and to enable isolation of staff at fault. The change management process SHOULD be a service for the IT function (as well as other business areas) to facilitate change. I should assist in ensuring change is implemented in the correct manner , with the correct considerations and as efficiently as possible. As intimated in the article, the process is so complex @ RBS that the staff involved in change spend so much time on the administration that (given difficulties with obtaining change windows and the volume of change that is required to keep the operation running), that administration takes away focus from the technical detail. Again as the article indicates each time there is a change related technical incident , the WHOLE of the IT business endures increased process/admin, in most cases complete lock down. Which again creates backlog of mandatory change (either technical to remain on supported versions, or regulatory to ensure tha bank is allowed to operate). The whole situation creates inevitable failure and only does more so as time goes on and a multiplier effect takes hold.
Both of the above can be explained by the management style at the bank, where it's extremely rare for management to be challanged by their direct reports once you get past the front line technical staff.
I dislike it in general when engineering principles are shoehorned willy-nilly into IT operations, but in this instance it's worthwhile. The analogy with aircraft at the end of the article echoes this.
Firstly, someone has actually carried out a proper root-cause analysis here, in terms of looking at the actual decision and process chain, not just individuals and software products. That's a refreshing change in this business.
The next step is to follow up that root-cause analysis with actual fixes and improvements.
In the aviation industry, while certain standards are thoroughly mandated, what you do underneath to achieve the right outputs is kind of moot, as long as you have your standards covered (e.g. triple redundancy, outputs in x format, etc etc). If there is a failure, then the regulator can and will pull an operating licence if there is a standards breach. Root-cause analysis will go right back to manufacturing processes, staff management/training processes and so on and so forth to base its airworthiness directives on to enforce the appropriate fixes.
Cumbersome processes have been stripped away with airworthiness directives - at one point, there was a dozen-step procedure for pilots to get through before they could evacuate themselves from a burning plane. I believe it is now 4 steps (shut down engines and fuel being the major components).
Of course, in this instance, the regulator has the power to enforce anything they like, with the operating licence as their ultimate fallback. Financial sector regulators seem to only concern themselves with accounting standards - ok in the days of paper - but this is not the reality now. Perhaps they need to start concerning themselves with all the operating aspects of a financial entity, including how transactions - people's livelihoods and life savings - are processed, secured and managed, beyond the basics of balancing the books.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
