back to article Oh no, RBS has gone titsup again... but is it JUST BAD LUCK?

And here we go again, another IT systems failure at RBS. RBS appears to have been having a remarkable run of high-profile core-system failures, but I suspect that it has been rather unlucky or at least everyone else has been lucky. Ross McEwan, the new chief executive of RBS admitted to us in a canned statement that "decades …

COMMENTS

This topic is closed for new posts.
  1. MrGoggle

    Ha Ha Ha

    Those people at the top, They are really in touch with their organisations. They think they are worth the millions of pounds that have spent on them to save money, yet they cannot seem to invest some of the money correctly into core systems that keep the business running. It seems like the technical people that develop and keep the systems running are the real masters of the universe, not the bankers.

    1. Anonymous Coward
      Anonymous Coward

      Re: Ha Ha Ha

      It seems like the technical people that develop and keep the systems running are the real masters of the universe, not the bankers.

      Yup, but the technical people move on to another job when they find that there is no career progression or salary increases in their current role. Either that or they get made redundant when their role is outsourced. The turnover in IT staff is a huge problem, as we're seen as a commodity or component that can be simply be replaced with another programmer, sysadmin or DBA.

      For instance, I had a role where I had architected and lead the implementation on a quite sophisticated system. This took three years, during which I got no salary increase. I then lead the support and enhancement of this core system, as well as implementing a few others on the underlying framework we'd written, for another two years. This included a painful merger, when we were verbally promised improved salaries if we stayed loyal. Then we were finally told there would be no salary increases.

      So with a salary eroded considerably by inflation, I quit. The DBA then quit. The other non-contract programmer then quit. It took them months to replace me, as they actually lowered the salary they were offering for my replacement. The person they finally got didn't understand the core system and completely screwed it up.

      1. James 51 Silver badge

        Re: Ha Ha Ha

        Sounds like they got what they paid for. ^^

      2. Shagbag

        Re: Ha Ha Ha

        "It seems like the technical people that develop and keep the systems running are the real masters of the universe, not the bankers"

        I'll just get my Battle Cat then....

    2. BillG Silver badge
      Pint

      Re: Ha Ha Ha

      > Those people at the top, They are really in touch with their organisations. They

      > think they are worth the millions of pounds that have spent on them to save money

      It doesn't matter what they are worth. It matters what they can get paid.

      Sorry, but as I get older I realize that I'd rather be wrong and rich, than right and poor.

      who is brave enough to argue for budget to go back and fix those things which aren’t broken

      As one business mentor told me, "nobody gets promoted for preventing screwing-up. Nobody gets promoted for taking preventative actions"

      1. Anonymous Coward
        Anonymous Coward

        Re: Ha Ha Ha

        It's no joke. In my professional role, I work with C-Level execs in banking, education, manufacturing and retail. All but a very few have even so much as the faintest clue about the department or organisation they are running and make frighteningly poor decisions having been fed a line of spurious ill-informed nonsense. It's not just only a case of 'the Emperors New Clothes' - his courtiers are also parading around with their backsides hanging out as well.

        There is an insidious level of greed and short-term opportunism in UK 'Big Business' today that leads to short-term penny pinching with no view to the long-term good of the whole business - all driven by a bonus culture and a complete lack of accountability. If you off-shore or out-source your IT systems, you deserve everything you get - unfortunately, you'll probably be long gone by the time everything comes tumbling down and no-one will be able to pick up the mess you left behind.

        1. BillG Silver badge
          FAIL

          Re: Ha Ha Ha

          It's no joke. In my professional role, I work with C-Level execs in banking, education, manufacturing and retail. All but a very few have even so much as the faintest clue about the department or organisation they are running and make frighteningly poor decisions having been fed a line of spurious ill-informed nonsense. It's not just only a case of 'the Emperors New Clothes' - his courtiers are also parading around with their backsides hanging out as well.

          Nowadays people don't become C-level executives by business skill, they do it by holding a black belt in corporate politics, period.

          And the courtiers have figured out that they can garner power for themselves by controlling the information they feed their bosses.

          In the end, in both the US and the UK, the corporate systems are broken because management no longer has a vested interest in the company's success. The rich get richer and everyone else gets screwed.

  2. James 51 Silver badge

    Title is too long.

    " It seems like the technical people that develop and keep the systems running are the real masters of the universe, not the bankers."

    But the attitude is IT is a cost centre, not a profit centre. Every penny spent on IT is wasted but spend millions on X manager or Y sales persion or Z broker, why look at how they are cutting costs/selling stuff/????

    Every bank and insurance firm in the land is in the same postion.

    1. P. Lee Silver badge

      Re: Title is too long.

      > But the attitude is IT is a cost centre, not a profit centre.

      It is a cost, not a revenue centre, but there are so many over-cost projects which don't deliver savings and/or are not at all robust.

      The general trend in s/w development appears to be "larger, more bloated and buggy" so rewriting is an unattractive option.

      1. Anonymous Coward
        Anonymous Coward

        Re: Title is too long.

        'The general trend in s/w development appears to be "larger, more bloated and buggy" so rewriting is an unattractive option.'

        Do one job, and do it well. MS has seen to the death of the Unix philosophy.

        Also, if these companies ran on F/OSS the code could still be maintained/ported/documented. The only sauce they need keep secret is the configuration/business rules that the code processes. £10 + £1 is still £11 whatever way you cut it.

        1. Anonymous Coward
          Anonymous Coward

          Re: Title is too long.

          "Also, if these companies ran on F/OSS the code could still be maintained/ported/documented"

          That's not the issue here, the code is maintained, it's just that it's been maintained to a point where it very rarely needs any active maintenance. Also, it would have been impossible for these systems, which date back 30+ years to have been made on FOSS as there just wasn't any then.

          These days, where is the FOSS CICS or the FOSS database to rival DB2 or Oracle? They just don't exist. They also certainly don't exist with anything like the track record to run core banking on. RBS do run large amounts of Linux on Z and Intel but these are not core systems. Also the requirement for support means that they can't just go about changing the code and a bank doesn't do OS development - it's way out of core business!

          1. Frankee Llonnygog

            Re: Title is too long.

            The first question bankers ask about anything is 'who owns the risk'. The right answer is 'not me'. Hence they buy core systems from companies with liability insurance

        2. Nigel 11

          Re: Title is too long.

          £10 + £1 is still £11 whatever way you cut it.

          ROFL. Whatever legal way!

          One of the oldest "victimless" frauds was dropping all fractional pennies into a personal account, instead of rounding or dropping them into the bank's own account.

          1. Anonymous Coward
            Anonymous Coward

            Re: Title is too long.

            Nah ... Superman or Lex Luther always notice eventually.

        3. JimC Silver badge

          Re: Title is too long.

          The code *is* being maintained/documented/ported... What its not is fully understood. I'm all for open source software, but it appears to me to have sweet fa to offer to this problem.

        4. Stephen Channell
          Facepalm

          do be serious

          The OS that run the mainframes used to be distributed as source (OS/360,OS/VS2,MVS), but customers didn’t want it, and bits of the IPL code that boots it is older than some pensioners. The code is not the issue, it’s the people. Fact is a 55 Systems Programmer can’t be bought with a promise of promotion; its cash or dash. A 40 C-something has no real understanding of what that guy does and doesn’t believe him anyway.

          We’re about to get one of those lessons in what a promise is really worth to a bright 20-year on (UK) minimum wage looking for a route overseas.. meanwhile HSBC & Barclays will pay what they need, and we’ll never hear about it.

      2. James 51 Silver badge

        Re: Title is too long.

        IT for big business is a cost in the same way that the car in F1 is a cost. You might save money by getting rid of it but you aren't going to get very far.

    2. Anonymous Coward
      Anonymous Coward

      Re: Cost centre, Profit centre

      One of he worst business concepts ever. Setting people apart and against each other. Anything that does that is just damn stupid management.

      1. Gartal

        Re: Cost centre, Profit centre

        Couldn't agree more, just look at what happened to Nokia with their stupid internal restructure in which each area of the business became competitors with the others.

        I bet the turds who originally came up with the idea made a quick $Z and ran off laughing.

  3. Andy The Hat Silver badge

    I applied for a job with a large retail group in the 80s. I found out that part of the job description was to transfer the entire stock database system to new hardware with a complete rewrite using a language as yet unspecified. When asked what the souce was they replied 'assembler' and I replied 'byeeee"

    1. Roo

      "When asked what the souce was they replied 'assembler' and I replied 'byeeee""

      I think you might have missed an opportunity there... Rock solid spec (ie: the code), writing code in the language of your choice... I think your fear might have let you down badly there. ;)

      I have found reading assembler written for "big" machines (eg: VAX, S/360 etc) by old school programmers (you know, the guys who inclued verbose comments describing the pseudo code for the program) to be a lot easier than reverse engineering a system written in a mixture of VB6, VBA, SQL, C#, C++ and Java for example...

      1. Pascal Monett Silver badge

        I completely agree. In assembler, it does exactly what it says on the tin. There are no mysterious side-effects to take into account. Even if there is ten times the amount of code to check, once you've analyzed a block and signed it off, you're SURE you know what it does.

  4. Pete 2 Silver badge

    Oh yeah!

    When Usenet was new, I saw a couple of posts discussing a kernel bug. The discoverer posted the method of replicating the fault - instructions on which programs to run and what options to use. This was followed by the warning that doing this would reliably kill your machine (A Sun platform IIRC).

    A few posts later, someone replied "Oh yes, it does - doesn't it".

    Could it be that someone within RBS had identified the cause of (one of their many) cockups, told his/her/its colleagues - one of whom then replicated the conditions ...

    1. Anonymous Coward
      Anonymous Coward

      not just faults

      I had a colleague trying to extract data from a live account system. He used somewhat unusual methods which effectively scraped the data he wanted on an individual account basis. The side effect of this was only noticed when someone a few desks down from me complained that they couldn't access the accounting system (which has about 20000 users across many centres/countries). I asked him to stop his routine and the system started responding again. He was convinced his code was not to blame, so we waited half an hour, noting no issue with the system, then ran his code again. This time we spent a few minutes on the balcony overlooking the main sales floor. Everyone's system was not responding. He finally conceded that it might be his fault. He stopped it, and it all sprang back into life. He has since moved on to be our main solutions specialist. In all fairness, he was grappling with an unfamiliar system designed by committee and patched too many times to be efficient.

      Maybe somebody in RBS is trying something equally silly.

      1. Pete 2 Silver badge

        Re: not just faults

        > He was convinced his code was not to blame

        And that's the problem. While his code worked, it sounds like it had many unforeseen (and on a production system: unforgivably untested for) side-effects: possibly including high I-O usage, unintended database locks, de-cacheing, or futzing (technical term) with the middleware - among other possibilities.

        Too many outfits I've worked at regard "testing" as merely checking the known input produces the expected output. They don't bother checking boundaries, error handling, scalability or resource-hogging. Since a lot of these other tests would require having access to a complete replica of the production system - with a simulation of all the users and their workloads - most companies can neither afford, nor organise such a massive testing operation.

        1. Yet Another Commentard

          Re: not just faults

          "Too many outfits I've worked at regard "testing" as merely checking the known input produces the expected output. They don't bother checking boundaries, error handling, scalability or resource-hogging."

          Indeed. During a SAP test at my old employer I was told off for going off the script and trying to break it by putting in stupid data (letters in value fields etc), or forcing it to do something it shouldn't. "We don't have time to test that" being the answer. Oddly enough, launch was an unmitigated disaster as the users all tried to circumvent the controls (successfully) or put in data in the wrong fields breaking reports and searches. Ho-hum.

        2. This post has been deleted by its author

          1. Scott Pedigo

            Re: not just faults

            What? You never wrote an interrupt handler? Just don't forget to turn them back on...

            1. Will Godfrey Silver badge

              Re: not just faults

              ... eventually

    2. Dances With Sheep
      Alert

      Re: Oh yeah!

      First rule of coding init....

      "Never test for an error condition you don't know how to handle" ;-)

  5. Anonymous Coward
    Anonymous Coward

    Ha Ha Ha bis

    While I do agree with the contents of the article... just look at things from the perspective of the bank - as a money raking machine - and not from an IT best practice perspective.

    The mistake is to think retail banks consider their consumer facing systems to be "mission critical"....

    Retail banks funnel money in to allow their investment bank buddies to play the stock market (and these systems get all the budget they need)

    Providing services to the customers comes second fiddle at best...

    So, If it ain't broke don't fix it... and keep it running until it breaks.

    If you're lucky you can run for years without any major additional investment in the system

    And when it breaks (because it will) then just put a contrite face on - fire a scapegoat CTO - give a few quids of compensation money to your punters - and invest (minimally) into a new system (or a patch.

    This "mess" will cost you much less than properly managing and upgrading your legacy systems....and will be forgotten by the punters in a few weeks....

    1. Yet Another Anonymous coward Silver badge

      Re: Ha Ha Ha bis

      That is an excellent strategy system until somebody comes along and tells you to merge 3 different high street banks - who all have systems like that - into one modern dynamic environment.

      And then tells all the staff that they will be outsourced when the job is finished....

    2. Anonymous Coward
      Anonymous Coward

      Re: Ha Ha Ha bis

      Verdad! They also take advantage of people being too busy or too lazy to switch banks. They always bank on angry client's anger dissipating quickly. In short they have us by the cojones. And you're right about IB money, customers exist to fund trades, because those fund real bonuses!

    3. Anonymous Coward
      Anonymous Coward

      Re: Ha Ha Ha bis

      ...or better still Outsource the management of your systems, ideally to the lowest bidder, then you can pretend all is well while they (the Outsourcer) does minimal management (no updates, no patches in case such changes introduce a service interruption) and produce lovely charts each week/month showing how SLAs have been maintained. Eventually the lack of investment will catch up with you and there will be a costly/ embarrassing outage but you will either a) be long gone with a hefty wedge in your back pocket or b) be newly into your role therefore not responsible for the mess. In either case, you can fall out with the Outsourcer and both part company at the end of your 3/4/5 year O/S deal and enjoy the thrills of retrospectively updating systems that are back-level by 5-10 years

  6. Anonymous Coward
    Anonymous Coward

    It strikes me...

    ...that there is a naming problem here. When Ross says "systems", he's not talking about the tin on the floor, or in many cases the middleware or OS. All of these things are easy to replace.

    What he's talking about is the "business system", which is the bespoke software and operational procedures. Much of this code is touched as little as possible.

    I think you have picked up on this, but most of the media have not. They are still thinking that the banks have mainframes from before the millennium.

    I can promise you this is not the case. Their machine rooms will be mostly filled by hardware less than 5 years old, with very regular maintenance and patching processes. If they did not, the FCA (the regulator that replaced the FSA) have the ability to suspend the bank's operating license. Believe me, this provides huge motivation to the bank's management to make sure maintenance happens.

    I got caught up with this at my last stint in one of the large banks about 6 years ago. After an internal audit to check compliance, it was decided that if there had been an external audit of the maintenance and patching regime, the bank would have been severely embarrassed by the results. Suddenly, getting the most recent OS patches and firmware updates onto as many machines as possible became the highest non-operational priority within the support organisations, and at one point, almost all of the support personnel within the part I was working in were involved in one way or another.

    In this instance with RBS, I heard on the grapevine, very indirectly, that the problem was with a piece of privileged third party monitoring software which interfered with the sysplex locking structures, causing database access to fail across the whole sysplex. It appears that they effectively had to restart part or all of the sysplex to restore the function. Whether they had to roll back the database(s) and re-apply transactions I do not know. Speculation is that this was a known but obscure fault.

    1. Anonymous Coward
      Anonymous Coward

      Re: It strikes me...

      That is not entirely true. While it is completely correct to highlight the business system as the main problem, a lot of banks ARE running legacy systems on ancient hardware. Various banking regulations actually prevent older data/processes from being updated. If you look at ebay you can see certain types of old hardware get pounced on for spare parts to keep these running.

      1. Anonymous Coward
        Anonymous Coward

        Re: It strikes me...

        It varies from bank to bank.

        After the Millennium, Barclays (my main experience) spent significant amounts of money and a lot of effort consolidating as many systems as they could onto a maintainable platform (they called it project "Stretch"). The aim was to eliminate as many of the old systems as they could.

        This spanned the Mainframe, and most of the UNIX platforms they ran.

        The main aim was to get all the systems running on a smaller number of larger, more cost-efficient systems that were virtualised/shared and could be bought in volume, and managed more easily (sound familiar - they were well ahead of the curve, and doing this before Hyper-V and VMWare were invented or regularly deployed). But a side effect was that the systems could be maintained more easily, and could be subject to regular OS updates,

        They did not completely succeed. Sometimes, the specifics of code that cold not run on more modern versions of the respective OS (I remember Sybase and Informix causing some problems). Sometimes they required specific hardware that no longer worked on modern systems (hardware encryption cards were the most serious problem IIRC). But where this happened, the applications were flagged as in serious need of replacement, and the internal businesses warned that they were in significant danger of losing their service and breaching audit requirements.

        But the penetration was more than 80% when I left that project in 2005, and got better after that time, I'm led to believe.

        In general, the only stuff that really got left behind was the international banking systems for minor countries (like African countries), where upgrading the remote systems was complex and difficult because of their location, so the local systems were kept largely in line and on the same hardware. But even this was being addressed. In all of these cases, there was an explicit risk evaluation done by the internal auditors, so the respective internal business managers were well aware of the risks they were taking.

        So legacy systems were the exception rather than the rule, and were known about. I'm sure that most of the other large banks in the UK are the same. Certainly not what is being implied by some commentators.

    2. Anonymous Coward
      Anonymous Coward

      Re: It strikes me...

      "I can promise you this is not the case. Their machine rooms will be mostly filled by hardware less than 5 years old, with very regular maintenance and patching processes. If they did not, the FCA (the regulator that replaced the FSA) have the ability to suspend the bank's operating license. Believe me, this provides huge motivation to the bank's management to make sure maintenance happens."

      Doesn't work like that. They have to have support for all equipment not a certain age limit. If a vendor agrees to support it then they can run kit into the ground which is why a lot of financial's are still chuking big bucks at MS to keep Win2K and Xp boxes going

      1. Anonymous Coward
        Anonymous Coward

        Re: It strikes me...

        I'll concede that. But if an old system is still under support, and the patches are the latest that vendor/support agent have, it meets the audit requirement. But you would hope that a system that is so old is (a) backed up by a sufficient spares pool, and (b) had most of the bugs rattled out of it over time.

        But you're not going to find an IBM 3090 still running anywhere in a UK bank, because it is has been easier and cheaper to migrate onto a more recent mainframe, even if you have to run the application in some compatibility mode because you can lo longer compile it.

        You will note that I used "mostly", and I did not say that the FCA mandated that the systems were younger. It's just that most banks can easily spot that maintenance is an FCA requirement, and the cost of maintaining older kit is expensive. Most of them do actually replace kit when the economics are right, especially with the vendors so keen on selling new kit. What the FCA do is make sure that they have a maintenance strategy, and that it is being followed.

        The banks can no longer conveniently forget that piece of critical kit sitting in the bottom of the rack in one of their old machine rooms, because they can quite realistically be forced to stop trading as a bank if the regulator believes they are seriously neglecting their responsibilities. This has sunk into the bank management. Being the chief executive of a not-bank when the licenses were suspended would seriously damage the chances of another high-profile job.

        I'm quite sure that this RBS incident will trigger an FCA investigation at some level, and there will be questions asked about the maintenance state of the systems involved.

  7. dotdot

    Cost versus benefit

    I hear the statement above re IT being a cost.

    Whilst its an interesting point , it's also a point worth referring to when you think about old IT and new / newer IT infra.

    Many are moving to a... use my kit , pay me money ..model , ie a charge based model.

    In which case IT for some is becoming a part of the charge base , if you make Money we all make money and not the poor relation of old.

    Worth noting , when anyone talks about IT and old kit... In future .. If its old its probably costing you a fair wedge (remember support costs on old kit , people) if that wedge is a cost , then your "wheels" and support within the org is poor, you're onto a loser and as ,quite rightly, the storage bod points out , it's only a matter of time.. Before those "wheels" come adrift .

    1. asdf Silver badge

      Re: Cost versus benefit

      >Many are moving to a... use my kit , pay me money ..model , ie a charge based model.

      >In which case IT for some is becoming a part of the charge base , if you make Money we all make money and not the poor relation of old.

      What could possibly go wrong with moving all of your mission critical business systems into the cloud? I am sure your provider has your interests at heart. The NSA snooping is just bonus.

      1. Nonny Mouse

        Re: Cost versus benefit

        I used to work for 2e2, can't see anything going awry with sticking mission critical stuff in the cloud myself.....

  8. Version 1.0 Silver badge

    Planning? We've heard of it...

    Isn't this a wider problem? Who writes code with the expectation that it will need to be replaced or migrated in ten or twenty years in the future? Look at all the recent grief given MS for daring to say that Windows XP will meet it's "Sell by" date next year - and really it should have been replaced several years ago.

    But computer programming is still in its infancy compared to Banking and we're still struggling to figure the whole thing out - sure, there are computer languages that have some modicum of portability and long term evolution but we've mostly thrown them out because the new kids on the block are sexier and offer mood of the moment features - who bothers to learn COBOL, FORTRAN, or ADA any more?

    Old stuff - only old farts write in that stuff - we use VB ... no, wait lets code the new ATM with Ruby or Java ... and so down the road we discover that both our new fancy languages have more holes than an aerosol sandwich. I'm not arguing for a wholesale return to writing banking applications in COBOL (assuming you're still with me and haven't down-voted this and moved on) but simply saying that we need to learn from the history of application writing - COBOL et al have legions of issues but the applications worked and were/are supportable for a long time ... do you really want to be debugging Java code that's twenty years old?

    But nobody ever thinks about the future when they write the code these days - just get it out of the door, collect the bonus and move on.

    1. Anonymous Coward
      Anonymous Coward

      Re: Planning? We've heard of it...

      "who bothers to learn COBOL, FORTRAN, or ADA any more?"

      I know COBOL. It was good at what it did. Would I want to write a GUI app in it? Hell no. Would I prefer it to the likes of Java for tearing through masses of formatted files/records? Hell yes.

      "do you really want to be debugging Java code that's twenty years old?"

      Even debugging Java code that's two years old makes me cry.

      1. Tom 7 Silver badge

        "who bothers to learn COBOL, FORTRAN, or ADA any more?"

        Software Engineers who are interested in making their code work with the above - or even fixing it. You can write unit tests and- fuck me - good comments in all the above. Just because a language is old doesn't make it immune from good software practices. But I bet the budget and management probably do.

      2. Destroy All Monsters Silver badge

        Re: Planning? We've heard of it...

        > Even debugging Java code that's two years old makes me cry.

        But why?

    2. Anonymous Coward
      Anonymous Coward

      Re: Planning? We've heard of it...

      "who bothers to learn COBOL, FORTRAN, or ADA any more?"

      Ada flies aircraft. No Ada, no fly (there are obscure alternatives, arguably safer and quite possibly lower cost ones too, certainly flight proven over many years, but because they are obscure and contractors therefore cannot be instantly recruited from the jobcentres, the High Ups prefer Ada).

    3. proud2bgrumpy

      Re: Planning? We've heard of it...

      The problem is that COBOL / FORTRAN are not *cool* languages any more. But they are absolutely suited to the purpose they were designed for: Processing transactions - ie shunting around text and numbers (with a little bit of simple maths) from one place to another. Because these systems are very stable - ie: the language itself is stable & subject to few changes and the business processes are stable, few changes are required, so these languages (COBOL etc) appear to be in decline because there is relatively little continued development effort required. In reality many, many more transactions are handled by this old code thanks to the increased capabilities of modern HW.

      So now we have a dearth of COBOL coders and a glut of Java/VBA/C/C++/C#/Javascript coders - so what happens, these (stable and working) systems get re-worked or interfaced to by Java/VBA/C/C++/C#/Javascript code and these Object Oriented languages are not optimised for this sort of simple transactional workload.

      Dependencies on 3rd part frameworks, overly complex syntax, deployment onto cheap wintel servers add unnecessary complexity and fragility. Simple truth is that pre-Windows GUI (and now Web UI) these systems ran on simple Green Screen Dumb terminals (remember them?) they were more efficient, the staff were more efficient and the maintenance of these systems was simpler, more effective and less costly. Whats more they do exactly the same thing today as 20 years ago - nothing has changed except the increased effort to achieve it.

      1. Anomalous Cowturd
        Thumb Up

        @proud2bgrumpy

        Welcome to elReg.

        Have an upvote for talking sense.

        Try not to make a habit of it! ;o)

      2. asdf Silver badge

        Re: Planning? We've heard of it...

        >Simple truth is that pre-Windows GUI (and now Web UI) these systems ran on simple Green Screen Dumb terminals (remember them?) they were more efficient, the staff were more efficient and the maintenance of these systems was simpler, more effective and less costly. Whats more they do exactly the same thing today as 20 years ago - nothing has changed except the increased effort to achieve it.

        You may have a point in some cases but not in general. Worker productivity the past few decades has went up across the board like no other time in history and much of this is due to software interacting more efficiently with people. For example, the GUI would never have become so ubiquitous if it was not useful to the layman. In fact computers didn't really take off with the layman until the GUI came around. There are lessons to be learned from the past but the paradigm of hardware being much more valuable than development time as was the norm then doesn't translate well today.

  9. Anonymous Coward
    Anonymous Coward

    All the major banks are just as worried about something like this happening.

    Most people would be surprised that excel is one of the primary inter bank exchange tools used to verify and transfer extremely large sums of money on a daily basis.

    1. 's water music Silver badge

      Excel and finance pros

      > Most people would be surprised that excel is one of the primary inter bank exchange tools used to verify and transfer extremely large sums of money on a daily basis.

      I would be surprised to find that there was /any/ human activity that isn't carried out with excel by at least one accountant or banker.

      1. Grikath Silver badge

        Re: Excel and finance pros

        Why would we be surprised? I've used it, and still use it for any number of transfer/conversion stuff I have to do. Peeps may not have your flavour of DB, but they will damn well have Office running somewhere. Whether it's oodles of money, or stock, or $data, there's a ton of things you can do with Excel (or any other decent spreadsheet) to make a visual check of the data easy on the eyes, convert whatever you get in to the $format you need to feed it to your own setup and make pretty graphs for the higher-ups who are not interested in how things work, as long as the numbers fall within their expectations. All in one go.

        Disparage them all you want, decently set up (and locked!) spreadsheets can save you a lot of headaches.

        1. Destroy All Monsters Silver badge
          FAIL

          Re: Excel and finance pros

          there's a ton of things you can do with Excel (or any other decent spreadsheet) to make a visual check of the data easy on the eyes, convert whatever you get in to the $format you need to feed it to your own setup and make pretty graphs for the higher-ups who are not interested in how things work

          The ONLY thing that that abortion (which used to be a so-so and half-arsed idea to make PCs useful back in the early 90s) is good at is shooting yourself in the foot. And I do hope the medication for the FSCKING UTTER IDIOTS OF EPIC POINTYHEAR UNIVERSITY who don't realize this is computed by their docs in Excel.

          88% of spreadsheets have errors: Research: Bad math rampant in family budgets and Harvard studies

          Close to 90% of spreadsheet documents contain errors, a 2008 analysis of multiple studies suggests. “Spreadsheets, even after careful development, contain errors in 1% or more of all formula cells,” writes Ray Panko , a professor of IT management at the University of Hawaii and an authority on bad spreadsheet practices. “In large spreadsheets with thousands of formulas, there will be dozens of undetected errors.”

          1. Anonymous Coward
            Anonymous Coward

            Re: Excel and finance pros

            Yes spreadsheets have some errors, if users aren't careful. Just the same as code. This should not be news.

  10. Gordon Pryra

    Banks legacy systems

    Its already been mentioned in a few posts in different ways, but the overall bank view is that the risk of replacing old working systems far outweighs the cost benefits.

    The cost if the system goes down is never really counted because the figure is scary high.

    Its a house of cards and the original developers have retired so the people in charge do NOT want to blow on it.

    1. Pascal Monett Silver badge
      Coat

      That's an interesting picture you paint there.

      I see the core systems becoming something of a religious practice in the coming decades. Maybe even with rituals and sacrifices to ensure proper functioning.

      And the sysadmin in a long red robe, holding a dead chicken and chanting in front of the blinkenlichten...

      1. Fatman

        RE: rituals and sacrifices

        Maybe even with rituals and sacrifices to ensure proper functioning.

        And the sysadmin in a long red robe, holding a dead chicken and chanting in front of the blinkenlichten...

        With manglement offered up as the sacrificial virgin!

        1. TopOnePercent Silver badge

          Re: RE: rituals and sacrifices

          "With manglement offered up as the sacrificial virgin!"

          How can management can't be virginal when they f**k everyone. Repeatedly.

  11. Anonymous Coward 101

    The comments in this article are rather interesting:

    http://blogs.telegraph.co.uk/news/iainmartin1/100248741/why-the-rbs-computer-keeps-saying-no/

    1. Anonymous Coward
      Anonymous Coward

      The article itself is complete arse, however. The only real interesting comment is that while the article complains that NatWest's systems were more advanced than RBS', the reason that they merged NatWest into RBS was because RBS used CICS/DB2, whereas NatWest used IBM's old hirachical database.

      Then the comments stray off into conspiracies like "it wasn't a computer glitch, they ran out of money". or "they took everything offline because they were hacked."

      1. Anonymous Coward
        Anonymous Coward

        The Euro....

        I seem to remember that the muti-currency handling ability of the RBS IMAS platform was a considerable factor in it's choice over the NatWest BOLP platform. At the turn of the century the financial world was damn-near-certain the the UK would drop the pound and move to the euro.

      2. QuiteEvilGraham

        Yes, it was arse. Very hard to see how someone could think that using IMS was more advanced than using DB2 in the modern world. Disclaimer - both IMS and DB2 are rock-solid, industrial strength database management systems, very well supported by IBM. Most likely RBS simply went with what they knew best. BTW, if you think it is hard to get experienced CICS/DB2 people these days, try getting good IMS people. It is extremely beloved by American insurance companies though.

        Their problems mainly stem from getting rid of people who knew and understood their core systems IMHO.

        Especially the one written in Assembler which has comments dating back to the late 60's. Good luck getting someone who can hack that for what RBS expect to be the going rate.

  12. Anonymous Coward
    Anonymous Coward

    Underinvestment? I was at RBS for seven odd years, vast amounts of money were spent on IT, new greenfield datacentres with machine rooms the size of football pitches were filled and extended, old mothballed datacentres from other company purchases were re-commissioned. Truly mind-boggling amounts of money were spent on IT. Underinvestment is not the problem RBS has, not until the financial crisis and then the under investment is more in staff than hardware and software. Letting your most senior staff go and re-employing them with far less experienced staff that's the investment problem. This behaviour didn't kick in until after the financial crisis, certainly after Fred went.

    Do RBS have legacy systems? sure. Do they have old obscure hardware, most certainly. Do they have old core codebases? Definitely. Have I been in meetings rowing with project managers about costs of projects, or saying that if we'd done this X years earlier it would be much cheaper/easier? Yes indeedy. The thing is that legacy systems can be run on modern supported hardware, the obscure hardware will be replaced over time, sometimes you just can't replace that obscure serial line encryption device or the old optical jukebox. Some projects come before others, you can't do everything at once. It's also legitimate to run old code, mostly because it's reliable and well understood. There are certainly areas that could have had more investment, but we're not looking at a systemic underinvestment across the board here. The things which have gone wrong are core and are the most up to date systems, what appears to have caused problems is human error. That's where the investment problem is and it's new, not decades old.

    1. Steve Todd

      Hardware and software are different investments

      You can spend huge sums putting new machines in to run old software, and it's the maintainability of the software that's the issue here. The trend has been towards making software LESS efficient, but that as a byproduct of making it easier to document and maintain. There is a limited supply of developers who understand CICS/COBOL still, and little new uptake. There's a big cost and little apparent benefit in migrating to something more modern (say UNIX/C++), but it's something the bank is going to have to do unless it wants to paint its self into a corner when it comes to staff skills.

      1. Anonymous Coward
        Anonymous Coward

        Re: Hardware and software are different investments

        Or maybe, and I'm thinking out of the box here, you understand, just running this up the flagpole to see who salutes, maybe the banks could consider training people to work on their systems ?

        I know, sorry, silly idea.

        1. Steve Todd
          Stop

          Re: Hardware and software are different investments

          Yes, it is a silly idea. Think about it from the point of view of a fresh graduate. Do you go in to a company which will teach you skills that are freely transferable to most other companies out there, or one that will teach you stuff that is pretty much ONLY used by them, and is very old tech. You'll get SOME uptake, but it's not going to attract the best and the brightest.

  13. phil45

    Been there, done that

    You need enough failures for non-technical senior management to understand that they can't continue with the old stuff. Once that point is reached, new development becomes the fire fight, not the cause of it.

    Operationally managing systems which are definitely going to crash and burn is tricky. You have to demonstrate to "senior" management that they're screwed, without screwing them yourself, and without getting screwed by outages. It requires considerable finesse and timing.

    That's assuming the cause is old systems, rather than outsourcing operations to somewhere with cheaper day rates, or straight operational incompetence. As this is a taxpayer owned company, can we use the FOI to pull their incident reports?

    None of this is limited to banks of course.

    1. Fatman
      WTF?

      Re: Been there, done that

      You have to demonstrate to "senior" senile management that they're screwed, without screwing them yourself, and without getting screwed by outages.

      FFS, please get it right???

  14. Bladeforce

    So many posts so much silliness

    Most people posting here regards IT as a cost, such a short sighted view. All it takes is a decent CEO who knows what is going on and take the longer view of IT always pays off in the future. It's only short sighted CEO's that cause the problems and bad management. Sems to be ingrained in business today short sightedness and clogging the magerial chain up with managers that have zero ability to man manage

    1. Anonymous Coward
      Anonymous Coward

      Re: So many posts so much silliness

      I'd say the solution to any company's IT is somewhere between "it's a cost we must keep to a minimum" and "any investment will always pay off".

      I've worked for companies who've made major investments but not known when to stop, costing millions more than was necessary to get the job done. I've also worked for companies who kept knackered old IT infrastructure going with no support because it just about got the job done, but crucially didn't cost. This is the skill of senior management and it's a thankless task. On the one hand you get the business guys really annoyed that they have to spend so much money, on the other hand you get the IT guys bitching about how you don't know your job and are totally incompetent, because they can't see that you can't spend all the company profits on IT.

    2. No, I will not fix your computer

      Re: So many posts so much silliness

      The best analogy is that IT is like a "shop front", you might be able to get away with no glass, or you might decide you want huge realestate, it's a cost which directly affects income, maybe a regularly maintained display to keep people interested, but if it's just a costly inconvienience you don't understand what it is.

    3. despun

      Re: So many posts so much silliness

      Whether it "pays off" on the future or not is irrevelant - it's a cost. To question this is financial illiteracy.

      1. Gartal

        Re: So many posts so much silliness

        It depends on how you define your accounting system as to whether it is a cost. I don't mean the software you use either, I mean the nuts and bolts of accounting.

        1. Getriebe

          Re: So many posts so much silliness

          "It depends on how you define your accounting system as to whether it is a cost. I don't mean the software you use either, I mean the nuts and bolts of accounting"

          Agreed. And more important is how you are writing that cost off and which side of the Balance Sheet it appears.

    4. Fatman

      Re: So many posts so much silliness

      Most people posting here regards IT as a cost, such a short sighted view. All it takes is a decent CEO who knows what is going on and take the longer view of IT always pays off in the future. It's only short sighted CEO's that cause the problems and bad management.

      Well shit, you finally touched on the problem, but not really.

      The problem, at least as I see it, is the obsession of Wall Street types with short term profits; over long term profitability.

      Wall Street types want their fucking money now; they won't be around in 5 years. Carl Icahn is a fine example of that breed. The other part of the problem is the Business Schools that reinforce this activity. "You must have better quarterly performance numbers, regardless of the financial climate."

      Wall Street types == Assholes!

    5. DropBear Silver badge

      Re: So many posts so much silliness

      Not that surprising at all - in all fairness, I consider replacing the timing belt on my car as strictly a cost as well, in spite of the fact that it most definitely wouldn't run without it; it's one of those things that are not conspicuously useful in daily use and it very, very rarely breaks - but if it does, I'll most definitely notice (and regret)...

  15. Anonymous Coward
    Anonymous Coward

    Millenium bug?

    > For example, a retail bank where I once worked had its core standing order system written in pounds, shillings and pence with a translation routine sitting on top of it – yet many of these systems were supposed to have been rewritten as part of the millennium-bug investigations.

    Not so much the millennium bug as Euro compliance. One bank we worked at had a relatively easy time with Euro compliance changes because *every* routine that did anything with money had already been patched through to a common 'override' routine that convert pounds and pence into pounds shillings and pence to keep the even older code happy. Only one routine needed intercepting!

  16. hairydog

    What is a bank?

    There was a time when a bank was a secure, imposing building with staff wearing smart suits.

    Now a bank is a computer system with a banking licence. But the people running banks think their suits matter more. So they spend money on suits and not computers.

    It goes without saying that they don't have a clue. But they do have our money.

    They'd be laughing all the way to the bank if they weren't there already!

    1. Destroy All Monsters Silver badge

      Re: What is a bank?

      Correct, though you need to add "ledger".

  17. Anonymous Coward
    Anonymous Coward

    Underinvestment is a problem in all departments.

    Its not just the software you have to worry about. I know of a factory that has been shut down for over a year. Built in the '80s and next to no preventative maintenance done. It all came to a head after several incidents that, more to luck than judgement, didn't kill double (approaching treble) figures.

    Several pipes containing gas had corroded and leaked. Equipment designed to detect leaks didn't work. Luckily someone noticed the smell on each incident.

    Place has been shut down long term while everything is replaced and upgraded.

    Yes, they made a shed load of money by saving on maintenance, but they have having to spend it all now, plus losses due to downtime. But it could have cost them a whole lot more.

    1. DropBear Silver badge

      Re: Underinvestment is a problem in all departments.

      I bet the management reaction at the time was "but why would we concern ourselves with those out-of-order detectors? It's quite clear the smell can readily be detected anyway?!?". And the sad thing is, I'm joking only halfway...

      1. Anonymous Coward
        Anonymous Coward

        Re: Underinvestment is a problem in all departments.

        I think the only reason they were shut down was because HSE ordered them to. Having heard other stories from that same site, safety was not their main priority.

  18. James Pickett

    " just patching it up and moving on."

    Just like Windows, then.

    1. Anonymous Coward
      Anonymous Coward

      Of course by "Windows" what you really mean is "any modern operating system"

      Or can you name some modern operating systems which don't need to be updated?

      1. Anonymous Coward
        Anonymous Coward

        Re: "modern operating systems which don't need to be updated"

        "Or can you name some modern operating systems which don't need to be updated?"

        Can YOU name an embedded OS ?

        Can YOU name a modern embedded OS? (whatever 'modern' might mean to you)

        Can YOU name a modern embedded OS which *does need* to be updated routinely like Windows still does?

        Thought not.

        (Hint: embedded Linux is just one, perhaps the best known one, of many potential answers. It's in your router, your smart TV, your NAS box, and many other less visible places. When did these things last *have to* be updated in general?)

        1. Anonymous Coward
          Anonymous Coward

          Re: "modern operating systems which don't need to be updated"

          We are talking about banking IT here, I hardly think that RBS are running anything on embedded OSes.

          Anyway, that's besides the point that most modern embedded systems I've used allow for firmware updates and those updates are pretty regularly released.

          Can you name a modern embedded OS which doesn't allow for updates? Even modern pacemakers allow you to update them, my car certainly can be updated. Looking round my sitting room, my Blu Ray player and TV can be updated as can my camera can but I don't think my wireless thermostat can.

        2. Anonymous Coward
          Anonymous Coward

          Re: "modern operating systems which don't need to be updated"

          "Can YOU name a modern embedded OS which *does need* to be updated routinely like Windows still does?"

          Can YOU name a modern embedded OS which is installed on as many devices as windows currently is in the world?

          Make a big enough target and people will aim at it.

          Embedded devices with internet connectivity will not be bomb proof. People are starting to exploit them:

          http://www.theregister.co.uk/2013/11/28/researchers_warn_over_connected_device_malware/

          And there are many modern bits of kit with no available updates as the manufacturer has moved on to newer devices.

        3. Anonymous Coward
          Anonymous Coward

          Re: "modern operating systems which don't need to be updated"

          "Or can you name some modern operating systems which don't need to be updated?"

          Can YOU name an embedded OS ?

          Can YOU name a modern embedded OS? (whatever 'modern' might mean to you)

          The Windows CE handheld computers I support haven't been patched in years.

  19. Stretch

    Banking industry are nothing compared to Insurance for legacy

    having worked for a few years in one of the biggest insurer's IT department I can tell you that the banks don't even come close in the "piles and piles of legacy" stakes

    mmmm CICS/COBOL. Job for life tbh.

  20. Stuart 16

    This isn't just finance, it is a problem in most organisations where accountants, for whom an Excel macro is high art, make or veto technical decisions.

  21. All names Taken
    Joke

    Maybe?

    Just like its punters RBS can now say "Sorry - can't pay you at the moment as my bank is acting up."

    "Can you come back next year?

    What? Tomorrow?

    I don't think so."

  22. SirDigalot

    They don't make them like that anymore

    could one of the issues be the fact that the old stuff was so robustly built and designed to last a lifetime (with the associated cost) and the fact for the most part these industries never really have ground breaking new discoveries be the actual downfall?

    we are always buying new stuff and trying to adapt our infrastructure to maintain parity or comply with new standards, it can be very frustrating to always be trying to change, but at least we do not have the legacy issue as of yet, although I am willing to bet in a few years as company size increases further and there is a larger requirement for profit, we will hit the same trap.

  23. big boaby

    Money problems

    Looks to me like they are running out of money. IT glitch is just an excuse.

    1. Jason 24

      Re: Money problems

      You appear to have commented on the wrong article, I think you should be over here - Telegraph

  24. Daz555

    Legacy systems? Show me the money. Stay current always. Nothing however beats being one of the few guys who knows something about that mission-critical piece of crap no-one dares even to reboot.

  25. Anonymous Coward
    Anonymous Coward

    we knew this would happen

    As someone who used to work in RBS for many years till redundancy a few years back this is no surprise. Everything was moved to india where unfortuantely they had no background on the systems they were taking over. Any knowledge of quirks would have been lost with those leaving the bank.

  26. timhowarduk

    Any chance you could remove "titsup" from the title?! If you did then I could send this very helpful article to my COO and CFO.

    1. Anonymous Coward
      Anonymous Coward

      @timhowarduk

      Summarize it and ask to present the basic premise of the article and provide some costs and benefits for your operation. Go on get involved and make a name for yourself

      If you sent this article to me I would would walk over and ask you to tell me what the relevance is to the business. If you sat there and used geek speak or spluttered, I'd hit you.

      Hey, its my caring and sharing management style.

      1. timhowarduk

        Re: @timhowarduk

        Thanks for reply and advice, though in fact I'm already involved. Too involved!

        In our case we have a system designed in the 80's and written in the 90's. It's getting harder and harder to keep running and we are down to myself and two other guys that understand it (between us). This distributed system handles international money transfers between over 80 countries and currencies and does end-to-end auditing to meet anti-terrorism laws in each country that sends money overseas. If this thing breaks, money doesn't flow, or if it does then no one knows what the money is for, aid projects stop, workers don't get paid, it's generally 'game over'

        It's taken five years of playing the risk management card to finally get work on replacement system(s) started.

        I'm hoping to see something new running before I retire. The article actually hits a few nails on head, and it's still shame that a good article can't be shared in a professional context due to the title.

  27. Anonymous Coward
    Anonymous Coward

    From what I can see banks basically have to do two things :

    1) Provide savings products and a prudent lending service which gives customers and businesses access to finances and the bank makes a tidy profit and not run the bank into a wall requiring a massive government bailout to prevent the collapse of the economy.

    2) Run a reliable transaction system that supports day-to-day personal and business transactions.

    RBS fails on both of those counts so, one would have to ask : Why is it still in banking!?

    This is pretty fundamental stuff!

    Their Ulster Bank subsidiary over here which operates in Northern Ireland and is also one of the largest consumer banks in the Republic of Ireland suffers these IT glitches too as their IT platform was centralised back to RBS HQ some years ago. (I was with them for a long time before that and there was never any issue!)

    When the RBS system went down in 2012, Ulster Bank customers faced nearly 6 weeks of disruption! The RBS and NatWest customers' issues were resolved a lot sooner. It was absolutely unprecedented over here and it really sent a lot of customers running to other banks. I mean can you imagine being without your current account and debit cards for over five weeks?!?!

    I'm not sure why the Ulster Bank situation took longer to resolve than RBS and NatWest, but I would speculate that they've merged they may have merged their IT infrastructure, but still all be running quite different software albeit out sourced to a single centre somewhere. So, I'm guessing the IT unit either didn't know how to work Ulster Bank's legacy software or that it failed differently somehow when the glitch happened.

    Clearly banks operate in some kind of parallel universe where this kind of stuff can happen and they remain in business thanks to government support.

    1. Anonymous Coward
      Anonymous Coward

      It's probably worth noting that Ulster bank uses Euros and GBP, while the back end systems are the same it's highly likely that there are a large amount of extra batches to run on a dual currency bank than there are on the single currency banks.

      1. Anonymous Coward
        Anonymous Coward

        To be honest, as a customer I couldn't care less how their batches operate.

        They left me without money for nearly 6 weeks so they won't be seeing a single cent or penny of my business EVER again.

        The good old Bank of Mattress and Bitcoins are starting to look like an ever more attractive option when you see how wonderful the mainstream banking industry is!

        1. Anonymous Coward
          Anonymous Coward

          if you don't care why the difference, why ask?

          Also, did you go to the branch and talk to them, I was under the impression that they were extending credit to anyone who needed it. Or did you just sit at home and bitch on the Internet?

          Oh, and good luck with Bitcoins and mattresses. One is seriously unstable and the other is prone to fire, flood, burglary...

          1. Anonymous Coward
            Anonymous Coward

            I simply moved all my accounts to an institution with a better record with IT systems as soon as their network became fully available again and all the errors on my accounts had been resolved back in 2012.

            Once bitten twice shy, as they say.

  28. Big_Boomer

    The market will soon start to see that not bringing their infrastructure into the 21st century may kill their business. RIM, Nokia, Sony are all learning it right now. The banks are the most fuddy-duddy conservative of all businesses so they will be the last to get their poo in a pile, but how many of them will have to go under before they realise that doing nothing is no longer an option?

    1. Anonymous Coward
      Anonymous Coward

      "The banks are the most fuddy-duddy conservative of all businesses"

      They are and they should be, look what happens when they stop being conservative and lend unchecked?

      Also, the banks were the first companies to use computing, way before any others, they understand it better than pretty much any other company. Rushing for the new shiny shiny will have your company always in flux and never settled. Slap it all on a z server and keep it running for decades, if it needs a new functionality, add a satellite system, never change what you have got that works, unless you absolutely have to.

  29. sisk Silver badge

    Pretty accurate. It's a problem even at small businesses. The scariest one I've seen is a local jeweler. His customer database won't run on anything newer than Windows 98 and there's no way to export the data anymore except as a report. Exporting the actual database requires a separate application that seems to be extinct. (I've no idea where he got that system, but I can see why the company that made it isn't around anymore.)

  30. Vociferous

    Right, and not right.

    The article doesn't really "get" business. To the IT guy, a 30 year old system which no one really understand, is a disaster waiting to happen, and it would make sense to spend money to fix it before it breaks. To a CEO, a 30 year old system which no one understands, is a technology unlikely to break in the average five years the CEO will be with the company, and spending money to fix it before it breaks will cut in to profit/bonus.

    What people really have to understand is that the management act to maximize their own short-term profit, not the best long-term interests of the company.

    1. sisk Silver badge

      Re: Right, and not right.

      What people really have to understand is that the management act to maximize their own short-term profit, not the best long-term interests of the company.

      No, that would be what bad managers do. Good managers realize that long-term company interests equal long term profits for them to. Sadly we have more bad managers in the world than good ones.

  31. Anonymous Coward
    Unhappy

    Barclays online banking down for a few hours

    Barclays online banking was unavailable last night (was after 00:30 this morning, still down at 04:00), no idea what happened, but it is back now. Nothing on their website to say why. Their website was up, just the online banking unavailable. A page stating that it was down and they were working on it was displayed when clicking on online banking login. I did email the newsdesk at el reg, not heard anything, not seen any articles yet.

  32. Anonymous Coward
    Anonymous Coward

    The biggest impediment to moving forward…

    … is the stuff you can't change.

    I was involved in updating a tracking system used by a mining company to track copper cathodes. It had been running well, but the owners felt the machines were a little old, and so wanted to move to something newer.

    11:09:35 up 1083 days, 22:40, 2 users, load average: 0.00, 0.00, 0.00

    That was the uptime of one of the two servers. Running Ubuntu 8.04. That was just before I typed 'sudo poweroff'.

    We were going to move the platform to Ubuntu 12.04. They wanted to move it to their VMWare infrastructure, so I set up a VM which would become their production system. So I set up a dev environment, compiled all our stuff, then found one module which needed the Oracle libraries. No worries, I try installing them.

    No chance, it needed an older version of libstdc++. So I go looking on Oracle's site for a newer release, find one, install it, then get everything built.

    Come commissioning day, we discover that version, the oldest one I can get which builds, won't talk to their Oracle server because the Oracle database is too old. I wound up getting them to download the Gentoo Install CD, booting the VM up off that, then I shelled in and did a P2V of one of their existing boxes, so that at least the hardware was looked after. We'll tackle the software another day.

  33. scrubber

    Banks?

    You ceased to be banks years ago, you are now IT centres. The sooner you realise this and invest appropriately, and promote and listen to IT people, the sooner you will overtake your competitors and enhance profitability.

  34. Anonymous Coward
    Anonymous Coward

    offshoring is the problem

    The cause is probably the same reason as last time and most major cockups i have been involved in the past 5 years. UK legacy programmers with decades of experience replaced in their hundreds (not retired, not retrained) by cheaper inexperienced Indian staff.

    My own personal experience is of 75% of the UK IT workforce (most of whom developed the system 15-20 years ago and were still supporting it) given 6 months, before they were made redundant , to train their Indian replacements who had left university and just finished a 2 week mainframe course. Most of the workers then had to find new careers with a very small percentage getting a new IT job (it took me 3 years to find another role in the UK with 200 people applying for every mainframe job). multiple that across banking, retail, insurance and utility companies and you have very small pool of mainframe people still working in this country. If these experience people were still looking after these systems then the problems would not have happened. these companies are moving too fast, too quick and put no value in experience.

  35. Anonymous Coward
    Anonymous Coward

    All computer programming code becomes 'legacy' the moment it hits the tin on a production system

    That applies to everything be it operating systems such as Z/OS, UNIX and Windows, databases such as DB2, Oracle and SQL Server; third party software packages such as SAP or in house bespoke applications.

    All will contain routines by people who resigned, retired or passed on to that great programming pool in the sky some time ago.

    The question is not the age of the code but how easy is it to support its continued execution without risking systemic failure .

    With regard to banking the basic computer algorithms of its core accounting ledgers have existed since at least the 1970s when these processes first started to be automated ( with the underlying rules having been applied clerically on paper systems for centuries before). I find it difficult to believe that a basic business process that seemed to run without to much problems in the days when programs were loaded from punch cards and data was keyed in on paper batches has now mysteriously become impossible to execute reliably 40 years later. I am afraid I do not believe that the 'legacy' technology is at fault since it should be perfectly possible to design, code, test and run this stuff on nearly any current major platform. However, that does depend on having staff sufficiently well trained, paid and motivated to do the job. On suspects it is this element that is ultimately behind these problems is human and organizational failure. And that is the fault of the people who are paid to manage the banks.

  36. sodium-light

    RBS has been under a concerted short attack for a few weeks now. Blatantly obvious media campaign and now the soft IT underbelly.

    Someone wants stock. It's a billion pound game so worth a lot of effort.

  37. Anonymous Coward
    Anonymous Coward

    Not just restricted to retail banks.

    I used to work for Visa Europe up until a couple of years ago and they're still running BASE I and BASE II which were designed in the 1970s. Yes they work and yes they rarely break but the last person in Europe who had intricate knowledge of BASE I retired in 2009-ish. To get some kind of advanced understanding you'd have to plough through ancient operating manuals as training courses for these clearing and settlement systems simply do not exist.

    And yet today, despite VE investing millions in its own clearing and settlement systems, BASE I and BASE II still somehow soldier on. And in the rare event when they break down...all hell breaks loose. I believe this is because Visa America refuse to dump the system despite sustained pressure from Europe and other major banks in the world who depend on Visa for their transaction services (and thus their reputations).

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019