This comment: "Furse should not resign, she should be sacked.." would come better from someone who is not interested in replacing senior executives at high cost, such as "Dominic Connor,...a headhunter."

Perhaps when he was "Dominic Connor, developing trading systems", he would have recopmmended that Furse stay, but employ someone different to develop the trading system?

Paris Icon for obvious reasons...

This article irritated me greatly. Now I can't comment on what happens at the very top level of such a large business, but as someone who has recently jumped ship from being a "geek like you" to a service continuity manager, I have to take issue, and especially with this bit in particular:

"If the DR site was working, why didn’t it take over? Can the LSE put paid to the rumour that they were running exactly the same software for both live and standby? If you are Clara Furse reading this, here’s a hint, two copies of the same software will probably crash at the same time, given the same inputs. That’s why grownups use multiple versions.”

Did Accenture tell you that? Did it sound like a luxury to the media beancounters you appointed? What a load of rubbish. Are you seriously suggesting that the reason you have a DR system, is merely in case the software crashes? And that the way to recover from a software crash is to have some "different" software on your DR system? If so, how different? A newer version? How about an older version? Should my production systems use Oracle on unix, and the DR system DB2 on mainframe?

No, the DR system should be identical to your production system in terms of it's design and functional behaviour wherever possible. OK, depending on the size of your business and the importance of the system in question you may have have lower capacity, or other limitations as part of your design, but these differences should be carefully considered, and may be chosen for reasons of cost, complexity or any number of other reasons, but the nutshell is that what you choose to make available in a DR situation has to mirror the functionality, if not the specification. You can't make arbritary changes to second-guess potential future problems.

To deliberately choose to run a different system in case there is some sort of bug, or error that could cause the design in question to fail is both impossible to do effectively, and not a function of service continuity, but of system design. This is important!

I would like to see what happens when there is a problem in terms of there being a real "disaster" (fire/flood/power cut/sysadmin-gone-mental), whereby the failover to DR systems would have worked, only didn't because the software was different. Your comment "but it’s obvious from this event that if the pathetically vulnerable St. Paul’s site is taken out, we can have no confidence in when the market will be back on line." misses the point that it's entirely possible that the DR site could have worked perfectly, if the cause of the fault had actually been the primary site being "taken out" rather than suffering from potentially poor design, or unexpected "input" (whatever you mean by that).

I note that there hasn't been much press coverage in sufficient details for me to understand what actually went wrong, but I know that if it were my systems, I would not want you to be working to fix it, not only do you misunderstand the concept of what service continuity is and how to effect it, but your "holier-than-though" attitude would surely waste peoples time and misdirect their efforts. You may understand when you become one of us "grownups", stop publishing these childish rants and behave like a (fully) responsible journalist...

While I do agree that the senior execs of the exchange should be held accountable, there is no reason for this type of failure to happen.

Depending on how much money you want to spend in creating redundant hardware, you can achieve the 6 9's of uptime.

Using IBM's IDS as a database in a configuration of local failover and then remote failover will give you the protection you will need.

Its unfortunate that IBM has the database technology, the hardware technology, yet not the marketing sense to go to the exchanges and pitch the idea.

Of course even if they were, the cost would be high and it would have to be justified against the risk. (Redundancy upon redundancy upon redundancy will give you a fairly high level of uptime, with planned down time allowed.

The LSE lives and dies as a network service. If the boss cannot understand that then it is time to go.

Spot on. The decision not to invest enough in IT had a business impact. So why not the front page of the FT asking her head but instead El Reg on a weekend?

PS. anyway the board should be removed for the disastrous merger with the most useless stock exchange of the planet, the Milan Stock Exchange, hardly a world-class destination with blue-chip public companies.

Skull because "off with her head"

That brought us endowment mortgage shortfalls, Northern Rock and the credit crunch, foreign-owned essential utilities, and the like... managers get the bonuses, customers carry the risk. Hmmm, trebles all round, unless your savings/pension just went down the tubes.

That aside, wasn't there an OS missing from the author's list of OSes of choice for reliable disaster tolerant systems, potentially with split site capability? Possibly even two OSes? Still, clueless headhunters are nothing new are they. Anyway, more by luck than by design, HP own the two relevant OSes these days, but once upon a time the world knew a little bit more about VMS and whatever the Tandem Nonstop OS is officially called, and indeed until relatively recently those two were the foundation for most of the world's most succesful stock exchanges. Then the LSE decided that a desktop-heritage OS was good enough for them, and obviously purely by coincidence, here we are today.

That being said, HP do have an interesting and relatively recent disaster tolerance video: "We demonstrated that IT services continued to be available for all of our operating system environments— HP-UX, Microsoft® Windows® Server 2003, Red Hat Enterprise Linux, NonStop OS, and OpenVMS." Have a look if you have a few minutes, and if you're associated with Ms Furse, send her this link:

http://www.hp.com/go/DisasterProof

Wrt dissimilar redundancy: don't diss it too much, it's got a few projects done which wouldn't have got approved without it. Stuff that flies, or goes bang, or must never go bang (nuclear power bang), springs to mind. It's not often seen in traditional business IT, but that's largely because dissimilar redundancy done right is *expensive*, and traditional IT beancounters usually prefer "cheap" to "done right".

I have a slightly different view via the safety embedded world, concerning some of the comments about why not having independent hardware and software redundancy/system integrity designed in:

You don't tend to get too many planes falling out of the sky due to software failure, and no redundant/backup capability.

Then of course I guess the LSE (and a few others whose entire business is dependent on the computers/networks working) don't have effectively three levels of hardware( different silicon/routing) running concurrently using three different software implementations developed by seperate teams working in isolation (only the system spec is common).

Is this expensive? Sure. But the news headlines are likely to be far worse if an Airbus decides to fall out of the sky (plus they'll stop selling planes and go bust if it happens often enough) than if some poor inside trader can't ditch his XL Group shares.

Maybe decent system integrity is too expensive for finance types.

Maybe the potential losses to the LSE through a system failure haven't seemed to be large enough for them to actively avoid the lack of system redundancy issue.

And maybe the odd sacking of a CEO of an information driven business, who doesn't have Tech Director/CIO on the board, to encourage the others, isn't a bad idea for the general health of GB plc.

Completely agree - the concept of deliberately running a different version of the codebase on the DR system is bizarre. A bug is fixed, a new version released - if I understand this article correctly the author would recommend running a version /known to be buggy/ on the DR system? I'm aware heterogeneity is broadly a good thing, but not in the middle of a disaster thanks very much. I want an identical handover, no surprises or gotchas.

Anyway when was the last time software caused something like this? Hardware, that's where it always goes wrong. OK, except for the metric/imperial mixup on the Mars Orbiter. But generally it's hardware...

*systems may go down as well as up.

The reason you have different versions is so when the DR comes up it will be fed the same scenario as the live site, and so you take down the recovery site as well.

Cisco systems are a good example of this, sometimes after a failure they get stuck in a loop electing a new leader, then automatically back off to avoid creating the loop and downtime grows each cycle.

I think you have confused service with systems.

"To deliberately choose to run a different system in case there is some sort of bug, or error that could cause the design in question to fail is both impossible to do effectively, and not a function of service continuity, but of system design. This is important!"

For example, Juniper or Cisco can operate with what the LSE are doing and it gives me great comfort to have the two systems interleaving and bypassing each other. On failure of one the other can still take over all the way to Linx for example or can be repaired while keeping the service running. The system is what lets people work, the service is what they provide to others.

Having worked on a project that is very close to the LSE's new trading platform I would have to agree with most of Dominic's views.

Accenture have to ability to convince senior executives that they know what they are doing because of their branding this forcing unnecessary hires for testing, support and development staff with little or no experience. Moreover, Accenture insisted on using .Net to develop most of the feed handlers. This made it easier to find developers but also had the effect of reducing performance and also blighting the order matching system with .Net garbage collection issues. Yes I know the system is supposed to be horizontally scalable but what’s the point if you can’t scale up since your non-horizontally scalable parts won’t for huge cost. Fortunately the LSE has been severing ties with Accenture over the last 2 or 3 years although this has just meant buying back staff from Accenture they had previously sold to them.

I must admit I never met Ms Furse during my stint at the LSE. Nevertheless, it seems a touch arrogant for her to appear to ignore competition from the likes of Turqoise and Chi-X. Looking at the share price now, perhaps a takeover back in 2005/6 might not have been a bad idea?

One slight error Dominic is that the LSE’s primary site is not at St Paul’s. The two main data centres are elsewhere with the St Paul’s acting as part of a Quorum.

The DR should be held back in line with a working version of production. When the production is updated and has settled, then the DR should be patched. In that scenario the DR is useful as a failover in ANY circumstance, not just "act of god" type problems. Having a business critical system down for more than a few hours? No work being done? Seems to quailify as a disaster to me! You pay for DR, use it! Ideally use DR as an offsite storage and offline reporting and processing setup, then you know it works when you need to fire it up in anger.

Sounds like the DR was not up to spec and I can imagine the cry went up much like places I have worked in, "Sorry but DR won't take the load and and you never allocated any money or time to get it up to spec, it was all a token effort to please the audtiors. I would concentrate on getting production back up ASAP!"

I am not entirely sure of the full spread of kit used at LSE but from the discussion I have had with colleagues we are under the impression that it was a showcase setup for a single platform technology. No one in their right mind uses one platform for business critical systems, the trendy term "best of breed" is investigated and you try enusre that you get a useful spread and mix of tech that plays well together. If something goes belly up, you only have to look in one place, not all over the bloody place to try to find the glitch. You never run all the same make of router at your gateways, if one goes down or get compromised, you know the other will keep you going to buy you time to find out what to fix. Run IIS to Oracle on Sun, or Apache on Sun to DB2 on IBM, whatever it takes to buy time if one goes out, especially when O/S patching, on ANY platform, is involved!

I've seen so many things messed up because the people who make the IT decisions are not qualified to make them. I could rant on about how reliable IBM's VM/MVS/VSE systems are, or how the NS here has tried to move off VMS for many years, spent milllions on outsourcing it (a bankrupt concept) got nothing for it and have decided they need to do it in-house etc. etc. etc. but it'd just be a rant so I'm off to write something trendy... what's next... oh drat - a Delphi 3 program. Sigh.

PS El-Reg web-site designer: I can only see 25 chars in the title box - it scrolls even though there's lots of white-space on the rhs. And insert std. rant about the font-size and fixed-width here - we really should have an icon for it.

Lods a bull ! It is not networks or Cisco stupid head hunter. You must have been a bad techie so had to move on to be a head hunter.

My DR site had a 12 hour outage a few weeks ago, IBM's Sampson house switched to generators, due to local leccy issues. Then the gennys overheated shut down so no n+1. Then they coudln't handle the load so the load was dropped.

End result whole building down for 12 hours. They should be sacked along with "Furse"

Lots of senior execs don't understand that they rely totaly on IT now.

I work for an Insurance company - and we are seriously creaking at the seams. Low morale and lack of balls by the IT Director (and understanding) means the business just doesnt understand what will happen when the systems fail. Who gets blamed, the poor person running that system day in day out that's been screaming for investment!

This Icon looks like a pick-pocket....

She should apply for a job with IR.

From IR's wiki:

"Significantly, several IT initiatives are being phased in to better handle ticketing, freight, rolling stock (wagons), terminals, and rail traffic, including the use of Global Positioning System (GPS) and Microsoft (MS) Windows Vista for train tracking in real time."

all aboard the 6.45 "meatgrinder express"!

I cannot agree with Dan Wilkinson more, this article is geek sensationalism, wrote by someone with a poor understanding of this issues present.

hang on.. You aren't seriously suggesting that you would use a different system at your DR site are you?? You HAVE to use the SAME systems at your DR site to provide continuity!!

Now, if you were saying that testing of new software needs to be done off the live systems, then yes you are right (naturally) but have written it extremely poorly. Testing of new software should be done off network (if you have the means) and at the very least on a test system.

Do not, under ANY circumstances, try to say that you should be using differing systems between your live and DR site...

Am baffled by this, but can only say that you have (hopefully!!!) mis-written what you mean.

It means data corruption from stupid programmers, infrastructure lockout from bugs in firmware, negotiation problems or expired licenses. Plus you never have the tools to conclusively prove what the problem is nor the leverage to force the vendors to investigate (They're like plumbers - "sorry mate, not our problem").

And there is rarely a "smoking gun" for the problem - could be OS, could be database, could be application. Had a 3 DAY outage on SAP caused by a bug in the update software, we had to hire our old consultants back as SAP refused to acknowledge it was their problem. (It was)

Even though I cannot agree with most of what the author says, (I do agree with Dan Wilkinson), there is an underlying truth about the author's rant.

People making big decisions don't have the foggiest more often than not, and having someone who's not tech savvy making decisions is bound for disaster.

About Accenture I can say that if you deal with them ask for people from their group who belong to Avanade or similar and you'll get what you pay for, (probably even more).

Remember:

No matter what any vendor will tell you, it is you who are responsible, people in charge should always know what they are dealing with.

Now where is Paris Hilton when you need her? She probably could run the place better than Furse. :P

I used to work for a City Institution which shall remain nameless. The IT department used to be a relatively small percentage (20%) of the workforce, but grew to be something like 50%. Every single member of the board, and every member of the Executive Committee, was a banker or financial expert. Not one knew the first thing about IT.

The CEO nearly had puppies at a company meeting in which one perspicacious employee described the company as a "software house", despite the fact that they were utterly reliant on IT and spent most of their time writing software, all without any hint of good archtiectural practice. The consequence was that they allegedly wasted a sum not unadjacent to £100M, mostly on consultants, in creating a new set of strategic systems that would never fly (not least because security was apparently viewed as an add-on rather than a deeply ingrained part of the system).

And the arrogance of the City rolls on, undiminished ...

If there was such a lack of issue at the outage, why are the executives paid a lot? Only people with important jobs get paid lots and if your system can be missed for a day and nothing untoward happen, why not save a few million each year and pay this bunch of monkeys peanuts?

I was once involved in troubleshooting a network outage at the LSE when I worked for a company that supported them. The "consultant" at LSE had called us to say that the network was in meltdown. One of my junior colleagues had taken the call and the LSE chap was screaming so loudly that i could hear him through the phone earpiece even though I was about 5 meters away. I took the phone from my colleague and introduced myself as an senior engineer and asked what was the matter. He said that the whole network was down and kept screaming "when is an engineer going to get here". I told him that an engineer was on his way but the fellow just kept on panicing and asking for the ETA of the engineer.

I then told him "what you need to do is calm down and then take a walk around the building inspecting all of the key components of the network". He agreed to do so so he hung up the call and 10 minutes later my colleague got a call to say we should cancel the engineer visit as he had "fixed the problem". He did not leave any explanation as to what the problem was, he just stated that he had managed to fix the problem. I could not rest until I knew what had been done to fix the issue so I called him to find out what he had done. He said that he had walked into a comms room and found a network device that was continuously rebooting. He switched it off and everything started working. He took the credit for fixing the problem and did not say thank you for our assistance. I am therefore not surprised to hear about outages on their network if they hire people like this.

Just one more example of the total lack of awareness of governance and operational risk management in business. And financial services seems to lead the pack in spite of all of the regulatory activity directed at it. The Peter Principle is alive and well at the C*O and Board levels . . .

Anyone who has worked on large-scale IT projects in the public or commercial sectors will have seen this time and time again. Senior management largely drawn from the ranks of marketing, sales and accountancy. If you're lucky (very lucky!) some of them might know enough and be honest enough to realise that they need advice and expertise from the technical staff, but that's not particularly common.

Of course, the IT industry itself is partly to blame. Almost all major projects are ridiculously oversold on a slippery mixture of snake oil and bullshit. That's why massive schedule and cost overruns are the norm rather than the exception. Unfortunately, as long as there's even one major supplier out there who will promise the world at half price by next Wednesday, everyone has to play the same game. So you end up with a bunch of salesmen and accountants having the wool pulled firmly over their eyes by another bunch of salesmen and accountants while the poor buggers who actually have to design and implement their badly-specified and insane pipe-dreams look on in a mixture of despair, resignation and mute fury.

Been there, seen too much of it. That's why I left "big IT" a couple of years ago - it finally reached the stage where I couldn't ignore my moral and ethical misgivings about the whole thing. Now I just wait for the day when something goes sufficiently wrong somewhere that someone big finally says they've had enough and sues one or more of their suppliers into near oblivion.

OK, so that isn't particularly likely, but it's going to need something on that scale to make the IT business grow up and get its collective act together.

(P.S. Nice to see that comments are working now. I originally tried to post this one sometime on Sunday, but in spite of all the above waffle, the comment system still insisted that I had only submitted a title and no actual comment text. Oops! On an article about IT cockups too...)