Independent inquiry into British air-traffic-control IT nightmare The UK Civil Aviation Authority (CAA) and the National Air Traffic Services (NATS) have today opened a joint independent enquiry into the IT cock-up which grounded hundreds of flights last week. On Friday London airspace was effectively shut down due to an IT failure at NATS. Sources told El Reg this was due to an IBM S/390 …
"there should have been a parallel system to failover to"
Parallelism doesn't help if it's based on identical hardware running identical software and the failure is the result of a software error. (Unless you have an independent system produced by an entirely segregated group of developers - some safety critical system work this way, but then you may need a third system to vote on which of the two disagreeing systems has actually failed.)
You clearly haven't read the leader in The Times.
http://www.thetimes.co.uk/tto/opinion/article4297212.ece?shareToken=8668b2ee1ce634234120e2f75d0eb424
From NATS mission critical, real time systems to GDS in one easy step...... I really do wish people had some sort of a clue before commenting, especially in a broadsheet.
I have just skimmed that article - all I can say of its author Matt Ridley is that he is a clueless c**t.
How on earth can you conflate what seems to be a "standard" but serious production issue to a problem with the development methodology? That somehow fixable by adopting the medialuvvie approach of those lunatics who do development at the cabinet office.
He has now raised the bar for idiotic technology reporting waaayyy above Rory Clueless-Jobsworth on the Beeb - something I hitherto thought impossible.
"Matt Ridley [...] is a clueless c**t."
You might want to see if you can find his Times article from a couple of weeks ago on how pilotless aircraft are going to be safer than the ones with pilots which we currently have.
The resulting Interweb demolition of his 'logic' by people with a clue is quite interesting.
I particularly liked this one:
"It's a lot more rational to remove bankers from the financial system than consider airline pilots the biggest danger to the public, and Ridley should check his opinions for reality as he so often advises others to do."
I suspect Matt (Viscount) Ridley had written the article before the NATS problem, but found it a convenient peg to introduce the subject, even though all Reg readers would know it's actually irrelevant.
The main point of the article is that government ought to stop (and perhaps is actually stopping) trying to deliver giant multi-year, multi-billion pound IT projects and instead adopt agile development methods, a sentiment I suspect many on here would agree with. As an article on a technical subject, written by a non-technician for a non-technical audience, I don't think it's too bad (if you delete the first couple of sentences mentioning NATS - agile development is not necessarily the best approach for safety critical systems).
Whether the GDS, civil service mandarins and (more importantly) politicians would be capable of successfully participating in agile development is another question entirely (A hae ma doots).
No apologist for Matt - I was one of the many criticising his earlier article about replacing pilots by automated systems (I use the same handle on The Times).
It doesn't just apply to safety critical systems.
It's not OK to agilely develop the benefits system and see if it works or not and potentially have people unable to pay to eat, clothe themselves and keep a roof over their head.
It's not OK to agilely develop the tax system and see if it works or not and potentially some paying too much, some not at all and the country unable to service itself.
It's not OK to agilely develop a patient facing system and have people's heathcare potentially jeopardised in 'user research'.
Yes. Agile can be great. I've worked on XP and Scrum projects myself. And GDS can do that beautifully on the three millionth reskin of Gov.UK.
But to say that we should use it for much of what Government (and public sector) IT does is madness, IMO.
Now, if you believe this should be in-sourced, then that's a different debate and one which is definitely worth having. And even if it should be outsourced, you can argue that the contractual model is wrong. But, for me, agile in much of this is plain wrong.
>It doesn't just apply to safety critical systems.
Yes it does, you just have to test them. Remember that test-driven-development part of agile?
>It's not OK to agilely develop the benefits system
Yes it is. Have the system do one simple part, roll it out, check it works, then move more and more functionality to the new system until you can turn off the old one. The existing model consists of gather all the requirements, spend 10years and umpteen billion developing a system, discover you have missed a case where somebody with more than one married partner changes sex and name in the same financial period - abandon entire system and start again.
>It's not OK to agilely develop the tax system
Again develop a a system that allows looking up the status of your current return, then adding returns for simple cases, then add more and more complex cases, If necessary. it might make sense to have some very weird cases done entirely manually or kept on a small part of the legacy system. That's the point of agile - have something working all the time.
Yes it is. Have the system do one simple part, roll it out, check it works, then move more and more functionality to the new system until you can turn off the old one.
Unfortunately it is never as simple as this. No politician agrees to spend millions re-designing the benefits system just to proudly announce that it now runs in Java hosted on AWS rather than COBOL on ICL mainframes. So changes are only funded when accompanied by benefits rules changes. This is not a bad thing in itself, but it does mean that something 'simple' and suitable for agile in the new system may not be in the old.
Meaning that I could agiley develop a new payments for mothers, for example, but somehow I need to disable the previous parents payments, or whatever they are called, from the previous systems, otherwise mothers will be paid twice. And you can't just stop the payment going into a bank account - letters have to be adjusted, line items on statements removed etc so that you don't get loads of people phoning up asking why they've only received one out of two payments.
And what if the new calculation needs to be used later on by the old system? Somehow it needs to get 'injected' into the calculation stream as if it had happened under the old approach. And so on.
In short, under the waterfall approach you spend a lot of time and effort developing the system up-front but decommissioning the old one is relatively painless. Under agile, developing probably costs less, but transitioning / decommissioning will be more expensive.
You pays your money and takes your choice. :-(
"It's not OK to agilely develop the benefits system and see if it works or not and potentially have people unable to pay to eat, clothe themselves and keep a roof over their head."
By your own definition such a system *would* be safety critical. The problem isn't the development methodology the problem is decision to deploy and the testing methodology. That and the fact that government contractors much like MPs can't - generally - tell their arse from their elbow.
You clearly haven't read the leader in The Times.
Thank-you for posting that link, I learned something very valuable. Namely that Mike Bracken is another (along with Liam Maxwell) person with insufficient IT skills and experience for the position of authority occupied, and needs to be avoided at all costs.
Although at least he doesn't directly suggest that safety-critical software be 'evolved' as the truly moronic journalist does:
start small, fail fast, get feedback from users early
Yes Matt Ridley, good idea, is that small planes or small numbers of people that you want to see crashing and dying during the "fail fast" phase?
@BearishTendancies - thanks for the link, it's quite mind-boggling:
"GDS is not just trying to make government services online as easy as shopping at Amazon"
So we can look forwards to a third-party software glitch to set all our tax bills to 1p?
What's the air-traffic control equivalent of setting prices to 1p - altitude to 1 foot perhaps?
Ok, so there should have been a parallel system to failover to.
There is a parallel system, and NATS failed over to it successfully when the initial problem occurred.
There wouldn't have been a publicly-visible incident at all, except for a second failure, this time restoring the network link between the two systems, and an aggravating factor, which is that once NATS downtime reaches a small critical threshold (I think the earlier Reg article said it's about 8 minutes), the overhead of delayed and diverted flights cascades into a major air-traffic interruption.
You can of course argue that there should have been a parallel network link. Or, perhaps better, two backup systems and redundant links among all of them.
There are three problems with that line of reasoning:
- Empirical research shows partitions will still continue to happen, and applications will still have downtime. There's a good article in the September issue of CACM on this topic.
- More components means more complexity and more opportunity for harder-to-diagnose failures.
- Greater costs, in physical assets and staffing.
The underlying issues are 1) air travel is oversubscribed - too many planes in the air, too many flights scheduled, too many people traveling for the system to have much margin for accommodating transient errors. And 2) complex systems are complex. They're expensive to change, particularly when you want the result to be better, by various metrics, than the one it replaces.
There's no simple fix. "They should have a parallel system" is just pointless hand-waving. (On the other hand, Vince Cable is clearly an ignorant jackass, since his diagnosis is orders of magnitude less accurate and useful.)
"too many planes in the air, too many flights scheduled, too many people traveling for the system to have much margin for accommodating transient errors."
Yes.
Also applies to managed traffic in city centres, and (increasingly in the UK) on motorways.
The introduction of computerised traffic management allows peak traffic flows to be increased a little bit, and/or journey times decreased a little bit, when things are working well. At off peak times the management is normally irrelevant.
But when the management fails, or some other unexpected event happens, the ensuing chaos tends to be worse than it would have been *without* the use of traffic management, in part because the working margins have been reduced beyond sensible levels.
"More components means more complexity and more opportunity for harder-to-diagnose failures."
A properly designed set of fault tolerant hardware and software will be more complex and have more components than a non-FT equivalent. [Same for high availability, but for now let's focus on FT]
However, if done properly, an FT system design will have the result that a component failure does not result in a system failure. The increase in complexity and number of components allows the system to accomodate hardware component failures, and thus *decreases* the overall system failure rate,
If done properly, the extra hardware and software will also include capabilities including immediate reporting of suspect field replaceable units, and capabilities for hot-swapping the failed FRU without needing system downtime.
On the other hand, doing that kind of thing right takes time and money and doesn't come cheap (ask any Tandem customer).
Software code doesn't deteriorate with age. The hardware has been updated regularly and presumably uses current technology albeit within a mature architecture. And hardware, AFAIK, wasn't relevant anyway.
Maturity is a way of reducing failure. Its been tested and tested in real life conditions, all previous observed errors should be fixed. Re-writing that code is going to re-introduce many more. As a plane passenger (who spent rather longer at Athens Airport than planned returning last Friday) the only reason to dump that code (rather than fix it) is to provide new significant functionality.
Thank God isn't as lame as politicians commenting on an unfortunate but hardly catastrophic event. Delays? Don't they ever use the M25? And as for safety. Killed in and by planes that same day: zero. Killed in and by motor vehicles, rather more than zero. Sort out the real safety problems please.
"Thank God isn't as lame as politicians commenting on an unfortunate but hardly catastrophic event. Delays? Don't they ever use the M25?"
The senior ones rarely travel anywhere other than in a convoy of blue flashing lights, using bus lanes and hard shoulders if the hoi polloi can't or won't get out of their way. Delays are things for little people. Remember when the berk Cameron when on holiday and forgot his passport? Luckily more blue flashing lights were deployed to ensure the git was able to go on holiday. I'll wager they won't be doing that for you or me.
All this talk "ancient systems" and "skimping" - all sounds like he wants to push for a nice new system. No doubt the usual "government IT" project suspects are rubbing their hands with glee at mega-££££ quotes in the off'ing. I'd have expected such talk from an old school Tory looking for a directorship, but not so much from a senile LibDem'er looking for the nearest retirement home.
The phrase "Air Traffic Control" illustrates the control freak mentality behind the whole current setup. What we need are just two relatively simple functions: a collision avoidance function, and a priority function for take-off and landing. Let all the other details sort themselves out automatically.
With that approach, NATS could indeed be replaced by an iphone on every plane.
An extra two hours sitting on a plane at Heathrow waiting for permission to push back. <sigh>.
Luckily, although there were a number of small people on board, there wasn't much disruption.
And it was a good job we all switched our phones on as ATC weren't telling the pilot anything apart from airspace closed.
I'm not happy about it but I've had worse delays due to mechanical failure, light dustings of snow and strikes.
And I agree. It's old tech but it's mature tech. No one has built, from scratch, a system to do what the current systems do with the volumes they handle.
And
No one died although a lot of people were p***ed off. including myself.
Obligatory knee-jerk reaction from people who know sod all about the subject but think they need to be seen to be on the side of the poor old travelling public. Politicians should obliged by law to keep their mouths shut until they've had it explained to them with pictures and short words exactly what happened and why they shouldn't overreact.
There may well be areas where more money could be usefully spent, but it'll never stop something like this happening; all systems are fallible. It's how it's handled that matters, and in this case nobody died. The disruption (or length of it) could have been reduced if the major airports weren't running so close to full capacity, but that's a different argument for the politicians to get excited about / duck depending on where their constituencies are.
Well, there's probably more processing power in the avereage, modern smartphone.
Merely being in service so long is no reasons to trust the programme. In fact, if any monotoring has been done, its deteriorating performance and increasing risk with growth in usage and changes in remote, peer software should have raised the alarm long before such a failure wsa likely.
With software, you never know if it is broken. All you do know is, you have not encountered the problemantic code path or conditions yet. Assuming that longevity equates to reliability is otherwise known as complacency.
However, lots of large firms seem to follow this philosophy as it is cheaper in the short term, whether software, hardware or processes. So we shall see more such cases where the root cause is managerial incompetence, putting big profit first and poor IT work.
Take your modern smartphone and put it where the US torture report dwells on.
also: IBM System z
Merely being in service so long is no reasons to trust the programme.
Could it be that the 5-minute generation has ARRIVED?
"Well, there's probably more processing power in the avereage[sic], modern smartphone."
Impresively off the mark. I will grant you that an actual S/390 platform is nowhere near as capable as a zSeries platform of any vintage (almost like running Windows on a 80286 these days). If this is true, the PHBs need to explain why that is what is being used. But you served up a massive failure concluding a smartphone is more capable than a S/390. If we compare the 80286 to an entry-level S/390, it is immediately obvious the 80286 is rudimentary. Comparing the 80386 is more sporting but it's still not even close.
Why are so-called IT people so humourless. It was a humorous comment. However, had you worked for any significant time, e.g. a couple of decades, in IT, you would be aware that a smartphone is phenomenal. I ran a departmental computing system supporting 250 local users and many offsite users (X25, those were the days), doing mathematical analysis, management, data storage on 1.5 megaBYTES of memory (later upgraded all the way to 2.5) and 350 MB of disc, later up to 700 MB for a few years on what was considered a pretty modern and effective system. My iPhone is so far ahead of that that it was unimaginable at the time and, though mainframes were powerful, few had more resources. Of course, they were not driving pretty pictures. But they were probably programmed more efficiently.
I notice lots of down votes. So I am reassured that my comments about bad management and IT struck home. Hanging on to ancient systems as the environment and load change is not some magic guarantee that you found all the bugs. As you probably stopped testing aeons ago, you are certain to get bitten, as in this case.
Down-voters, what makes you so sure that old software is bug free even as load and usage vary? What makes you so sure that age means that every code path and algorithm have been tested?
I am not saying that age alone is a good reason to change. But I am saying that it is no guarantee of perfection.
Dunno which clueless numpty downvoted you but your recollections are pretty much on the ball, from May this year.
http://spectrum.ieee.org/riskfactor/computing/it/report-claims-u2-to-blame-for-us-west-coast-airtraffic-outage
It even made the Daily Mail, twice. Linking to the DM would deserve a downvote, so instead here's one of the articles based on Associated Press coverage:
http://www.csmonitor.com/USA/Latest-News-Wires/2014/0506/Why-U-2-spy-plane-shut-down-LAX-air-traffic-control-system
This post has been deleted by its author
It's just as well that Prestwick looks after OCEANIC traffic, otherwise a sizable chunk of world air transportation would have been compromised.
Twenty-five year old software, overloaded computers and an American contractor with a history, little wonder the system collapsed.
Better to use AMSterdam or PARis then take the train.
It seems that a faulty flight plan caused the FP server to fall over.
Looks like poor error checking of input data. But it takes a good programmer/designer to produce good error checking: he or she has to scope out a range of error cases surrounding what is valid input, otherwise bad data just crashes the program. Which is what may have happened here.
That means hiring a good designer/programmer rather than the cheapest contractor.
It's not difficult for a competent developer to pick up another programming language, provided it's not one of the evil (aka "esoteric", though the common meaning of that adjective fits JOVIAL) ones, like INTERCAL or Unlambda.
I've never actually written an ALGOL or JOVIAL program, but I've read plenty of ALGOL code (including some discussions of the man or boy test) and it's not a difficult language. The syntax is relatively expressive and should seem comfortable to anyone with decent experience in the large and prominent family of ALGOL descendents (Pascal, C, etc).
JOVIAL permits inline assembly, but S/390 assembly isn't difficult either. It's a pretty CISCy architecture and assembly conventions and APIs are very well documented.
The "it's written in an obscure programming language" argument just doesn't count for much. There are plenty of mainframe programmers around (the "skills shortage" is largely an effect of employers not wanting to pay reasonable rates), and any S/390 or z systems programmer should be able to learn JOVIAL in short order. I can think of four - no, make that five, maybe six - people off the top of my head that I believe could be competent maintainers of a decent-sized JOVIAL application in a month or so. And I'm not counting myself, because I'd have to brush up on some of my mainframe skills first.
According to Vince Cable, NATS's software is ancient. As someone who wrote some of Swanwick's software, I must also be ancient. This is worrying as I'm at least 20 years away from the state retirement age. What will I be by then, prehistoric?
(BTW, I didn't write the bit that broke - or at least I don't think so. The trouble is being sooo ancient, I can't remember if I worked on that bit or not)
"The trouble is being sooo ancient, I can't remember if I worked on that bit or not"
Surely it'll be in the change history?
A couple of years back I needed to dig out some stuff from the early 1980s, used in ground test versions of safety critical stuff that flew. Turns out I must have written much of it because my name was scattered throughout the file headers. Lot of water under the bridge since then.
i recall the reply I was given when question what seemed to be old model numbers etc for equipment for a safety critical system updates, it was along the lines of the system being "at the cutting edge of obsolescence" as everything had to be tried tested and certified over a space of years. Basically old was good as we knew it worked
And when they've finished with NATS IT systems they can start on the systems controlling nuclear, they're even older so must be prime for failure, in fact, using their logic, it's a miracle we need electricity for lighting at all, we should be self sufficient in glow in the dark radioactives by now.
.. visiting the Prestwick end of NATS in the 1990's. At that point it was running on PDP11s, with unbelievable amounts of life support. When I asked "Why don't you port it to something modern, like a Z80?" I was told that the problem is that the system has to be ratified by the British CAA, the merkin FAA, the french, spanish, german, yadda yadda. Getting anything agreed is practically impossible. They were looking fwd to the opening of the new Atlantic house, but freely admitted it would be newer tech, but not new tech.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
