back to article Airbus confirms software brought down A400M transport plane

Airbus has confirmed the crash that stalled its A400M program was caused by engine control software. However, according to Handelsblatt, the problem wasn't that the software is buggy. Rather, someone in the final assembly process installed the software incorrectly. Marwan Lahoud, Airbus' chief strategy officer, told the …

Page:

  1. Anonymous Coward
    Anonymous Coward

    Why can you install software incorrectly on an aircraft?

    I would have thought that somewhere in millions of quid spent on this a check on having the correct software properly installed would have been built in on boot?

    Or maybe I'm just old fashioned in my attitudes?

    But a sad loss of life, RIP.

    1. Yet Another Anonymous coward Silver badge

      Re: Why can you install software incorrectly on an aircraft?

      CAPA Report:

      Add line "remove disk 1" before line "insert disk 2" .....

    2. Destroy All Monsters Silver badge
      Holmes

      Re: Why can you install software incorrectly on an aircraft?

      software properly installed would have been built in on boot

      Checksums don't work for every problem. Wrong units (pound vs. kg) and configuration with off-by-one errors in the engine serial numbers come to mind.

      1. Anonymous Coward
        Anonymous Coward

        Re: Why can you install software incorrectly on an aircraft?

        >>Checksums don't work for every problem. Wrong units (pound vs. kg) and configuration with off-by-one errors in the engine serial numbers come to mind.

        What do checksums have to do with anything? Checksums are to address data corruption, not software configuration.

      2. Anonymous Coward
        Anonymous Coward

        Re: Why can you install software incorrectly on an aircraft?

        >>Checksums don't work for every problem. Wrong units (pound vs. kg) and configuration with off-by-one >>errors in the engine serial numbers come to mind.

        An off-by-one error on the engine count might be more severe.

      3. BillG
        Megaphone

        Re: Why can you install software incorrectly on an aircraft?

        the problem wasn't that the software is buggy. Rather, someone in the final assembly process installed the software incorrectly.

        Having worked for a defense contractor, let me translate: The software was buggy.

        You never, ever admit fault to your customer, ever, as this can and will affect your ability to get new contracts and also scuttle the contractor's stock price. If pressed you only admit to human error (which is impossible, see below).

        To make this more clear, the final assembly process is the most solid part of the process. The final code is complied, after which it is run through an automated code check which can take hours. In final assembly, or rather when the code is loaded into each computer, a series of diagnostic tests and simulations are run to verify both HW and SW operation. These are composed of test vectors simulating actual operating conditions. These are all Go/NoGo tests as simple as a green light means pass and a red light means fail.

        Upon power-up (here, before each flight) all systems run some built-in self-test (BIST) diagnostics. They are not just standalone tests, they depend upon inputs from other systems on the plane that share the same parameters. A failure locks the system and prevents operation. See how this can't be an isolated failure?

        For a failure such as this I have seen two reasons for failure: either the software was buggy, or somehow the system was tricked (hacked or accident) into running the factory test simulation code. In one case (not a plane) the operator accidentally pressed a secret key combination (holding down 3 keys simultaneously, released, then pressing one key within three seconds) that forced the system into diagnostic mode, with tragic consequences.

        Maybe things are different with non-U.S. contractors, as the above process is extremely expensive.

    3. Anonymous Coward
      Anonymous Coward

      Re: Why can you install software incorrectly on an aircraft?

      Software is common across different engines. The most likely "incorrectness" of the install is installing the wrong parameter tables. From there on the plane (which is 100% fly by wire) is asking the engine to deliver take off thrust (as was in that case) and it delivers a paltry fraction of it. With the expected results.

      This still does not excuse Airbus from not having checksums (or other means) of verifying the correct install stored in the main avionics and not interrogating the engine that it has the correct ones.

      1. BitDr

        Take Off Thrust vs Position of Throttles...

        Instead of calling for take off thrust the system simply needs to monitor the position of the throttles. If it only knows the relationship of those to the position of the throttles in the fuel system then there is no confusion in defining what "take off thrust" means in aircraft to aircraft.

        Anyway, it's all water under the bridge now, RIP.

        1. Cynic_999

          Re: Take Off Thrust vs Position of Throttles...

          I doubt that the error was in the sensing of the flight control position (i.e. that takeoff thrust was being demanded). It was more likely using the wrong engine performance or sensor parameters which resulted in incorrect engine control settings being used to try to achieve the commanded power.

          As an example, if you were to set the parameter on your programmable home thermostat so that its temperature input is interpreted as degrees Celsius, but your digital room thermometer is sending degrees Fahrenheit, your house is going to be a lot colder than you are commanding it to be.

        2. IanGD

          Re: Take Off Thrust vs Position of Throttles...

          That is the distinction between an EEC and a FADEC. The FADEC does not allow any overriding input from the operator (Pilot). The EEC is a subcomponent of a FADEC System and for many reasons a FADEC has become the choice moving forward. It is no longer possible for a human to calculate and control an engine according to all the variable parameters within the environment that modern jet engines operate.

    4. g e

      "extremely expensive"

      Whew. At least compo to the bereaved seems cheaper than redesigning shit....

      Classy.

      1. Anonymous Coward
        Anonymous Coward

        Re: "extremely expensive"

        "At least compo to the bereaved seems cheaper than redesigning shit...."

        I believe that's called the "Ford Pinto" decision. Widely documented, e.g.

        http://philosophia.uncg.edu/phi361-metivier/module-2-why-does-business-need-ethics/case-the-ford-pinto/

        1. Anonymous Coward
          Anonymous Coward

          Re: "extremely expensive"

          In aircraft safety, it is known as the Tombstone Imperative.

        2. MyffyW Silver badge

          Re: "extremely expensive"

          "A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one."

          - Edward Norton, Fight Club

    5. MrXavia

      Re: Why can you install software incorrectly on an aircraft?

      I am guessing the engine control software is generic, why write a very complex piece of software more than once? Then its just configured to the engine/airframe...

      I can see how poor configuration could cause a stall on takeoff...

    6. asdf
      Megaphone

      Re: Why can you install software incorrectly on an aircraft?

      Really with QA on something like an aircraft the rule should be no one person or even team (except pilots I suppose) could cause catastrophic failure even if they tried intentionally. If its possible as is the case here obviously a whole lot of people in management need to find a new line of work. People will make mistakes especially groups of them (none of us is as dumb as all of us) but if the process doesn't account for it the process is garbage.

    7. cray74

      Re: Why can you install software incorrectly on an aircraft?

      "I would have thought that somewhere in millions of quid spent on this a check on having the correct software properly installed would have been built in on boot?"

      The test systems for aerospace hardware often contend with different software versions in the gear they're testing. I'm on an aerospace company's program for a targeting system and we have different firmware versions for different customers and legacy versions of the system, which means the test rigs are necessarily flexible - they don't scream if you've got (for sake of argument) v1.01 firmware installed instead of the latest v1.03. And it's real easy to let your eyes glaze over when the boot-up data scrolls past and you've got the chance to spot "v1.01" instead of "v1.03." The real meat of the test data is whether the system works and all the flagged functions say, "pass," "working," and so on.

      The A400M's engines go through a lot of tests. I can guess at some of the objective evidence that would be required to make Airbus and Europrop happy. At a minimum:

      1) The ECU was tested and certified by its maker before it was sold to Europrop (admittedly probably without the software, which would be Europrop's or Airbus's responsibility to install);

      2) Europrop tested the all-up engines on a test cradle before it sold the engines to Airbus;

      3) Airbus would've powered up all 4 engines on the A400M long before the plane was allowed to take its first, fatal test flight since big aircraft don't get near a runway until their many systems are tested;

      4) On that fatal flight, the engines powered up and made all the blinken lights in the cockpit glow happy colors without spitzensparken

      The software that crashed the plane did go through check after check, and it worked. Up until the most important check.

      Airbus has noted problems with the A400M program's organization: it treats production, development, and retrofitting as separate programs, rather than an integrated single program. That's an environment where version control problems are going to proliferate. I could very easily imagine the poor bastard who "incorrectly configured" the software was just using an out-of-date manufacturing process plan that said something like, "get the USB stick with software version 1.01" because no one had flowed down process changes to the MPP hardcopy at his workstation to say, "get the v1.03 stick." Or something like that.

  2. x 7

    Likely to raise a shudder among the military fraternity.....

    they've been painfully through this before multiple times, e.g. problems with the FADEC software in the Chinook. Sounds like lessons still have not been learned in some high places.

    1. Anonymous Coward
      Anonymous Coward

      Isn't there an Osprey crash evey other week also

      1. Destroy All Monsters Silver badge

        Not really and the Osprey has some problems due to physics which are only software-fixable in games.

      2. asdf

        They cut down on the disaster that was the Osprey though several redesigns and whole lot of trial and error in blood. Always hated that pork program myself but unlike the F35 at least it won't cost us over a trillion dollars.

  3. Peter Prof Fox

    Is there any reason for the Spandsh to block the black bod data?

    [Our world] No

    [Their world] Yes

    Discuss, using only one side of the paper.

    1. Yet Another Anonymous coward Silver badge

      Re: Is there any reason for the Spandsh to block the black bod data?

      Evidence, transparency, proper legal procedure.

      As opposed to the more traditional response to a military aircraft accident. If pilot==dead then {pilot error.}

      Because suggesting that the aircraft was at fault might impact the manufacturer's profitablilty which would put the defence of the realm (and jobs in a marginal constituency) at risk.

      1. Destroy All Monsters Silver badge

        Re: Is there any reason for the Spandsh to block the black bod data?

        Because suggesting that the aircraft was at fault might impact the manufacturer's profitablilty which would put the defence of the realm (and jobs in a marginal constituency) at risk.

        I hate to bring this message but some Yuropean countries have the hots for their "friend's" F-35, so your "risk" is already in the intestinal zone.

        Evidence, transparency, proper legal procedure.

        You know, nowadays you can actually *copy* black box contents while maintaining evidence, transparency and proper legal procedure and getting the engineers to work.

    2. Anonymous Coward
      Anonymous Coward

      Re: Is there any reason for the Spandsh to block the black bod data?

      [Our world] No

      [Their world] Yes

      Discuss, using only one side of the paper.

      Let's correct that for you:

      [Our world] Yes

      [Their world] Yes

      There's likely to be a court case over this accident. The investigation in Spain is being conducted by their judicial authorities, as is normal in their (and many other country's) Inquisitorial/Roman system of justice.

      Regardless of the judicial arrangements having the evidence, especially if it is damning, made publically available seriously prejudices any court case in any jurisdiction. How could a trial be 'fair' afterwards? That would deny the victims and their families their day in court. Thus anyone pressing for public disclosure of the data (evidence) is being stupid and supremely selfish towards the victims.

      I bet you take photos of other people's car crashes, you jerk.

      However...

      The fact that Airbus are confirming the cause of the crash indicates that they have no intention of contesting whatever legal case is coming up. They're going to take what's coming on the chin, which is the proper thing for them to do. That is to the benefit of the individuals who carried out the installation (who are no doubt completely devastated by the crash): such an admission means that the responsibility is now aimed higher up the managerial chain.

      1. Blofeld's Cat
        Unhappy

        Re: Is there any reason for the Spandsh to block the black bod data?

        "The fact that Airbus are confirming the cause of the crash [...] means that the responsibility is now aimed higher up the managerial chain."

        From dealings with another large company, I suspect that any blame will go up the chain until it hits a committee or two, and then dilute into a fog of collective responsibility.

        Full marks though for not just blaming the pilots or some lowly technician.

      2. SkippyBing

        Re: Is there any reason for the Spandsh to block the black bod data?

        But, not releasing the black box data to the aviation accident investigation authority responsible is in contravention of international treaty (Annex 13 of the International Treaty on Civil Aviation, I think) for aviation accident investigation. Aviation accident investigation is purely to prevent reoccurence and not to accord blame. There is a slight question over whether this was a civil or military accident, however as it was being flown on a test by an Airbus crew I'd think it was civil at this stage as it hadn't been handed over to the Spanish Air Force.

        Of course it's an interesting move by Airbus to confirm the cause of the crash without the data from the black boxes as that may reveal other information.

        1. Anonymous Coward
          Anonymous Coward

          Re: Is there any reason for the Spandsh to block the black bod data?

          You're right, but some countries consider airplane crashes a "criminal investigation", especially if there are victims.

          In Italy it happened in the Linate crash too - it was managed by Italian "Carabinieri" and magistrates without a clue about properly managing a crash site, debris where removed immediately and piled up at the helipad, while access to the site by the investigators sent by the airplane company and its national agency was forbidden.

          When an airplane flown with Alitalia colors (but rented from a Romanian company with its crew) went off the runaway in Rome some time ago (without victims, luckily), the Alitalia logos where removed immediately with white paint, clearly "contaminating" the crash site evidences.

          Usually data are shared with the investigators of all interested parties, and of course the manufacturer is the one that is best equipped to analyze some data like engines performance, and fixing the problem is far more important than hiding it, actually hiding issues and having more crashes it's not what an airplane manufacture aims for, it's already difficult to sell planes with a good reputation, selling models known to crash is pretty impossible.

          Just some states have outdated rules and are obsessed with police/magistrate control on everything, just because of fear.

    3. Paul Hovnanian Silver badge

      Re: Is there any reason for the Spandsh to block the black bod data?

      Not block. But the Spanish authorities might be having trouble reading it, resulting in delays.

      This was a military transport, ordered by Turkey. Odds are that CVR and FDR data is encrypted. So extra steps may need to be taken with Turkey's cooperation to get a 'plaintext' copy.

    4. Graham Bartlett

      Re: Is there any reason for the Spandsh to block the black bod data?

      Usually I'll let typos go, but "black bod data" is too good to resist. Are we talking Grace Jones or Rampage Jackson though?

  4. Michael Thibault

    >Airbus will still need to satisfy customers that it can ensure that software installs don't go wrong in future.

    I can't think of more than one way they can do that.

  5. julianh72

    "Partly filled with wrong"

    When I read this:

    "Handelsblatt beats Google's translation with the sentence “Die Software für die Steuerung der Motoren sei bei der Endmontage falsch aufgespielt worden” "

    Well ... I had to see what Google Translate offers up:

    "The software for controlling the motors had been partly filled with wrong during the final assembly"

    "Partly filled with wrong" - my new catch-phrase for every time somebody cocks something up!

    1. frank ly

      Re: "Partly filled with wrong"

      My God, it's full of wrong!

      1. seven of five

        Re: "Partly filled with wrong"

        No, actually is was only partly filled with wrong. :)

        As the plane afterwards was completely fubared the must be some rounding issue...

        1. The First Dave

          Re: "Partly filled with wrong"

          Indeed, the plane wasn't completely borked - IIRC three engines shut down, but the fourth kept running, and one does rather wonder why that one was ok?

    2. Johan Bastiaansen

      Re: "Partly filled with wrong"

      Google Translate has this one wrong.

      Software aufspielen simply means, installing software.

      So the software wasn't installed properly. That's all the article said.

      The rest is speculation.

  6. Uberseehandel

    Impossible Testing Scenario

    I have worked on many multi national projects. To cater for customers who want to interface with equipment in their own language, which might be Mandarin Chinese, Sanskrit, Standard Arabic, Bulgarian or what ever, software tends to be hugely parameter driven, to an unsafe extent. Also, to allow for minor differences in customer requirements, parameters are adjusted,

    This gets unmanageable really quickly, for example, just 20 Yes/No options have over a million different combinations.

    One application I know of has over 24,000 lines of parameters, most with multiple values. This is inherently unsafe, there are not enough hours in the foreseeable future to test all the possible combinations, prior to release of the product.

    I have seen people involved in major acquisitions demand "modifications", which, inter alia, complicate the parameter file(s), merely to justify their own role as part of the acquisition team.

    1. Anonymous Coward
      Anonymous Coward

      Re: Impossible Testing Scenario

      Agree with the complexity issue though - have seen an automotive comms stack with something like 62 on/off options :-(

      However, you could argue that you only need to test the configurations that are actually used/shipped.

      1. maffski

        Re: Impossible Testing Scenario

        And then you need a configuration management system to ensure no-one can try to ship an untested configuration.

        And then you need to test the configuration management system...

        1. Anonymous Coward
          Anonymous Coward

          Re: Impossible Testing Scenario

          And then you need to test the configuration management system...

          Just use Rational ClearCase: No one will want to touch that - even using a bargepole with Prince Philips stuck on the end of it!

    2. Alan Brown Silver badge

      Re: Impossible Testing Scenario

      Language issues are best dealt with by Locale files - and those locale files are best translated by someone bilingual, then vetted by at least 3 other people (I've just finished making ~1500 corrections to a UK-english locale which was originally translated from french by a guy who learned english at high school...)

      It's the fiddling with other parameters which gets really messy, real fast - and leads to Chinook crashes.

      1. Tom 7

        Re: Impossible Testing Scenario

        Even my software can ask what hardware there is present before loading the wrong stuff.

        They used to call them sanity checks.

        1. Stoneshop
          Holmes

          Re: Impossible Testing Scenario

          They used to call them sanity checks.

          And surely you know the sanity check is correct and infallible.

    3. chris 17 Silver badge

      Re: Impossible Testing Scenario

      if this condition send error message [lang][51]

      the reason for the error will be the same in every language. the letters used to describe the error to humans should just be mapped.

    4. Graham Bartlett

      Re: Impossible Testing Scenario

      As to "inherently unsafe", that depends on whether parameters do need testing in combination. Some will, some won't. No testing strategy, not even for safety critical software like medical equipment, requires testing of every possible combination of individual states. If you think that's a problem, you don't understand ALARP.

      Good architecture should minimise linkages so that you *can* trace what can affect what. If you don't have good enough engineers to design a robust architecture, then you have bigger problems than just the testing.

      1. Anonymous Coward
        Anonymous Coward

        Re: Impossible Testing Scenario

        Fortunately, testing of avionics and nuclear plants does require the testing of all possible combinations - things like full MC/DC, etc.

        However, even though that test all the paths through the software it doesn't cover all possible combinations of paths through the software...

  7. jibanes

    I don't buy it.

    The "misconfigured" software didn't prevent the plane to take off nor didn't turn a warning light on?

    Also, all this was deduced without the blackbox logs?

    1. Alan Brown Silver badge

      Re: I don't buy it.

      "Also, all this was deduced without the blackbox logs?"

      It's known what software was installed on the aircraft and simulators exist for a reason.

      1. asdf

        Re: I don't buy it.

        >It's known what software was installed on the aircraft and simulators exist for a reason.

        Yeah to be used before live testing I would assume.

        1. Alan Brown Silver badge

          Re: I don't buy it.

          Supposed to be used before live testing, but often used to fly a profile based on known weather/software/etc or even replay black box data to see what the pilots would have seen.

          It's creepy flying a profile (more like riding along) where you know people died and without the benefit of hindsight usually impossible to avoid making the same mistakes that caused it.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like