back to article Airbus confirms software brought down A400M transport plane

Airbus has confirmed the crash that stalled its A400M program was caused by engine control software. However, according to Handelsblatt, the problem wasn't that the software is buggy. Rather, someone in the final assembly process installed the software incorrectly. Marwan Lahoud, Airbus' chief strategy officer, told the …

  1. Anonymous Coward
    Anonymous Coward

    Why can you install software incorrectly on an aircraft?

    I would have thought that somewhere in millions of quid spent on this a check on having the correct software properly installed would have been built in on boot?

    Or maybe I'm just old fashioned in my attitudes?

    But a sad loss of life, RIP.

    1. Yet Another Anonymous coward Silver badge

      Re: Why can you install software incorrectly on an aircraft?

      CAPA Report:

      Add line "remove disk 1" before line "insert disk 2" .....

    2. Destroy All Monsters Silver badge
      Holmes

      Re: Why can you install software incorrectly on an aircraft?

      software properly installed would have been built in on boot

      Checksums don't work for every problem. Wrong units (pound vs. kg) and configuration with off-by-one errors in the engine serial numbers come to mind.

      1. Anonymous Coward
        Anonymous Coward

        Re: Why can you install software incorrectly on an aircraft?

        >>Checksums don't work for every problem. Wrong units (pound vs. kg) and configuration with off-by-one errors in the engine serial numbers come to mind.

        What do checksums have to do with anything? Checksums are to address data corruption, not software configuration.

      2. Anonymous Coward
        Anonymous Coward

        Re: Why can you install software incorrectly on an aircraft?

        >>Checksums don't work for every problem. Wrong units (pound vs. kg) and configuration with off-by-one >>errors in the engine serial numbers come to mind.

        An off-by-one error on the engine count might be more severe.

      3. BillG
        Megaphone

        Re: Why can you install software incorrectly on an aircraft?

        the problem wasn't that the software is buggy. Rather, someone in the final assembly process installed the software incorrectly.

        Having worked for a defense contractor, let me translate: The software was buggy.

        You never, ever admit fault to your customer, ever, as this can and will affect your ability to get new contracts and also scuttle the contractor's stock price. If pressed you only admit to human error (which is impossible, see below).

        To make this more clear, the final assembly process is the most solid part of the process. The final code is complied, after which it is run through an automated code check which can take hours. In final assembly, or rather when the code is loaded into each computer, a series of diagnostic tests and simulations are run to verify both HW and SW operation. These are composed of test vectors simulating actual operating conditions. These are all Go/NoGo tests as simple as a green light means pass and a red light means fail.

        Upon power-up (here, before each flight) all systems run some built-in self-test (BIST) diagnostics. They are not just standalone tests, they depend upon inputs from other systems on the plane that share the same parameters. A failure locks the system and prevents operation. See how this can't be an isolated failure?

        For a failure such as this I have seen two reasons for failure: either the software was buggy, or somehow the system was tricked (hacked or accident) into running the factory test simulation code. In one case (not a plane) the operator accidentally pressed a secret key combination (holding down 3 keys simultaneously, released, then pressing one key within three seconds) that forced the system into diagnostic mode, with tragic consequences.

        Maybe things are different with non-U.S. contractors, as the above process is extremely expensive.

    3. Anonymous Coward
      Anonymous Coward

      Re: Why can you install software incorrectly on an aircraft?

      Software is common across different engines. The most likely "incorrectness" of the install is installing the wrong parameter tables. From there on the plane (which is 100% fly by wire) is asking the engine to deliver take off thrust (as was in that case) and it delivers a paltry fraction of it. With the expected results.

      This still does not excuse Airbus from not having checksums (or other means) of verifying the correct install stored in the main avionics and not interrogating the engine that it has the correct ones.

      1. BitDr

        Take Off Thrust vs Position of Throttles...

        Instead of calling for take off thrust the system simply needs to monitor the position of the throttles. If it only knows the relationship of those to the position of the throttles in the fuel system then there is no confusion in defining what "take off thrust" means in aircraft to aircraft.

        Anyway, it's all water under the bridge now, RIP.

        1. Cynic_999

          Re: Take Off Thrust vs Position of Throttles...

          I doubt that the error was in the sensing of the flight control position (i.e. that takeoff thrust was being demanded). It was more likely using the wrong engine performance or sensor parameters which resulted in incorrect engine control settings being used to try to achieve the commanded power.

          As an example, if you were to set the parameter on your programmable home thermostat so that its temperature input is interpreted as degrees Celsius, but your digital room thermometer is sending degrees Fahrenheit, your house is going to be a lot colder than you are commanding it to be.

        2. IanGD

          Re: Take Off Thrust vs Position of Throttles...

          That is the distinction between an EEC and a FADEC. The FADEC does not allow any overriding input from the operator (Pilot). The EEC is a subcomponent of a FADEC System and for many reasons a FADEC has become the choice moving forward. It is no longer possible for a human to calculate and control an engine according to all the variable parameters within the environment that modern jet engines operate.

    4. g e

      "extremely expensive"

      Whew. At least compo to the bereaved seems cheaper than redesigning shit....

      Classy.

      1. Anonymous Coward
        Anonymous Coward

        Re: "extremely expensive"

        "At least compo to the bereaved seems cheaper than redesigning shit...."

        I believe that's called the "Ford Pinto" decision. Widely documented, e.g.

        http://philosophia.uncg.edu/phi361-metivier/module-2-why-does-business-need-ethics/case-the-ford-pinto/

        1. Anonymous Coward
          Anonymous Coward

          Re: "extremely expensive"

          In aircraft safety, it is known as the Tombstone Imperative.

        2. MyffyW Silver badge

          Re: "extremely expensive"

          "A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one."

          - Edward Norton, Fight Club

    5. MrXavia

      Re: Why can you install software incorrectly on an aircraft?

      I am guessing the engine control software is generic, why write a very complex piece of software more than once? Then its just configured to the engine/airframe...

      I can see how poor configuration could cause a stall on takeoff...

    6. asdf
      Megaphone

      Re: Why can you install software incorrectly on an aircraft?

      Really with QA on something like an aircraft the rule should be no one person or even team (except pilots I suppose) could cause catastrophic failure even if they tried intentionally. If its possible as is the case here obviously a whole lot of people in management need to find a new line of work. People will make mistakes especially groups of them (none of us is as dumb as all of us) but if the process doesn't account for it the process is garbage.

    7. cray74

      Re: Why can you install software incorrectly on an aircraft?

      "I would have thought that somewhere in millions of quid spent on this a check on having the correct software properly installed would have been built in on boot?"

      The test systems for aerospace hardware often contend with different software versions in the gear they're testing. I'm on an aerospace company's program for a targeting system and we have different firmware versions for different customers and legacy versions of the system, which means the test rigs are necessarily flexible - they don't scream if you've got (for sake of argument) v1.01 firmware installed instead of the latest v1.03. And it's real easy to let your eyes glaze over when the boot-up data scrolls past and you've got the chance to spot "v1.01" instead of "v1.03." The real meat of the test data is whether the system works and all the flagged functions say, "pass," "working," and so on.

      The A400M's engines go through a lot of tests. I can guess at some of the objective evidence that would be required to make Airbus and Europrop happy. At a minimum:

      1) The ECU was tested and certified by its maker before it was sold to Europrop (admittedly probably without the software, which would be Europrop's or Airbus's responsibility to install);

      2) Europrop tested the all-up engines on a test cradle before it sold the engines to Airbus;

      3) Airbus would've powered up all 4 engines on the A400M long before the plane was allowed to take its first, fatal test flight since big aircraft don't get near a runway until their many systems are tested;

      4) On that fatal flight, the engines powered up and made all the blinken lights in the cockpit glow happy colors without spitzensparken

      The software that crashed the plane did go through check after check, and it worked. Up until the most important check.

      Airbus has noted problems with the A400M program's organization: it treats production, development, and retrofitting as separate programs, rather than an integrated single program. That's an environment where version control problems are going to proliferate. I could very easily imagine the poor bastard who "incorrectly configured" the software was just using an out-of-date manufacturing process plan that said something like, "get the USB stick with software version 1.01" because no one had flowed down process changes to the MPP hardcopy at his workstation to say, "get the v1.03 stick." Or something like that.

  2. x 7

    Likely to raise a shudder among the military fraternity.....

    they've been painfully through this before multiple times, e.g. problems with the FADEC software in the Chinook. Sounds like lessons still have not been learned in some high places.

    1. Anonymous Coward
      Anonymous Coward

      Isn't there an Osprey crash evey other week also

      1. Destroy All Monsters Silver badge

        Not really and the Osprey has some problems due to physics which are only software-fixable in games.

      2. asdf

        They cut down on the disaster that was the Osprey though several redesigns and whole lot of trial and error in blood. Always hated that pork program myself but unlike the F35 at least it won't cost us over a trillion dollars.

  3. Peter Prof Fox

    Is there any reason for the Spandsh to block the black bod data?

    [Our world] No

    [Their world] Yes

    Discuss, using only one side of the paper.

    1. Yet Another Anonymous coward Silver badge

      Re: Is there any reason for the Spandsh to block the black bod data?

      Evidence, transparency, proper legal procedure.

      As opposed to the more traditional response to a military aircraft accident. If pilot==dead then {pilot error.}

      Because suggesting that the aircraft was at fault might impact the manufacturer's profitablilty which would put the defence of the realm (and jobs in a marginal constituency) at risk.

      1. Destroy All Monsters Silver badge

        Re: Is there any reason for the Spandsh to block the black bod data?

        Because suggesting that the aircraft was at fault might impact the manufacturer's profitablilty which would put the defence of the realm (and jobs in a marginal constituency) at risk.

        I hate to bring this message but some Yuropean countries have the hots for their "friend's" F-35, so your "risk" is already in the intestinal zone.

        Evidence, transparency, proper legal procedure.

        You know, nowadays you can actually *copy* black box contents while maintaining evidence, transparency and proper legal procedure and getting the engineers to work.

    2. Anonymous Coward
      Anonymous Coward

      Re: Is there any reason for the Spandsh to block the black bod data?

      [Our world] No

      [Their world] Yes

      Discuss, using only one side of the paper.

      Let's correct that for you:

      [Our world] Yes

      [Their world] Yes

      There's likely to be a court case over this accident. The investigation in Spain is being conducted by their judicial authorities, as is normal in their (and many other country's) Inquisitorial/Roman system of justice.

      Regardless of the judicial arrangements having the evidence, especially if it is damning, made publically available seriously prejudices any court case in any jurisdiction. How could a trial be 'fair' afterwards? That would deny the victims and their families their day in court. Thus anyone pressing for public disclosure of the data (evidence) is being stupid and supremely selfish towards the victims.

      I bet you take photos of other people's car crashes, you jerk.

      However...

      The fact that Airbus are confirming the cause of the crash indicates that they have no intention of contesting whatever legal case is coming up. They're going to take what's coming on the chin, which is the proper thing for them to do. That is to the benefit of the individuals who carried out the installation (who are no doubt completely devastated by the crash): such an admission means that the responsibility is now aimed higher up the managerial chain.

      1. Blofeld's Cat
        Unhappy

        Re: Is there any reason for the Spandsh to block the black bod data?

        "The fact that Airbus are confirming the cause of the crash [...] means that the responsibility is now aimed higher up the managerial chain."

        From dealings with another large company, I suspect that any blame will go up the chain until it hits a committee or two, and then dilute into a fog of collective responsibility.

        Full marks though for not just blaming the pilots or some lowly technician.

      2. SkippyBing

        Re: Is there any reason for the Spandsh to block the black bod data?

        But, not releasing the black box data to the aviation accident investigation authority responsible is in contravention of international treaty (Annex 13 of the International Treaty on Civil Aviation, I think) for aviation accident investigation. Aviation accident investigation is purely to prevent reoccurence and not to accord blame. There is a slight question over whether this was a civil or military accident, however as it was being flown on a test by an Airbus crew I'd think it was civil at this stage as it hadn't been handed over to the Spanish Air Force.

        Of course it's an interesting move by Airbus to confirm the cause of the crash without the data from the black boxes as that may reveal other information.

        1. Anonymous Coward
          Anonymous Coward

          Re: Is there any reason for the Spandsh to block the black bod data?

          You're right, but some countries consider airplane crashes a "criminal investigation", especially if there are victims.

          In Italy it happened in the Linate crash too - it was managed by Italian "Carabinieri" and magistrates without a clue about properly managing a crash site, debris where removed immediately and piled up at the helipad, while access to the site by the investigators sent by the airplane company and its national agency was forbidden.

          When an airplane flown with Alitalia colors (but rented from a Romanian company with its crew) went off the runaway in Rome some time ago (without victims, luckily), the Alitalia logos where removed immediately with white paint, clearly "contaminating" the crash site evidences.

          Usually data are shared with the investigators of all interested parties, and of course the manufacturer is the one that is best equipped to analyze some data like engines performance, and fixing the problem is far more important than hiding it, actually hiding issues and having more crashes it's not what an airplane manufacture aims for, it's already difficult to sell planes with a good reputation, selling models known to crash is pretty impossible.

          Just some states have outdated rules and are obsessed with police/magistrate control on everything, just because of fear.

    3. Paul Hovnanian Silver badge

      Re: Is there any reason for the Spandsh to block the black bod data?

      Not block. But the Spanish authorities might be having trouble reading it, resulting in delays.

      This was a military transport, ordered by Turkey. Odds are that CVR and FDR data is encrypted. So extra steps may need to be taken with Turkey's cooperation to get a 'plaintext' copy.

    4. Graham Bartlett

      Re: Is there any reason for the Spandsh to block the black bod data?

      Usually I'll let typos go, but "black bod data" is too good to resist. Are we talking Grace Jones or Rampage Jackson though?

  4. Michael Thibault

    >Airbus will still need to satisfy customers that it can ensure that software installs don't go wrong in future.

    I can't think of more than one way they can do that.

  5. julianh72

    "Partly filled with wrong"

    When I read this:

    "Handelsblatt beats Google's translation with the sentence “Die Software für die Steuerung der Motoren sei bei der Endmontage falsch aufgespielt worden” "

    Well ... I had to see what Google Translate offers up:

    "The software for controlling the motors had been partly filled with wrong during the final assembly"

    "Partly filled with wrong" - my new catch-phrase for every time somebody cocks something up!

    1. frank ly

      Re: "Partly filled with wrong"

      My God, it's full of wrong!

      1. seven of five

        Re: "Partly filled with wrong"

        No, actually is was only partly filled with wrong. :)

        As the plane afterwards was completely fubared the must be some rounding issue...

        1. The First Dave

          Re: "Partly filled with wrong"

          Indeed, the plane wasn't completely borked - IIRC three engines shut down, but the fourth kept running, and one does rather wonder why that one was ok?

    2. Johan Bastiaansen

      Re: "Partly filled with wrong"

      Google Translate has this one wrong.

      Software aufspielen simply means, installing software.

      So the software wasn't installed properly. That's all the article said.

      The rest is speculation.

  6. Uberseehandel

    Impossible Testing Scenario

    I have worked on many multi national projects. To cater for customers who want to interface with equipment in their own language, which might be Mandarin Chinese, Sanskrit, Standard Arabic, Bulgarian or what ever, software tends to be hugely parameter driven, to an unsafe extent. Also, to allow for minor differences in customer requirements, parameters are adjusted,

    This gets unmanageable really quickly, for example, just 20 Yes/No options have over a million different combinations.

    One application I know of has over 24,000 lines of parameters, most with multiple values. This is inherently unsafe, there are not enough hours in the foreseeable future to test all the possible combinations, prior to release of the product.

    I have seen people involved in major acquisitions demand "modifications", which, inter alia, complicate the parameter file(s), merely to justify their own role as part of the acquisition team.

    1. Anonymous Coward
      Anonymous Coward

      Re: Impossible Testing Scenario

      Agree with the complexity issue though - have seen an automotive comms stack with something like 62 on/off options :-(

      However, you could argue that you only need to test the configurations that are actually used/shipped.

      1. maffski

        Re: Impossible Testing Scenario

        And then you need a configuration management system to ensure no-one can try to ship an untested configuration.

        And then you need to test the configuration management system...

        1. Anonymous Coward
          Anonymous Coward

          Re: Impossible Testing Scenario

          And then you need to test the configuration management system...

          Just use Rational ClearCase: No one will want to touch that - even using a bargepole with Prince Philips stuck on the end of it!

    2. Alan Brown Silver badge

      Re: Impossible Testing Scenario

      Language issues are best dealt with by Locale files - and those locale files are best translated by someone bilingual, then vetted by at least 3 other people (I've just finished making ~1500 corrections to a UK-english locale which was originally translated from french by a guy who learned english at high school...)

      It's the fiddling with other parameters which gets really messy, real fast - and leads to Chinook crashes.

      1. Tom 7

        Re: Impossible Testing Scenario

        Even my software can ask what hardware there is present before loading the wrong stuff.

        They used to call them sanity checks.

        1. Stoneshop
          Holmes

          Re: Impossible Testing Scenario

          They used to call them sanity checks.

          And surely you know the sanity check is correct and infallible.

    3. chris 17 Silver badge

      Re: Impossible Testing Scenario

      if this condition send error message [lang][51]

      the reason for the error will be the same in every language. the letters used to describe the error to humans should just be mapped.

    4. Graham Bartlett

      Re: Impossible Testing Scenario

      As to "inherently unsafe", that depends on whether parameters do need testing in combination. Some will, some won't. No testing strategy, not even for safety critical software like medical equipment, requires testing of every possible combination of individual states. If you think that's a problem, you don't understand ALARP.

      Good architecture should minimise linkages so that you *can* trace what can affect what. If you don't have good enough engineers to design a robust architecture, then you have bigger problems than just the testing.

      1. Anonymous Coward
        Anonymous Coward

        Re: Impossible Testing Scenario

        Fortunately, testing of avionics and nuclear plants does require the testing of all possible combinations - things like full MC/DC, etc.

        However, even though that test all the paths through the software it doesn't cover all possible combinations of paths through the software...

  7. jibanes

    I don't buy it.

    The "misconfigured" software didn't prevent the plane to take off nor didn't turn a warning light on?

    Also, all this was deduced without the blackbox logs?

    1. Alan Brown Silver badge

      Re: I don't buy it.

      "Also, all this was deduced without the blackbox logs?"

      It's known what software was installed on the aircraft and simulators exist for a reason.

      1. asdf

        Re: I don't buy it.

        >It's known what software was installed on the aircraft and simulators exist for a reason.

        Yeah to be used before live testing I would assume.

        1. Alan Brown Silver badge

          Re: I don't buy it.

          Supposed to be used before live testing, but often used to fly a profile based on known weather/software/etc or even replay black box data to see what the pilots would have seen.

          It's creepy flying a profile (more like riding along) where you know people died and without the benefit of hindsight usually impossible to avoid making the same mistakes that caused it.

  8. Cliff

    Combined with the Android Bitcoin article

    Perhaps random.org going HTTPS only meant all the fuel got sent to one engine?

  9. JJKing

    Does this meant they ran Setup.com instead of Install.exe?

    Absolutely bloody tragic consequences and I hope the families sue the crap out of them for this inexcusable and criminal cockup.

  10. paulc
    WTF?

    Engine ground runs?

    you would think that they'd do some proper engine ground runs first before sending it down the runway for it's first flight...

    1. Anonymous Coward
      Anonymous Coward

      Re: Engine ground runs?

      Obviously don't know their process, but I've been involved in a project where special test modes are used to test the hardware in production.

      These don't perform a functional tests of the software (as that should be done as part of the development process), but do allow all the mechanical / electrical / electronic systems to be tested. Testing like this means that aspects can be tested in isolation, making it much easier to find any issues (e.g. by not having to have all the features required for a closed-loop control system to be working when testing).

      The final step on the production line was to load the configuration data for the particular customer. This step was then verified as part of the final QA before shipping (e.g. verify that the configuration data checksum matched the checksum on record for the customer).

    2. seven of five

      Re: Engine ground runs?

      This model is flying since 6 years and since at least two years operational for the French army - they have around a dozen operational, iirc. It was Spains first, though.

      1. fandom

        Re: Engine ground runs?

        The crashed plane was going to be Turkey's third A400M

  11. Anonymous Coward
    Anonymous Coward

    Testing? What's that for?

    One well known vendor of safety critical stuff I'm familiar with is keen on model based systems engineering and similar entirely sensible tools.

    Unfortunately their idea of testing is to test solely against the model, on somebody's desktop/datacentre PC, not test against whatever eventual implementation of the model ends up being programmed on the aircraft subsystems. It's so much more cost effective that way, obviously, and what could possibly go wrong between the model and the stuff on the aircraft.

    Expect more of this A400M kind of thing in the years to come, and not just on military trials, unless attitudes and processes return to being driven by engineers, testers, etc, (as they used to be in the early days of safety critical software) rather than being dictated by cost-driven MBA types who think Integrity is a piece of productivity software (and who ruthlessly silence anyone who dares question the new improved approach).

  12. Anonymous Coward
    Joke

    Extrapolating wildly from my SETUP experiences

    They were in a hurry and didn't spot the pre-checked "ASK toolbar" selection...

    1. Anonymous Coward
      Anonymous Coward

      Re: Extrapolating wildly from my SETUP experiences

      Just to disarm Flanders before he posts. Blah blah you should be more respectful of the dead. Blah blah blah I am such an uptight a s s clown, etc.

  13. SteveK

    Of course, it could also be argued that the actual cause of the *crash* was [reportedly] flying into power lines and/or an electricity pylon whilst attempting an emergency landing in a field.

    1. Anonymous Coward
      Coat

      I've now amended the report to say Pilot Error. Thanks you've just saved us millions.

      Love

      Airbus - Legal Department.

    2. paulc

      and just what was the 'root cause' that required them to perform a forced landing?

      1. Jess--

        leaving the ground

  14. saif

    Fly-by-wire problems

    A recurring theme, where modern flight control systems control mechanical devices totally with software and electronics rather than using them to assist or resist mechanical actions. One suspects even if the pilots realised the plane was behaving unexpectedly, they would either have to trust the electronics to recover or probably not be able to anything about it in time anyway.

    1. Anonymous Blowhard

      Re: Fly-by-wire problems

      Fly-By-Wire has proven itself to be far safer than the old mechanical systems; check the statistics for modern FBW airliners:

      http://www.fearofflying.com/resources/safest-airliners-and-airline-safety.shtml

      In this case the software problem was in the FADEC, which replaces the role of the "flight engineer" (planes haven't had flight engineers for about thirty years); FADEC is like the engine management software in your car (unless you have a car that is pre-1990s that is) so it's virtually impossible to replace with mechanical systems and still have the same efficiency and reliability.

      Going back to the pre-FBW systems is likely to introduce far more safety problems than you remove so whilst we know that improvements can be made, it won't be by abandoning modern avionics.

      1. naive

        Re: Fly-by-wire problems

        Perhaps you are right about the safety of FBW. The site clearly indicated the clear advantage Boeing has over Airbus in safety issues.

        As a layman, it is pretty hard to imagine somebody manages to create a situation in which software starts shutting down the engines in mid take off.

        Somebody would think that when steel wires and hydraulic pipes for management of the essential controls are replaced with copper wires, there should be some brain behind it.

        1. Anonymous Coward
          Anonymous Coward

          Re: Fly-by-wire problems

          "when steel wires and hydraulic pipes for management of the essential controls are replaced with copper wires, there should be some brain behind it."

          There has been brain behind it in the past.

          Safety critical stuff (software in particular) *was* rocket science when it first routinely arrived on airliners almost thirty years ago, when engineers ruled the roost. And it stayed that way for a long time, and rightly so.

          Thirty or so mostly accident free years later, the new dynamic young people in charge of the business seem to be starting to get complacent. Nothing's gone wrong [the missing word here is 'yet'], but we need new methods to keep our costs down and improve performance all round. Methods, tools, processes, are being introduced from other parts of the software world. methods that lose much of what has kept safety critical systems as reliable as they have been.

          New people in the regulatory authorities think that if there's a defined process and the paperwork follows the process, everything will be OK. The auditors are no longer subjec t matter experts, they're paperwork experts. Well guys, there's a reason engineering-based things like code reviews and extensive (not exhaustive, but extensive) testing of a representative setup have been considered vital in the past. They may or may not discover errors, but they should help *prevent* them, **IF the right people do them for the right reasons** (which does not mean because they're on the project ticklist).

          This stuff can be fixed. It needs to be fixed before too long.Fixing it will involve managers listening to engineers, designers, testers, etc. Otherwise at some time in the next few years, there'll be an inquiry and lessons will be learned and all the usual guff. The lessons will likely already have been known, but will have been ruled as outdated by Bright Young Things on their way up the 'career advancement by cost reduction and schedule compression' pole.

          1. Alan Brown Silver badge

            Re: Fly-by-wire problems

            "Fixing it will involve managers listening to engineers, designers, testers, etc"

            The jobsworthian "tickbox" mentality goes _well_ beyond aviation and engineering. People do stuff by rote without understanding why they're doing it and that in many fields the tickboxes are merely a guide to ensure the job's being done right, not an end in themselves.

            Then they get promoted to management and the tickboxes become more important than the end target.

  15. kmac499

    History repeating

    I seem to remember that the UK Chinooks had problems with their engine software, (termed FADEC Full Authoity Digital Engine Control I think) Basically a fly by wire throttle pedal which took loads of other parameters into account when setting the engine power.

    A possible contribution to the '94 crash in scotland where a load of anti terrorist experts died en route from Northern Ireland. Initially blamed on the pilots..

    I think this was also a case of 'not invented here', where the MOD decided we couldn't use the off the peg Boeing system and had it rewritten. Jobs for the boys I suspect..

  16. ciaran

    Every engine the same, but different

    I saw in a different forum what the installation problem was.

    Its a problem with the motors. Depending on where they're attached to the wing they have to behave differently. You have to first install the software and then add the settings. If you mix them up its very difficult to detect it on the ground. Once in the air its difficult not to notice.

    1. AndyS

      Re: Every engine the same, but different

      Sounds interesting, any chance of a link to the forum?

  17. cambsukguy

    "Die Software für die Steuerung der Motoren sei bei der Endmontage falsch aufgespielt Worden"

    Translates as "The software for controlling the motors be incorrectly been played during the final Assembly" using the Bing engine.

    No fluent German/English people here willing to weigh in?

    Will have to wait until the BBC report it - they have a ridiculous number of multi-lingual people working for them.

    1. Johan Bastiaansen

      Software aufspielen

      Software aufspielen simply means, "installing software".

      So the software wasn't installed properly. That's all the article said. The rest is speculation.

    2. sgp

      Aufspielen means "to play" as well as "to install".

      No points for guessing the correct translation.

      1. Anonymous Coward
        Anonymous Coward

        >>Aufspielen means "to play" as well as "to install".

        >>No points for guessing the correct translation.

        So you are... not awarding points... for... Bing guessing... the wrong translation?

        Can somebody please translate your sentence?

    3. supine

      I'm no fluent speaker but my translation would be:

      "The software for controlling the motors was incorrectly deployed during final installation."

  18. Anonymous Coward
    Anonymous Coward

    DLL hell?

  19. imanidiot Silver badge

    If bad install can bring down a plane

    Then I'm not quite convinced its not ALSO bad programming.

    1. Anonymous Coward
      Anonymous Coward

      Re: If bad install can bring down a plane

      Indeed, smacks of blaming the user with the usability of the software is poor.

      Seems like it should be made very difficult to configure this software incorrectly.

      1. Cynic_999

        Re: If bad install can bring down a plane

        "Seems like it should be made very difficult to configure this software incorrectly"

        How? There are a load of parameters that the software & hardware has no way of checking, and must thus rely on the installer inputting the correct data. As a trivial example, if one of the parameters dictates the position of the engine on the aircraft, the electronics/software has no way to check that the wires physically go to the left wing rather than the right wing. The only checks that can be made is that all the parameters are consistent with each other - and if they were parameters that were designed to work with a slightly different engine type or configuration, they will be self-consistent and pass all the "sanity checks".

        1. Anonymous Coward
          Anonymous Coward

          Re: "the software & hardware has no way of checking"

          "The only checks that can be made is that all the parameters are consistent with each other "

          Confident? Final answer?

          Here's a thought for you. How hard would it be to have the parameters (and maybe nothing but the parameters) programmed into something that's associated with the *engine* and follows the engine (does not follow the controller).

          An engine-attached widget with a standard interface that can be used by the engine control to read the parameters. A widget which can be (and should be) programmed and verified independently of the engine control itself.

          Hint: it's not very hard. Ask people in the industry about what a Data Entry Plug is, and where they're used, and how long they've been using them that way. Or read about Data Entry Plugs (aka rating plugs) on the Interweb.

          Please don't try to defend the indefensible by making things up. It's not helpful.

          1. Alan Brown Silver badge

            Re: "the software & hardware has no way of checking"

            "How hard would it be to have the parameters (and maybe nothing but the parameters) programmed into something that's associated with the *engine* and follows the engine "

            Not very (and this is already done), but as some have already pointed out the desired characteristics of the engine vary depending on which side of the aircraft it's on, whether it's inboard or outboard and whether it's driving a reversing reduction gear or not.

            Ideally the engine would recognise which position it was in and pick up the right settings automatically, but people insist on individually manually programming things "because it's more reliable" and that opens up the possibility of mistakes.

            Let's not go into Murphy's Law (If an aircraft part can be installed 2 ways and one way is the wrong way, someone will do it) - which amongst other things is exactly the reason that the Genesis probe crashed into the Utah desert (someone installed a gravity sensor upside down) and that there was much excitement about Spirit and Opportunity camera images until it was discovered the calibration files had been swapped over...

            It's better to design it right and make _sure_ it can't be installed the wrong way up. This applies as much to software as hardware, because the wetware is fundamentally unreliable.

            1. Anonymous Coward
              Anonymous Coward

              Re: mistake-proofing

              "better to design it right and make _sure_ it can't be installed the wrong way up."

              Very wise words (and the rest). The once-much-worshipped Japanese even had a name [1] for that concept.

              An example: if an item needs fixing to another item, and the relative position matters, make the design such that (1) the necessary orientation is obvious, and (2, just in case obvious isn't enough) make the wrong orientations impossible.

              E.g. If the orientation of a circular or square part matters, and you're attaching it with screws or similar, have the holes at irregular positions so that the correct orentation is the only one which is possible. On many occasions I've seen stuff like this follow the "do everything neat and regular" principle which frequently introduces opportunities for confusion (4 possible orientations, only one of which is correct) or worse.

              It's not really rocket science is it. Just 'common sense, properly applied'.

              [1] poka-yoke, from the (in)famous Toyota Production System and thus from all the finest LEAN toolkits. But LEAN (and its abuse by those who just tick boxes without understanding why the concepts matter) is another story for another day.

        2. Anonymous Coward
          Anonymous Coward

          Re: If bad install can bring down a plane

          >>How? There are a load of parameters that the software & hardware has no way of checking, and must thus rely on the installer inputting the correct data.

          I hope you're not in charge of designing anything important.

          Let's say you need to configure some software using the user's birth year. According to you, it sounds like the only way you can do that is ask the user for his birth year and if he's wrong, everything is eff'ed and there's no way you could have prevented the problem.

          But what if you ask for birth year AND age and compare the two, as a sanity check? That should sort out of a lot of problems. And if you wanted to triple check, ask for the user's social security number and arrange some way to confirm the year with the government. etc.

          I'm sure similar double/triple/quadruple checking of configuration parameters is possible when installing software for a plane.

  20. Henry Wertz 1 Gold badge

    Wrong calibration

    "Instead of calling for take off thrust the system simply needs to monitor the position of the throttles. If it only knows the relationship of those to the position of the throttles in the fuel system then there is no confusion in defining what "take off thrust" means in aircraft to aircraft."

    Yes there is confusion, "take of thrust" might be like 30% throttle on one engine, and 75% thrust on another engine (I doubt it's as low as 30%, but for sake of argument). These vendors tend to order an airframe and then order some engines to be put on seperately... why, I don't know, but that's what they do, so it's not like there'd be an "A400M calibration". This kind of thing does actually apply on cars too -- some car with 1.8L engine, 2.2L engine, 3.0L engine option, they will usually all 3 use the exact same computer, with a calibration ROM (in the past a physical ROM chip, now flashed in parameters) saying "this is a 4-cylinder, here is the fuel curve, here's the ignition timing curve, here's an "fast idle" versus temperature curve", and so on. Although I've never heard of GM f'ing this up by putting the wrong calibration in.

    Besides confusion over "how much throttle" with electronic throttle control, there'd also be confusion over fueling (I'm sure underfueling or overfueling a jet engine jacks up your power just like it would on a car engine), the different engines may even be expected to run at different RPM to produce some amount of thrust.

  21. Toska

    German here. Proper translation: "During final assembly the engine control software had been installed incorrectly."

    It doesn't mention parameters and lacks any specifics about how they managed to install the software "incorrectly". In a German military forum (augengeradeaus.net) there has been some speculation by current and former military personnel that either the incorrect revision of software had been installed, or that (indeed) some parameters were used that were incompatible with whatever software they had been using during the maiden flight. Or a mix of both.

    However: Airbus is awfully silent about what did lead to the inability to retract the flaps after take-off, which the doomed crew had reported by radio.

    A "software installation problem" during final assembly is awfully convenient for Airbus, too. With that they say: "The software is not the problem. The installation was. (i.e.: human error) All we need is better Q/A.". And also: "The engines themselves aren't the problem."

    But fact of the matter is: Airbus has (since long) lost the ability to integrate turboprop engines into airframes. Or to design airframes around turboprops. For the A400M they had to start on a clean sheet for that. It would not be surprising if the engines of the A400M (and their integration into the system as a whole) contain some unaddressed issues that haven't yet been fully ironed out.

    The German ministry of defence also doubted the statement of Airbus and said that if the software was glitching, the engines should have continued to operate on their last power setting instead of shutting down.

    Bottom line: Airbus is in all out damage control mode and it can be speculated that the entire (sad) truth hasn't come out yet.

  22. daddyo

    And we are assured that Entertainment Bus is isolated from flight control

    On the other side of the pond, a security researcher is in BEEG trouble for either patching into a commercial airliner's engine controller system from his seat back and making the aircraft yaw, or he just made up that story and tweeted it, and got kicked off the flight and then corralled by the FBI with some major league criminal charges. http://www.theregister.co.uk/2015/05/19/airplane_hacking_panic_why_its_a_surely_a_storm_in_a_teacup/

    There is substantial dust-up as to whether it could have happened, even if he didn't actually do it on a real flight.

    And the use of the CRM 114 discriminator absolutely prevents any errors and ensures General Ripper's commands are adhered to. Does the ARINC 629 provide the same protection.

    1. Chris Miller

      Re: And we are assured that Entertainment Bus is isolated from flight control

      Military transport aircraft (the dedicated ones, rather than airliners painted grey) don't usually have IFE.

  23. Dagg Silver badge
    Pint

    I blame Agile

    This is what happens when you use agile to manage a project. You get something just good enough based on requirements that wander all over the place with QA that only tests the best cases....

    I need a beer!

  24. Will Stephenson

    That's amazing. I wonder where Google gets the 'filled up' from though - I don't know of any uses of 'spielen' meaning 'to fill'.

    'aufspielen' is the proper German vocab for 'to install' in the context of software. It could include both installation and configuration. It has nice retro connotations of installing software by sticking a reel of on a magnetic tape drive and threading the header around the rollers.

  25. Munkstar

    A first for Airbus? The admission that is.

  26. ian 22

    "Last week, Airbus CEO Tom Enders had complained that the company needed the black box data, which was being withheld by Spanish agencies, to conduct its analysis of the crash."

    Aha! So that's Enders game is it.

  27. Anonymous Coward
    Anonymous Coward

    Tony Hoare in the Turing Award lecture in 1980 said...

    "there are two ways of constructing a software design: One way is to make it so simple that there are *obviously* no deficiencies and the other way is to make it so complicated that there are no *obvious* deficiencies"

    The FADEC systems I saw in the 1980s and early 90s were in category 1: as simple as possible (e.g. no loops, no compilers, etc).

    Times have apparently changed. Engineering best practice has apparently been replaced by cost effective, productive, etc. For now.

    https://www.cs.fsu.edu/~engelen/courses/COP4610/hoare.pdf

    Longer version (same concept, different presenter):

    http://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf (55 slides)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like