back to article Aviation's been Boeing through a rough patch: Software tweaks blamed for Airbus A220 failures

Software alterations have been fingered as the cause of Airbus A220 airliner problems that led to at least three emergency landings after excessive vibration caused engines to fail, according to reports. Financial newswire Reuters reported that "recent changes in engine software ... may have caused parts that compress air …

  1. Anonymous Coward
    Anonymous Coward

    Call to all coders

    Er, guys, can you please stick with mobile phones and leave airplanes alone ?

    1. DougS Silver badge

      Re: Call to all coders

      I imagine there may have been some coders on the MCAS team who tried to tell their managers that not having a redundant sensor reading was a disaster waiting to happen, but the guys with the MS Project timeline to meet ruled the day.

      1. Anonymous Coward
        Anonymous Coward

        @DougS - Re: Call to all coders

        No, those who tried to blow the whistle were engineers or pilots.

        1. DougS Silver badge

          Re: @DougS - Call to all coders

          You have a full list of everyone who tried to blow the whistle, do you?

          1. Anonymous Coward
            Anonymous Coward

            Re: @DougS - Call to all coders

            No, why? Are you on that list ?

  2. Yet Another Anonymous coward Silver badge

    Just wondering

    Which federal agency supervises the testing of US built aircraft engines?

    1. DougS Silver badge

      Re: Just wondering

      I'm not sure aircraft engines are independently tested, they are part of an aircraft and need to be tested as part of that aircraft.

      1. Crazy Operations Guy Silver badge

        Re: Just wondering

        Engines are independently tested, twice even. Once in approval of the engine itself, then a second certificate indicating its suitability with a specific aircraft type as part of the aircraft's type approval. The manufacturer also receives certifications of their own indicating that their manufacturing processes are sound, they have appropriate testing facilities, they perform their due diligence in auditing suppliers of components, the employees building the engines have the appropriate certifications of their own, and so on. Parts of the supply chain also need to be certified (Although those certificates are handled by other agencies).

        1. Yet Another Anonymous coward Silver badge

          Re: Just wondering

          So exactly the same process as Boeing's software ?

          1. Anonymous Coward
            Anonymous Coward

            Re: Just wondering

            "So exactly the same process as Boeing's software ?"

            Sounds like it. After all, everyone trusts the suppliers, otherwise there'd be no such thing as self certification (applies outside the aircraft industry too).

          2. Crazy Operations Guy Silver badge

            Re: Just wondering

            The problem with MAX's is that Boeing was allowed to determine whether they could reuse the type certification from the -700/800/900 model series. This is normally allowed when the proposed modifications do not modify the aircraft's performance and handling characteristics significantly like when the 900ER was rolled out. However the MAX series, which used larger engines that were moved forward, produced significant changes to handling, which are corrected with the MCAS system (Which was approved as a pilot aid, rather than as a critical component to keeping the aircraft properly trimmed). Boeing never should have been allowed to make that determination.

            The new type certificate would have required much more rigorous testing that would've likely caught the problem and redundancy would have been mandatory, or the FAA could've reached the conclusion that the aircraft is unsafe by design and denied Boeing approval until they redesigned the system so that the aircraft can perform properly without no matter what failure mode the MCAS system experienced. This would have also required that pilots operating the MAX series to have received a new round of training that would have included awareness of the MCAS system and troubleshooting issues with it.

            In my opinion, any aircraft that requires electronic system making calculations to maintain stability is an aircraft that needs to stay on the ground until it it gets recycled. I'm okay with fly-by-wire systems since they are designed to fail-safe so that the controls directly modify the position of the control surfaces instead of being put through a computer that prevents moving the control surfaces to unsafe positions.

            1. swm Silver badge

              Re: Just wondering

              Some new military airplanes are unstable and cannot be flown without the software to stabilize them - the pilot's reaction times are not fast enough.

              1. Anonymous Coward
                Anonymous Coward

                Re: Just wondering

                That is true, but those airplanes all come with ejector seats.

                1. DougS Silver badge
                  Devil

                  Re: Just wondering

                  I wish commercial airlines did. Then if you had an annoying seat mate you could reach across and pull their ejection lever!

                  1. Anonymous Coward
                    Anonymous Coward

                    I wish commercial airlines did.

                    Hey Mom, what does this dooooooooo........

                  2. herman Silver badge

                    Re: Just wondering

                    Hmm, except that if you would do that, you would lose your arm and likely a few other miscellaneous body parts as well.

                    1. Michael Wojcik Silver badge

                      Re: Just wondering

                      "System testing revealed some issues not apparent in unit tests."

              2. MrReynolds2U

                Re: Just wondering

                Aye... Eurofighter/Typhoon for example.

                Designed to be aerodynamically unstable for maximum manoeuvrability.

            2. sanmigueelbeer Silver badge

              Re: Just wondering

              The problem with MAX's is that Boeing was allowed to determine whether they could reuse the type certification from the -700/800/900 model series.

              The job is with the FAA. Unfortunately, a lot of FAA "staff" are not really FAA staff. A lot of Boeing certification are made by Boeing staff seconded to work at FAA. So the FAA certification are handled by Boeing staff. It is not really "ideal" but trying to maintain talent is getting more and more difficult with an always-shrinking budget.

              The issue with the Boeing MAX was that Boeing knowing & deliberately kept key documents away from the FAA so the MAX can be re-certified quicker (two- to three years faster). FAA was keen to "rubber stamp" the MAX.

              This would have also required that pilots operating the MAX series to have received a new round of training that would have included awareness of the MCAS system and troubleshooting issues with it.

              Read this: Boeing Was ‘Go, Go, Go’ to Beat Airbus With the 737 Max. This explains why Boeing did what they did: Boeing needed to bring MAX out NOW (and not 3- or 4 years later) in order to compete with the NEO. This means Boeing had to cut a lot of corners (which resulted in death).

              But y'know what is sad about this? No one goes to jail.

              So much for America's much vaunted "do the crime, do the time".

              NOTE: U.S. FAA revokes certification of Xtra Aerospace after Lion Air crash

              1. Anonymous Coward
                Anonymous Coward

                Re: Just wondering

                Really, had the pilots involved ever learned how to use the trim button instead of trying to muscle the plane's controls it would have been a big annoyance, as occurred in the first incident, rather than the deadly disasters of the second and third incidents. Even the second incident was ok until the captain handed off control to the first officer to flip through the operations book rather than, wait for it, just flying the plane. Declare a mayday and radio to company headquarters for the chief pilot to dig though the book. Or ask any other pilots for help while concentrating on keeping the plane above the ground.

                Fortunately, MCAS will be fixed, even if pilot attitude and training never will be.

                1. SW10

                  Re: Just wondering

                  had the pilots involved ever learned how to use the trim button instead of trying to muscle the plane's controls

                  Remembering always that MCAS was introduced “on the quiet” and that its purpose was to change trim behind the the pilots’ backs

                  https://theaircurrent.com/aviation-safety/what-is-the-boeing-737-max-maneuvering-characteristics-augmentation-system-mcas-jt610/

                2. imanidiot Silver badge

                  Re: Just wondering

                  "Really, had the pilots involved ever learned how to use the trim button instead of trying to muscle the plane's controls it would have been a big annoyance, as occurred in the first incident, rather than the deadly disasters of the second and third incidents."

                  Please read the incident reports again before spouting non-sense. The pilots (correctly) diagnosed a trim issue and set the trim cut-outs to OFF. Which means the electric trim switches on the yoke don't do anything anymore either. Due to the heavy down trim already applied however (and the huge control forces thus required) the manual trim was also jammed pretty much solid at that point. The pilots then (incorrectly) decided to try re-engaging the trim cut-outs at which point another MCAS cycle occured and put them into the ground.

                  Handing off control to the FO is normal crew resource management and entirely according to training. Either pilot should be competent in flying the plane. There is/was no reason to assume the hand-off from Captain to FO had any influence on the outcome of the flight.

                  As to contacting someone else, Aviate, Navigate, Communicate. They had their hands full with the first bit, why would they complicate that further by trying to communicate the problems to someone far away unfamiliar with the problems encountered?

            3. Electronics'R'Us Bronze badge
              Mushroom

              Re: Just wondering

              The problem with the MAX is that MCAS is a 'bolt-on' to an otherwise manual control aircraft and not a fully tested and designed flight control system (the first Boeing civil aircraft to get that was the 777 IIRC). MCAS is necessary to meet Part 25 rules for certification (without it, the aircraft violates a rule that a given stick force results in a linear change in AoA at all points in the flight envelope).

              As such, it was not properly evaluated or DAL rated (it can move flying control surfaces to the point the pilot has insufficient control authority to override it - it should be DAL A) although it would not pass the sniff test for any competent (and independent) safety authority (regulatory capture is a real problem).

              Modern military aircraft are indeed unstable and require flight control computers (the B2 has no vertical stabiliser, for instance and the Typhoon is controlled by canards) but that instability has other advantages (in the case of the B2 it removes an entire potentially radar reflective surface).

              The engineering failures with MCAS are astounding and disturbing but ultimately it seems that getting the aircraft out for sale trumped any safety issues.

      2. anothercynic Silver badge

        Re: Just wondering

        Of course they are independently tested! How do you think are they built? You don't expect Rolls-Royce, CFM, GE or Pratt & Whitney to own one model of each jet that their engines are mounted to, do you?

        And yes, the engines are also tested as part of the first flight for each airframe. Yes, *every* plane gets flown at least twice (once by manufacturer crew, once before acceptance by the client crew) before it gets handed over.

    2. Danny 2 Silver badge

      Re: Just wondering

      "Which federal agency supervises the testing of US built aircraft engines?"

      I'm guessing Boeing.

      Boeing pilots' messages on 737 MAX safety raise new questions

      Forkner said in one text message, “I basically lied to the regulators (unknowingly).” The other employee responded that “it wasn’t a lie, no one told us that was the case” of an issue with MCAS.

      1. anothercynic Silver badge

        Re: Just wondering

        Boeing is not a federal agency.

    3. Reg Reader 1

      Re: Just wondering

      I was wondering the same thing. But in a world of unrestricted free market capitalism there is less or no governmental oversight. People will say to this that the market will sort itself out, but when there are only a few big players and they are all doing the same thing then there is little chance of change because there are no alternatives.

      1. Anonymous Coward
        Anonymous Coward

        Re: Just wondering

        "Free market" is just the cover story for what is now fully crony capitalism. I'm just trying to figure out if companies capturing the government is the same as government controlling all the companies.

        Seems like what you wind up with is communism or fascism either way, Total control by the few, whatever you call it.

        Name a law or regulation in the past few decades that wasn't easy for a big corp to comply with using only their pocket change, but which costs enough to ban upstart little guys. I doubt you can.

        Anyone, can, for example, do the same tax dodges the big tech (and other) outfits do - that stuff is legal at present. But to be legal, you have an office in the various countries involved in doing the shuffle. Which is far past the means of most taxpayers.

        Actually, it's a very long list.

  3. tip pc Bronze badge

    Both engines?

    Did this fault affect both engines or just 1?

    FADEC rules should mean that both engines control systems are independently developed of each other so would be interesting to understand if both engines where impacted the same.

    1. Crazy Operations Guy Silver badge

      Re: Both engines?

      Assuming both FADECs are on different software versions, but I wouldn't be surprised if both have had the same modifications made to prevent a thrust mismatch between engines.

      Given that the PW1000 engine series is a geared-turbofan design, I'm assuming the software change is to expand the range where the lower gear ratios are used, but accidentally deviating outside of the safe range for air stream densities entering the compressor stage and causing an over-pressure situation causing cavitation on the compressor blades. Or maybe a back-pressure situation causing the fan's vanes to cavitate.

      But thats the problem with squeezing your margins for efficiency, sometimes you end up with your performance curves crossing to points outside of your new margins...

      1. Anonymous Coward
        Anonymous Coward

        Re: Both engines?

        " I'm assuming the software change is to expand the range where the lower gear ratios are used"

        And I'm assuming that it doesn't work like that. But ICBW, citation welcome.

        Afaik the gear ratio is fixed at design time. The geared turbofan has no equivalent to "changing down" (or up) like there is in road vehicles.

        1. Wellyboot Silver badge

          Re: Both engines?

          I think it's a straight 3:1 ration between the 'jet' & 'fan' parts of the engine to allow the fan blade design to be optimised for lower rotational speeds where the fan tips stay subsonic. But ICBW also.

        2. Crazy Operations Guy Silver badge

          Re: Both engines?

          Its less a gear change like a transmission and more of a governor like you'd see in modern turboprops. I too may mistaken, I am going off of what engineers have explained to me by some GE's turbine engine R+D engineers that had worked on the LEAP (We are members of the same EAA chapter). I tend to trust them as they hold zero loyalty to their employer and instead equip their craft with engines from multiple different manufacturers even though a GE engine would suffice.

          They also seem to have permission to grab whatever they want from the GE Boneyard, one of them has a Velocity V-Twin with a Rockwell-Collins ProFlight flight deck that came off a King Air).

          1. Crazy Operations Guy Silver badge

            Re: Both engines?

            I looked back on the chapter's internal forums, apparently they were talking about a proposed engine that was shelved early in development and they were talking about starting on it again as an alternative to a variable-diameter bypass design they've been working on.

          2. anothercynic Silver badge

            Re: Both engines?

            LEAP works differently to the PW geared TF. The PW uses a planetary gear box to reduce the rotational speed for the fan compared to the low-pressure turbine that drives it (and the LP compressor right behind it for the core).

            RR did the right thing when they developed the RB211 (from which all modern Trent engines are derived). That was the first commercial three-spool design (and three shafts within each other), so that the HP turbine drives HP compressor, IP turbine drives the LP compressor and LP turbine drives the fan with *no* gearing needed. They patented that if I remember correctly.

            PW is trying to do the same thing with two spools... it's a design that's been described as too complex and too heavy, and now too problematic, but hey...

      2. boltar Silver badge

        Re: Both engines?

        "causing an over-pressure situation causing cavitation on the compressor blades. Or maybe a back-pressure situation causing the fan's vanes to cavitate."

        Err, wtf are you talking about? How exactly do you get cavitation in a gas? Cavitation is specifically a liquid rapidly turning into a vapour/gas due to sudden pressure loss so unless the blades are bathed in neat JA1 I fail to see how this can be occuring.

      3. Electronics'R'Us Bronze badge
        Holmes

        Re: Both engines?

        The FADECs are very probably identical with redundancy and dissimilarity internal to the FADEC itself. This piece of kit is probably manufactured by one of the usual suspects.

    2. Anonymous Coward
      Anonymous Coward

      Re: Both engines?

      "FADEC rules should mean that both engines control systems are independently developed of each other so would be interesting to understand if both engines where impacted the same"

      Can you imagine how much that would cost? Look at the MCAS saga for how cost-sensitive Boeing think their customers are.

      Can anyone find readers somewhere in DO254, DO178, etc that mandates dissimilar redundancy?

      1. SkippyBing Silver badge

        Re: Both engines?

        My understanding is that the engines have dual control channels so if one fails you can swap to the alternate one. These control channels may be developed independently, but each engine has both.

        This is all based off the engines in the type I work with so may be completely irrelevant in this case.

      2. Anonymous Coward
        Anonymous Coward

        Re: Both engines?

        There are also papers (can't find a reference at the moment) that argue N-diversity development may not be as much help as was once thought.

        In the past, projects where this has been done have all started from the same set of requirements, meaning that up-front design issues are faithfully reproduced in all implementations. Writing the software is the "easy" part - getting the requirements correct isn't!

        N-diversity may help reduce the risk of an implementation defect causing a failure, but those sort of defects are generally caught during testing - especially when behavioural differences between implementations are observed.

  4. Androgynous Cow Herd

    "Clearly, any time you get an issue like this, we're on top of it. "

    "because for Gawds sake you don't want to be under it..."

  5. Wellyboot Silver badge

    >>>this would mean running at hotter temperatures, thus increasing maintenance costs<<<

    This almost certainly involves burning extra fuel, with extra maintenance why would airlines even consider that? This is a modern airliner and hardly has to drag itself into the sky.

    P&W are risking their reputation messing with the engine like this so early in its development, the FAA has already reduced the hub life limit on these engines after corrosion was found. Airbus could easily switch to another engine like the latest CFM56 used on B737max & A320neo types.

    1. Steve Davies 3 Silver badge

      re: Switching Aircraft Engine Types

      is nigh on impossible for Aircraft that are already out there flying.

      Some Airlines prefer to stick with on engine supplier so the maker has to soak up the cost of putting that engine onto their airframe and modifying all the associated control systems. That's why 'launch engine' is such a kudos point for an engine maker. They'll work closely with the airframe maker to get it right but sometimes... things go a bit wrong.

      The R-R Trent engines on the Quantas A-380 is a good example. Only after the aircraft had seen a good number of operational hours did problems develop. A lot of midnight oil was burned in Derby fixing that problem.

      Then there is the whole other elephant in the room and that is spare parts. It is so, so easy to buy spares that are later found to be fakes. These can and do cause crashes. Authenticity of supply is a real problem in the Commercial Aviation world where Airlines are always looking to save a few £££/$$$ etc.

      1. Wellyboot Silver badge

        Re: re: Switching Aircraft Engine Types

        I was thinking of the 500ish outstanding A220 orders that could possibly be switched to the newest CFM engines, that engine is on other current Airbus models and I don't think it'll cause positioning problems here like it did with the 737max.

        Either way PW is on the hook for a large wedge of cash to get these engines fixed.

        1. werdsmith Silver badge

          Re: re: Switching Aircraft Engine Types

          The design of the 220 is just a renamed Bombardier C series so there is not likely to be much engineering compatibility.

        2. anothercynic Silver badge

          Re: re: Switching Aircraft Engine Types

          Aviation companies hate heterogenous fleets.

          It's a nightmare for their maintenance operations. Some however *have* done this... Qatar Airways was one who told PW to take a running jump after the 'unacceptable' 6 minute shutdown/startup delay requirement for the PW engines on the A320neo.

          If an operator already has a CFM fleet, it's easier to switch to that than if they didn't...

      2. Anonymous Coward
        Anonymous Coward

        Does the airline even own the engine?

        A lot are effectively sold "as a service", with telematics continuously streaming engine performance data "back home" so that maintenance issues can be identified before they lead to in-service failures.

        The manufacturer is responsible for all the maintenance costs (and repairs), with the intent being that the "rental" income is greater on average over the deployed fleet.

    2. Peter X
      Stop

      I would've thought the easiest/safest solution at this point, is to revert the software update? I suppose that would mean admitting an "issue", but ploughing ahead looks like worse idea to me.

    3. Phil O'Sophical Silver badge

      This almost certainly involves burning extra fuel,

      Not necessarily. If you burn fuel at lower than stiochiometric ratio you get hotter combustion, it's a problem in lean-burn car engines because it increases NOx output.

  6. Anonymous Coward
    Anonymous Coward

    Are these engines running Windows 10?

    Just checking.

    1. Korev Silver badge
      Coat

      Re: Are these engines running Windows 10?

      Not sure; but I'm pretty sure they use the Jet database underneath...

  7. hammarbtyp Silver badge

    Usual sw solution

    Did they try switching them on and off again?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019