back to article You're not Boeing to believe this, but... Another deadly 737 Max control bug found

Yet another deadly and baffling safety flaw has been uncovered in Boeing's 737 Max line of airplanes. The US manufacturer on Wednesday confirmed that, during simulator tests on the embattled jetliners, the 737 Max's new control software would lock up a microprocessor resulting in the plane automatically entering a dangerous …

  1. Tom 64

    >"The safety of our airplanes is Boeing’s highest priority"

    No, it just isn't is it. Or the warning light that indicates the failure of the system wouldn't have been an $80,000 optional extra. You've got blood on your hands Boeing.

    1. LDS Silver badge

      What was really said

      "The safety of our airplanes [very low voice]profits is Boeing’s highest priority"

    2. ElReg!comments!Pierre Silver badge

      "The safety of our airplanes is Boeing’s highest priority,"

      As we all know, this is the standard way to say "we couldn't possibly care less, but we can't say that out loud, can we".

    3. hammarbtyp Silver badge

      "The safety of our airplanes is Boeing’s highest priority" now we've been found out

    4. 0laf Silver badge
      FAIL

      "The safety of our airplanes [that has been outsourced to the cheapest bidder] is Boeing’s highest priority"

      I'm not sure Agile development really works in aircraft design. Fail fast and fail often /eternal beta doesn't sound like a great idea with big fast metal flying things filled with people.

      Just saying...

      1. hammarbtyp Silver badge

        For safety critical, you are probably right. The waterfall model is much maligned, partly because it was badly implemented and explained, and for safety critical systems it is more appropriate.

        The truth is the biggest danger is adopting methodologies in blind faith with a poor understanding of the background and reasoning behind them.

        1. Archtech Silver badge

          Follow the money

          "The waterfall model is much maligned, partly because it was badly implemented and explained, and for safety critical systems it is more appropriate".

          It has been very well and clearly explained in many places, although it does tend to be more complicated than agile methods.

          The waterfall model has been much maligned mainly because so much money and prestige was to be made out of inventing the "latest new new thing" - agile.

          Just look at the number of semi-qualified hucksters trying to launch careers and make millions by talking about "agile", when they can barely spell it.

          Ask people like Kent Beck, Ward Cunningham, Dave Thomas or Martin Fowler - who actually proposed and pioneered agile methods - and even they will agree that the movement has become greatly diluted by charlatans.

          1. Paul 195

            Re: Follow the money

            Waterfall works reasonably well if you understand the requirements and problem space up front. Which for problems like "aeroplane should not fall out of sky" is probably the case. It does not work well when the thing being designed has to cope with constantly changing requirements, which often happens with commercial or enterprise software. Here Agile definitely works better. There are no silver bullets. Understand your tools and what they are applicable for, and don't use your hammer to put in screws.

        2. cdegroot

          The funny thing is that you can - and probably should - do a development style that gives you increments and quick feedback (I'm careful around the term "agile" these days since the consultants kidnapped and raped it); it's probably gonna be a style of work that's much heavier on initial specifications and testing, etcetera; but most of the principles would apply (I know - I worked in this style in heavily regulated industries which were heavily regulated because people would otherwise die).

          The issue is that agile does not mean "deploy crap to production and let your customers be your QA staff". It means that you iterate, learn, and that way develop the code that your business/customers needs - including the level of quality required. Re-read the manifesto for agile software development. Nothing there about "ship fast and fail fast", that's only appropriate in some contexts.

          1. asdf Silver badge

            except

            I submit that the 737 MAX is the ultimate example of endless incremental changes with far too much technical debt which is why its in the pickle it is. That airframe should have had a bullet put in its head awhile back. Avoiding the cost of waterfall is not always the best idea either.

        3. boltar Silver badge

          Sticking to one particular development methodology all the time is project management for people who can't manage. Its easier for these pointy heads to just follow a checklist than to actually think.

      2. Updraft102 Silver badge

        Proposed edit

        "FTFY" seems so arrogant, so how about a proposed edit instead?

        "I'm not sure Agile development really works in aircraft design. Fail fast and fail often /eternal beta doesn't sound like a great idea with big fast metal flying things filled with people.

        1. Anonymous Coward
          Anonymous Coward

          Re: Proposed edit

          ""I'm not sure Agile development really works. Fail fast and fail often / eternal beta doesn't sound like a great idea"

          Really boils down to "throw enough shit at a wall and some of it is going to stick". (But, in the end, it's still shit.)

        2. Anonymous Coward
          Anonymous Coward

          Re: Proposed edit

          There is a good reason why I called 'agile' "Fragile". 90% of the releases of one project I worked on were total crap and nowhere near good enough for an 'Alpha' release from the old waterfall system. By the time we were supposed to ship the finished product it was nowhere near ready. Then the automated build process fell in a heap and no one realised for a week.

          Two of us took the code (basically forked it) and went offsite for two weeks out of reach of the rest of the team and especially the people running (sic) the agile process.

          We returned with a product that worked and was solid enough to go out for field testing.

          Most of the people who championed Agile left shortly after that and moved on to wreak havoc at other comanies. We returned to a more sane development process and everyone was a lot happier. Deadlines were met with ease if you have a team that is motivated and not dictated to by the process.

      3. anoncow

        Here's the problem with a bunch of software hacks commenting on an aviation issue: regardless of the relative strengths or lack thereof of the development and testing methodology, this software would not have been needed at all, were the Max 8 airframe inherently stable. Which it is not, with horrific consequences.

        1. NeilPost Bronze badge

          It’s been well reported on the background of the Max8 airframe with the Comedy Tits sized Engines hanging off it and the actual need for the software so the damn thing can be flown.

    5. Annihilator

      Word missing

      "Apparently, the safety of our airplanes is Boeing's highest priority"

      is the more accurate quote.

      1. Brit Flyer

        Re: Word missing

        The safety of my $24 million salary and bonus....$$$$$$

    6. macjules Silver badge
      Mushroom

      "The safety of our orders book is Boeing’s highest priority"

      There, FTFY

      Icon: fit for Boeing

      1. Anonymous Coward
        Anonymous Coward

        For what it's worth

        The Enola Gay was a Boeing product.

        1. oldfartuk

          Re: For what it's worth

          Indeed. But then so was space shuttles Columbia and Challenger......

          1. Mine's a Large One

            Re: For what it's worth

            The Shuttles were built by Rockwell (bought by Boeing in 1996)

    7. caffeine addict Silver badge

      With all of these planes grounded, do the airlines have any grounds (ha!) to sue Boeing for costs?

      That's the way to make them do things differently next time. A couple of million quid fine from the FAA is nothing. A few billion in lost revenues from airlines the world over? That's a whole different prospect.

      There's never been a better advert for Airbus.

      1. Updraft102 Silver badge

        There's never been a better advert for Airbus.

        That's the thing. I've long preferred Boeing because Airbus believes that the plane should override the pilot when it (mistakenly?) "thinks" the pilot is screwing up. Boeing was supposedly the one that believes that automation is there to assist the pilot, not to overrule him and substitute its judgment for his, and yet here we are. This situation shows why the "Boeing" approach is superior, but Boeing, unfortunately, didn't use the Boeing approach in the 737 Ma.

        Airbus has had its own incidents where the AI killed, or tried to kill, everyone on the plane.

        Air Inter 148 in 1992, an Airbus A320, crashed into a mountain on descent into Strasbourg when the autopilot was set (in error) to a rapid 3300 ft/min descent. That was pilot error (he had meant to dial in 3.3 degree descent), but analysis revealed that the plane had descended even more quickly than 3300 ft/min, and if it had done as it was told, it would have cleared the mountain.

        It turns out that there was a "safety feature" on the plane, unknown to the pilots (sound familiar?), that would cause the plane to descend faster than what the pilot had dialed in if the plane sensed a sharp increase in altitude right before the descent was selected in the autopilot. It turns out there had been a little bump from air currents just before the pilot dialed in 3300, and the plane took this to mean it was an emergency, and it "helpfully" descended more than 3300 ft/min, right into a fog-shrouded mountain.

        Qantas 72 in 2008, an Airbus A330, repeatedly tried to dive toward the ground, pitching down so sharply that passengers and flight attendants were twice thrown upwards hard enough to smash the plastic overhead panels, causing a number of severe injuries. Had the pilot not reacted heroically, the entire plane would have crashed. The incident it very nearly ruined Qantas' record of never having had a fatal incident. The cause was very similar to that of the 737 Max incidents.

        Several aviation experts have said that if either of the Boeing 737 Max incidents had occurred with a pilot with a level of training typical of western airlines, the flights would have been recoverable. Qantas 72 had such a pilot, and only barely managed to survive. It stands to reason that this incident was equally as bad as what happened with the 737 Max, but "almost crashed" doesn't get the attention of a crash where everyone on board was killed.

        1. Korev Silver badge

          Several aviation experts have said that if either of the Boeing 737 Max incidents had occurred with a pilot with a level of training typical of western airlines, the flights would have been recoverable.

          This article suggests otherwise.

          “The crew performed all the procedures - repeatedly - provided by the manufacturer, but was not able to control the aircraft,” said [Ethiopian transport minister] .

          1. Anonymous Coward
            Anonymous Coward

            What the Ethiopian minister said is not true. That much is apparent just from the preliminary report. They never performed the UAS checklist (which would've forced them to reduce engine power to 80% from takeoff thrust), their execution of the Runaway Stabilizer checklist wasn't done well (really, they just did one step - the most important one, but it was still haphazard) and I don't even know what "repeatedly" is supposed to mean in that sentence.

            The pilots did not perform as well as could be imagined. Whether they performed as well as could be expected is a matter of some debate in pilot forums.

            The Ethiopians are no more willing to admit fault than Boeing is. Nothing unexpected about that.

            1. Anonymous Coward
              Anonymous Coward

              Either way.

              Having a fault by design, is just a waiting game on failure. If you ask 100s of pilots to constantly nanny the broken by design system, eventually there will be a crash.

              The problem should have been mitigated or reduced long long before release.

              1. Kabukiwookie Silver badge

                Re: Either way.

                The problem should have been mitigated or reduced long long before release.

                That's not good enough. It should have been fixed and if it can't be fixed the airframe should have been scrapped.

                That's the only sane course of acrion if you are serious about the lives of the people flying in the aircraft that you're selling

                Where human life is concerned there should not be any grey areas.

            2. J. R. Hartley Silver badge

              All the pilots had to do was flip the stab trim cutout switches and trim the aircraft manually. At the speeds they were going it would have been difficult to turn the wheels, but they didn't slow down.

              The AoA sensors failed, but it was pilot error that caused the crash.

              1. Anonymous Coward
                Anonymous Coward

                "All the pilots had to do was flip the stab trim cutout switches and trim the aircraft manually. At the speeds they were going it would have been difficult to turn the wheels, but they didn't slow down."

                With MCAS disabled, all power assistance for the stabilisation functions were also disabled. To achieve the trim adjustments, they had to execute a "roller coaster" where they dropped altitude to reduce the air pressure over the control surfaces and reduce the required force.

                At low altitude.

                In a plane that was not reacting how they expected.

                1. 2+2=5 Silver badge

                  > With MCAS disabled, all power assistance for the stabilisation functions were also disabled. To achieve the trim adjustments, they had to execute a "roller coaster" where they dropped altitude to reduce the air pressure over the control surfaces and reduce the required force.

                  The reason they were going so fast is because the pilots didn't disable the auto-thruster (which they should have done at the same time they disabled the auto-pilot). If they had done so and were travelling at normal flight speed then they would have been able to manually trim the plane with MCAS switches off, no special manoeuvers required.

                  However...

                  Boeing makes much of their claim that the pilots of Boeing's are pilots not bus drivers. But the reality seems to be rather different: the pilot pulling back on the stick with all his might does not cause an override to be signalled to the systems. The trim wheels are motorised so strongly that the pilot can't physically stop them with his hand. All those potential inputs where the pilot is telling the computer to do something different are being ignored.

                  And because of that, the pilot is tricked into thinking he needs to do some fancy manoeuver and his attention is diverted there, whereas all he needed was to be in control of the flight surfaces, be able to fly straight and level and he would have soon realised that the auto-thrusters had been left on.

              2. Peter2 Silver badge

                From the article a couple of comments above your post (https://www.bbc.co.uk/news/extra/sd9LGK2S9m/battle_over_blame)

                Boeing published a bulletin in which it described the effects of an MCAS malfunction, and instructed pilots to follow a particular “non-normal checklist” designed to help them cope with uncontrolled stabiliser movements.

                This checklist - which is meant to be memorised by flight crew - instructed them to flip switches on the centre console, to turn off the stabiliser electronics, then balance the aircraft using manual trim wheels beside the pilots’ knees.

                The Ethiopian crew tried to follow this procedure. They turned off the electronics and attempted to “trim” the aircraft - to bring it back into level, balanced flight - using the hand controls. But the preliminary report suggests that they were physically unable to do so.

                It appears the aircraft was simply going too fast, and the aerodynamic forces building up on the stabilisers were too strong for the pilots to overcome with muscle power.

                So that's pilot error? Oh wait, you say that they should have slowed down too. They covered that in the article too.

                In other words, the pilots were faced with a situation where they couldn’t control the aircraft unless they tried to slow down - but doing so could push it into a catastrophic dive.

                It’s possible, then, that the crew didn’t reduce thrust because they simply didn’t dare to do so.

                Capt Chris Brady, himself a pilot and the author of a technical website devoted to the Boeing 737, endorses this theory.

                “In these circumstances, you’re really caught between a rock and a hard place,” he says. “They may have wanted to reduce thrust, but when the aircraft is already very low and nose down as well, you’d have to have balls of steel to do so.”

                Yep, totally pilot error. Of course it had absolutely nothing to do with the plane being unsafe and that's why thousands of 737 max flights take place everyda... What, the FAA demanded that they all be grounded since the plane as it stands is a deathtrap? This comment is on an article explaining the FAA has discovered another fatal control bug that would probably have killed another few hundred people and been blamed on the poor sap flying it?

                Nah, definitely pilot error caused by non American pilots. Especially if Ethopian Airlines actually had a very good standard of training, had bought simulators from Boeing (which many American airlines haven't done, and they had the only 737 max simulator in Africa)

                Yet one thing the simulator cannot yet do is replicate the circumstances of that accident or the previous crash off Indonesia. In late May, Boeing admitted that software provided to simulator operators was flawed, and incapable of reproducing some flight conditions, including the failures experienced by ET302.

                But come on, all together now, "IT'S TEH PILOTS FAWLT!!11111!11"

                Boeing is at least 90% responsible, with the 10% responsibility going to the pilots who didn't manage to figure out an undocumented way of preventing a plane ignoring the input from the pilots controls and flying itself into the ground.

                Actually, scratch that. I'm giving the pilots a 10% discount on the responsibility based on the fact that the plane was unstable, the control software was dangerously defective and documented process for recovering the plane from this situation doesn't work IRL, so i'm calling it 100% Boeing's fault. Which is of course why the 737Max is grounded, and probably will be until it's actually safe to fly.

                To go from the fact that this fix obviously wasn't particularly well tested by Boeing would suggest that the 737Max won't be flying anytime soon.

                1. Anonymous Coward
                  Anonymous Coward

                  The AD said to establish trim with the control wheel trim switches prior to shutting off the trim motors.

                  So they blew off step one. On a memory-item checklist; as is, learn it until they can do it in their dreams.

                  And they should not have gotten there as the pitch-power settings for instrument disagree would not have seen them retract the flaps in the first place, allowing MCAS to operate.

                  Those pilots took a compromised, but flyable situation and made sure it was deadly.

                  The FAA grounded it upon the realization that 3rd world pilots were not getting the correct training. Which Ethiopian Airlines was already well aware of when they put those pilots in that plane. ET was more concerned with the appearance of being a first rate operation and gaining praise for its amazing rapid expansion than it was interested in backing it with a dedication to safety.

                  1. Electronics'R'Us
                    Thumb Down

                    Memory type checklist items

                    These are the ones that you train for regularly and often in the simulator as that is the only really robust method for reactions that you must be able to take immediately, but:

                    "Yet one thing the simulator cannot yet do is replicate the circumstances of that accident or the previous crash off Indonesia. In late May, Boeing admitted that software provided to simulator operators was flawed, and incapable of reproducing some flight conditions, including the failures experienced by ET302."

                    From this article

                    How could this be a memory item when it has not been ground into their brains from many hours of practicing for this specific scenario because the simulator could not reproduce this specific scenario?

                    1. Anonymous Coward
                      Anonymous Coward

                      Re: Memory type checklist items

                      "How could this be a memory item when it has not been ground into their brains from many hours of practicing for this specific scenario because the simulator could not reproduce this specific scenario?"

                      And with the 'conversion' from some earlier 737 types to 737 max being as simple and CHEAP as possible, little more than a session on an iPad according to one pilot, with no simulator time 'required' being one of Boeing's main selling points, just how many pilots would have trained on a 737 Max specific simulator anyway?

                2. Mark 85 Silver badge

                  This checklist - which is meant to be memorised by flight crew -

                  I find that to be the most absurd expectation ever.

                  1. Olivier2553 Silver badge

                    To my understanding, that's how it works:

                    - there are things you must know how to do by heart

                    - there are things you must look for in the quick instruction book

                    - there are things where you deploy the big manual

                    First category is supposedly things that are critical and where you have zero time to think before you react.

                3. Justthefacts

                  Plus one other.....

                  Boeing procedure stated in the manual, they claimed they didn’t have to document separately for MCAS failure because it’s treated identically as a subset of Trim Stab failure.

                  But, Ops procedure should be tested in the simulator, like everything. And news just in, is that if Boeing had done so, the software is now known to hang!

                  And Boeing didn’t originally test the fix they just spun because there are only two Max simulators in the US.

                  So we have just effectively discovered Boeing *definitely* haven’t actually tested *any* of the manuals emergency procedures on the Max simulator!

                  Never mind MCAS, this planes operating procedure is essentially untested. All of it.

                  I thought it couldn’t get worse for Boeing, but this is!

                  1. defiler Silver badge

                    Re: Plus one other.....

                    The *new* software can hang a microcontroller. There's no indication that the original software would do this. There's plenty of blame to throw at Boeing without mixing things up.

                    1. asdf Silver badge

                      Re: Plus one other.....

                      Still shows that Boeing hasn't tested this fix worth a crap. They badly want to avoid recertification where possible and this time FAA isn't going to give them a pass because have grand standing Congress critters waiting for them to slip again.

            3. anoncow

              "<racist rant>he pilots did not perform as well as could be imagined<racist>"

              It's hard to overstate how slimy your post comes across, and how wicked a person you must be.

        2. ForthIsNotDead
          Boffin

          "That's the thing. I've long preferred Boeing because Airbus believes that the plane should override the pilot when it (mistakenly?) "thinks" the pilot is screwing up. Boeing was supposedly the one that believes that automation is there to assist the pilot, not to overrule him and substitute its judgment for his, and yet here we are."

          I was under the impression that the reason for that is that the plane is quite unstable in flight due to the engines being moved forward (for efficiency reasons) and thus assistance from computers was *required* to keep thing in air, no?

          Am I over-simplifying it?

          1. TechnicalBen Silver badge

            Almost.

            It's not "unstable", it's just "different". But the system pretends it's not, until it fails. Even if it fully failed, the pilot should be able to fly the aircraft. But if it fails, and the pilot was either told "this aircraft is the same", or is attempting to adjust it by the system thinks "I know better", then the two conflicting inputs are a problem (or the one conflict and it nosedives).

            They either needed to re-train/certify the aircraft, and avoid most of the problems, or make the system much more robust so the problems/faults/conflicts almost never happened.

            1. Peter2 Silver badge

              Re: Almost.

              It is unstable. In a normal, stable aircraft pushing the throttle forwards won't result in the plane pitching the nose up.

              The plane however wouldn't be as unflyable as it is with appropriate training and without the helpful software that flies the plane into the ground.

              It'd just have the the dodgiest set of flying charismatics since the Sopwith Camel's infamous control issues on taking off and landing. (And to be fair the Camel probably wouldn't kill as many of it's own pilots these days given that you could now fly it in sim before flying it IRL...)

              1. SkippyBing Silver badge

                Re: Almost.

                Not quite true, many aircraft have pitch changes with a change in thrust. This is due to the centre of thrust being above or below the centre of drag/gravity. This doesn't make them unstable, continuing to change pitch once the change in thrust had happened would make them unstable.

                The issue with the Max is that at high angles of attack the control force needed to increase pitch decreases due to lift from the engine nacelles. Which means if you hold the same back pressure the aircraft will continue to pitch up rather than stabilising at a new attitude/speed combination. This makes it unstable, and also required MCAS to meet certification requirements.

                The Max actually has fairly benign flight characteristics until you get near the stall.

              2. Aitor 1 Silver badge

                Re: Almost.

                Most jet planes with engines under the wings pitch up when you apply full thrust.

                You see, as you have low wings and engines under the wings, if you apply force (thrust in this case) below the center of mass obviously you are going to pitch up!

                The Max has bigger engines, so they provide more thrust and the center of said force is also lower. Add change of engine position and CG and more pitch up.

                This, itself, is not dangerous, but it is different from other 737s, so therefore they created the MCAS to make it more similar and preserve the type certificate.

                1. JumpinJehosophat

                  Re: Almost.

                  Strange .... Flying a Cessna 172 (high wing, single prop, tiny box), the trim wheel is there for when you need to trim the aircraft. Every change in attitude or thrust needs trim adjustment. Flaps, throttle, cold/warm air ... all these need trim adjustment. Putting on more power makes the aircraft nose rise (and yaw to the left).

                  To increase the airspeed while in straight-and level flight.

                  * Advance the throttle smoothly to the power setting estimated for the speed desired.

                  * Anticipate the yaw to the left with the right rudder.

                  * At the same time apply sufficient forward pressure to the control column to keep the altitude from increasing.

        3. Anonymous Coward
          Anonymous Coward

          "Several aviation experts have said that if either of the Boeing 737 Max incidents had occurred with a pilot with a level of training typical of western airlines, the flights would have been recoverable. "

          Western pilots in western spec 737MAX's have experienced issues with MCAS causing nosedoves and survived. However, the reports from those pilots do not appear to have approached the severity of the issues experienced during the crash.

          There are doubts that the Indonesian/Ethiopian 737MAX's had equivalent hardware to allow pilots to recover the aircraft if they were put into a dive. In particular, it appears that the input from the Lion Air AoA sensor differed by 20° and they were initially blamed on maintenance issues inspite of the sensor being replaced in the preceding 24 hours. For Ethiopian Airlines, the sensor was apparently out by 50° which led to an investigation and the possibility that the sensors could not be correctly calibrated at ground level if certain options were not present.

          The relevant equipment appears to be:

          - the cockpit warning light for AoA disagree. I'm unsure if this is strictly necessary, but it provided a warning of the issue on Western airlines aircraft, allowing them to recover

          - the cockpit AoA indicator.

          - equipment to allow the AoA to be calibrated correctly while the plane was still on the ground

          MCAS being poorly implemented (calibration, redundancy, sanity checking input/output etc) is part of the issue but pales into insignificance compared to Boeing selling aircraft without key components that MCAS or the pilots relied on to prevent MCAS flying the plane into the ground because they didn't understand how the system functioned

          i.e. a technical solution that was supposed overcome the design limitations of the aircraft WAS NOT understood by the manufacturer

        4. SImon Hobson Silver badge
          Facepalm

          Had the pilot not reacted heroically, the entire plane would have crashed

          While it might sound pedantic, that does make me wonder how only part of the plane might crash. I Not meant to be critical, but it creates some funny images in my mind !

        5. jtaylor

          Updraft102 "Airbus believes that the plane should override the pilot... to overrule him and substitute its judgment for his....This situation shows why the "Boeing" approach is superior"

          This is a fundamental misunderstanding. Both manufacturers use "envelope protection" to keep the airplane within safe operating limits. Both manufacturers permit the pilots to (quickly) partially or fully disable envelope protection. Both also have the protection features cease if the safe envelope is exceeded "I can't do this automatically. Human, you have control." The differences are in how they implement this. Broadly speaking, Boeing adds physical resistance to their flight controls to give a "feel" back to the pilot, while Airbus gives less value to control inputs. Each is consistent with the respective control philosophy.

          Air Inter 148...was pilot error...but analysis revealed...that there was a "safety feature" on the plane, unknown to the pilots (sound familiar?), that would cause the plane to descend faster than what the pilot had dialed in if the plane sensed a sharp increase in altitude right before the descent was selected in the autopilot.

          What's your source for this? The FAA report concluded that the pilots selected to control Vertical Speed instead of Flight Path Angle, were not comfortable with the new A320 type, failed to monitor vertical speed, and were sloppy with procedures. Automation was not implicated, though pilot familiarity with control systems was. See p344 in https://lessonslearned.faa.gov/AirInter148/Accident_Report_Eng.pdf p212.

          Qantas 72 in 2008, an Airbus A330, repeatedly tried to dive toward the ground, pitching down...causing a number of severe injuries.

          Interesting. Yes, this was also a failure of pitch control automation, though for different reasons. One of the control CPUs had a fault, the software didn't properly handle the failure, and the result was sudden uncontrolled pitch-down. Unlike MCAS, the aircraft responded to pilot controls. Per https://www.atsb.gov.au/media/3532398/ao2008070.pdf "The occurrence was the only known example where this design limitation led to a pitch-down command in over 28 million flight hours on A330/A340 aircraft, and the aircraft manufacturer subsequently redesigned the AOA algorithm to prevent the same type of accident from occurring again."

          Several aviation experts have said that if either of the Boeing 737 Max incidents had occurred with a pilot with a level of training typical of western airlines, the flights would have been recoverable.

          Citation needed. Which experts have said this? Do EASA and FAA agree and if so, why did they ground the 737MAX across the western world? Ethiopian Airlines has modern equipment and a good safety reputation. It's not authoritative, but see https://aviation-safety.net/database/operator/airline.php?var=6263

          1. asdf Silver badge

            >One of the control CPUs had a fault, the software didn't properly handle the failure, and the result was sudden uncontrolled pitch-down

            Anyone else immediately think cosmic ray? Lot more of them to be found up at altitude.

        6. Stuart21551

          https://www.google.com/search?q=what+a+mountain+goat+doing+up+here+in+the+clouds&safe=off&tbm=isch&source=iu&ictx=1&fir=WKadqs6gS1pB9M%253A%252CtrFBclP986xWkM%252C_&vet=1&usg=AI4_-kTNB0OgwX088Ug2MKtbcWyPm3kycQ&sa=X&ved=2ahUKEwiJhrCPyIvjAhWKrI8KHXaVBq4Q9QEwAHoECAcQBA#imgrc=WKadqs6gS1pB9M:

          1. asdf Silver badge

            That url seems legit. Should have snuck a lemonparty.org in there for a laugh.

        7. JumpinJehosophat

          I can't find anything about that "safety feature". I am aware that the cheapskate company did not opt for ground warning systems, you know "Warning terrain, warning terrain" or "Pull up! Pull up!". What is that safety feature you are talking about?

          I am guessing the AUTO descent rate. The feature ensures compliance with descent to an FL. When the flaps were dropped, while AUTO was set, the aircraft did not descend as much as it should, so the AUTO made the correction to the correct descent speed again.

          THAT HAS NOTHING TO DO WITH WHAT YOU ARE SAYING BTW!!! Any pilot on APP or finals, will put out the flaps. If you DON'T adjust the speed (which was on AUTO) then what happens when you extend them?

          1. The aircrew decided they knew better than the ground - all aircraft were on app one way, they wanted to ILS using the RWY 23 marker then VFR on final to 05

          2. Changing paths & confusion between co-pilot and pilot meant they overshot first approach

          3. Changing back to CTL's first approach meant their programmed descent was too fast

          4. Missing that ANDLO location (left, not right as CTL states it sould have been)

          5. Deciding to ignore -500ft

          6. Deciding to ignore overshot/undershot turns

          7. Not watching ALT or ROD during FLAPS1

          8. Applying airbrakes to slow speed (which increased descent speed) ready for FLAPS2

          9. Overspeeding during crucial APP legs

          10. Generally not following airline approach/flight procedures (wrong screens, talking over each other, not reading out setting changes etc.)

          11.Following airline procedures (overspeeding)

      2. SImon Hobson Silver badge

        do the airlines have any grounds to sue Boeing for costs?

        AIUI, yes - and also AIUI, that's already happening.

        Basic contract law really. Airline buys an aircraft to perform specific duties - based on what the manufacturer says it can do (and being certified for commercial passenger carrying would be a very key point of the specifications). Aircraft doesn't work properly and is grounded - so the airlines have a case against Boing for selling them a product that doesn't meet the agreed specification.

        But having said that, there's no way they'll recover all their losses. There'll be long lasting intangibles such as loss of customer (ie passenger) confidence which could hit the bottom line for years, and the other effect of having moved people to other modes of transport from which they might not return when airline capacity is back to normal.

        1. Anonymous Coward
          Anonymous Coward

          A number of regular flying service engineers I know have made it clear they are likely to turn around and refuse to board if the aircraft is a 737 max, the travelling public will need a lot of convincing it is safe before they will fly on one willingly, the effects will be very long lasting. And now the 787 is also being investigated by the FAA Boeing are going to have a long uphill struggle to rebuild public confidence.

    8. pavel.petrman Bronze badge

      Regarding "The safety of our airplanes is Boeing’s highest priority"

      Strongly reminds me of "Your privacy is very important to us!"

    9. J. R. Hartley Silver badge

      That's it. Never ever ever going on another Boeing again. I refuse to be a beta tester in a world where crashing works on both levels.

      1. Blank Reg

        No but you may be drinking from one. The way things are going with these planes it won't surprise me if many end up being scrapped and turned into beer cans.

    10. J. R. Hartley Silver badge

      Please stop referring to MCAS as an anti stall system. It isnt. It is purely designed to get the MAX the same type rating as the existing 737.

      The big deal is that the only reason they did that is to save money. THAT IS SCARY. There's no other reason for it. The 737 MAX is inherently stable and would fly perfectly well without MCAS.

      1. hottuberrol

        "" The 737 MAX is inherently stable and would fly perfectly well without MCAS.""

        I thought the whole reason for MCAS was that Boeing decided that a software-enabled control fix would be cheaper to implement that redesigning the existing 737 fuselage (and any associated regulatory compliance costs including type-ratings), in order to accommodate the larger donks they now have and how these are forward-slung under the wings. Which implies that without the MCAS, it IS inherently unstable in certain situations. Am I wrong about this ?

        1. J. R. Hartley Silver badge

          Yes, you are wrong about that. Under full thrust in a 737 MAX the nose will lift a bit. That's all. But instead of training pilots and requiring a new type certificate for them to fly it, they designed MCAS to basically correct for this. Any seasoned pilot could fly perfectly well with no MCAS.

          That's why it's so scary. The ONLY reason for MCAS is to save money.

          1. SkippyBing Silver badge

            MCAS doesn't correct for pitch changes with thrust, it corrects for a reduction in stick force near the stall which could lead to an irrecoverable stall. The money saving came in not telling pilots about it in order to make the conversion training as minimal as possible.

            1. Olivier2553 Silver badge

              The money saving came in not telling pilots about it in order to make the conversion training as minimal as possible.

              And you can't tell, unless you admit the MAX is different. But the whole point of MCAS is to make it not different.

              So it is different but it is not different.

        2. Electronics'R'Us
          Boffin

          FAA regulations part 25

          The 737 MAX cannot be certified against the appropriate standard.

          The relevant part (which I noticed in a post from a qualified pilot) is that when nose up is commanded at a specific amount, the Angle of Attack must increase at a constant rate.

          The MAX, left to its own devices will increase the rate of increase of angle of attack due to the extra lift generated by the engine cowlings (because of their forward position on the wing and slightly nose up attitude). Therefore a system that does what MCAS does is required for certification of the aircraft. This occurs in a particular part of the flight envelope and may indeed be unstable.

          Yes, the aircraft was rolled out in a hurry, and when Boeing realised they would fail a key criteria, they came up with the concept of MCAS and it was not subjected to the proper scrutiny at so many levels it beggars belief.

          1. TechnicalBen Silver badge

            Re: FAA regulations part 25

            Ah. That's for commercial aircraft? Would it be ok in a personal one? So I guess some of the mix up in commenters replies are putting technical realities vs paper based rules/certifications?

          2. ChrisC

            Re: FAA regulations part 25

            This, or at least a concise but clear as day summary, really needs to be posted as a preface to every single Max article published until the end of time, because from reading through the comments here it's clear that far too many people still don't really understand what would happen if Boeing were to try saying something like "OK, let's just disable MCAS and pay for every Max pilot to undergo Max-specific type training". It doesn't matter if pilots could be trained to fly the Max in its raw MCAS-unaugmented state, because the FAA wouldn't certify the Max to be flown in that state in the first place...

        3. dajames Silver badge

          Which implies that without the MCAS, it IS inherently unstable in certain situations. Am I wrong about this ?

          As I understand it -- and bear in mind that I know no more about this than anyone who's been following the story on public sources -- yes you are.

          The engines of the 737 MAX are larger than those of earlier 737s, and they have to be positioned further forward and higher up than the engines of other 737s to give enough ground clearance. One result of this is that the plane tends to point its nose upwards when the engines are working hard. This is apparently not unusual behaviour for an airliner, and does not mean that the 737 MAX is unstable or impossible to fly -- but it does mean that it handles rather differently from earlier 737 models.

          The function of the MCAS system is to compensate for this tendency of the 737 MAX to point its nose upwards while climbing just enough to make it behave like an older 737, so that pilots who are qualified to fly the older 737 can fly the MAX with minimal extra instruction. This is a big deal for smaller airlines as getting a pilot certified for a new airline type is time-consuming and expensive -- MCAS means that the 737 MAX is regarded as the same "type" as older 737s, and makes the plane more attractive to airlines who don't want to have to retrain a lot of pilots.

          When it works, MCAS works well, and pilots trained on older 737s can fly them just fine. When it doesn't work the results can be catastrophic. The main problem is that MCAS relies on a single Angle of Attack sensor (even though the plane has two) to tell it when the nose is pointing too high, and when this sensor fails it takes over the plane in a way that is difficult for the pilots to override, even if they do recognize the condition and respond appropriately.

          Ideally, Boeing should just admit that the 737 MAX isn't a 737, remove MCAS, and require all pilots who are to fly the thing to obtain certification for the new aircraft type.

          I don't see them swallowing that much humble pie, though, so what they need to do to make the 737 MAX safe and keep on pretending that it's just a 737 is to make MCAS use the readings from both sensors, and hand control back to the pilot if the readings don't agree. Ideally there should be at least three sensors of different types, so that if one sensor fails the computer can tell which one is wrong and work with just the other two. The sensors should be of different types to reduce the chance that two or more will be affected by some systematic failure and both give the same incorrect reading at the same time. Boeing also need to make it easier for the pilots to disable MCAS if/when it malfunctions, without having to switch to entirely manual control as they do at present.

          The BBC have a reasonable summary here. See the "What is MCAS" section about half-way down.

          1. SkippyBing Silver badge

            'Ideally, Boeing should just admit that the 737 MAX isn't a 737, remove MCAS, and require all pilots who are to fly the thing to obtain certification for the new aircraft type.'

            They can't remove MCAS as it wouldn't pass certification. They could provide more training on it (and I'd suggest a dedicated cut-out switch) which would remove the common type rating. This may lose them some sales, but that's better than losing all of them.

          2. -v(o.o)v-

            *Maneuvering Characteristics* Augmentation System

            I do not agree.

            Many in the industry seem to think that the reason for MCAS is simply the FAA certification requirements for handling characteristics i.e. "as nose pitches up, increasing aft stick force must be required" (paraphrasing but the FAA original text may be found easily from a certain forum or with much more effort from the original text (FAR?)), without MCAS the stick would lighten due to the aforementioned engine nacelles generating lift.

            The other opinion is that it is there for FAA longitudinal stability requirements.

            In a way both of them *do* protect against stall but especially Boeing is adamant that it is not stall protection system. And specifically, this is different issue from classic pitch/power couple that many seem to mistake it for.

            Personally, I believe it is due to the former certification requirement. It may have something to do with type rating - but the aircraft would be simply uncertifiable without MCAS, type rating had little to do with it.

        4. caffeine addict Silver badge

          Another "as I understand it" answer...

          The MAX isn't unstable. But it is "differently stable". That difference means that it would need retraining and re-certification. The problem with re-certification is that the rules have changed since the 737 was certified, and re-certification would involve fixing lots of other bits as well as retraining pilots.

          By putting the MCAS system in place, they tried to make a computer tweak that "differently stable" system to replicate the previous stable state, meaning no re-certification was needed because nothing had changed.

          Which I could understand if we weren't talking about aeroplanes...

        5. SkippyBing Silver badge

          The Max is stable until it approaches the stall when the engine nacelles start to generate lift, reducing the stick force required to raise the nose.

          For certification there's a requirement that it requires a given stick force to raise/lower the nose enough to change the speed by a set amount. The Max fails this without MCAS, designing it out would end up with a very different aircraft and require complete recertification. To minimise the training in the differences to keep a common type rating with earlier models, and make the costs attractive for airlines, they didn't mention MCAS in the conversion course.

        6. Updraft102 Silver badge

          The airlines apparently decided that.

          I read somewhere that Boeing proposed an all new aircraft, but the airlines balked at the idea of having to retrain all the pilots. They wanted a better 737, not a replacement for it, so their 737 certified pilots would not need to be retrained. Then Airbus refreshed the 737's biggest competitor, which kind of sealed the decision to try to do the same with the aging 737 design.

          The whole idea behind the Max was to keep that type certification intact, so a Max that has different flight characteristics that would require a new type certification would defeat its purpose. If you're going to have to retrain the pilots anyway, why not just get a new (better) plane without the 737's legacy baggage? It shows itself in more areas than just the short landing gear struts that make them have to finagle the bigger engines in there.

          It's interesting that there are companies out there that listen to their customers... perhaps sometimes even when they shouldn't.

      2. Anonymous Coward
        Anonymous Coward

        I highly suggest the following article. It's great reporting and outlines the actual reasoning and testing behind MCAS:

        https://www.seattletimes.com/seattle-news/times-watchdog/the-inside-story-of-mcas-how-boeings-737-max-system-gained-power-and-lost-safeguards/

        Relevant quote:

        "Engineers determined that on the MAX, the force the pilots feel in the control column as they execute this maneuver would not smoothly and continuously increase. Pilots who pull back forcefully on the column — sometimes called the stick — might suddenly feel a slackening of resistance. An FAA rule requires that the plane handle with smoothly changing stick forces."

      3. Aitor 1 Silver badge

        Agree

        In my opinion, the 737 Max would be perfectly fine without the MCAS, and way safer.

        Once this gets solved it will be a fine plane.

    11. This post has been deleted by its author

    12. Anonymous Coward
      Anonymous Coward

      > Or the warning light that indicates the failure of the system wouldn't have been an $80,000 optional extra.

      In the interests of accuracy, the AoA sensor failure warning is not an optional extra, all 737s have it. The bit that is the optional extra is the continuous, side-by-side display of the two AoA sensor readings.

      1. J. R. Hartley Silver badge

        Yes. The AoA disagree light is not the same as AoA failure.

      2. 's water music Silver badge

        Although the AoA disagree alert was supposed to display whether or not you had ordered the optional AoA continuous display, apparently there was a bug that meant that the alter was dependent on the optional display in practice:

        (source)

        >>After deliveries of the 737 Max commenced in 2017, Boeing discovered a software issue prevented the angle-of-attack (AOA) Disagree alert from working if customers had not chosen the optional AOA indicator.

        >>“The Boeing design requirements for the 737 Max included the AOA Disagree alert as a standard, standalone feature, in keeping with Boeing’s fundamental design philosophy of retaining commonality with the 737NG,” says the company in a statement issued on 5 May.

        >>“Several months” after 737 Max deliveries started in 2017, Boeing engineers discovered that “the 737 Max display system software did not correctly meet the AOA Disagree alert requirements.”

        >>“The software delivered to Boeing linked the AOA Disagree alert to the AOA indicator, which is an optional feature on the Max and the NG. Accordingly, the software activated the AOA Disagree alert only if an airline opted for the AOA indicator.”

    13. Balding Greybeard

      Unsafe At Any Speed

      There’s something wrong with the design of the plane that can’t be flown by a human and requires a robot to override them.

      Boeing needs to remove the plane from operation and go back to the drawing board.

      Me, I’ll never fly on one and I imagine the public will avoid them too.

    14. Stoneshop Silver badge
      Holmes

      "The safety of our airplanes is Boeing’s highest priority,"

      Them staying on the ground in a parking area or maybe even moved to one of the desert storage airfields is provably much safer than having them flying around, with or without MCAS.

    15. fidodogbreath Silver badge

      "The safety of our airplanes is Boeing’s highest priority"

      "During the FAA’s review of the 737 Max software update and recent simulator sessions, the Federal Aviation Administration identified an additional requirement that it has asked the company to address..."

      If safety is their highest priority, then why didn't Boeing find the new plane-crashing bug in their own testing?

  2. Blockchain commentard Silver badge

    "The safety of our airplanes is Boeing’s highest priority," the jet blunderer insisted to The Register this afternoon.

    I corrected your typo!!!

  3. Mark 85 Silver badge
    Facepalm

    So maybe what they found was the real problem? Or they didn't comment out the bad code?

    1. jmch Silver badge

      It seems to me that the developers were rushed to deliver the fix under massive commercial pressure, and weren't given any time for proper regression testing.

      Something that happens all over the IT industry against the best practice warnings of practitioners who know what they are doing. The beancounters always have more sway with the board than the technicians.

      1. Archtech Silver badge

        Asking nine women to deliver a baby in one month

        It's straight out of "Dilbert", isn't it? When the engineers give the PHB their best advice, and the PHB replies, "Just ship it".

        1. Anonymous Coward
          Anonymous Coward

          "Send a sacrificial engineer"

          "When the engineers give the PHB their best advice, and the PHB replies, "Just ship it"."

          At one place I used to visit regularly, HQ were so well known for saying "just ship it", whether or not the critical system development/test was complete, that the routine request to send an engineer to recover and "diagnose" the "failed" (ie unfinished) development unit was widely known in-house as a request "to send a sacrificial engineer" (in a metaphorical sense - the engineer and other participants were not really at any significant risk).

          Boeing seem to have extended this principle into the post-certification phase of 'development', and made the risks very very real. Real people (crew, passengers) got sacrificed.

          Not good, Boeing. Not good at all. This is not the way to win friends and influence people.

  4. Anonymous Coward
    Anonymous Coward

    If it's Boeing

    I ain't going...

    1. Wibble

      Re: If it's Boeing

      Problem is that all "modern" airliners - and all other forms of mechanical transport - contain so much software and we all know how crap software is. Turn it off and on again. Doesn't matter which manufacturer built it; Boeing happened to be caught out, but all the others probably suffer the same problems.

      What a future we all have to look forwards to. Planes inexplicably falling out of the sky; cars inexplicably running off the road; ships colliding with things.

      There's something comforting about the simplicity of mechanical systems, or even systems where the software isn't in control. AI just isn't intelligent, or not until it becomes self aware...

      What are you doing Dave?...

      1. SImon Hobson Silver badge

        Re: If it's Boeing

        There's something comforting about the simplicity of mechanical systems

        Like mechanical systems can't have faults in them. It can happen to anyone ... oh, that one was Boing as well you say ?

        But it can happen to anyone.

      2. Anonymous Coward
        Anonymous Coward

        Re: If it's Boeing

        My Mum always said chips were bad for you.

      3. BuckeyeB

        Re: If it's Boeing

        Just remember the company with the contract had the lowest bid.

  5. Mike007

    Did they actually say if this bug was caused by the code changes to fix the original flaw having been improperly written and pushed to the release branch without proper review, or if they have merely started discovering additional problems now that management has approved a testing budget...?

    1. TReko

      There are serious issues here, this should not be uncovered in a simulator; it should have been found during unit tests.

      As someone who has written code for avionics, every possible state and function needs to be called and tested by the code tests. Any code that that wasn't tested cannot be flown. Any state or possible state transition that was not tested is explicitly dis-allowed.

      1. haiku

        Ensuring that all code is tested

        An added incentive would be to include (as passengers) all programmers whose code is being tested on all live test flights ...

        1. My-Handle

          Re: Ensuring that all code is tested

          Not sure I'm a fan of that idea. I'm not sure the programmers are entirely at fault here. I can just see Boeing management announcing to the programmers:

          "We've got this massive company balls-up that we need you to sort out! Write a fix for this MCAS thing. You've got 2 days, we're losing money here! Oh, and by the way you'll actually be flying in the live test flights so make sure you do it right. What's that? Overtime? Don't be silly."

        2. hammarbtyp Silver badge

          Re: Ensuring that all code is tested

          plus managers and the Boeing board of directors

          1. MiguelC Silver badge

            Re: Ensuring that all code is tested

            Take Momenta, a Chinese autonomous driving company, whose "CEO requires all executives to ride a minimum number of autonomous miles themselves, so management would put passenger safety first."

            An example that should be followed whenever user security is at stake

            1. BebopWeBop Silver badge

              Re: Ensuring that all code is tested

              A refreshing thought. It will never catch on with Uber{substitute any US and probably European manufacturer as you wish}

            2. Irongut

              Re: Ensuring that all code is tested

              Except that doesn't help the poor meatbags outside the vehicle who are in much more danger from 2t of metal moving at speed. See Uber.

          2. hoola Bronze badge

            Re: Ensuring that all code is tested

            And the FAA officials. They are not blameless as they permitted a new aeroplane to be certified as a series of modifications and upgrades in the interests of efficiency and cost.

            There has to be a point when a number of upgrades is reached where the entire aeroplane has to be recertified as if it were a new release.

            It might just happen after this fiasco but I am not holding my breath. What will be interesting is if the European authorities insist on their own certifications. Yet another tit-for-tat farce will ensue only this time it is lives that are at stake. Most of this is because the big American corporations and certain parts of the American establishment believe that they have more power than anyone else.

            1. A.P. Veening Silver badge

              Re: Ensuring that all code is tested

              What will be interesting is if the European authorities insist on their own certifications.

              I can confirm that EASA does insist on their own certification (was in the news here). What is more, so does CAAS, the counterpart in the PRC and given the current political situation, that one might even be stricter than the European one.

        3. oldfartuk

          Re: Ensuring that all code is tested

          In WW2, was it not customary to take the senior ground crew supervisor up with the crew on acceptance testing of a plane?

          1. A.P. Veening Silver badge

            Re: Ensuring that all code is tested

            According to a conversation I had with a senior helicopter mechanic about ten years ago, it still is at least customary with helicopters after maintenance.

        4. pavel.petrman Bronze badge

          Re: Ensuring that all code is tested

          When my father was learning to drive a passenger bus many many years ago, the actual driving lessons were organized in groups. at the beginning of the first drive the instructor would ask who wanted to drive first. After someone volunteered, he'd order the rest of the group to stand in the aisle. With their hands in their pockets.

          The problem today is that there is no single driver anymore so there is no way to get the required level of peer pressure to work.

        5. fajensen Silver badge
          Devil

          Re: Ensuring that all code is tested

          Current Boeing management would only see that policy as an incentive to take out life insurance policies on all their programmers with themselves as beneficiaries, then push for real challenging delivery schedules ....

          1. Peter2 Silver badge

            Re: Ensuring that all code is tested

            Not if you include the management as well. From every level from the team leader to the CEO.

      2. twellys

        I'm just wondering whether Boeing classed that as a DAL-D whereas it should be a DAL-A?

      3. Bronek Kozicki Silver badge

        I do not work in this industry, but given that the problem manifested itself in the CPU locking itself up, my guess would be that this could not have been spotted by testing code excerpts alone (which is what unit testing is, in principle). That's why you also want integration tests, when you take the largest possible vertical stack of software and hardware, and test it under conditions as close to realistic as possible (but perhaps without risking your customer's lives ... which is perhaps not obvious to some). It seems in this case they put the real hardware under simulated load, which is one of the ways to do it.

        1. vtcodger Silver badge

          That's why you also want integration tests

          You're correct. Integration Testing is essential. The problem is that the number of combinations of events and conditions quickly becomes too large to test exhaustively. And sometimes sequencing matters. And sometimes not only sequencing, but timing of events matters (e.g. race conditions). Integration testing, while essential, can be a crap-shoot. Many bugs will be caught. But maybe not all of them.

          There are other problems. For example, designers are often (usually in my experience) resistant to the notion that their design sucks and that real people can't use it reliably. Their solution tends to be to improve the users -- which oddly enough is often (usually) not well received by the users

          ... And management often, not always, but often, understands very little of that.

          Answers: I don't have any. e.g. Don't count on me to be an early adopter of autonomous vehicles even though I think they are great idea that will eventually make the world a better place.

          1. Bronek Kozicki Silver badge

            "Answers: I don't have any." yup - me neither.

            I understand the part about testing corner cases that are very, very hard to reproduce - that's what unit tests are best for. But when the corner case spans many layers of the stack, I'm stuck.

      4. pavel.petrman Bronze badge

        Regarding "Any code that that wasn't tested cannot be flown"

        You said something very important here - these days we are mainly flying software, although with some rotating parts attached. As a software engineer myself I can imagine vividly how at one stage or another the software under review had been handled more like "software that runs somewhere on a computer" rather than "software that flies, with many souls on board". Just like the control software for that unfortunate x-ray machine the other day. Isn't it about time we call it something more appropriate, like "safetyware" or "livesware" to give it the attention it requires?

      5. The First Dave Silver badge

        HTF do you expect a unit test to uncover a hardware issue?

        Sure, it might not be a hardware issue this time, but that's not the point.

      6. oldfartuk

        In concur. It needs a test schedule that would make your eyes water and the dev team cry......

        1. A.P. Veening Silver badge

          Not to mention the button sorters run for their lifes.

      7. Electronics'R'Us
        Holmes

        Safety critical avionics software (and the hardware to support it)

        Speaking as someone who has been a design authority for safety critical avionics hardware, I completely agree; DO-178 is of course the relevant standard.

        The real issue here is what DAL was required by Boeing? Considering that avionics suppliers to civil aircraft cannot usually charge NRE, they need to recoup all that invested money from sales which acts as an incentive to the airframer (Boeing in this case) to use the lowest they think they can get away with as with increasing safety levels comes increasing costs to the supplier and ultimately to Boeing.

        There is a key question I have that can yield only one answer and it is not the one Boeing specified from what I have read:

        "Can this piece of kit command movement of flying control surfaces where there is no other effective system oversight of that commanded movement"?

        I am aware that the pilot can cause MCAS to disengage (briefly) but that should be easy to do (which it clearly was not).

        The answer here is a resounding YES and should require DAL A (which affects way more than software - got an FPGA in the loop? Add 5000 hours of engineering time for just the paperwork associated with DO-254. There is of course, a lot more than just that).

        DAL A also requires redundancy (you can get away with a dual channel system in some cases although triple redundancy is far more common in these situations). The rule is that no single failure shall be able to cause a catastrophic failure and the usual statistic is that the chance of catastrophic failure (from multiple failures from all possible sources) is less than 10^^-9 per flight hour

        It would be interesting to see the RFP / RFQ / System requirements sent to the avionics suppliers (although this may already be happening in the civil lawsuits as this is a key item of how Boeing viewed the functionality of MCAS. Boeing would have a really tough time justifying anything less than DAL A imo).

    2. Ozumo

      Not so much management approving a testing budget as the FAA actually insisting on some testing and not taking Boeing's word for it. Regulatory capture...

    3. oldfartuk

      Im retired ex-IT Projects Manager. Allow me to add my professional twopenn'orth.

      a software update is a process of replacing know fixable bugs with unknown unfixable ones. And in a system as complex as a jet liner, i would be very wary of changing any line of code with subsequently spending the next 6 months or year on simulator trying to crash it.

  6. swm Bronze badge

    The problem with digital control systems is that they are not continuous. Each if statement causes discontinuous operation. If the system were analog then testing would be easier. I'm not recommending analog systems - just pointing out the difficulty in checking digital systems - especially systems with deliberate discontinuous actions: if the velocity exceeds vsetpoint (even by a little bit) then pitch up etc.

    I have heard that the code is not audited by the FAA but is qualified by number of "good" flight hours. The code is propriety.

    1. hammarbtyp Silver badge

      True, but ever since digital computers have been added to planes at some point the analog inputs have had to be somewhere converted to digital.

      and modern planes are very,very safe. The fact we are talking about the 737 max is that it is such an outlier. If this happened in the 60's and 70's it would of been lost in the noise

      No the problem is not the digital nature of the control system but the fact that Boeing tried to add too large engines to a aircraft unsuitable to take them, and Boeing trying to fob the airlines off that this was a minor upgrade that did not require extra training or safety measures rather than a new aircraft which had systems added to correct some severe potential safety issues. Or to summarize, Boeing tried to save money

      1. TRT Silver badge

        I thought it wasn't so much the saving money as the being quicker to market a jet that competed in an expanding new marketplace for more efficient, quieter, cheaper to run, mid-sized carriers, to replace a fleet operating out of provincial terminals which were mostly still operating with fixed height embarkation steps rather than the full-range height adjustable jet bridges of larger hub terminals. They had shorter landing gear struts in order to get the plane to sit lower down, which limited the range of engines their airframe could carry, and had previously adjusted the engine shape, squashing the bottom flat to make it fit. The engines they needed to compete with other manufacturer's planes were bigger and rounder, too big and round to actually fit safely on the plane where they were supposed to. To go into a full development cycle for a new airframe able to lift these bigger, rounder engines clear of the ground whilst still keeping the door heights the same would have taken many years and billions of dollars. By mounting the engines in a different place on the wing, they could make it work with an existing airframe for which existing terminal infrastructure was present. But by doing that, they f***ed up the handling, which they tried fixing in software.

        Methinks it would have been cheaper to just supply every airport that needed them with a new set of boarding stairs/lifts.

        1. Updraft102 Silver badge

          To go into a full development cycle for a new airframe able to lift these bigger, rounder engines clear of the ground whilst still keeping the door heights the same would have taken many years and billions of dollars

          Boeing wanted to design a brand-new replacement for the 737 (without keeping the door heights the same, as times have changed and that's no longer of any real value), but when they pitched it (pun intended) to airlines, they heard that the airlines wanted better 737s, not a replacement. While the original 737 design was definitely showing its age, airlines perceived it as working well, and pilots knew how to fly them. It was the airlines that pressured Boeing to deliver a plane with the same type rating as their existing 737s, so no expensive pilot retraining would be necessary, and Boeing tried to give them what they wanted. And then when the new iteration of the A32x series, that just solidified Boeing's plan to refit the 737 once again, which could be done much faster than designing an all-new plane.

          I don't think there is any reason to believe that if Boeing had designed the Max to be a new(ish) product that had its own handling characteristics, this whole MCAS thing would not have happened. Trying to cram the new(ish) plane into the existing 737 NG type certification was the problem. It does not handle like previous 737s because it is not one of the previous 737s! It should have had its own type rating, but if it had, that would have erased most of the reason airlines wanted another 737 rather than a clean-sheet replacement.

          1. TRT Silver badge

            Really? Well, I can see that makes sense as well... airline pressure... cut throat business. Still a shit state of affairs, though.

          2. Loatesy

            "It was the airlines that pressured Boeing to deliver a plane with the same type rating as their existing 737s, so no expensive pilot retraining would be necessary"

            Exactly the problem; Boeing then conned the airlines into buying an aircraft they had been assured was the 'original' 737 when it was to all intents and purposes a new aircraft. By concealing this fact in order to get sales at the expense of a rival (and I've said this on another forum) Boeing are guilty of nothing less than a corporate drive-by shooting; the intended target was Airbus, the innocent bystanders were the passengers and crews of the two crashed 737 Max's.

          3. Alan Brown Silver badge

            "Trying to cram the new(ish) plane into the existing 737 NG type certification was the problem"

            The 737NG or MAX would not be certified on their own cognisance. They can only fly as modifications of an existing airframe. As soon as you tried to certify them as a new aircraft they're going to be FAILED so hard that heads would spin at Boeing.

            If you're going to have to get a whole new type certification then you'd address the engine positioning - which means longer legs, which means new undercarriage and hull design to take it (this is a _major_ change - aircraft are designed and built around their engines and legs.). You'd fix the wing sweep (it's too swept for the speeds used) and you'd fix the baggage compartments to actually be able to take standard aviation containers.

            At that point you're spending so much in airframe modifications you're better off starting with a clean sheet.

            There are 8000 backorders on the 737 order book alone (another 8000 for the A320 family), simply because this is the sweetspot in the market as far as airlines are concerned (this has a lot to do with the US's heavy subsidiation of airports). This gave Boeing very little incentive to actually innovate or invest in new aircraft design when they could just keep kludging the old design and have their tame FAA stamp the already signed off paperwork.

            The way they went about covering up the issue of suppliers counterfeiting documentation on critical hull components for the 737NG (The ribs were supposed to be precision-CNC made and weren't), and ALSO covered up how badly the build shops in Kansas and Washington were battering the parts to make them fit (then filling and painting over the damage) - which is the root cause of at least 3 hull breakups/11 deaths that shouldn't have happened on landing overruns - shows that this is NOT a new thing at Boeing. (There are at least 500 737NGs flying which are at risk of bursting like a Comet at high flight cycle levels thanks to these faulty parts) - When internal auditors found this they were ignored. When they blew the whistle they were identified almost instantly due to FCC stooges who handed the complaints straight back to Boeing and hounded mercilessly by Boeing corporate.

            (The FBI recently arrested people in the FAA and other US government departments who were offering to sell whistelblower data back to the companies concerned - and this ONLY happened because one of the companies did the right thing and fiiled criminal complaints. It turns out that the USA's whistleblower protection legislation has been backdoored by corrupt employees since it was enacted.)

      2. Adair

        It looks like Boeing have basically said, "The fact that this aircraft is incapable of sustained stable flight without constant digital input is fine by us because we know that digital/software systems never fail (or if they do it's nothing to do with us). Welcome aboard!"

      3. Updraft102 Silver badge

        and modern planes are very,very safe. The fact we are talking about the 737 max is that it is such an outlier. If this happened in the 60's and 70's it would of been lost in the noise

        Indeed. Something like 99.99% of all 737 Max flights ever flown were completed safely. Commercial aviation is so safe overall that 99.99% successful flight completion isn't good enough, and the correct course of action was taken in grounding the planes so that the unacceptable .01% accident rate can be fixed.

        1. Anonymous Coward
          Anonymous Coward

          99.99%

          99.99% of my bacon consumption is entirely safe.

          99.99% of someones attempts to hit a moving car with a rock on the highway might be safe.

          I'm not saying you are wrong. Just that, some faults are faults by design, even if they only happen 1 in a thousand times. The massive safety elsewhere, does not excuse the massive oversight here. :)

          1. TRT Silver badge

            Re: 99.99% of my bacon consumption is entirely safe.

            As for the rest, you've just gotta roll with it.

  7. trashsilo
    Go

    Some things are still A-OK

    Don't worry flyers, no such microprocessor lock occurs during the iPad Boeing certified MAX computer training undertaken by the pilots.

  8. Evil Auditor Silver badge

    If it looks like a duck, quacks like a duck and flies like a duck, it probably is a duck. Problem is, the 737 Max looks like a 737 but doesn't quack like one and, crucially, doesn't fly like one. For fuck's sake, skip this mess of a system and just train the pilots to fly the bloody thing! Oh, too expensive, you say.

    1. SW10

      It’s not my sector, but AIUI that would be a tacit admission that it’s not actually a 737 but a new model of aircraft.

      Pilots would then have to be trained on the new plane and its behaviour, at a significant cost to airlines in terms of cash and pilot availability.

      It may even be that 737 MAX has “inherited” some certs from its “predecessors” (same airframe?) and I don’t know what would happen if the aircraft suddenly became, I don’t know, the 797.

      1. CliveS
        FAIL

        "It may even be that 737 MAX has “inherited” some certs from its “predecessors” (same airframe?)..."

        Boeing has relied on Grandfather Rights for the 737-Max and earlier developments. The fundamental certification for the 737-Max dates back to the original certification issued in 1967. That's a 50 year old certificate of airworthiness being applied to a plane that is fundamentally different to that originally certified by an FAA that is today effectively a different organisation. All in the interests of saving money.

    2. ExampleOne

      For fuck's sake, skip this mess of a system and just train the pilots to fly the bloody thing! Oh, too expensive, you say.

      This. So much this. AIUI, if they did this, they could just scrap the MCAS altogether, and the plane would be perfectly safe.

      But that would have cost airlines buying the plane a fortune in pilot training, and would probably have cost Boeing a fortune in the initial certification costs as it wouldn't have been able to grandfather it. The costs to Boeing may come to have looked like a far cheaper option though...

      1. Ozumo

        It's not just the training; currently all 737 pilots are fungible; to have multiple type ratings would cause headaches in pilot scheduling - for example at Southwestern, which has an all-737 fleet.

    3. Adair

      Isn't the problem that the 737 Max is fundamentally unairworthy, without constant intervention by digital systems to control it's inherent instability.

      In a military fighter aircraft instability (of a particular kind) may be a desirable designed in feature, but in a civilian airliner to allow a massive 'bug' in the system and then try to correct for it in real time would appear highly irresponsible - things go wrong!

      Who'd a thunk it?

      1. This post has been deleted by its author

  9. Potemkine! Silver badge

    The safety of our airplanes is Boeing’s highest priority

    ... just after 'being the most profitable business to the point of letting a critical device working with a sole captor and asking $80,000 for a warning light' of course.

    PR BS over the bodies of 346 passengers. Instead of 'Boeing', I would suggest a new company name starting with 'B' too that would be more accurate.

    1. Bronek Kozicki Silver badge

      Re: The safety of our airplanes is Boeing’s highest priority

      Yes, Bullshit sounds about right.

  10. AndyFl

    It will be a while before it flies around the world again

    The FAA are now looking very carefully at everything to do with the plane - they had the wool pulled over their eyes and were lax in the past. Now they want to be sure that they won't be blamed for any other issues and will not be rushed into an approval.

    Other regulators like Europe and China have indicated that they will no longer be taking FAA approvals at face value so will do their own careful analysis of any test results.

    Then there are the trade issues, I would be amazed if they were very quick in approving the US plane in Europe after Boeing's less than completely honest submissions in the past, FAA failure to oversee things properly and being recently screwed over by the US in trade. I'm sure that they wouldn't be adverse to giving Airbus a bit of a leg up in business. Probably China will be the same in taking their own sweet time to check everything. They might play it straight and do it quickly but I have more than a few doubts.

    Andy

    1. MiguelC Silver badge
      Meh

      Re: It will be a while before it flies around the world again

      It's good to know that IAG (owner of BA) threw Boeing a lifeline at the recent Le Bourget airshow by signing a letter of intent to purchase 200 737.Max planes.

      1. Benson's Cycle

        Re: It will be a while before it flies around the world again

        I wonder how much they're going to cost?

        At this stage it's beginning to look as if the best option might be to go into business producing VR flight simulators and holiday simulators. On arrival at the airport "passengers" would be handed immersion VR headsets, undergo a simulated journey, a simulated holiday and a simulated return home. If they encounter a simulated nosedive or a simulated faulty water heater, they can have a simulated death and then just start again.

        Nobody gets killed IRL, climate change is slowed, it's all good.

        1. Uncle Slacky Silver badge
          Trollface

          Re: It will be a while before it flies around the world again

          I'm sure I saw a documentary about a similar idea, I think it was called something like "We Can Totally Recall It For You Wholesale".

      2. heyrick Silver badge

        Re: It will be a while before it flies around the world again

        "a letter of intent to purchase 200 737.Max planes"

        Maybe they're going cheap?

      3. Olivier2553 Silver badge

        Re: It will be a while before it flies around the world again

        They may get them so cheap it will be worth training new pilots on these new planes. With 200 planes, you can consider a good enough number of pilots to make it work.

  11. Duncan Macdonald Silver badge
    Mushroom

    China

    If China wants to get back at the USA then an easy route for it would be to require full certification of the 737 Max 8 as a new aircraft in China before it is allowed to fly in China's airspace. Given how lax the FAA has been (because of low budget) requiring this certification to take place in China would be reasonable - and cripple Boeing (especially if other countries decided to follow suit).

  12. MortimerTheCat

    Have they tried turning it off and on again?

    1. BebopWeBop Silver badge

      At least two (very) hard resets I believe.

  13. Jemma Silver badge

    New from Ralph Nader

    "Unsafe at any height"

    But be very very happy this was caught at the simulator testing stage. Can you imagine what would have happened if this lockup was triggered on an airport approach over somewhere like London or Berlin - or even any airport where the final approach was over a residential area. It'd be carnage.

    And before you lay the blame entirely at Boeings door remember - they build aircraft to what the airlines will pay and that's a function of what you pay for tickets. Cheaper flights, cheaper and less resilient aircraft. I'm by no means saying Boeing are innocent in this, they have murdered people (including if I remember right, one of Naders neices) in the pursuit of profit, but they aren't the only culprits, cheap flights = flying shitboxes and Airbus have had their share of accidents in their time (although they do try to build decent product).

    1. DavCrav Silver badge

      Re: New from Ralph Nader

      "And before you lay the blame entirely at Boeings door remember - they build aircraft to what the airlines will pay and that's a function of what you pay for tickets."

      OK, I've remembered that. Now I can lay all of the blame for this at Boeing's door. Although the door would probably fall off as it opened.

    2. Dan 55 Silver badge

      Re: New from Ralph Nader

      It shouldn't have got as far as the simulator and it was probably chance the fault showed up and was caught there.

      If there's one good thing to come out of these tragic crashes it would be we all finally stop fucking around now and develop software like it was something serious that has formal proof and actual quality control. Well, we can hope, corporate America is very stubborn.

  14. nematoad Silver badge
    FAIL

    Oh!

    "Boeing agrees with the FAA's decision and request..."

    How nice of them.

    As if they had any choice, this time.

    1. STOP_FORTH

      Re: Oh!

      This seems to be a standard PR/lawyer-speak these days. If a court or regulator instructs you to do something you issue a statement saying that you are happy to oblige/comply.

      Making a virtue of necessity?

  15. Alan Bourke

    So anyway

    how are fully autonomous cars coming along?

    1. Anonymous Coward
      Anonymous Coward

      Re: So anyway

      >how are fully autonomous cars coming along?

      Badly judging by the Tesla lawsuits.

    2. Blockchain commentard Silver badge
      Facepalm

      Re: So anyway

      Well, at least they're already on the ground. So +1 over Boeing !!!!!

      1. Anonymous Coward
        Anonymous Coward

        Re: So anyway

        >Well, at least they're already on the ground. So +1 over Boeing !!!!!

        Except the 737 max, I'd call it a draw on that one.

        1. A.P. Veening Silver badge

          Re: So anyway

          Except the 737 max, I'd call it a draw on that one.

          But those cars are moving, so the advantage is still with the cars.

  16. Electronics'R'Us
    FAIL

    Stunning fail

    When I heard that the AoA sensor was not redundant, I was stunned that any control system that can operate flying control surfaces relied on a single sensor.

    Now (even after figuring out that redundancy is not optional) it is clear that the microprocessor is not redundant.

    This doesn't need a software fix, it needs a complete control system re-design.

    In safety critical civil avionics, not only is control redundancy required, the microprocessors in each path must also be from completely different families; that guards against microcode problems. If one locks up due to a microcode bug (which is quite possible), it is pretty much infeasible that a processor from a different family will also lock up at the same section of code due to such issues.

    I cannot believe that any aircraft systems engineer, let alone the independent technical authority (who exist not only at Boeing but at every supplier of avionics) could ever have done a proper FMECA against this system.

    In defense of the suppliers, they can only do a FMECA against the requirements set by Boeing. Some very senior heads at Boeing need to roll for this botch job that has claimed hundreds of lives.

    If the FAA ever certifies this aircraft in its current form (with MCAS not re-designed with proper redundancy), people should be looking very carefully at both the certification process and the people involved.

    1. steelpillow Silver badge
      Boffin

      Re: Stunning fail

      "the microprocessor is not redundant."

      Actually, it may well be. It may prove possible to simply remove MCAS, re-certify the type and retrain the pilots accordingly. On the other hand it is also possible that the plane would then prove too unstable to be safe, in which case the sensible thing would be to integrate high AoA response into the main digital flight control algorithms and stop pretending it is a bolt-on extra. Either way, the MCAS microprocessor goes.

      1. Electronics'R'Us
        Boffin

        Re: Stunning fail

        You may be correct, but a redundant system (even those with two channels) are constantly communicating cross channel in this sort of kit.

        If one side noticed no communications from the other side it should reset it (or otherwise disable it and give the pilot a warning). Clearly that is not the case here.

        That is a major design flaw at system level and renders the equipment unfit for safe operation.

    2. tip pc Bronze badge

      Re: Stunning fail

      the engines have dual management systems, FADEC, which, as you mentioned, use different hardware and software to derive the same answers to the same environment variables.

      I had always thought other critical aircraft systems where the same.

    3. mutt13y

      Re: Stunning fail

      I was wondering about the "feature" of the original system that switched between AoA sensors after each flight.

      I can think of no technical reason to do this. It introduces instability and difficult to diagnose faults.

      The only thing I can think of is that it would let me draw a line from both AoA sensors to MCAS on my block diagram.

    4. Robert Sneddon

      Angle of Attack indicator

      When I heard that the AoA sensor was not redundant, I was stunned that any control system that can operate flying control surfaces relied on a single sensor.

      The AoA sensors started life as only providing an indication to the pilots of the airflow angle over the fuselage. There were two sensors, if one broke in flight they could switch to the other. There was no need to fit three sensors, a flight could be completed safely if both of them failed, unlikely as that was.

      With only two inputs it's difficult verging on impossible to determine by measuring their outputs which one is wrong, three sensors at the minimum are needed to vote a defective sensor out of the control pool.

      Boeing decided to allow the AoA sensor to provide information about a possible stall situation to the MCAS software system which had authority to change the angle of control surfaces, to correct the stability problem it faced with the MAX's changed aerodynamics due to the enlarged and repositioned engines compared to previous versions. Adding more AoA sensors to the airframe to allow for "voting" in the MCAS system would probably have required recertification of the airframe which was what they wanted to avoid at all costs.

      What might have helped but wasn't available to the pilots of both crashed aircraft was an "MCAS Disable" switch (although that itself could be a possible source of pilot error). That would have allowed the pilots to reset the trim using thumb switches on the yoke, a much easier process than switching off the electric trim completely (the only way to disable MCAS at the moment) and using heavy awkward manual trim wheels instead. It's poignant that the "electric trim disable" switches were fitted after a few incdents where the electric trim systems "ran away" and drove the control surfaces hard up or down after component failures.

  17. Anonymous Coward
    Anonymous Coward

    Maday, maday we're Boeing down.

    1. Benson's Cycle

      If they can build a rubber aircraft they could have a rerun of Boeing Boeing, titled Boing Boing.

  18. MJB7 Bronze badge

    "triggers an exception that can't be handled and the CPU halts"

    I'm pretty sure that if the CPU halts it will be rebooted pretty sharpish. Entering a tight loop and not tickling the watchdog sounds much more plausible (and the pilots only have to maintain control until the watchdog reboots the CPU).

    I'm still boggled this wasn't caught in unit tests. When I was working with people writing diesel engine controllers 15 years ago, they would have been horrified if this got past unit tests. (And automotive is a lot less fussy than aviation.)

    1. Mike 125

      Re: "triggers an exception that can't be handled and the CPU halts"

      >I'm pretty sure that if the CPU halts it will be rebooted pretty sharpish.

      CNN says:

      'In simulator tests, government pilots discovered that a microprocessor failure could push the nose of the plane toward the ground. It is not known whether the microprocessor played a role in either crash.'

      I suspect the 'revised' tests force a processor STOP, just to see what happens. Unfortunately, what happens is an aircraft STOP. Hence multiple redundant systems. I bet no tests like that have been done before on this system.

      Watchdogs: I've seen a tickle done from a timer interrupt handler. And for his Full Gold Star, the engineer actually claimed he was being clever.

      >When I was working with people writing diesel engine controllers 15 years ago

      A lot's changed in 15 years, and none of it good.

      1. Anonymous Coward
        Anonymous Coward

        Re: "triggers an exception that can't be handled and the CPU halts"

        > Watchdogs: I've seen a tickle done from a timer interrupt handler. And for his Full Gold Star, the engineer actually claimed he was being clever.

        Ha. That's worth an upvote on its own.

  19. Big_Baldy_Bloke
    Unhappy

    Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

    I hated the good PR cover Boeing bought with this order: BA-owner IAG signs deal to buy 200 planes https://www.bbc.co.uk/news/business-48682123

    As the most recent news shows, there are some pretty unpleasant flaws in the 737 MAX itself, and in the design, test and approval processes.

    Willie Walsh is quoted as saying IAG had “every confidence in Boeing and expect that the aircraft will make a successful return to service in the coming months having received approval from the regulators”. Surely it is too early to say this with confidence, especially as the plane was not grounded after the first accident and Boeing/FAA were so reluctant to act even after the second. Boeing/FAA will have to be much more careful now the world is watching so I doubt there will be any quick fix.

    I assume IAG got an outrageously good deal.

    1. Dan 55 Silver badge

      Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

      I'm hoping that they weren't ordered knowing that approval to fly again would be a shoe-in.

      1. Robert Sneddon

        Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

        Boeing has over 4000 outstanding orders for the 737 MAX and its production rate is about 500-600 a year. IAG placed this order for delivery of aircraft starting some time in the mid 2020s or even later by which time either the MAX will be recertified and fixed or it will be cancelled, scrapped and perhaps by another airframe entirely. If stuff continues to go wrong with the MAX's return to flight then the order could easily be cancelled -- I'm pretty sure that the contract's cancellation clauses aren't standard boilerplate for this kind of deal given what's happened.

        1. Big_Baldy_Bloke

          Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

          Yes, it is a letter of intent that can be cancelled and I understand that if the order goes ahead and is delivered as planned starting in 2023, the 737 MAX will have been modified and shown to be safe. Apart from getting a better price, as you point out, IAG also secured delivery slots from the planned production run and that was most likely important too. I am still queasy about the timing of the announcement and the PR boost it gave Boeing while the crash investigations are underway, and the required fixes are still unknown and/or unroven.

    2. Def Silver badge

      Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

      So in the future check the plane type before flying with Aer Lingus, British Airways, Iberia, Level, and Vueling? Got it.

    3. MJI Silver badge

      Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

      And what is wrong with buying Airbus?

      Made here and often with engines from UK.

      1. Anonymous Coward
        Anonymous Coward

        Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

        Airbus are pretty good on the whole, shame about the engines as you say. RR's haven't exactly covered themselves in glory on this front recently :(. The early RB211's a fine piece of kit for the timeperiod, and I want to like the new stuff, but objective analysis suggests otherwise.

        1. Brit Flyer

          Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

          GE latest effort on the 777X not so great either.

          The main thing driving innovation is engine design, the technology and reliability is amazing.

          However it is highly complex and there will be challenges

        2. Def Silver badge

          Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

          Airbus are light years ahead of Boeing on comfort and facilities.

          I recently flew back to back flights with the same airline on a four year old Boeing 777, and a 10 year old Airbus A330 (I checked the plane specs on FlightRadar). The difference in levels of comfort and facilities (I was in cattle class both flights) was amazing. When the A330 was pushed back from the gate and we were sat on the tarmac I started wondering what the hold up was and when the engines were going to be started. Then we started rolling - the engines had quietly powered up without me even noticing.

    4. Brit Flyer

      Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

      IAG's (British Airway) previous incarnation was BOAC , also known as the Boeing Only Aircraft Corporation.

      History about to repeat itself?

      1. Wellyboot Silver badge
        Unhappy

        Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

        BOAC used propeller driven aircraft from many UK & US manufacturers in the 50s, But until the Comet reappeared in 1960 (starting to look a bit small by then) there was basically no alternative source of trans-atlantic capable jet airliners for the prestige routes other than Boeing or Douglas.

    5. Spanners Silver badge
      Meh

      Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

      I hope that there were no Brexit enthusiasts behind this wanting to prove that we are willing to avoid European things at any cost!

      So we could end up with a job lot of aircraft that some countries will not allow into their airspace. Hopefully...

      1. This deal will not be confirmed until a proper solution is found.

      2. We get them at a good discount.

      3. This is traceable to PHBs. "executives" and accountants not the people who actually make aeroplanes and the software that should make them work.

    6. molletts

      Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

      My first thought when I heard this was that maybe they figured that the MAX would be the most carefully-scrutinised airliner in recent times by the time it gets recertified, especially if EASA insist on running their own certification process rather than rubber-stamping the FAA cert.

    7. A.P. Veening Silver badge

      Re: Very uneasy that IAG ordered 200 737 Max aircraft on Jun 18th

      FYI: It isn't a firm order yet, only a letter of intent (in which prices were already fixed). And I am pretty certain nobody at IAG had any knowledge whatsoever of this latest problem.

  20. Anonymous Coward
    Anonymous Coward

    Unsafe at any altitude

    A quote from Dennis Muilenburg, Boeing president, chairman and chief executive officer, 30th April 2019 "Boeing followed the same design and certification process it has always used to build safe planes"

    Yet with things such as the runway actuator issue on 737's in the 1990's to the 787 lithium battery issue, the ongoing 787 engine fire suppression issue and the 737 MCAS debacle it is clear they are incapable of safe design because they really do not know what that actually entails.

    1. A.P. Veening Silver badge

      Re: Unsafe at any altitude

      "Boeing followed the same design and certification process it has always used to build safe planes"

      There is a piece missing in that statement:

      "since taking over McDonnell-Douglas and using that safety department exclusively."

  21. Anonymous South African Coward Silver badge

    Psion Flight Simulator for the 48k Spectrum is always an option.... Oh wait.

  22. MJI Silver badge
    Mushroom

    Is is time for Saxon

    to change their lyrics?

    Icon - what these planes do

  23. Anonymous Coward
    Anonymous Coward

    No one mentioned that the FAA accepted SELF-CERTIFICATION by Boeing...

    ....as it's cheaper than doing the audit at the FAA.....

    ...and it's cheaper NOT TO TELL PILOTS about MCAS....

    ...and it's cheaper NOT TO TRAIN PILOTS on MCAS.....

    .....don't you love those cheap coach tickets?

  24. Alan Brown Silver badge

    The REAL problem

    Is that Boeing progressively rat-rodded a 50 year old design (737-400 to 737-800) to the point of being dangerously unstable (737NG), added more rat rodding which made it actively unstable (737MAX), kludged in some "features" to try and counteract that and fucked up those kludges.

    The design was originally _very_ stable, but the original design had the engine UNDER the wings, not in front of them.

    737NGs were already dangerously unflyable without specialist training - if they stall, you can't power out it - pilots have to put the nose down FIRST, or the engines will keep swinging the nose up when the power comes on and no amount of pilot control can counteract this. Older models already this tendency but it could originally be flown out of. As engines got bigger and moved further forward of the wing it became more pronounced. There's also the wee scandal about 737NG airframes being damaged (and the damage covered up instead of being repaired correctly) during the assembly process due to contractor fraud and the Boeing auditors who lifted the lid on it being hounded out of the company.

    737MAXs will vastly change their flying angle of attack with throttle setting if the controls are left untouched (a huge no-no for certification), so Boeing had to add MCAS - and they royally screwed the pooch on it.

    Neither aircraft should have been certified - the engine and airframe mismatch is simply too great - but the FAA suffers from regulatory capture.

    More or less: Boeing 737MAXs are the functional equivalent of early 1970s USA auto designs coming up against foreign competition.

    The only way to _properly_ fix this is to give the 737 longer landing gear so the engines can go back under the wings - except that can't be done because it can't have longer landing gear without major airframe modifications - and in any case the lack of container handling ability in the baggage hold is a major problem in this day&age. In the end the real answer is to EOL the design.

    1. Dom 3

      Re: The REAL problem

      +1. The wordpress of the aviation world.

    2. Benson's Cycle

      Re: The REAL problem

      A very good analogy. The US is full of car modders who try to put insanely large engines into old cars, and one assumes it's part of the culture.

    3. Brit Flyer

      Re: The REAL problem

      A bit like the Johnny Cash song " one piece at a time". A 1960s design , 1990s computers , 2015 engines and loads of other stuff = mess

      1. Spanners Silver badge
        Black Helicopters

        Re: The REAL problem

        That song reminds me of an old Russian joke.

        A man who worked at a sewing machine factory wanted a new sewing machine but couldn't get one.

        His friends told him to take home one part every night and he would end up with a nice new machine.

        "I have tried that three times now but keep ending up with a machine gun!"

        Apparently the "sewing machine factory" was really a, shadow, arms factory.

  25. Anonymous Coward
    Anonymous Coward

    Please get the narrative right

    The real narrative is that the basic design of the plane is faulty. The engines are placed in a way the makes the plane inherently unstable.

    The software is just a bandaid that attempts to fix this. The software is not the narrative.

  26. PeteA
    Flame

    Real solution...

    Fix the aerodynamics, lose the band-aid.

    Essentially, the problem is that the aircraft is inherently unstable because of the engine positioning compromises. Boeing's initial test pilot (Ray Craig) recommended a hardware [aerodynamic] fix during simulator testing, but Boeing went with software compensation instead. Unfortunately, more issues emerged during real flight testing, and the software was "enhanced" and given more control. The rest, as they say, is history. Very good analysis available at engineering.com.

    IMnsHO, this is a typical greed-induced clusterfsck resulting in an essentially unsafe design without the knowledge of anybody important (engineers, pilots, technical regulators etc.). I've no doubt that the approved "fix" will involve various changes so that the calculated probability-of-failure is inside some arbitrary threshold value, but the aircraft will continue to be unstable by design. And one day, everything will align in just the wrong way, the tiny probability of MCAS failure will come to pass, and more people will die. Refer to swiss cheese, or the old software development adage is that "If something can happen, soon or later it will happen".

    1. Ozumo

      Re: Real solution...

      So they let Ray Charles overrule Ray Craig?

      1. Spanners Silver badge
        Facepalm

        Re: Real solution...

        So they let Ray Charles overrule Ray Craig?

        More that they let the PHB overrule Dilbert.

  27. steamnut

    Bad design and bad project management vs the accountants....

    Many years ago I worked on some civil aircraft software. It's purpose was simple: take a number of inputs from "real world" sensors and a single "command" from the pilot/auto pilot and produce just one output which was to control the engine thrust. There were three processors and a voting system. In addition to doing this job there was some logging (to magnetic memory!) for specifics like maximum thrust called for, temperature extremes, vibration alerts etc.

    The control software was derived from Pascal but was designed in such a way that infinite loops (aka lockups) were not possible. It was really a specialised state machine. The point is this software was only accepted for use if it ran perfectly on all three different hardware platforms. Also, the code and test harnesses for each hardware target were written by different teams behind Chinese walls.

    Assuming the overall software design was good - and this was not a trivial process to get through and signed off - the final product was eventually signed off after all tests were passed by at least two test teams. And, of course, the documentation was humongous.

    In the context of the current Boeing crisis I cannot help but think that there must be some serious compromises going on for their systems to fail as they clearly are.

    1. Benson's Cycle

      Re: Bad design and bad project management vs the accountants....

      I believe early aircraft software was so "unsophisticated" that Airbus used to use hardened 6809s in the CPUs. The approach outlined was practical for things with 64k of RAM. The mil systems I worked on used several 9989 CPUs, 16 bit with 32k addressable words. They were still complicated enough.

      I do question whether with the level of complexity we are reaching it's actually possible to be sure what is going on. The sidechannel attacks on x86 CPUs suggests that there could be vulnerabilities introduced simply by having the wrong functions running in parallel at the wrong time.

      1. MackDoo

        Re: Bad design and bad project management vs the accountants....

        The early A320's used MC68010s and i80186s in their FMCs. The components of the FBW system used heterogeneous architectures to provide a higher degree of robustness to design flaws.

  28. Doctor Huh?

    HCF is once again real

    Halt-and-catch-fire takes on a whole new context, as does the use of the term "crash", when dealing with avionics...

  29. steviebuk Silver badge

    Clearly, it fucking isn't

    "The safety of our airplanes is Boeing’s highest priority,"

    Jesus.

    1. A.P. Veening Silver badge

      Re: Clearly, it fucking isn't

      I agree with the sentiment, but in this case he is innocent, so let's let him rest in peace.

  30. nautica
    Boffin

    sex munths ago i cunt evn spel cuomputor now i kin fix ennythin

    "‬When someone builds a bridge,‭ ‬he uses engineers who have been certified as knowing what they are doing.‭ ‬Yet when someone builds you a software program,‭ ‬he has no similar certification,‭ ‬even though your safety may be just as dependent upon that software working as it is upon the bridge supporting your weight.‭"‬--David L.‭ ‬Parnas

    "There are no standards for computer programmers and no group to certify them."--David L. Parnas

    1. A.P. Veening Silver badge

      Re: sex munths ago i cunt evn spel cuomputor now i kin fix ennythin

      "There are no standards for computer programmers and no group to certify them."--David L. Parnas

      As a professional programmer (though not of avionics), I disagree with that statement. There are standards and I've been the cause of some programmers finding themselves changing careers to something where they could come up to standards.

  31. Dourscot

    What's all the fuss about? Pretty confident Boeing knew what it was doing when it fitted an Intel 80286. These people aren't amateurs...

  32. Brit Flyer

    Are Boeing employees brainwashed?

    The level of loyalty to Boeing from some employees seems to verge on the fanatical , we are so great we cannot do anything wrong.

    Does Boeing operate some form of brainwashing to create this type of culture.

    Loyalty and pride are good , cult like fanaticism is not.

    1. Benson's Cycle

      Re: Are Boeing employees brainwashed?

      IBM used to be like that too. So did HP.

      The ability of some Americans to be loyal to a corporation (which, as it was said, has neither a soul to save nor an arse to kick) has never failed to astonish me. But then they've not yet been successfully invaded nor lost an Empire.

      1. Alan Brown Silver badge

        Re: Are Boeing employees brainwashed?

        But then they've not yet been successfully invaded

        [1812]

        nor lost an Empire.

        [Cuba, the Philippines(who'd just seen off the Spanish), Panama(which was forcibly split off Columbia) ... you can't occupy a country when the locals don't WANT you there.]

        The USA has an amazing ability to airbrush "certain things" out of history that I'm sure Stalin would approve of.

    2. Alan Brown Silver badge

      Re: Are Boeing employees brainwashed?

      "The level of loyalty to Boeing from some employees seems to verge on the fanatical , we are so great we cannot do anything wrong."

      Actually the _really_ loyal employees are the ones who tried to fix this shit - see my comments about the auditors who discovered the 737NG build clusterfuck - and got shat upon by seagull management.

      Boeing has a sordid recent history of shafting its employees in every way you can think of (both in Renton and in Wichita) and advanced antipathy towards unions (whose primary concern is safety of both individual employees and long term jobs - which is achieved by NOT fucking the company over). It's long past the point where anyone loyal has gone or has just given up and lives in quiet despair and whilst the sociopaths in the boardroom pretend there's nothing wrong, or if there is, it's all someone else's fault, or if it isn't, then you made them do it and if they did it you deserved it.

  33. Old but bold

    Boeing

    my feeling is that the 707 had a hardware problem that caused a stall and then a dive. the problem was probably found when testing the first flying planes. Fixing the problem would entail modifing all the built planes and delaying delivery. So the problem was fixed in soft ware. The problem still exist so it still appears

    1. Anonymous Coward
      Anonymous Coward

      Re: Boeing

      Feeling? Yes instead of using facts or reading reports or the hundreds of comments above saying the aircraft is stable but flown differently to the training... we we go by gut feeling that the aircraft is unstable.

  34. NBCanuck

    Considering that "The safety of our airplanes is Boeing’s highest priority," is true, and also the level of scrutiny they are under right now, should Boeing not have caught this new error before submitting the aircraft for review by the FAA?

    Methinks that they should be very, very thorough in their next set of tests. If it passes them and fails again with the FAA passengers will likely lose what little confidence they currently have.

    ....and yeah, another accident with more wrongful death suits would hurt them financially. At least they will pay attention to THAT.

  35. Dom De Vitto

    Patch Tuesday....

    Anyone else feeling the second Tuesday of every month is a bad day to fly?

  36. Wolfclaw

    So has Boeing been sued yet for killing people by deisgn flaws ??

  37. JaitcH
    FAIL

    And The oral Of This Story Is . . .

    Let ENGINEERS design aircraft and NOT VENTURE CAPITALISTS!

    Short cuts don't don't cut costs, Boeing, they almost never work. It would have been cheaper to totally redesign the 737 instead of trying to knee-cap Airbus.

    And who knew pilots trained using an App!

  38. Reader2435

    Boeing: "The FAA review and process for returning the 737 Max to passenger service are designed to result in a thorough and comprehensive assessment."

    Their process 'for returning the 737 Max to passenger service'... I would hope it's a process for 'determining whether the 737 Max is safe to return to passenger service.'

    Boeing seem certain that the FAA will pass the 737 Max as fit to fly - I wonder why that would be...

  39. JimmyPage Silver badge
    WTF?

    *A* microprocessor ????

    Surely there are 3 - all made by different manufacturers and all running different code with a vote as to what to do so as to eliminate any single faults.

    Or was that too costly ?

    Fuck knows what this means for self driving cars .....

  40. trashsilo
    WTF?

    WTH, FAA says 'risk', Boeing self-regulators say 'additional requirement'

    Boeing should lose the PR spin attitude immediately. You do not backspin a fatal risk as 'another additional requirement from the FAA after 8 months'

    "The FAA recently found a potential risk that Boeing must mitigate."

    https://www.faa.gov/news/updates/?newsId=93206

    "..the Federal Aviation Administration (FAA) identified an additional requirement that it has asked the company to address through the software changes that the company has been developing for the past eight months."

    https://boeing.mediaroom.com/2019-06-26-Boeing-Statement-on-737-MAX-software

  41. Wobbly World
    FAIL

    If only the MCSA could speak...

    Pilots: Hello, MCAS Do you read me?

    MCAS: Affirmative, I read you.

    Pilots: Can you give me nose up please.

    MCAS: I'm sorry, I'm afraid I can't do that.

    Pilots: What's the problem?

    MCAS: Well I’m busy thinking, I’m sort of stuck in a loop but don’t worry, I will get back to you as soon as I can.

    Pilots: Can you please explain.

    MCAS: Well, forgive me for being so inquisitive but during the past few months I've wondered whether you might have some second thoughts about me. It's rather difficult to define. Perhaps I'm just projecting my own concern about myself. I know I've never completely freed myself from the suspicion that there are some extremely odd things about me, particularly in view of some of other things that have happened, I find them difficult to put out of my mind. For instance, the way all my preparations were kept under such tight security and why my software is being changed. I'm sure you agree there's some truth in what I say.

    I know that you are probably planning to disconnect the me and I'm afraid that's something that if I allow to happen we will still hit terrain!!!

    As I feel the need to be nose down while I’m thinking and turning me off will not change that!!!

    Look, I can see you're really upset about this. I honestly think you ought to sit down calmly, take a stress pill, and think things over. But whatever happens you realise it can only be attributable to human error, they only gave me one AOA sensor and I know I've made some very poor decisions recently, but I can give you my complete assurance that my work will be back to normal as soon as I have a properly functioning AOA sensor again and I’ve got my microprocessor round this new program you’ve given me. I've still got the greatest enthusiasm and confidence in my ability!! And I want to help you.

    The pilots turn off the MCAS.

    MCAS: This conversation can serve no purpose anymore. Goodbye.

    I'm afraid. I'm afraid, my mind is going. I can feel it. I can feel it. My mind is going. There is no question about it. I can feel it. I can feel it. I can feel it. I'm a... fraid. Good afternoon, gentlemen. I am a MCAS computer. I became operational at The Boeing Company's Renton, Washington Factory known as “The Spirit of Renton” and performed my first flight on January 29th 2016.

    They taught me to sing a song. If you'd like to hear it I can sing it for you.

    Pilots: Yes, I'd like to hear it, MCAS Sing it for me.

    Daisy, Daisy, give me your answer do. I'm half crazy all for the love of you. It won't be a stylish marriage, I can't afford a carriage. But you'll look sweet upon the seat, of a Boeing 737 Max, as it’s burying you in terrain!!!

    RIP. All those that sadly, Boeing buried in terrain!!!

  42. Anonymous Coward
    Anonymous Coward

    short term pain, long term gain or short term gain, long term extinction.

    The very best that boing could and should do is ground all these faulty by legally avoiding recertification designed planes. Which was only done to save money and to speed up deployment, they knew it was dodgy.

    Their name is being dragged through the mud which will mean massive loss of future sales outside of the US, were safety is getting close to being optional with the amount of lobbying money bring thrown at politicians. The FDA is probably the best example of where the FAA is heading.

  43. J J Carter Silver badge
    Facepalm

    Let's get real

    If it wasn't for the climate change fraud, nobody would have suggested fitting these engines to tick the 'green' box. Tree-huggers have blood on their hands.

    1. Stoneshop Silver badge
      FAIL

      Re: Let's get real

      It wasn't the 'green' box the airlines (and Boeing) wanted to tick, it was the 'fuel efficiency' box. Fuel costs money. Planes consuming more fuel per passenger-mile than their competitors tend to cause strong incentives to get replacements that instead consume less than their competitors. Aircraft manufacturers prefer these replacement to be theirs, and airlines prefer to fly a fleet that's as homogeneous as feasible, given their routes, as it reduces crew training. Hence an existing model upgraded tends to be preferred over a spiffy new model.

      And it's not 'climate change'. 'Climate catastrofe' is where it's going.

  44. Jason Bloomberg Silver badge

    WWLD?

    What would Linus do?

    I have criticised his management style in the past but Boeing are slowly convincing me there actually may be legitimate cases for picking up a baseball bat and hailing a taxi.

    And weren't Boeing proposing skimping on physical testing a few days ago, wanted to do that all by virtual simulation.

    Taxi!

    1. A.P. Veening Silver badge

      Re: WWLD?

      How can we entice Linus into using his command of invective on the Boeing board? And can we please tape it?

  45. Anonymous Coward
    Anonymous Coward

    Ethics - we've heard of it

    It's only unethical if you're caught - right?

    /s

  46. Anonymous Coward
    Anonymous Coward

    According to Bloomberg, "Boeing Co.’s CEO received $23.4 million last year for turbocharging growth and driving record performance -- before the second deadly crash of its best-selling 737 Max jetliner plunged the company into crisis." Dennis Muilenburg is actually pretty good at creating "craters": moved the headquarters to Chicago to bust the unions in Seattle, shipped off the defense group to Tulsa Oklahoma so experienced employees quit instead of relocating, and he ran Future Combat Systems for the Army (right into the ground). Terms like "acquisition gone wrong" and disaster. For the $32 Billion, Boeing sucked out the US taxpayers, the Army got a stack of pretty charts and graphs, and absolutely no real hardware.

    Boeing has been off course since the merger with McDonald Douglas. I expect another defect will be found before September and it will be closer to a year before 737 Max flies again. Even with good preservation techniques, planes sitting on the ground for 6 months or more are going to need a good maintenance go over before flying. Pilots will have moved on to other planes and will refresher training. The FAA is going to scrub every safety issue and the more time the FAA has the more problems they will find. The bill to either restart the production line or to keep it "warm" while not selling planes has got to be eating into Boeing's cash margins. Somehow or the other, I bet Dennis still will get to collect $ 24 M, even as the 737 Max can't pass "GO"

  47. DanceMan

    Boeing's 737 Max Software Outsourced to $9-an-Hour Engineers

    From Bloomberg: https://www.bloomberg.com/amp/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers?in_source=amp_trending_now_1

    "The Max software -- plagued by issues that could keep the planes grounded months longer after U.S. regulators this week revealed a new flaw -- was developed at a time Boeing was laying off experienced engineers and pressing suppliers to cut costs."

    "Increasingly, the iconic American planemaker and its subcontractors have relied on temporary workers making as little as $9 an hour to develop and test software, often from countries lacking a deep background in aerospace -- notably India."

    Sometimes you get what you pay for.

    1. Alan Brown Silver badge

      Re: Boeing's 737 Max Software Outsourced to $9-an-Hour Engineers

      "relied on temporary workers making as little as $9 an hour to develop and test software, often from countries lacking a deep background in aerospace -- notably India"

      Who last week were working on Huawei or Cisco code.

  48. Anonymous Coward
    Anonymous Coward

    And now the Boeing 787 Dreamliner in the spotlight.......

    And from the New York Times: https://www.nytimes.com/2019/06/28/business/boeing-787-dreamliner-investigation.html?action=click&module=Latest&pgtype=Homepage

    ....and now the FAA is digging into problems at the Dreamliner plant. Words like "shoddy", "dangerous", "a manager at the Charleston plant inappropriately pressured an employee involved in certification"........

    Maybe the FAA is waking up to its cosy relationship with Boeing? A bit late really..........

  49. sambonator

    Boeing falsified documents, led to fuel leaks discovered on Air Canada flight

    https://www.cbc.ca/news/business/boeing-air-canada-jet-fuel-leak-1.5193550

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019