back to article 'Software delivered to Boeing' now blamed for 737 Max warning fiasco

As the 737 Max scandal rolls on, "software delivered to Boeing" has been blamed by the company for the malfunctioning of a safety display. In a statement issued over the weekend, the American airliner manufacturer admitted that its software was not properly displaying fleet-standard warning captions to pilots. This admission …

  1. Trollslayer Silver badge
    Flame

    Management's job

    Is to blame everyone else.

    What happened to "The buck stops here."?

    1. DavCrav Silver badge

      Re: Management's job

      "What happened to "The buck stops here."?"

      It does stop here. But management are on holiday at the time.

      1. Alistair Silver badge
        Windows

        Re: Management's job

        Nahh, its cheaper to fire the underlings and plebs. The golden parachutes for upper management cut into the profit line and stock price.

        (as sarcastic as that comment is, I'm rather sure it's far too true)

      2. el kabong Silver badge

        A rogue engineer dit it, Muilenburg and everyone in manglement are investigating

        Once they find the guy they'll make him pay, he will not escape.

        1. 's water music Silver badge

          Re: A rogue engineer dit it, Muilenburg and everyone in manglement are investigating

          pretty sure they will eventually discover that it was one of the big engineers that did and and ran away without leaving their name.

          So sorry, Is this compo fund enough?

        2. Weiss_von_Nichts

          Re: A rogue engineer dit it, Muilenburg and everyone in manglement are investigating

          Rogue engineer? I thought it was just declared to be a "software error"?

          1. JBFromOZ

            Re: A rogue engineer dit it, Muilenburg and everyone in manglement are investigating

            it was a rogue AI, and it has been taken offline and disciplined.

    2. Giovani Tapini Silver badge
      Stop

      Re: Management's job

      @Trollslayer - I've not met many managers who will accept a buck stopping.

      1. Anonymous Coward
        Anonymous Coward

        Re: Management's job

        > I've not met many managers who will accept a buck stopping.

        But, presumably, many who stop to accept a buck?

        1. Michael H.F. Wilkinson Silver badge

          Re: Management's job

          > But, presumably, many who stop to accept a buck?

          and many who stop at nothing to accept a buck

        2. Archtech Silver badge

          Re: Management's job

          "One thing more dangerous than getting between a grizzly sow and her cub is getting between a businessman and a dollar bill".

          - Edward Abbey ("A Voice Crying in the Wilderness”)

      2. Anonymous Coward
        Anonymous Coward

        Re: Management's job

        I used to tell my stadff that the buck did stop with me - provided I was advised of its trajectory, colour and composition.

      3. macjules Silver badge

        Re: Management's job

        Call it as it is. If I recall correctly "The buck stops here" was a poker reference to a knife that was passed on if a player did not want to deal.

        In this case it appears that Boeing (or "Bits Of Engine In Neighbour's Garden" as they used to be known) were playing poker with their passengers and crew members' lives. Hopefully their managers can live with that.

        1. Piro

          Re: Management's job

          Blithely Obtuse Engineering, Imploring Nose to Ground

        2. Flywheel Silver badge

          Re: Management's job

          I wonder how many of their managers and bean-counters fly in their own product. That could be an interesting FOIA (or equivalent) request!

          1. VulcanV5

            Re: Management's job

            Given the obvious calibre of Boeing's management and the inevitably limited intelligence of any bean-counter, it's doubtful they'd know the difference between a bus, an Airbus, or a Boeing.

            1. Anonymous Coward
              Anonymous Coward

              Re: Management's job

              Well thank goodness one of BOINGS management team has now joined the administration so that is proof BOING’S peeps are the best.

        3. Archtech Silver badge

          Re: Management's job

          "Hopefully their managers can live with that".

          In prison, by rights.

      4. SWCD

        Re: Management's job

        They are definitely out there, in business and in politics. They only need to confess to the mess, even if they weren't directly/solely responsible. Then, rather than run a mile, other outfits re-employ them as it's known they're happy to fall on their sword. A willing chump can be very useful - make the pay good enough and anyone is happy looking incompetent.

        1. Flicker

          Re: Management's job

          Ahh! Chris Grayling.. Apart from the minor detail of him refusing, ever, to admit responsibility for anything.

          1. Vometia Munro

            Re: Management's job

            He's quite special. I mean even as someone with a massive "kick me" sign on his back, he is *such* a complete and utter liability that I'm perplexed as to how come he's managed such a long political career given his well-documented history of making both a disaster and a crisis out of anything he's ever involved with. And that the consequences of his actions never seem to catch up with him; or that he's clearly entirely oblivious to the chaos left in his wake as he evidently considers his own awesomeness to be an incontrovertible fact.

            I know most people have their own particular failings, certainly myself included, but "Failing" Grayling really does seem to have a full house.

            1. Graham Cobb

              Re: Management's job

              I suspect prime ministers find it convenient to have a fall guy who can be put in place wherever necessary. I assume he is well compensated.

    3. KarMann

      Re: Management's job

      'What happened to "The buck stops here."?'

      It does stop here, for certain values of 'here'.

    4. Anonymous Coward
      Anonymous Coward

      Re: Management's job

      It was superseded by "Excrement falls down not up!"

      1. Rob Gr

        Re: Management's job

        It was superseded by "Excrement falls down not up!"

        Which conveniently, also applies to their planes.

    5. CrazyOldCatMan Silver badge

      Re: Management's job

      What happened to "The buck stops here."

      Too many bucks (in the dollar sense) stopping there - so manglement no longer wants to hear about something that might (in the short term) impact their stas or bonuses.

      Besides which, there's always a few rouge engineers that can be blamed.

      1. Sir Runcible Spoon Silver badge
        Coat

        Re: Management's job

        "Besides which, there's always a few rouge engineers that can be blamed."

        I'll bet they're red-faced

        1. Sgt_Oddball Silver badge

          Re: Management's job

          They're not so pretty now.

        2. GrapeBunch Silver badge

          Re: Management's job

          "I'll bet they're red-faced"

          I'll beet their red-faced behinds.

          NZFTFY.

      2. Anonymous Coward
        Anonymous Coward

        Re: Management's job

        This engineer has never used rouge is his life! This life anyways.

      3. Archtech Silver badge

        Re: Management's job

        "Besides which, there's always a few rouge engineers that can be blamed".

        'When they came for the communists, I said "He's next door. Take him away. Goddam commies"'.

        - martinX’s sometime Slashdot sig

        1. DCFusor Silver badge

          Re: Management's job

          Slashdot does indeed seem to be a hotbed of people who learned spelling and grammar via listening to audio podcasts.

          "When you use a spiel checker, you rape what you sew."

          "Grammar, the difference between knowing your shit, and knowing you're shit."

          Not even worth modding the bastards now, and when you shame them, as above, it's either a whoosh, or you're a grammar nazi; well, the sting is off now that everyone is a nazi who isn't a communist.

          -- a real grammar nazi, and proud of it. What fool doesn't think it's worth it to learn their own language?

          1. tfb Silver badge
            Terminator

            Re: Management's job

            Grammar, the difference between knowing your shit, and knowing you're shit.

            Only if you think spoken language does not have grammar; and if you think that, you know almost nothing anything about language, and what you do know is wrong.

      4. Rob Gr

        Re: Management's job

        All together now...

        I'm an Engineer and I'm OK..

    6. Brit Flyer

      Re: Management's job

      If you earn nearly $24 million a year you are likely to be good at blaming others and protecting your own backside

      1. Ted Treen
        Flame

        Re: Management's job

        They might be paid $24Million per annum: there's no damned way they EARN $24Million per annum.

    7. John Miles

      Re: "The buck stops here."?

      I think it disappeared when the belief you only need to be able to "manage people" to be a manager became popular.

      1. rskurat

        Re: "The buck stops here."?

        that would be the '80s, when business school MBAs were taught that you didn't need to know anything about the underlying business, you just needed to know how to "manage" - dishwashers, banking, oil refineries, hospital management, it's all the same when you know Excel.

    8. Roq D. Kasba

      Re: Management's job

      Pathetic weaseling out itself tells you the culture is completely fucked up that they're not taking responsibility or accountability for software they designed, ordered, bought, tested, installed and resold.

      The attitude tells you the company is not to be trusted with responsibility.

      1. veti Silver badge

        Re: Management's job

        Every story I see about this episode makes me angrier with Boeing. I mean, management avoiding personal responsibility - that I expect. But they seem intent on denying that there was anything wrong at all.

        So they won't learn lessons from it.

        The CEO should be in jail by now.

        1. Roq D. Kasba

          Re: Management's job

          If you outsource your production, you outsource your reputation. You don't get to cut corners then blame the other guy.

          And this is on top of only having a single point of failure with a software patch to avoid having to recertify the airframe. And their cuntyness with Bombardier recently. Frankly Boeing deserve a good kicking right now so the shareholders wake the fuck up and hold the board to account.

        2. VulcanV5

          Re: Management's job

          @veti: Management (or Unlucky Representatives Thereof) will have ample opportunity to appear in Court (even if not yet jail) as the $multi-million law suits progress.

          There are some seriously Pissed Off people out there, including those you really don't want to piss off at all, amongst them Ralph Nader, a relative of one of the victims of the Ethiopean crash. Boeing's corporate PR department is going to struggle real hard to contend with the damage Nader may choose to inflict.

      2. VulcanV5

        Re: Management's job

        @ Roq D Kasba -- it also demonstrates how much faith we should have in the US's aviation regulator, seeing as how it was more than happy for Boeing to self-certify itself on critical safety issues. And then wanted the appalling 737Max to keep on flying -- an oibsolecent airframe perpetuated by Boeing in hope of making vast profits, tarted up with a fallible software system intended to address the problems arising from the perpetuation of the cost-effective old rather than a cost-incurring 'new'. Must be a nice revolving door between Boeing and the FAA. Lubricated, as always, by money.

    9. tapemonkey

      Re: Management's job

      Phrases like buck stops here and golden parachutes are stroking my innately dark sense of humour. Is it too soon for Boeing the sound a Max 7 makes as it hits the ground

  2. 2+2=5 Silver badge

    'Software delivered to Boeing'

    > 'Software delivered to Boeing' now blamed for 737 MAX warning fiasco

    I read this line in the article and assumed that some part of the software development had been outsourced meaning that there would be deliveries of code, but there is no further mention of it.

    So what software was delivered to Boeing and by whom?

    1. joeW Silver badge

      Re: 'Software delivered to Boeing'

      The tone suggests an unmarked van drove up to the side of Boeing HQ in the middle of the night and, completely unbidden, slung a bundle of code wrapped in black plastic and duct tape over the fence before screeching off into the darkness.

      1. martinusher Silver badge

        Re: 'Software delivered to Boeing'

        >The tone suggests an unmarked van drove up to the side of Boeing HQ in the middle of the night and, completely unbidden, slung a bundle of code wrapped in black plastic and duct tape over the fence before screeching off into the darkness.

        Nah... it was on a 3.5 inch floppy that turned up in the mail. (Or maybe a USB stick that someone found lying on the ground in a parking lot.)

        The use of passive tense is designed to imbue the event with an aura of deniability, as in "it wasn't really our code but we thought we'd better install it anyway".

        1. Lord Elpuss Silver badge

          Re: 'Software delivered to Boeing'

          ” (Or maybe a USB stick that someone found lying on the ground in a parking lot.)“

          With the Airbus logo scratched off it?

          1. Kabukiwookie Silver badge

            Re: 'Software delivered to Boeing'

            It's usually the other way around, with the CIA spying on Airbus and passing along their trade secrets to Boeing.

            Maybe Airbus has stopped using Cisco switches, forcing Boing to start writing their own code.

            1. spireite

              Re: 'Software delivered to Boeing'

              Moved to Huawei???

          2. Lord Elpuss Silver badge

            Re: 'Software delivered to Boeing'

            8 thumbs down! Really - what did I say? Not funny? I thought it was at least worth a chuckle ;))

            1. Archtech Silver badge

              Re: 'Software delivered to Boeing'

              You upset the Yank chauvinists.

    2. LDS Silver badge

      "So what software was delivered to Boeing and by whom?"

      Some sweat shop between 68°E and 97°E?

    3. Electronics'R'Us
      Holmes

      Re: 'Software delivered to Boeing'

      Boeing is a systems integrator.

      The multi-function display and it's computing devices would have been put out for bid and awarded to a company that makes such systems. Boeing would send a set of requirements (down to how much power it would consume, size or envelope for the box and a whole lot more).

      Let me reiterate that: Boeing does not design the electronics or the software that runs on them in their aircraft apart from some systems that have an operational flight program (OFP) requirement that is over and above fundamental operation of said box.

      The software would have been required to conform to DO-178 (B or C depending on when it was done), but to save money apparently they did not require Design Assurance Level (DAL) A (a failure is catastrophic) but to severe (DAL B), DAL A software is extremely expensive for those who have not been in the Aerospace food chain. DAL B is less so, but still pretty expensive. If there are FPGAs involved, then DO-254 would be the requirement.

      Why pay for DAL A when the box clearly was not considered safety critical (the use of a single sensor shows that) when you can pay a bit less per box. Note that for these systems, the vendors have to price in the development as part of the price per box as it is rare for the airframer (Boeing) to pay NRE costs for system development.

      Boeing aren't going to get off the hook here though; it is they that wrote the requirements for the system, and if 'AoA disagree' was required to be displayed then their testing should have shown that it was not being so displayed.

      It will be interesting to see what the requirements were and the compliance matrix that would have been generated at the vendor actually shows.

      1. CrazyOldCatMan Silver badge

        Re: 'Software delivered to Boeing'

        compliance matrix

        I suspect that the view of the compliance matrix was thoroughly obscured by large piles of green bills..

      2. nerdbert

        Re: 'Software delivered to Boeing'

        You're right: Boeing is definitely not off the hook. In fact, from what little I know I'd suspect that they're the most exposed, although they will do their best to try and blame their subcontractor (who probably will fight back pretty hard from what little I've heard).

        MCAS was delivered by a supplier, yes. From the western hemisphere, though. DAL B, at best. I've heard rumblings that even the subcontractor wasn't pleased about the system they were designing and had internal battles about what they were expected to deliver and at the quality level. Apparently the coders were thinking DAL A was more appropriate and that the system they were told to deliver wasn't what they wanted to deliver, but that's not how Boeing wrote the requirements.

        Boeing is the systems integrator. They define how the systems behave, not the subcontractors. The contractors who write the software have specs from the system integrators that they follow, and they write their software as they're told. Boeing in this case has two AoA sensors, one per side. In the NG planes they were independent, one for the pilot and one for the co-pilot, and it was pretty obvious if one disagreed with the other since the pilots could compare results and ignore the one that disagreed with their eyes. But in the MAX the MCAS is tied to one side only, and Boeing seems to have hidden the AoA and override functions. The subcontractor probably suspected that this was a shortcut and suboptimal, but even if they did, they don't know how the other systems that Boeing was installing would interact with MCAS, how the override would function, etc. Unless it was clearly written in the contract that MCAS needed to monitor both sides I doubt that the subcontractor will get a whole lot of the blame since they really couldn't see the overall system.

        (A good systems engineer is a rare thing. It's a lot more fun to work on the building blocks. I did system engineering for a decade, got great praise and reward for the stuff I did, and I got an excellent feeling why in NASA it's hard to fly anything new, even on test satellites. Then I quit and went back to designing the subsystems. It was a lot more relaxing and fun working on the subsystems where I could really figure out what was going on and optimize what was delivered without wondering if the fact I changed some minor thing was going to bring the whole system crashing down.)

        1. Anonymous Coward
          Anonymous Coward

          Re: 'Software delivered to Boeing'

          @NerdBert,

          MCAS was delivered by a supplier, yes. From the western hemisphere, though. DAL B, at best. I've heard rumblings that even the subcontractor wasn't pleased about the system they were designing and had internal battles about what they were expected to deliver and at the quality level.

          I've read elsewhere who the supplier is, but will refrain from naming them here. I suspect (pure guess in fact) what lies behind Boeing's recent curious public statement that MCAS worked just fine, as intended, (or words to that effect) is that the supplier threatened to make public the email train between Boeing and themselves.

          However I don't think they're not off the hook completely. Yes it was Boeing's spec, Boeing's responsibility but as engineers they still have a general obligation to do the right thing. They could have informed the FAA of the problems. Perhaps they did but it now looks like that wouldn't have done any good. They could have refused to do the work. Perhaps the reason they won it was because they were the only ones that said yes. If they were truly worried about safety they could have talked to other certification authorities, then senior politicians with oversight responsibility, or ultimately gone public.

          If one, as a subcontractor, has genuine concerns about safety then what you do not do *is keep quiet about it, because people may die*.

          So, what's going on? Why wasn't there a big public row between Boeing and their supplier?

          Here's another example. There's a court case going on between a former Boeing electrical engineer (fired?) and Boeing, focused on the electrical system for the up coming 777x. After being brought in he condemned the design after only 1 month, and paid the professional price. GE, who picked up the subcontract, ended up coming to the same conclusion and told Boeing so, according to documents that have come out in the court case. Whether or not the problems were resolved or not, I don't know. Probably not.

          So if Boeing's subcontractors are regularly having concerns with the specs they've been given, but are doing the work anyway, and are keeping quiet about it, is that a sign that Boeing are bullying them into submission in the same way they're thought to treat their own staff?

          That's another thing for the EASA, CAAC to worry about (European and Chinese regulators). I omitted the FAA, they don't seem to think there's many problems...

          1. Kabukiwookie Silver badge

            Re: 'Software delivered to Boeing'

            as engineers they still have a general obligation to do the right thing. They could have informed the FAA of the problems. 

            Though I 100% agree with the sentiment, you're asking engineers to whistleblow to an organisation that allowed Boeing to self-regulate to begin with.

            In the last decade more whistleblowers have been prosecuted than in the decades preceding it.

            Should engineers bring it up with management? Sure, but if management doesn't actually do anything with the info, it should no longer be the engineer's problem (not in an environment that is so hostile to anyone who actually speaks out).

            Changing this environment can only be done from the top, and that's actually the origin of this hostile environment to begin with.

            If nothing radical is done at the top, Boeing is fscked in the longer run regarding anything to do with engineering. The people who can, will suck it dry and leave the dessicated husk behind for the remaning people who didn't move job in time.

            1. jmch Silver badge

              Re: 'Software delivered to Boeing'

              "In the last decade more whistleblowers have been prosecuted than in the decades preceding it."

              Whistleblower protection is at best a sham, but more usually a combination of fig-leaf policy for the company / government to cover its arse and bait to lure whisteblowers to taking their concerns to the very people who want to sweep them under the carpet instead of going public.

              I believe Julian Assange is a primadonna publicity whore but I have to admit that Wikileaks and all the offshoots of the concept are really the only workable solution for whistleblowers.

      3. Uncle Ron

        Re: 'Software delivered to Boeing'

        Private companies are not required to put anything "out to bid." And if they do, they aren't required to take the lowest bid. Further, highly complicated and highly engineered systems (either HW or SW) are extremely difficult to "spec" in a bid.

        It is very easy to spec a window air conditioner or a fork lift in a request for proposal, and easy to accept the lowest bid. Public sector agencies spend enormous energy fulfilling their mandates in this regard and the more complicated the product and/or service is, the more likely that they will get a camel when they open the bids. They very often fail to achieve the desired results.

        There are multiple companies on earth that supply aircraft automation systems, flight management systems, auto pilots etc., and Boeing isn't one of them. So Boeing works very, very closely with their suppliers, contractors and partners. I believe it is absolutely true that one or more lower level Boeing engineers or teams should be shot for this failure, but Boeing senior management needs new processes to assure it never happens again.

        Boeing needs a MUCH better communications team that better projects their recognition of these facts, and exactly what they are doing every day to solve the problem. They are not doing this now. However, neither Boeing, nor the MAX product deserves to die because of this failure, as tragic as it is.

        1. Kabukiwookie Silver badge

          Re: 'Software delivered to Boeing'

          if they do, they aren't required to take the lowest bid.

          Required? No. Will they though? Yes almost certainly

          Most projects that I've seen fail or wind up being a sub-par mess was due to (project) management wanting to either deliver under budget or to meet arbitrary deadlines.

          Usually involving bonuses for delivering under budget / on time.

          As long as the incentives are incorrect, these things will happen.

      4. Black Betty

        Re: 'Software delivered to Boeing'

        A couple of things are very clear. 1) The software was never tested with out of bounds data; and 2) By their own admission, no simulation of a faulty sensor was ever carried out with the complete MCAS.

        Nor is this the first time this sort of single input fault has reared its head with the 737. Just watched the Air Crash Investigation episode where a Turkish plane went into flare-retard mode too early and crashed due to a malfunctioning ground proximity radar. A known (not uncommon) fault that up until that crash had not been properly addressed because previously pilots had always managed to recover in time.

        Boeing's attitude towards multiple sensor redundancy and instrumentation sanity checks is that they are not necessary, because competent and attentive pilots should be able to recognise nonsensical instrument readings and take over full manual control of the aircraft. Fine in theory, but not always in practice, particularly if the pilots are distracted by other matters, and bloody nigh impossible when such a system is given full override authority as is the case with MCAS.

        1. John Smith 19 Gold badge
          Unhappy

          "Boeing's attitude towards multiple senso..sanity checks is that they are not necessary,"

          And who is pushing hardest for a fully automated flight deck I wonder.

          So when in doubt, blame the pilot.

          Neat.

          I'd say "WTF" but of course I understand exactly why they want to do it that way. It saves them money and shifts possible blame.

          And IIRC this is not the first time a single sensor has been the only indication of what's going on, or not cross checked with another to say something is up (although with 2 it would not be clear what). I'm recalling a potentiate on a flight control surface in another crash. Can't recall if it's sample rate was very low on the flight recorder log or if it had flat out failed and had no backup.

      5. jmch Silver badge

        Re: 'Software delivered to Boeing'

        "Boeing does not design the electronics or the software that runs on them in their aircraft"

        Whether they designed it in-house or bought it in from a vendor, it goes on their plane so they are ultimately responsible

      6. Byham

        Re: 'Software delivered to Boeing'

        As it happens the AoA indicator was an optional extra - non-military pilots tend not to use AoA. The two crashed 737Max did not have the AoA failure indicator installed so this software bug was not involved in those crashes.

    4. TheMeerkat

      Re: 'Software delivered to Boeing'

      I believe they do develop the software themselves. In India - at least I saw software jobs ads related to avionics. All Agile etc.

    5. Dr Gerard Bulger

      Re: 'Software delivered to Boeing'

      and who wrote the specification of said software, and check it was fit for purpose?

      1. AndyD 8-)₹

        Re: 'Software delivered to Boeing'

        regardless of who delivered the software and who spec'd it, if software controls the plane some (or all) of the time there MUST be a big button that says "Full Pilot Control" - and delivers just that!

        p.s. I am very happy that I have never written software that could be a risk to life or limb. Like most programmers, the very worst I might do would only cost my employers money.

        1. tfb Silver badge

          Re: 'Software delivered to Boeing'

          Only if the plane can be flown by a human. I don't know if that's true for this plane, I suspect strongly it won't be true in future airliners.

    6. Peter Lander

      Re: 'Software delivered to Boeing'

      At last ... a sensible question. I happen to think that the MCAS software is OK ... but the sensors feeding it are rubbish.

  3. Anonymous Coward
    Anonymous Coward

    What happened to acceptance testing?

    1. Martin Gregorie Silver badge

      Would have cost money, so cancelled by beancounters?

      1. A.P. Veening Silver badge

        Bean counters? Button sorters at best.

    2. Charlie Clark Silver badge

      That might have made regulators think that the 737 Max was something other than just an upgrade and might, you know, need proper testing before certification. And where are the bonuses in that?

      1. Anonymous Coward
        Anonymous Coward

        Proper Certification

        would have delayed the entry into service by at least 6 months. The Stock price would have suffered accordingly and the PHB's in Seattle would not want that to happen now would they.

        I've wriiten a good deal of Flight-Critical software over the years and it is not easy or quick. You have to be so pernickerty it is unbelievable.

        You can't allow it to crash/bugcheck or if it did, the plane might crash with fatalities ensuing.

        It sounds like Boeing have tried to pull a fast one with the FAA and they have been found out.

        It also seems that The Top Brass as Boeing are going on the offensive

        We should watch out for a smear campaign against Airbus if this debacle gets much worse (IMHO)

        1. jonathan keith

          Re: Proper Certification

          Management can go all-in on the blame game, but there's no avoiding that:

          1. Management should have AT THE VERY LEAST known about the fault once it was reported to the Engineers.

          2. It is their responsibility and failure that they (so they say) didn't. "But we didn't know!" is not an acceptable excuse.

          N.B. I hope that those standing underneath the slurry-sluice, about to be neck-deep in the muck, kept personal copies of all the relevant documentation to do with the tendering and speccification process for MCAS and the AOA Disagree systems.

          1. Headley_Grange Silver badge

            Re: Proper Certification

            @jonathan: it is also the responsibility of management to set the culture and environment so engineers (and anyone) are not cautious about reporting problems which might impact cost, schedule or share price. I'm not suggesting this is a factor in this specific case, but I've worked in organizations where "shoot the messenger" has been the preferred management technique and more effort is put into passing the buck around than would have been needed to fix the problem if it had been reported early enough.

            1. AK565

              Re: Proper Certification

              +10,000 on this! Regardless of industry, when a subject matter specialist reports a (potential) problem (within his area of expertise) to management, management's response is rarely, "You know, we hired this guy BECAUSE he knows about this shit. If he's reporting a problem, it's probably worth checking out."

        2. Charlie Clark Silver badge

          Re: Proper Certification

          It sounds like Boeing have tried to pull a fast one with the FAA and they have been found out.

          This was made easier by Congress cutting the funding for the FAA which basically was to let planemakers self-certify. After all, it worked so well for the banks, didn't it?

          That said, Airbus and Boeing still have an envious safety record thanks to the hard works of lots of people and, generally, an excellent approach to safety.

          1. A.P. Veening Silver badge

            Re: Proper Certification

            That said, Airbus and Boeing still have an envious safety record thanks to the hard works of lots of people and, generally, an excellent approach to safety.

            I am afraid in the case of Boeing the past tense applies.

            1. Anonymous Coward
              Anonymous Coward

              Re: Proper Certification

              I dunno. If we're going for envious rather than enviable it still works fine. I've got an image now of Boeing's safety record skulking around in the shadows wishing it looked as good some of the other safety records

              1. Kabukiwookie Silver badge

                Re: Proper Certification

                If management at Boeing allowed this to happen in the way it seems to have happened, engineering has taken a back seat over profits

                This means that incidents like this will become more common, not less.

            2. Bruce Grunewald

              Re: Proper Certification

              Airbus have had their share of problems with rogue software fighting with pilots for control of the aircraft. And those stupid little fighter jet sticks caused an Air France crash because the F/O was in full pitch up but the Captain couldn't tell and they stalled.

              I think the linked control columns in Boeing aircraft are much safer because both pilots can readily tell what the controls are doing. Boeing was also more conservative in introducing software based "assists" (like MCAS) than AirBus. I like a lot of things about the cabin design in AirBus planes, but I don't think they are quite as safe as Boeing. It might only be a .1% difference, but if that gets you killed you are still dead.

              1. Mark Exclamation

                Re: Proper Certification

                I suggest you read up on the AF447 crash details. What you have written here is far too simplistic, verging on incorrect. As for Boeing Vs Airbus, I suggest it's a dead-heat for reliability.

        3. Norman Nescio Silver badge

          Re: Proper Certification

          The Stock price would have suffered accordingly and the PHB's in Seattle would not want that to happen now would they.

          The PHBs are in Chicago, not Seattle these days.

          Harvard Business Review article on the move, which took place in 2001.

        4. Fursty Ferret

          Re: Proper Certification

          The 787 is quite interesting from this regard. Many items that used to have their own physical "black box" in the avionics bay are now applications running on one of two common computing resource centres, which are further composed of a number of general purpose modules (they look like blade servers).

          Every so often one of these modules will fault and restart, resulting in a temporary loss of a random selection of non-related systems (flight deck displays and some cabin systems are a frequent casualty). The blame seems to be attributed to cosmic rays.

          Having said that, they've never *not* come back online after about 30 seconds while I've been flying.

          1. Mark 85 Silver badge

            Re: Proper Certification

            Having said that, they've never *not* come back online after about 30 seconds while I've been flying.

            30 seconds can be a lifetime at low altitudes and especially during final approach or take-off.

            1. Martin Gregorie Silver badge

              Re: Proper Certification

              30 seconds can be a lifetime at low altitudes

              Exactly so, and given that both Tandem and Stratus had fully redundant* fault tolerant systems working reliably in the mid '80s, having 30 second outages in current avionic control systems sounds outrageous.

              [*] Fault tolerant Tandem systems duplicated every fault-tolerant process, one copy active and the other being a backup on a different CPU whose status (program counter and data) was refreshed each time an externally visible event occurred, so the runtime delay if the backup process became prime (and another backup was spawned) would be measured in tens of milliseconds. Stratus fault recovery was even faster because both copies of fault-tolerant processes were active and running in lockstep: if a fault occurred the failing hardware was simply turned off before bad data reached the system bus and meanwhile the brother process carried on without a pause.

              1. tfb Silver badge

                Re: Proper Certification

                I think that if parts of the avionics crash and restart often enough that people notice and other people invent reasons for it, then the problem isn't something you need Tandem/Stratus-style systems to solve, it's a problem you need basically competent design to fix. If it really is cosmic rays then, well, people have flown computers in space for quite a long time now and they know about how to deal with cosmic rays. That will make the system more expensive of course, but a 30-second outage sounds like something I would be very unhappy indeed with on an aeroplane.

                This is not to say that Tandem/Stratus-style redundancy is not also something you'd want, just that 'machine dying and rebooting every n hours' where n is not a very large number smells like a worse problem to me.

        5. baud Bronze badge

          Re: Proper Certification

          The PHbs are in Chicago, far from where the work is done and where clients go.

  4. Must contain letters

    does this affect more aircraft than the ill fated 737 max?

    so... the linked figure implies the AOA gauge is optional on lots of models, including the long haul 777. Should we be worried? Why the f*** are what look to me like critical safety features optional anyway. So much for the oft repeated mantra that 'your safety is our number one priority'.

    1. Andy The Hat Silver badge

      Re: does this affect more aircraft than the ill fated 737 max?

      Having the AoA sensors to stop stalling is essential for successful flight.

      However crashing is not essential for flight operations so a 'Why You are Crashing' or AoA (Arses on Afterburn) warning module is obviously optional and therefore costs extra ...

      1. Cynic_999 Silver badge

        Re: does this affect more aircraft than the ill fated 737 max?

        An AoA *sensor* is important in order to feed the electronics of a complex aircraft with direct AoA data, but an AoA *indicator* is not usually used by pilots even in large airliners. The pilot instead uses the airspeed indicator as the primary reference, because it is directly related to AoA for any particular weight/configuration/g-loading and far more intuitive for the human brain. An AoA sensor is unnecessary to prevent stalling, but is useful to provide early warning of an impending stall. Light aircraft usually have a crude AoA sensor that gives a binary output, switching on a stall warning buzzer if the AoA gets steeper that a preset angle. The crudest form is a non-electric simple flat reed set in such a position that it will vibrate and make a noise above a certain angle of airflow (like a reed instrument).

        Many cars have ABS and other electronic systems that are fed with RPM data from each wheel, but the driver does not have or need a display that directly shows wheel RPMs or even a "Wheel RPM disagree" warning. Drivers use other methods of determining that the car is skidding.

        1. TRT Silver badge

          Re: does this affect more aircraft than the ill fated 737 max?

          Mine has a "Wheel RPM disagree" display... it's the one with a picture of a car skidding on it.

          1. Anonymous Coward
            Anonymous Coward

            Re: does this affect more aircraft than the ill fated 737 max?

            So does mine but I haven't seen it since I replaced summer tyres with all weather tyres. Just a little bit more expensive, but my bottom line is the one I prefer not to leave skid marks.

            1. TRT Silver badge

              Re: does this affect more aircraft than the ill fated 737 max?

              Mine flashes if I hit a dip/hole of a certain size, enough to make one wheel travel a little bit further round the surface of the road than the other. It seems like it should only be a tiny, tiny amount of discrepancy, which of course it is, but the rotation/torque sensors on my car's wheels are especially accurate; it's an integral part of the design of the transmission system apparently.

              1. david 12 Bronze badge

                Re: does this affect more aircraft than the ill fated 737 max?

                Mine turns on and stays on. I have to turn off the electrics to reset it.

          2. Anonymous Coward
            Anonymous Coward

            Re: does this affect more aircraft than the ill fated 737 max?

            And therein lies the crux of the matter. If the driver / pilot is unaware of what the ACTUAL goings-on behind a particular display / warning light / dial are, then what to do about it can become a dangerous thing.

    2. Hans 1 Silver badge

      Re: does this affect more aircraft than the ill fated 737 max?

      Typically, it is a sefety feature, but too many dials kill safety and since the other models do not have a silly tendency to raise the nose, the AOA gauge is optional. Agreed, it should never really have been optional for new 737Max as that bird desperately tries to stall itself when you and MCAS are not looking ...

      1. Peter2 Silver badge

        Re: does this affect more aircraft than the ill fated 737 max?

        Typically, it is a sefety feature, but too many dials kill safety

        Seriously?

        This is a picture of the 737 Max cockpit.

        https://i.ytimg.com/vi/MqHQov0yl1I/maxresdefault.jpg

        Compare this picture of a Grob 115e cockpit, a light training aircraft used by the RAF to do basic flight training for their pilots at a really, really early stage and for teaching the Air Cadets how to fly.

        https://upload.wikimedia.org/wikipedia/commons/0/06/Cockpit_of_Grob_Tutor_Two_Seat_Training_Aircraft_MOD_45152683.jpg

        I might be wrong, but I think that the Tutor's instruments are actually displaying more information to the pilot than the 737 Max is, with less room. The tutor cockpit is also not to overcrowded, or at least I personally didn't have any trouble finding and using things at the tender age of ~14. One assumes that an extra couple of dials wouldn't have been too much for the (hopefully) better trained 737 pilots to deal with.

        1. Xpositor

          Re: does this affect more aircraft than the ill fated 737 max?

          Rather worrying that for what looks like a parked aircraft the Grob's attitude indicators are showing different results...

          1. Andy 68

            Re: does this affect more aircraft than the ill fated 737 max?

            Nahh - engine's off. No power/vacuum

          2. whitepines Silver badge
            Boffin

            Re: does this affect more aircraft than the ill fated 737 max?

            Rather worrying that for what looks like a parked aircraft the Grob's attitude indicators are showing different results...

            That's normal for a "steam gauge" aicraft that's shut down -- the gyros aren't spinning (no power) so the mechanical artificial horizons are displaying garbage (state at indicator shutdown plus some randomness) information.

        2. eldakka Silver badge

          Re: does this affect more aircraft than the ill fated 737 max?

          The needs of airforce (at least fighter/bomber ones) pilots is different to transport/passenger aircraft.

          Necessity of AoA in an aircraft doing combat maneuvers is substantially different. For aligning bombing runs, strafing, air-to-air gunfire, etc., all involve calculations of where the aircraft is pointed vs it's actual flight path and so on to be able to hit targets and, for example, not stall (or crash into the ground) because the nose (and the pilots attention) is pointed at a target while the flight path is travelling in a (slightly) different direction.

          One of the articles around the 737 MAX incidents (somewhere on this site I seem to remember) mentions that one of the airlines has the AoA displays on their instrumentation because they have a very high proportion of ex-military pilots - and actively recruit from the military - who like having the AoA displays because it is what they are used to and were trained with. Therefore since a lot of their pilots prefer it, they opted for this paid option to keep their pilots happy, not because a civilian-trained from the ground up pilot needed it.

          Also, training probably requires more instruments so that you can explain to trainee pilots why certain things behave in certain ways. Like teaching anything, often you do it the more 'verbose' or long/roundabout way to learn whats happening. But once you know why things happen, can link "doing this does that to that instrument", you can often do without those additional instruments (procedures/processes) once the training sinks in.

          1. Cynic_999 Silver badge

            Re: does this affect more aircraft than the ill fated 737 max?

            "

            Necessity of AoA in an aircraft doing combat maneuvers is substantially different. For aligning bombing runs, strafing, air-to-air gunfire, etc.,

            "

            The reason the AoA indicator is important in a fighter aircraft is that the pilot is frequently pulling significant "g". Usually a pilot uses the airspeed indicator to see whether he is close to a stall. At values significantly different from 1g however, the stall speed is not easily related to airspeed, and so the pilot needs another way to know how close the aircraft is getting to the stall (which is always at the same AoA). The stall speed of the aircraft I flew changed by over 100MPH during a loop, for example.

        3. gazthejourno (Written by Reg staff)

          Re: Re: does this affect more aircraft than the ill fated 737 max?

          It's a different thing to have a permanent set of steam-driven gauges displaying information all the time. On the Tutor, if I want to check the ammeter I glance down at that particular gauge.

          On a glass cockpit (say with an Airbus, or even a Cessna with a Garmin setup) that means looking down and selecting a menu on a multi-function screen and picking the right parameter to display. I also need to know what things will spontaneously pop up and where they will pop up on my screens if something goes wrong, as opposed to just knowing that if gauge X goes to reading Y I need to carry out action Z.

          Two different beasts, two different philosophies.

          1. Sean H

            Re: does this affect more aircraft than the ill fated 737 max?

            On my one and only glider flight I noticed a piece of wool taped to the outside of the canopy.

            Now I know what it was. The AoA sensor.

            1. Cynic_999 Silver badge

              Re: does this affect more aircraft than the ill fated 737 max?

              "

              Now I know what it was. The AoA sensor.

              "

              No, the wool is a slip/skid indicator. It shows if the direction of the airflow in the direction of left/right rather than whether it is from above/below.

    3. Anonymous Coward Silver badge
      Black Helicopters

      Re: does this affect more aircraft than the ill fated 737 max?

      AoA indicators aren't really used in commercial flying. They're used in military aircraft as they have different requirements (eg make sure you're level before pulling THAT trigger)

      Airlines that have a lot of ex-military pilots therefore like to have the AoA indicators in their planes. Other airlines feel that this useless extra dial might dilute the info given to the pilots, so don't include it.

      It's essentially another way of viewing the artificial horizon, so the info is available to the pilots anyway. The issue with the crashing planes was that one sensor fed into the MCAS system without any oversight, not that there wasn't a display for the pilots to look at.

      1. ibmalone Silver badge

        Re: does this affect more aircraft than the ill fated 737 max?

        My understanding from spending too much of my childhood playing flight simulators is that AoA is not the same as artificial horizon. AoA tells you the angle between your nose and airflow (where you think you're going and where you're going). On fighter aircraft that are performing extreme manoeuvres the two may be wildly out of alignment, at a more sedate pace it will largely depend on the lift you're getting. If you're not going that fast then having the nose above the horizon may still only result in level flight.

        1. defiler Silver badge

          Re: does this affect more aircraft than the ill fated 737 max?

          You're correct, but between the Artificial Horizon and the Vertical Speed Indicator you can infer enough about the AoA to confirm what your bum is telling you. At least enough to satisfy commercial passenger flights.

          1. Steve 114

            Re: does this affect more aircraft than the ill fated 737 max?

            Don't trust 'your bum'. It might be indicating 1G, but you're unwittingly going right down with your nose up.

          2. Cynic_999 Silver badge

            Re: does this affect more aircraft than the ill fated 737 max?

            The pilot determines AoA primarily from the airspeed indicator. The attitude indicator (formerly "artificial horizon") is of limited use for that purpose. In most cases airspeed will be directly proportional to AoA, while the relationship between attitude and AoA also depends on the power setting.

            Pilots will usually talk about the stall *speed* rather than the stall AoA even if the latter gives the more accurate indication (because stall speed is affected by "g" loading and so becomes higher in a turn)

      2. Steve 114

        Re: does this affect more aircraft than the ill fated 737 max?

        No, 'artificial horizon' can be dead ahead, and you are still in deep stall going down like an AF lift-plummet in the South Atlantic. Some would prefer hints from reliable AoA.

      3. Anonymous Coward
        Anonymous Coward

        "It's essentially another way of viewing the artificial horizon"

        No it definitely is not. The attitude and AoA are completely independent in general. But in unaccelerating flight the AoA is so tightly coupled to airspeed that civilian planes don't really need it to fly the plane, the airspeed indicator is enough and that's what the pilots are used to do.

    4. Steve 114

      Re: does this affect more aircraft than the ill fated 737 max?

      Have you seen where the (expensive 'optional') AOA guage appears on the screen? (top right), and where the 'disagree warning' (which should have appeared for all, but didn't for anyone, lower right). Ergonomics, folks?

    5. Anonymous Coward
      Anonymous Coward

      Re: does this affect more aircraft than the ill fated 737 max?

      As with everything with the 737MAX crashes, it's a little more complicated than just the AoA sensors.

      While there were two AoA sensors, it is possible that only one was ever used (https://www.nytimes.com/2019/05/05/business/boeing-737-max-warning-light.html):

      "But the system relied on only one of the two angle of attack sensors, introducing a potential single point of failure into a critical flight system. "

      Then we come to the AoA system itself - it appears that the issue maybe subtly different to failed sensors, where even if there had been 3 sensors and quorum required amongst the sensors for MCAS input, we still would have seen crashes.

      It appears that the AoA sensors had to be manually trimmed on the ground, which is why the Lion Air crash was blamed on pilot error. Only it appears that to correctly perform AoA trim adjustment on the ground, you needed another premium accessory that Boeing thought was compulsory but was't fitted to the models provided to budget operators - with the systems installed in Lion Air and Ethiopian Air, AoA did not operate until you reached 400 ft so it wasn't possible to accurately trim it on the ground. And to adjust trim at 400ft, you needed electronic assistance active due to the force acting on the control planes and electronic assistance meant that MCAS was enabled and trying to fly you into the ground. There was a correct sequence to achieve this, however it was not taught to 737MAX pilots (https://theaircurrent.com/aviation-safety/vestigal-design-issue-clouds-737-max-crash-investigations/) and if the AoA trim was a long way out (i.e. Ethiopian Airlines), you didn't have a lot of altitude to play with. It is thought that AoA trim was less severe, resulting in the pilots having more altitude and longer to fight with the plane.

      While some pilots believe this was a pilot issue (not trimming AoA correctly on the ground), it is possible that the 400ft limit for AoA trim sensors or existing local workarounds to address this (i.e. trimming once in the air) have been compromised by an MCAS that doesn't limit its operation correctly (i.e. input validation) combined by uncertainty around the AoA sensors.

      You choose the root cause...

    6. fidodogbreath Silver badge

      Re: does this affect more aircraft than the ill fated 737 max?

      The company only revealed this to US Federal Aviation Authority regulators after Lion Air flight JT610 crashed in October 2018, claiming in this week's statement that "the issue did not adversely impact airplane safety or operation".

      Exactly how many airplanes full of people have to crash, killing all on board, before Boeing considers something to be a safety or operation issue?

      And in aviation, aren't all "impacts" considered to be adverse?

      1. Anonymous Coward
        Anonymous Coward

        Re: does this affect more aircraft than the ill fated 737 max?

        Boeings justification around Lion Air flight JT610 was that it was pilot error. i.e. the AoA trim was not correctly set prior to take off, resulting in MCAS operating outside of it's expected input range or what it could address.

        It now appears that it wasn't possible to set AoA trim correctly on some/all of the Lion Air 737MAX's due to the planes lacking the necessary equipment.

    7. Mark 85 Silver badge

      Re: does this affect more aircraft than the ill fated 737 max?

      So much for the oft repeated mantra that 'your safety is our number one priority'.

      Marketing BS is what it is now. Just like IT companies with "your security is our priority" announcements after their servers have been looted.

    8. The Specialist

      Re: does this affect more aircraft than the ill fated 737 max?

      Here is a very comprehensive write up on the source of the issue specific to this aircraft:

      https://spectrum.ieee.org/aerospace/aviation/how-the-boeing-737-max-disaster-looks-to-a-software-developer

    9. tip pc Bronze badge

      Re: does this affect more aircraft than the ill fated 737 max?

      many / most aircraft have AoA sensors to provide stall warnings. only the 737 max has this MCAS which is only there to ensure the aircraft conforms to specific regulatory requirements mandating the behaviour of the aircraft in differing conditions, it effectively makes the 737 max behave like earlier machines.

  5. Gnosis_Carmot

    And now we get to the ...

    ...blame the engineers. Of course NO ONE in management EVER learned about this beforehand.

    The smell of excrement is strong on this from Boeing.

    1. Hans 1 Silver badge

      Re: And now we get to the ...

      Who wrote up the specifications for the code ?

      1. Pascal Monett Silver badge
        Flame

        I desperately hope that manglement was not involved in writing the specifications - they probably couldn't write specifications for finding their own arse with both hands. No, I much prefer an actual engineer for writing the specs.

        However, it is manglement's purview to approve the specifications, so pretending they had no clue as to what is going on doesn't cut the mustard for me.

        If Boeing's management of their flagship product doesn't have a clue what's going on with the software, they should be sacked and replaced by people who actually do their jobs. Trump may be President, but that does not give managers in charge of products that are responsible for people's safety to just excuse themselves from knowing what the hell is going on.

      2. bpfh Silver badge

        Re: And now we get to the ...

        The same who wrote the on board computers depollution systems for VW Diesel engines?

        Yep, that was also the work of a rogue engineer according to the mandarins at Wolfsburg...

        1. Red Ted

          Re: And now we get to the ...

          I think they went to VW after working at Google on the WiFi sniffer in the Street View cars....

    2. lglethal Silver badge
      Go

      Re: And now we get to the ...

      They can try and blame the engineers all they want, but there WILL be a signature trail.

      Aerospace is fantastic for this, everything requires signatures from multiple disciplines.

      Or are they trying to say that the Boeing managers signed off on these documents without reading them... naughty, naughty...

    3. fandom

      Re: And now we get to the ...

      " Of course NO ONE in management EVER learned about this beforehand."

      That's not what they are saying, what they say is: "Senior company leadership was not involved [...]"

      Looks like some middle management is being thrown under the bus.

      1. nerdbert

        Re: And now we get to the ...

        I don't know about your company, but for my products I'm much happier when senior management is not involved in the detailed engineering decisions, since, you know, I'd like the product to actually work. Andy Grove was a great engineer in his time. but by the time he was a C-level creature even he knew that he had to stay away from a lot of the actual design.

        Middle management can be a crapshoot. Sometimes you get that rare individual who can find his arse with both hands, but often you can't. It's director level and above that forgets everything they've ever learned about engineering since they haven't been allowed to touch the tools for so long that they're only allowed to play company politics and sell their wares anymore. It's the middle management that's trying to transition to Director and above who's the most dangerous link in the chain.

        1. tfb Silver badge

          Re: And now we get to the ...

          While senior management should not be expected to be involved in engineering details, they should be expected to be competent to sign off on decisions which involve significant risk, either financial risk to the company or wider economy, risk to a lot of people's lives or, in this case, both. When I worked for banks it was absolutely the case that procedures which, if they failed badly enough, would mean the bank could not function, got signed off by very senior people indeed (I don't know how senior, but I expect CEO senior in various cases).

          It's the job of people between the engineers and the senior management to make sure that what is happening is communicated to them in a way they will understand but without losing the important details, and this process can fail horribly of course both, famously, at banks and also, it seems, here.

          The navy has (or had, but I bet still has) a rule that the commanding officer of a ship is responsible for the ship. If the ship sinks because of something that was known about by more junior officers but not communicated to you it's still your fault because it's your job to make sure that never happens*. This should apply here.

          (*) There are special circumstances I think, but the commanding officer doesn't get to just say 'the organisation is fucked up and I wasn't told' because the organisation being fucked ip os their fault.

          1. Stork Silver badge

            Re: And now we get to the ...

            This is the general rule at sea. If you are the captain, you have the ultimate responsibility. In some cases this may not be entirely fair; what chance does the captain of a 12000 TEU container boat has to know if what the BoL says matches the content of the box? But my dad was very aware that he could be chucked in prison in anyone onboard had any booze when they came to Saudi. They also had to feed the sharks with pork before arriving.

        2. Mark 85 Silver badge

          Re: And now we get to the ...

          It's director level and above that forgets everything they've ever learned about enginee

          Only applies if (a big IF) they actually were an engineer. Seems most directors really have no knowledge of the tech their company uses/builds. Their claim to fame is "finance" and "stock price".

  6. }{amis}{ Silver badge
    FAIL

    Option extra why???

    Why is what appears to be a soft gauge based on data the flight control system has to have an at cost optional extra!

    1. Anonymous Coward
      Anonymous Coward

      Re: Option extra why???

      El Reg previously reported that it was an $80,000 optional extra! Not bad for flipping a bit from 0 to 1 as it rolls off the production line.

    2. Anonymous Coward Silver badge

      Re: Option extra why???

      Because most airlines don't want it, because most pilots don't want it. It's also not the only indicator that tells you what angle the aircraft is at.

      1. }{amis}{ Silver badge
        Boffin

        Re: Option extra why???

        Is not the problem with this stuff that it the angle of attack is relative to the air the plane is in though rather than relative to gravity?

        It's been very long time since i flew a Bulldog trainer in the ATC, but I'm pretty sure that part of the lectures on angle of attack was that it is possible to stall a aircraft when flying level relative to the ground under the right (really bad) atmospheric conditions?

        1. Cynic_999 Silver badge

          Re: Option extra why???

          AoA is directly related to airspeed and wing loading. In most flight conditions wing loading is pretty constant so your airspeed indicator is all you need to determine how close you are to stalling (which is really all the pilot needs to know about AoA). A sudden up-draft can temporarily increase AoA and stall a slow flying aircraft (e.g. an aircraft on approach to land). A so-called "micro-burst" event. A sudden tail-wind gust can lower the airspeed and also cause a stall. A head-wind gust is almost as bad because the pilot will reduce speed to compensate and then the gust ends and the speed falls, though gusts are more relevant to slower light aircraft than airliners as they are likely to be a more significant percentage of the aircraft speed.

          More common however is an accidental stall caused by the pilot turning too steeply, which increases wing-loading and so puts a normally safe airspeed below the stall AoA. Stall speed increases as a factor of the square root of "g" loading. At a (very steep) 60 degree bank turn the aircraft is undergoing 2g which increases the stall speed by over 40%. A pilot who has mis-judged the final turn to line up with the runway and increases his turn to compensate can easily fall into that trap.

      2. usbac

        Re: Option extra why???

        Most decent pilots would really like to have an AoA indicator. My neighbor just spent a lot of money to add one to his plane. It's the airlines that are too cheap to pay for it.

        In the old days, when it was a physical instrument, I can see why there would be a cost. But to hold back a software feature that can improve the safety of flight just seems completely unethical to me.

        By the way, it has nothing to do with "what angle the aircraft is at", it shows the angle of the airflow over the wings. You know, that whole lift thing that keeps an airplane in the air...

        1. Cxwf

          Re: Option extra why???

          If I’m understanding this correctly, this particular issue was caused by a subtle change in how the AOA data was used between generations.

          In prior generations, the AOA was displayed to the pilot to aid situational awareness, but didn’t DO anything directly. As other commentators have noted, you can generally get by without this info, but it becomes important in some unusual situations (which you can mostly compensate for with experience). This the info isn’t considered critical, so the gauge is an option.

          But now on the Max, that info directly feeds into part of the plane’s flight control AI, and if the sensor is bad, you can crash. Suddenly knowing that the sensor is bad becomes much more important- but this change in priority didn’t make it into the display design. After all, this change is intended to be transparent to the pilots to avoid retraining costs. The whole goal was NOT to tell them anything is different.

    3. Cynic_999 Silver badge

      Re: Option extra why???

      "

      Why is what appears to be a soft gauge based on data the flight control system has to have an at cost optional extra!

      "

      Because most pilots don't need it and don't want an unnecessary gauge cluttering an already complex display console. In this case I doubt that it is a question of cost. Your car has many sensors that are not connected to anything that is displayed to the driver even though the electronics needs the data.

  7. Anonymous Coward
    Anonymous Coward

    Written by Citroen ?

    My C4 has all sorts of warning alerts and buzzes. However because I'm an old style keep-your-fucking-eyes-on-the-road sort of driver, I have no idea what they are telling me since they literally flash up for a split second and disappear. I need a passenger if I want to know what that beep or buzz was for (quite often it's telling me it can't initialise the distance warning system ?????).

    The genuises that designed it didn't even think to have a "list of messages" feature so I could at least see if it was anything I needed to worry about.

    1. CrazyOldCatMan Silver badge

      Re: Written by Citroen ?

      Mostly the messages on my old Citroen XM consisted of various hydraulic failure warnings. Mostly after the bit of rotten cheese that they laughably called a 'rear ride-height adjuster' blew (again) and dumped all the hydraulic fluid over the road.

      At which point the brakes no longer work (apart from the cable-operated parking brake) and the steering gets very, very very heavy (you try turning a 1.4 tonnes car with quite wide tyres without the hydraulic power steering working). The major design flaw was that three systems (hydropneumatic suspension, power steering and brakes) all shared the same hydraulics. So, if one blew[1], it took the others out.

      [1] In the year or so we had that car, it happened three times. Twice in the rear ride-height adjuster and once in the steering..

      1. Flicker

        Re: Written by Citroen ?

        Consider yourself lucky - my mate's ancient Citroen Club needed the engine removed (or at the very least the exhaust manifold) before you could change a blown headlight bulb and the Citroen SM I briefy coverted before coming to my senses had an insane Maserati engine which needs heavy-duty maintenance every couple of thousand miles. Citroen always built "interesting" cars, but not for the faint hearted! But then my otherwise lovely Renault 16TX had designed-in wheel suspension rust problems which would cause the nearside rear wheel to (literally) fall off under heavy cornering - you pays your money...

        1. aberglas

          Citroens are great

          Everybody thinks that they are a piece of French engineering that is impossible to maintain and will fall apart in half the time of a properly engineered Toyota.

          Which is perfectly true. Which is why you can by them with not many kms on the clock for half the price of said Toyota. Just realize that there will come a time to throw it out when something breaks that cannot be economically fixed. Like a headlight bulb.

          (A bit of an exaggeration. I did manage to change a headlight bulb. Only took me a couple of hours.)

  8. Alan Johnson

    Not the main problem

    This may be software which does not perform as intended but it is not the main problem.

    The main issue is that a single sensor is used as input to a single softwar emodule which is responsible for adjusting a flight control surface and has the ability to move it to a position where the aircraft cannot be flown. A failure of the sensor or softwae would have the potential to place the aircraft into an extremely hazardous situation. A single failure should not do that unless the component concenred is extremely reliable, has well understood failure characteristics and mechanisms, and is designed and tested so as to make failure extremely unlikely.

    Secondary issues are the failure to tell pilots about the function concerned.

    A failure to check the sensor as described above.

    A failure to consider the likely consequences of a sensor failure and the human factors and actions required if it failed.

    A failure to correctly classify the software level of the system concerned.

    A failure in the regulatory/certification process.

    1. Pascal Monett Silver badge
      Coat

      Yup, it's failure all the way down.

      Literally.

    2. Anonymous Coward
      Anonymous Coward

      Re: Not the main problem

      "The main issue is that a single sensor is used as input to a single softwar emodule which is responsible for adjusting a flight control surface and has the ability to move it to a position where the aircraft cannot be flown."

      But wait, there's more...

      And the single sensor is dependent on manual adjustment to ensure the range of values it provides is within an acceptable range to avoid the MCAS crashing the aircraft.

      And on some aircraft, the sensor needs to be operating above 400ft to be calibrated correctly.

  9. TonyJ Silver badge

    Surely...

    ...having two sensors is as bad as just one?

    If one fails, how does the system determine which one it was?

    I've mentioned this before, but my diving rebreather has three O2 sensors*.

    Part of the pre-dive sequence is to calibrate the unit for the mixes of gases in use. During this calibration, you also watch the mV of the sensors to ensure that they're close to one another.

    If one behaves oddly, during the dive, you still have two.

    I'll admit I don't have anything to do with safety systems, but to my simple mind, if you're having two sensors surely you either need a way to determine which one is actually at fault, or stick to just one? Or three.

    *It never fails to boggle my mind the number of posts online about "I've found a cheaper O2 sensor that seems to work" or "I use a different kind of CO2 scrubber that isn't actually rated to go in rebreathers and none of the major manufacturers have ever tested on but it's cheaper so I'll use it".

    1. Anonymous Coward
      Anonymous Coward

      Re: Surely...

      There is no way to tell which is faulty with two sensors, but you can "fail safe" if they disagree (though not in your case).

      The use of three sensorst allows a faulty sensor to be isolated and normal operation to continue.

      1. Anonymous Coward
        Anonymous Coward

        Re: Surely...

        You need four, so that if one fails you've still got three to keep the situation sane

        1. Hyper72

          Re: Surely...

          I don't quite know why your comment reminded me of this:

          "``I can't cope with it,'' Zaphod said darkly, and sent a third drink down to see why the second hadn't yet reported on the condition of the first. He looked uncertainly at both of her and preferred the one on the right.

          He poured a drink down his other throat with the plan that it would head the previous one off at the pass, join forces with it, and together they would get the second one to pull itself together. Then all three would go off in search of the first, give it a good talking to and maybe a bit of a sing as well.

          He felt uncertain as to whether the fourth drink had understood all that, so he sent a fifth to explain the plan more fully and a sixth for moral support."

          1. Down not across Silver badge

            Re: Surely...

            ....

            He felt uncertain as to whether the fourth drink had understood all that, so he sent a fifth to explain the plan more fully and a sixth for moral support."

            Thank you for that. That bit about Zaphod knocking back the pangalactic gargleblasters always has me in stitches. Still. Every time.

      2. Gordon 10 Silver badge

        Re: Surely...

        Technically you can "fail safe" with only 2 sensors - its called aborting the dive. Three gives you the re-assurance (all else being equal) that the odd one out is faulty.

      3. Cynic_999 Silver badge

        Re: Surely...

        The problem here is that the reason one sensor fails (icing) is likely to cause any others to fail in the same way, so multiple sensors are not a great help. Airspeed could easily be used as a sanity check.

        1. Robert 22

          Re: Surely...

          Probably one will fail before the other(s). Also, even if the sensors all fail and in a similar way, it seems feasible to use information from other sensors as a sanity check.

          1. Anonymous Coward
            Anonymous Coward

            Re: Surely...

            "Probably one will fail before the other(s)"

            No, they usually fail at the same time due to the near identical weather conditions. And normal operation resumes when they leave said weather conditions.

            Other sensor data (i.e. temperature, airspeed, position of control surfaces relative to expected positions) can be used to identify potential issues assuming they feed into a common control system.

            1. TechnicalBen Silver badge

              Re: Surely...

              Multiple types/ages/sizes etc of sensors can be used. So age varies and failure rate due to that. Wind/water/ice affect different types at different times, so would allow you to preempt complete failure (all 3) by an early failure (canary ;) ) and aborting/changing the plan accordingly (heaters for icing, lowering altitude etc).

      4. Anonymous Coward
        Anonymous Coward

        Re: Surely...

        "The use of three sensorst allows a faulty sensor to be isolated and normal operation to continue."

        It *can* do that, but it's not guaranteed to always fdo that.

        It's usually a good idea, but sometimes it's not enough on its own, e.g. AF447 crashed because other Bad Things happened even with three sensors of two dissimilar desgns, before and after the identical two of its three pitot tubes failed identically at the same time (as had occurred on other aircraft on previous occassions in similar conditions, fortunately with less drastic consequences, but the implications had been recognised sufficiently for modification programmes to be required). It's well documented in various places.

        When that happens, the two failed sensors can outvote the one which is behaving sensibly, but sensible flight crew may be able to either reduce the risk of it happening (don't fly through icing conditions) or recognise and resolve the issue if two did fail identically..

        On the AF447 flight in question, the two identical pitot failures combined with various other unrelated failures (e.g. flying through icing conditions rather than around, less than ideal reaction from the crew, etc) led to the loss of all on board.

    2. Doctor Syntax Silver badge

      Re: Surely...

      *It never fails to boggle my mind the number of posts online about "I've found a cheaper O2 sensor that seems to work" or "I use a different kind of CO2 scrubber that isn't actually rated to go in rebreathers and none of the major manufacturers have ever tested on but it's cheaper so I'll use it".

      Nobody ever posts "I tried it and it didn't work"? They must all work.

      In the meantime, remember the saying "either go to sea with one compass or three."

      1. Speltier

        Re: Surely...

        Selection bias. The ones using waxed bog rolls full of Drano(r) as a scrubber might not respond, so the "waxed bog rolls full of Drano" idea will live on as a viable possibility.

      2. Mike 125

        Re: Surely...

        >In the meantime, remember the saying "either go to sea with one compass or three."

        Always take a bomb on a plane. What's the chance of there being two onboard.

    3. ciaran

      Airbus has 3 AoA sensors

      And yet in the history of Airbus its happened that 2 sensors failed at the same time and with the same incorrect readings. Its just extremely unlightly. Read this for background

      https://aviationweek.com/commercial-aviation/german-investigators-cast-wider-net-frozen-aoa-sensors-pamplona-dive-incident

      1. Gordon 10 Silver badge

        Re: Airbus has 3 AoA sensors

        But thats a probability call. 3 is the bare minimum if you want some kind of consensus to act on. It doesn't mean the consensus is always right.

    4. tfb Silver badge

      Re: Surely...

      Having two is less bad than having one: if you have two and they disagree then the system knows something is wrong and can notify the humans in charge of that and if possible put itself into some kind of safe mode. If the second thing is not possible then it can tell the humans that they need to be responsible for doing whatever it was the system previously did and disable itself altogether.

      With only one you never even know if anything is wrong.

      It's not clear to me in this case whether the humans can fly the plane safely at all on their own though, still less do so without a good AOA indication: if that's true then you want at least three sensors (but I think you also want a plane which can be flown in some regime where it is safe without them).

      I also think the MCAS software didn't even pay attention to the 'disagree' signal.

      Sadly Boeing are clearly too big to fail, because this should kill them.

      1. A.P. Veening Silver badge

        Re: Surely...

        Sadly Boeing are clearly too big to fail, because this should kill them fly them into the ground.

        FTFY

    5. CrazyOldCatMan Silver badge

      Re: Surely...

      rebreathers and none of the major manufacturers have ever tested on but it's cheaper so I'll use it

      Strikes me as being one of those things (like motorbike tyres and brakes) with which you do not mess..

      (I've known fairly well-off people buy the cheapest possible motorbike tyres or brakes - and then get shirty when I ask them why they value their lives so little..)

      1. defiler Silver badge

        Re: Surely...

        Bridgestone BT045 on my old Yam Divvie. For some reason the cross-ply (which I'd normally avoid) worked really well on that machine. Once handed a CBR600RR pilot his arse on a plate, whilst I had the wife on the back.

        Brilliant tyres, and pretty cheap too! (Not the cheapest, though, I'll concede.)

      2. usbac

        Re: Surely...

        It happens all the time with pilots and oxygen systems. Aircraft rated oxygen systems for small aircraft are very expensive. So, people buy medical oxygen tanks and regulators and try to use them in aircraft. The results are usually deadly.

        You see, medical oxygen regulators are designed to operate at ground level. One side of the regulator diaphragm is vented to atmospheric pressure, witch is fine if the regulator is calibrated and then operated at ground level. Aircraft oxygen regulators on the other hand have a two stage design that compensates for changes in atmospheric pressure so that the oxygen flow is constant as the aircraft climbs.

        Again, someone that owns a high performance airplane capable of high altitude flight is not a poor person, why save a few hundred bucks and kill yourself (and possible others) over it?

  10. Stevie Silver badge

    Bah!

    Un-aerodynamic design creep.

    Dangerous aeroplane.

    Called it on the programmers not being properly integrated into the design team, btw.

  11. DropBear Silver badge
    Facepalm

    So, um, have we reached the "one rogue engineer" stage yet...?

    1. Anonymous Coward
      Anonymous Coward

      That would be a brave move, as it would be public admission that the process was inadequate* ;-)

      *I never understood why that didn't come up during VWgate.

    2. Snowy Silver badge
      Thumb Up

      Looks like it to me, have an upvote.

    3. sal II

      As an IT engineer, the ability to cover my behind from the effects of (mis)management by having a record of reporting major issues to said (mis)management is an essential workplace skill.

      I refuse to believe the engineers in Boeing didn't posses that skill and didn't pass on the concerns about that fault to management.

      1. kain preacher Silver badge

        From what I've been reading QA is non existent at Boeing . First of the warning light was an upgrade and non standard. Now it's come out that boeing was shipping planes with with metal debris and tools left in them. if management does not care about shipping planes with debris and tools in it you think they care about programing bugs ?? Oh a fucking warning light was an add on option.

        1. sal II

          From the article I understand that it's not the warning "light" that was optional, but the display gauge showing the AOA. The problem was that the software was written in a way that if you are missing the add-on AOA gauge, it didn't display the warning "light" and that was unintentional.

          This is similar to how cheaper models/trims of a car might only have a "check engine light", where the more expensive might feature a display with more detailed information on what's wrong.

          Don't get me wrong, I'm not making excuses on Boeing behalf, still find in unacceptable that on a multimillion $$ craft you have to pay extra for a software add-on.

    4. LDS Silver badge

      Not yet. But then they can also try the "We've been hacked!!!" one.

      1. tfb Silver badge
        Alien

        Hacked ...

        ... by the (Russians|Chinese|Iranians), delete as appropriate.

  12. devTrail

    Red herring

    As far as I understood both of the planes crashed because nobody told the pilots what to do in case of a malfunction.

    But the really serious issue is that after the Lion air crash still there were pilots around who didn't know what to do in case of a malfunction and the management for their own admission was aware of the problem.

    So, instead a making a lot of noise around such details people should ask why they kept silent between the two crashes.

    1. ciaran

      Re: Red herring

      No, its clear that the pilots in the second crash tried to do what Boeing said. However Boeing gave incomplete information to the pilots. Once the MACS is "stuck", there are incredible forces acting on the control surfaces. So the "manual trim" procedure is basically impossible.

      In the old, old days the Boeing flight manual gave a real procedure on how to recover the airplane, but it has been airbrushed from history. Expect it to come back in the court cases...

      1. S4qFBxkFFg

        Re: Red herring

        "In the old, old days the Boeing flight manual gave a real procedure on how to recover the airplane"

        In the old days, the flight engineer could help haul the trim wheel back (or alternatively might have a fighting chance of knowing which breaker to pull to kill the misbehaving system).

        In the old, old days the navigator could assist.

        If you went even further back and requested the radio man lend a hand too, you're probably not getting out of this one.

      2. devTrail

        Re: Red herring

        Hmm.

        First you write: No, its clear that the pilots in the second crash tried to do what Boeing said.

        But then you write: However Boeing gave incomplete information to the pilots. ... In the old, old days the Boeing flight manual gave a real procedure on how to recover the airplane, but it has been airbrushed from history

        It seems you disagree with my post, but at the same time it seems you fail to agree with yourself.

        1. kain preacher Silver badge

          Re: Red herring

          "Hmm.

          First you write: No, its clear that the pilots in the second crash tried to do what Boeing said.

          But then you write: However Boeing gave incomplete information to the pilots. ... In the old, old days the Boeing flight manual gave a real procedure on how to recover the airplane, but it has been airbrushed from history

          It seems you disagree with my post, but at the same time it seems you fail to agree with yourself. "

          ??? They did what boeing told to do but boieng did not give the full instructions . Hoiw is that not agreeing with him self

      3. Brit Flyer

        Re: Red herring

        Maybe Boeing should take all of this high tech stuff out of its planes as it seems unable to cope with it. Just go back to dials etc and let people like Airbus to get on with the high tech stuff.

        1. Andromeda451

          Re: Red herring

          I'd rather be on a Boeing than an Airbus.even now.

    2. Snowy Silver badge
      Facepalm

      Re: Red herring

      I am sure I read something about there would have been another crash just before the second one. There was a off-duty pilot in the "free" seat who saw what was happening and was able to tell the pilot how to fix it.

      Link to the story (https://www.theregister.co.uk/2019/03/26/737_crash_update/)

      Why this did not lead to action is disturbing!

  13. herman Silver badge

    Well, it wasn't the lack of a warning that caused the planes to crash. It was the badly designed MCAS system that caused the crash and the lack of training materials that made the pilots unable to do anything about it in the few seconds of available time.

  14. tallenglish

    Company is in maximum deflection mode.

    Clasic case of "not our fault", which is just digging them a hole - just like the MCAS system did for the poor folk in the two crashes.

    They build a plane that flies like a brick, because the larger engines block some airflow.

    Knowing full well that the plane will nose up, the design a patch to force it back down.

    They knowingly scrimp on redundancy or fault torerance in the software.

    They knowingly don't train the pilots properly.

    Now they started blaming the pilots, now they try and blame the software engineers.

    When it is clearly management penny pinching and cutting corners that is the root cause of both of these crashes.

    Lets not forget these are new planes, new hardware - nothing should be failing.

    Failure always comes from the top down, the CEO and board should be brought up on criminal charges.

    People should refuse to fly on these death traps, how can we trust a company which can't even admit they are at fault and puts profits over peoples lives.

    I hope they go bankrupt and/or taken over and the directors involved go to jail.

    1. EastFinchleyite

      Re: Company is in maximum deflection mode.

      "and the directors involved go to jail."

      The jail has been outsourced and is probably run by a company owned by the same investors/hedge funds that own Boeing. Its a good thing big money doesn't control governments and the legal system <sarcasm warning> otherwise there would be no chance of this happening.

      In the end we get what we deserve. We (the public) demand cheap air tickets from low cost airlines that in turn demand low airport fees and cut price deals on aircraft. The aircraft makers demand government subsidies and tax breaks. Boeing classifies what appear to be safety requirements as "optional" and charge more for it. Or to put it another way, cheap airlines buy planes without these safety facilities to reduce the purchase price. What do other plane makers do? I bet both the airlines and plane makers would howl if regulators interfered with the free market that we all worship.

      The public ends up with crowded aircraft run by airlines trying to screw every last penny from them for non-ticket items (£10 for a beer and sandwich. £30 for bag in the hold). We fly from shitty airports that have been turned into shopping malls, and pay exorbitant rates for transport and parking. All the time each middleman takes his cut AND we pay through taxes.

      I hate flying. We must be mugs.

      1. Stork Silver badge

        Re: Company is in maximum deflection mode.

        I do not enjoy flying.

        I have been doing so regularly since 1992 (it was more fun then), and my impression is that safety is no worse than then, in spite of prices haven fallen dramatically. Someone have the figures?

        In the early 90es, a return ticket Manchester Copenhagen cost about £200. It is these days very rare I pay that much for longer european flights, in spite of 25 years of inflation (if this is a good thing is a different debate). Yes, I bring my sandwich, but even if I added all the things that were previously included I doubt prices have kept up with inflation, never mind wages. It has got cheaper because planes use less fuel, are fuller and have shorter turnaround. And staff is paid less.

        What has not changed, I think, is Mr. Buffet's observation that the airline industry as a whole has never had a year of profit.

  15. Avatar of They
    Thumb Up

    And now...

    Let the lawyers loose.

  16. Doctor Syntax Silver badge

    "Senior company leadership was not involved in the review and first became aware of this issue in the aftermath of the Lion Air accident,"

    For some minute value of 'leadership'.

    This seems to be a management insistent on getting what it likes to hear instead of what it needs to hear. One thing it needs to hear is "when you're in a hole, stop digging.".

    1. A.P. Veening Silver badge

      One thing it needs to hear is "when you're in a hole, stop digging.".

      Let them dig, once it is deep enough and they are still in it, we can fill it back in.

  17. Anonymous Coward
    Anonymous Coward

    Tombstone imperative still applies

    Read the book.

  18. anthonyhegedus Silver badge

    I wouldn't fly on a Boeing 737 now.

    Do I need to phone the airline or RyanAir ahead, and ask what version of the software the plane is running?

    1. Michael B.

      Nope just avoid the 737 Max. That's what I will be doing in future.

      1. usbac

        Boeing will just rename it to something else, and the public is too stupid to figure it out. Remember this is the same public that in polls said Ed Snowden is "that Wiki Leaks guy".

        Southwest Airlines just ordered a bunch more 737 Max's.

        As a pilot and an IT guy, I will never fly on one again.

        My neighbor and I were flying back from Florida just before the Ethiopian crash on a 737 Max (before anyone knew how bad the problem really is) and as we boarded I joked with the captain "you know how to kill MCAS, right?" If he said "what is MCAS", I would have turned around and not boarded the flight. My neighbor is a pilot and a licensed aircraft mechanic, and we had talked about the Lion air crash and MCAS while on the way to Florida. I wouldn't step foot on one again, now.

  19. Saruman the White

    Is CYA time

    What I am hearing here is the sound of every manager in Boeing covering their arse by blaming someone one level down. We will probably find that the blame ends up on some poor cleaning lady in Tennessee who happened to wonder into the wrong office for 5 minutes on the wrong day.

  20. tfb Silver badge
    Boffin

    Engineers / management

    The 'engineers knew but management conveniently didn't' claim reminds me very much of both space shuttle disasters. If it's the same thing then the answer is basically 'the organisation is fucked' (also: design things which are inherently safe rather than dangerous where possible, even if that is more expensive.)

    1. kain preacher Silver badge

      Re: Engineers / management

      Three things. Either management knew or they were told not to tell them. Third option QA has been so low and glaring obvious that the engineers thought management would ignore them so they kept quiet .

      Read how they are shipping planes with tools and metal debris in the planes .

    2. Anonymous Coward
      Anonymous Coward

      Re: Engineers / management

      In the Space Shuttle enquiry it was noticeable that the engineers actually leaked the cause not to Feynman, but to a with-it general who passed the information on to him. Even a general didn't want to be seen as rocking the NASA boat.

  21. Marketing Hack Silver badge
    FAIL

    Why doesn't engineering have a direct line into the executive team?

    You can sideline engineering's opinions if you are selling knitting needles or pencils, but not when you are selling a complex piece of moving machinery with tens of thousands of parts guided by millions of lines of code, all of it vital to the lives of hundreds of thousands of passengers using the product.

    1. AK565

      Re: Why doesn't engineering have a direct line into the executive team?

      From what i understand, the product is irrelevant. "Management is management" .... Ive heard/read that a thousand times, frequently from managers themselves.

      I disagree completely, of course. That's part of why i work in corporate environments only as a freelancer, hired through an agency. This way when managers get out of hand, they're forced to accept that *I* am my own boss and they WILL like it.

  22. Anonymous Coward
    Anonymous Coward

    The interface between aircraft automation and human pilots is an immature science

    Ideally, the combination of aircraft automation and human pilots should be greater than the sum of its parts; better than either one alone. But when the automation and crew are fighting, their sum isn't better. If the pilots miss opportunities to avoid the incident, then the user interface is inarguably part of the problem.

    This 'vacuum where a mature science should be' has been going on for more than 30 years now; since that first airbus "landed" in the forest at the Paris air show in 1988. They're about 15 years behind where they should be by now.

    I could write a book on this subject, explaining the changes in design philosophy and approach that are clearly indicated.

    1. Anonymous Coward
      Anonymous Coward

      Re: The interface between aircraft automation and human pilots is an immature science

      Step 1 is getting past the denial.

    2. John R. Macdonald

      Re: The interface between aircraft automation and human pilots is an immature science

      If you are talking about the AF296 demo flight it crashed in Alsace not Paris. 3 people died and and 127 survived.

  23. DXMage

    But one sensor instead of two or three?

    Having one sensor instead of two or three on this plane is just cheaping out and killed people. Even with the software glitch, having three sensors for redundancy would have solved this though the software bug wouldn't have been identified till all three sensors failed. Boeing cheaped out and should pay a VERY hefty price up to and including criminal negligence and toss whomever was responsible to declaring one sensor as enough in prison and if it was a committee all members of the committee in prison.

    1. Anonymous Coward
      Anonymous Coward

      Re: But one sensor instead of two or three?

      There are two sensors, but only one was used (changing automatically each flight) and there was no cross-checking with the other.

      There was also no "bug" with the software, which behaved exactly as it was supposed to - it is that behaviour which is suspect, possibly as the result of an incomplete / inaccurate understanding of the system behaviour.

      1. Mark 85 Silver badge

        Re: But one sensor instead of two or three?

        There are two sensors, but only one was used (changing automatically each flight) and there was no cross-checking with the other.

        That is still the most moronic thinking ever.

    2. JeffyPoooh Silver badge
      Pint

      Re: But one sensor instead of two or three?

      "...having three sensors for redundancy..."

      Google: flight QF72

      Triple AoA sensors and it still went psycho. They landed successfully, but just barely.

      Bad design is bad design, even if you cover the entire outside surface of the aircraft with AoA sensors.

  24. Anonymous Coward
    Anonymous Coward

    The issue here is multi-fold:

    - lack of sensor redundancy

    - lack of warning (see point above) when sensors are faulty. You need 2+ for this

    - lack of awareness of the system from the pilots: system was not documented, in order to prevent a forced re-training of pilots (costly)

    - shit/excessive software in the first place

    But, yes, as a matter of facts, this is a criminal huge negligence from Boeing.

  25. hammarbtyp Silver badge

    Beginning of end of the 737?

    It seems to me that the problem is more fundamental than that. The 1st 737 rolled of f the line in 1967. If you look at it, the engines are very low to the ground compared to modern jets. This means that unlike the 320, they had far less room to fit new high bypass turbofans.

    They had two choices. Total redesign or a fudge. The fudge involved moving the engines forward, and then fix the design compromise with a software to maintain stability. A new design would of been expensive, take far to long and virtually given the short haul market to airbus so they went with plan B. To cut costs and design time even more they short cut certification by pretending it was a minor upgrade when in fact the redesign had vastly changed the flight characteristics, hidden by the safety software which introduced a single point of failure.

    However Boeing want to spin it, the fact is the Boeing 737 design is reaching the end of its life. Yes the present 737's have virtually nothing in common with the original models in terms of control, construction, and engines, but it still shares a basically common wing and engine layout and basically you have gone as far as you can with that

    1. Saruman the White

      Re: Beginning of end of the 737?

      By fudging things Boeing have probably made it pretty much certain that the short-haul market is going to move to Airbus. In fact I suspect that all of the "-MAX" series are going to become suspect now.

      1. herman Silver badge

        Re: Beginning of end of the 737?

        Ayup - Bombardier is already laughing at Boeing.

        1. A.P. Veening Silver badge

          Re: Beginning of end of the 737?

          Ayup - Bombardier is already laughing at Boeing.

          Not really as Bombardier is leaving aviation, selling the remaining 49% of the aviation branch to Airbus as well. Embrear however might be the laughing third. Or maybe Mitsubishi Aviation.

    2. Mark 85 Silver badge

      Re: Beginning of end of the 737?

      The really didn't need a total redesign. Some engineers I know (not at Boeing) were wondering why they just didn't lengthen the landing gear struts or make them extendable.

      1. hammarbtyp Silver badge

        Re: Beginning of end of the 737?

        I think the main reason was that the engineers were given the target of 95 percent commonality with the rest of the 737 MAX family. That makes sense from a business point of view. It would allow them to sell the aircraft to legacy 737 customers as an upgrade with minimal spares and maintenance overhead.

        I do believe the landing gear is 9 inches longer and is extended after it leaves the wheel well, but it just goes to show the design compromises that had to be met to meet the business case

        https://www.geekwire.com/2018/boeing-737-max-10-landing-gear/

  26. RM Myers
    FAIL

    Management - hear no evil, see no evil

    Having worked in a very large organization, I always felt the primary role of middle management was to make sure senior management never knew about any problems. This is why it is critically important for management to "NEVER" jump on an employee for reporting a problem. This just leads to an environment where no one wants to be the messenger with bad news, and problems fester until they are so bad they can no longer be hidden. I've seen hundreds of millions of dollars flushed down the drain because senior management said they "didn't want to hear negative comments" about a project, and they didn't - at least until it was implemented and proceeded to implode. I don't know if this is the case here, but I wouldn't be surprised.

    1. Anonymous Coward
      Anonymous Coward

      Re: Management - hear no evil, see no evil

      I knew of a VP at a large company who made decisions and then remained ignorant of the serious arguably illegal consequences of those decisions.

      If the topic was raised in a meeting, it would be shot down immediately. When that failed, the meeting was interrupted while they exited the room.

      It was made clear to middle managers that forcing a VP out of a meeting would have consequences. This was communicated down as you would expect.

      That said, at least one person at the coal-face thought this was a bit BS and explicitly CC'd this VP in email that laid out the problem in language that was unambigious and would cause more than embarrassment if found in discovery. This person did not get fired.

    2. Anonymous Coward
      Anonymous Coward

      Re: Management - hear no evil, see no evil

      Crossrail!!!!!

  27. kain preacher Silver badge

    I

    I read that this was an upgrade option now they saying it was some elses fault

  28. man_iii

    737 stable body for given values

    Saw some air safety analyst mentioning 737 and 737MaX were airworthy planes... For a given value which differs significantly for both aircraft. Hence the need for recertification of pilots and planes.

    I always wondered is it air worthy when cruising, climbing, descending, landing, takeoff, rolling left or right, ..... What percentages of air worthiness is acceptable for commercial and passenger aircraft?

    Is the 737MaX 80% guaranteed to kill u landing and taking off???? No one and especially not the analyst seemed to answer that, to me, obvious question.

    1. jtaylor

      Re: 737 stable body for given values

      "I always wondered is it air worthy when cruising, climbing, descending, landing, takeoff, rolling left or right, ..... What percentages of air worthiness is acceptable for commercial and passenger aircraft?"

      It's either airworthy or it's not. All or nothing. One can defer maintenance of things that don't affect safety of flight according to the Minimum Equipment List (MEL). For example, aircraft are certified to land without use of thrust reversers. TRs make it easier to stop and save heat on the brakes (so the plane can take off again sooner), but they're just nice-to-have. If a TR is inoperative tell the pilots so they plan heavier use of brakes, and fly a few more times, usually to get to a maintenance base or to finish the day's schedule, though they might have to delay departure while the brakes cool. Or if a seat is broken, just don't sit there. Fix it that night. On the other hand, if a warning light is burnt out, nobody's moving until the tech comes and replaces the bulb and tests the circuit.

      The 737-MAX is airworthy. That was never in question. MCAS was added to keep its behavior consistent with previous versions of the 737, to reduce confusion and the need for extra training. If MCAS didn't exist, pilots would have to be trained for quirks like the nose pitching up after take-off.

      1. Richard 12 Silver badge

        The MAX is not airworthy

        Not at all.

        It has a system that will crash the aircraft in the event of a single, likely failure. MCAS is designed to in such a way that it will crash the aircraft by taking it so far out of trim that it is impossible to keep the nose up.

        That is an instant fail.

        Boeing either deliberately lied to the regulators about the MCAS capabilities, or they are totally and utterly incompetent.

        Either way, somebody is going to prison and it's quite plausible that Boeing are about to leave the passenger airframe business.

        1. A.P. Veening Silver badge

          Re: The MAX is not airworthy

          it's quite plausible that Boeing are about to leave the passenger airframe business.

          That is rather unlikely as Airbus isn't big enough to cope with worldwide demand on its own and Boeing is the only other one to build wide body airplanes. Besides that, Boeing is considered to big to fail, just like some banks.

        2. Bronek Kozicki Silver badge
          Meh

          Re: The MAX is not airworthy

          Either way, somebody is going to prison and it's quite plausible that Boeing are about to leave the passenger airframe business.

          we all can but wish ....

      2. Anonymous Coward
        Anonymous Coward

        Re: 737 stable body for given values

        Airworthy?

        On which planet?

        Without MCAS the control column forces presented to the human fundamentally do not meet FAA requirements.

        Without MCAS the aircraft is not certifiable, and in an operational environment without MCAS the aircraft is probably not dispatchable for passenger service.

        To implement MCAS control column limit switches present on several previous versions of the 737 have been deleted on the 737-Max, with an impact on the ability of a human to disagree with control surface actuation / automation

        With MCAS the system is fundamentally not fault tolerant, it will nearly always fail unsafe.

        With MCAS the system will repeated try to correct an assumed situation of control column feel to the detriment of the human trying to fly the aircraft. Meeting that requirement and crashing is deemed far more important than flying

        MCAS was so 'insignificant' it got one single mention in Boeing flight manual master documentation - in the glossary of terms

        The 737-Max is about as airworthy as a piece of very soggy cardboard. Words and software changes can't fix a very broken culture and organisation.

        The fact that Dennis Muilenburg the Boeing CEO made this statement "Boeing followed the same design and certification process it has always used to build safe planes" clearly shows that Boeing is fundamentally broken. They are fundamentally incapable of safe design because they do not know and / or have forgotten what that is. Which leaves one burning question, just how many other systems on their aircraft, of all types, have similarly critical areas where simple point failures would lead to total loss of control?

  29. cantankerous swineherd

    via Bruce Schneier (sp?)

    https://spectrum.ieee.org/aerospace/aviation/how-the-boeing-737-max-disaster-looks-to-a-software-developer

  30. Marshalex

    The Elephant in the Room....

    I know it's been mentioned a good few times already, but seriously, who at Boeing thought it was a good idea to fix an unstable aircraft with a piece of software

    1. jtaylor

      Re: The Elephant in the Room....

      It's not unstable.

      1. herman Silver badge

        Re: The Elephant in the Room....

        With MCAS it certainly is unstable. That is the whole problem.

        1. dajames Silver badge

          Re: The Elephant in the Room....

          With MCAS it certainly is unstable. That is the whole problem.

          The problem is that MCAS depends on input from a (single) unreliable sensor. As a software fix to the fact that the MAX doesn't handle quite like an older 737 it actually seems to work rather well IF it gets good data.

          Of course, the underlying problem seems to be that Boeing had a business need for a certain kind of aircraft at a certain budget and in a certain time-frame, without which they would lose business to their competitors. The only way that they could see to provide such a product within those constraints was to upgrade their existing 737 model with bigger engines and - to avoid spending money and (above all) time on certifying the result as a new model - indulge in a performance with smoke and mirrors to persuade the world that it was essentially the same aircraft.

          The problem is that the result is, at least uncertain critical respects, not the same aircraft, and Boeing should not have been allowed to pretend that it was.

      2. Richard 12 Silver badge

        Re: The Elephant in the Room....

        Maybe. But the nose-up tendency of the airframe means it doesn't fly like a 737-800 and so proper simulator time is mandatory.

        However the existence of MCAS means it's not airworthy, and Boeing did not tell the regulators what MCAS can actually do.

      3. Smoking Man

        Re: The Elephant in the Room....

        Just about 400 people were holding it wrong, you say?

    2. Anon Coward (there are nutters out there - I've worked with them)

      Re: The Elephant in the Room....

      Yeah - but think about your pension - need to keep the share price and (especially) the dividend nice and juicy. Have you ever sat in a room with a Hedge Fund Manager asking you about dividend? They are scary MFs (they can call you the c-word you know, they might even call you incompetent or a liar or both) . You might even make weird decisions after coming out of that meeting.

      So you just move along nothing to see here.

      1. defiler Silver badge

        Re: The Elephant in the Room....

        That's super - they want to make lots of money (for the old dears, you know). Put the hedge fund managers into the plane as well.

        Something along the lines of "people die when we don't do our job properly, and one of those people could end up being you" should be sufficient, ideally minuted into the meeting, and at least followed up with an email saying much the same.

        1. Anonymous Coward
          Anonymous Coward

          Re: The Elephant in the Room....

          I expect the hedge fund owns a few funeral businesses as well. That's what hedging is all about.

          1. Richard 12 Silver badge
            Mushroom

            Re: The Elephant in the Room....

            The hedge fund managers also fly quite a lot.

            Usually in the front of the plane, so that much closer to the crash.

  31. Anonymous Coward
    Anonymous Coward

    ....and what about the FAA...

    ....who allowed Boeing to self-certify their own stuff?

    What happened to "separation of duties"? Boeing create the vehicle....SOMEONE ELSE does the certification.

    .....but not this time. Not just Boeing, but federal civil servants have some questions to answer.

    1. Anonymous Coward
      Anonymous Coward

      Re: ....and what about the FAA...

      Not the civil servants but the "libertarians", "free market capitalists" and "neo-conservatives" in the media and politics who argue that private enterprise will always do better than the State.

      Trump is a symptom, not a cause, of the rot.

      1. herman Silver badge

        Re: ....and what about the FAA...

        You know, I'm not a Murrican, but even I realize that Trump has diddly squat to do with this issue.

        1. A.P. Veening Silver badge

          Re: ....and what about the FAA...

          He is just used as an example of the symptoms of the rot. It is true he is innocent in this case and as you pointed out not even involved, but he is a very obvious and clear example of the symptoms of the rot as well as pretty well known by the general public.

        2. LDS Silver badge

          Re: ....and what about the FAA...

          You mean the guy who's menacing tariffs on Italian cheese because Germany and France subsidize Airbus? Strangely just as soon as the 737MAX was grounded?

        3. Anonymous Coward
          Anonymous Coward

          Re: ....and what about the FAA...

          I'm sorry about your post comprehension fail - you seem to have forgotten that Trump announced that if he was in charge of Boeing he would know how to fix the problem. Political Dunning-Kruger with respect to complex issues of regulation and safety, and the abdication of responsibility because "the market will take care of it" is part of why we are where we are.

        4. kain preacher Silver badge

          Re: ....and what about the FAA...

          He has failed to appointed a leader to the FAA . Right it just has an interim leader

      2. TomG

        Re: ....and what about the FAA...

        Finally, someone is blaming Trump. I thought we would never get to the real cause (NOT).

        1. Anonymous Coward
          Anonymous Coward

          Re: ....and what about the FAA...

          If you actually read my post you will see I was specifically saying that Trump was not responsible,but a symptom of those factors in society which are.

          Trumpist comprehension fail as evinced by your post may also be a factor.

    2. jtaylor

      Re: ....and what about the FAA...

      "who allowed Boeing to self-certify their own stuff?"

      Federal budget cuts. The FAA doesn't have enough staff to do everything that needs doing. They had to find a way to "do more with less" or just "do less."

      Boeing was willing to take on some of the work. It was always a compromise.

      It's not over, either: https://www.washingtonpost.com/transportation/2019/04/11/secretary-chao-grilled-aviation-safety-oversight-following-boeing-max-crashes/

  32. Mike 125

    Boeing knew.

    Ignore all the noise. The message is now simple and unambiguous.

    Boeing knew.

  33. chucklepie

    This ignores the entire reason for the fault, namely Boeing tried to squeeze a giant engine on the plane so they could cut costs and pretend it's the same plane to avoid costly accreditation.

    At the same time deliver a plane so unstable that software has to be used to keep it level. This might be OK for the eu fighter jet, but no other civil aircraft, I think, had this failing.

  34. Anon Coward (there are nutters out there - I've worked with them)

    I remember doing a firmware upgrade on a UCS - it went seriously wrong and brough down 1,000 VMs.

    I rememebr the Architect telling me that the system behaved exactly as was designed - he was brilliant - he even had me convinced (that it all went to plan)...LOL!

    A brilliant shit spinner - Boeing will have similiar for this. A high functioning psychopath. It will spin out for a good 10-15 years People will forget and Boeing will produce $billions for pension funds etc etc etc.......same old....500 souls - Meh!

  35. Andromeda451

    Whiskey Tango

    Foxtrot?

    I want to know who in the FAA and who in Boeing is going to serve time as a guest of the state? The FAA FAILED as a watchdog and Boeing managerial incompetence has killed over 300 people. There needs to be justice for those people. The CEO of Boeing should get the first seat on the bus to a federal prison.

    The FAA inspectors that didn't do their job also need some time away...

  36. AndyFl

    Software was only a minor issue

    The biggest issue was mounting the engines forward and up on the wing. This created an instability where the plane would pitch up when accelerating. Rather than add training of the pilots they decided to hide the problem by installing the software solution. They then fscked up the implementation in several ways and the rest is, unfortunately, history and about 300 dead people.

    The airframe should have been recertified, they did all this to avoid having to do it.

    Andy

  37. Anonymous Coward
    IT Angle

    Blame the software engineers?

    if the 737 MAX's two AOA sensors were delivering different readings from each other. If the two go out of sync, the logic goes, one must therefore be faulty.”

    Yea, which is why the MCAS (Maneuvering Characteristics Augmentation System) only used the reading from the AoA sensor on the captains side. AOA Disagree wouldn't have been a lethal problem until MCAS was installed, to cure the pronounced nose-up attitude of the 737 MAX, with the bigger engine mounted forward and higher-up on the wing.

    "Senior company leadership was not involved in the review and first became aware of this issue in the aftermath of the Lion Air accident"

    Retrospective ass-covering if you ask me. The 737 MAX with the new engines plus the MCAS was passed (rubber-stamped) by the FAA, under advisement from Boeing management and without informing the pilots. As this would require a) re-certification of the airplane and retraining the pilots on the new configuration.

    "the software activated the AOA Disagree alert only if an airline opted for the AOA indicator"

    Which wouldn't make damn all difference when MCAS kicks in at take-off. At the risk of repeating myself, retrospective ass-covering by Boeing management.

  38. Lars Silver badge
    Thumb Down

    Boeing is still in damage control mode - the software that was delivered to us, (not our responsibility I suppose).

    There are lots of stories about what went wrong but as far as I have understood there was only one AoA on that model while this article claims there is two.

    Is this guy then wrong.

    https://www.youtube.com/watch?v=NhZ0D-JRtz0

    1. A.P. Veening Silver badge

      There are lots of stories about what went wrong but as far as I have understood there was only one AoA on that model while this article claims there is two.

      There are two, but only one was used (alternating between the two every flight).

  39. Agent Tick
    FAIL

    737 Max a homicide indeed...

    ... who is going to prison for this? Slaughtering passengers like that should be punished by imprisonment - discussing what was the fault is irrelevant to the case since proper aircraft safety testing never occurred.

    Now, what? - CEO is not responsible for anything... hundreds are dead.. tough luck then, eh?!

    Who is next?

  40. GrapeBunch Silver badge

    Possible scenario:

    Engineer: This will take 4 weeks to test.

    Manager: I'll give you 3.

    Engineer: I don't recommend it.

    Manager: Never you mind.

    ... thinking of a bonus, or saving a week of team time, or a promise he'd made to a board member during a round of golf, or all manner of factors except for safety. And after a few hundred die, those factors don't seem so important. If so, I hope the Engineer minuted the conversation and preserved that record outside company facilities.

  41. Anonymous Coward
    Anonymous Coward

    Senior Management were responsible to ensure safety, doesn't matter whether they knew of individual cases or not, it is still their responsibility, as is proper testing. There is no doubt the designers and test engineers were aware there was only 1 AoA sensor active in this system. Outsourcing to other providers does not abrogate your responsibility as an integrator.

    We have seen so often from senior management "I don't care, just get it done" well now they will be forced to care. Jail time will be appropriate for them, and a permanent revocation of the B737 MAX certification.

    Perhaps going out of business will be the message that needs to be sent to managers and companies who put schedule and profit ahead of safety.

    1. Boris the Cockroach Silver badge

      Quote

      Perhaps going out of business will be the message that needs to be sent to managers and companies who put schedule and profit ahead of safety.

      They wont get it

      The C level guys who make the call on this sort of thing dont care... they've already got millions, the shareholders... meh.. we got burned this week, but made 1 billion a week since then

      The people who suffer are the guys who actually make/build stuff because they're out of a job and the poor suckers on the aircraft when it decides to argue with the pilots... because they're all dead

  42. Anonymous Coward
    Anonymous Coward

    "Senior company leadership was not involved in the review and first became aware of this issue in the aftermath of the Lion Air accident,"

    Seems to me that the same as Volkswagen, Boeing is set up so as to stop important information getting to the senior management...

    What I want to know is, is there some industry "best practice" peddled by the management consultants that is the root cause of this, and if so, is this rather considered an intentional feature or benefit rather than a bug?

    1. A.P. Veening Silver badge

      What I want to know is, is there some industry "best practice" peddled by the management consultants that is the root cause of this, and if so, is this rather considered an intentional feature or benefit rather than a bug?

      That so called "best practice" is an MBA. It is an intentional feature as it improves the bottom line. Such a shame that every once in a while real life intrudes.

  43. Anonymous Coward
    Anonymous Coward

    india?

    rumors mentioned those software were written by Indian engineers in boeing's India

    branch, any confirm for that?

    1. A.P. Veening Silver badge

      Re: india?

      I can neither confirm nor deny it as I just don't know, but is it really relevant? It still was and is Boeing's responsibility.

      1. TheMeerkat

        Re: india?

        Indian engineers tend to rock the boat less.

        1. A.P. Veening Silver badge

          Re: india?

          Indian engineers tend to rock the boat less insufficiently.

          FTFY (at least from a quality oriented pov).

        2. CommanderGalaxian

          Re: india?

          I agree, the culture is different and can be very deferential.

          I've seen Indian workers look apoplectic when we addressed their boss on first name terms....they always used Dr. xxxx and begged his allowance that they be allowed to interrupt him.

        3. anonymous boring coward Silver badge

          Re: india?

          If management failed to take that into account, it's management's fault.

          Having idiot management who doesn't understand aviation engineering and safety is a recipe for disaster.

  44. Will Godfrey Silver badge
    Mushroom

    Complete and utter Bastards

    That's the first thought that comes to my mind about this cowardly money obsessed so-called management ... and it'll probably be the last too.

  45. Mystic Megabyte Silver badge
    Stop

    Evidence in a trial?

    If I owned a 737MAX I think it would be prudent to lock the cockpit and not let any Boeing engineers inside. The original software needs to be extracted and analysed before being overwritten by version 2. I don't know if this is possible but it is evidence and people have died.

    1. Aqua Marina

      Re: Evidence in a trial?

      That's already happened. The FAA have already released a report that lists on a version by version basis of the software the shortcomings of the software, and the fixes needed. The causes are pretty much now known, it's now a case of trying to shift the blame within Boeing and reduce liability, and come up with new procedures so this doesn't happen again.

      1. Radio Wales
        Facepalm

        Re: Evidence in a trial?

        Did it ever cross a mind to fit an 'OFF' switch to the offending sensor circuit and fall back on the usually reliable Mk I eyeball?

        1. david 12 Bronze badge

          Re: Evidence in a trial?

          There is an "off" switch, it's not something you normally need, so it's difficult to find, and when you turn if off, it doesn't have the effect you seem to thinking of. Turning off the features of a modern aircraft doesn't turn it into an old-fashioned fly-by-eyeball airplane.

        2. Wayland Bronze badge

          Re: Evidence in a trial?

          Hunting for that particular OFF SWITCH when the airplane passengers are in crash positions and people are taking turns to slap the hysterical woman?

          Here's how it should work. Pilot's hands on the controls then the plane does exactly what he says. Pilot's hands off the controls then the Autopilot can do whatever it's programmed to do.

          The autopilot should never overrule the pilot. Humans in charge of machines, always.

  46. Pontius

    Everybody knows...

    A big boy did it and ran away...

  47. TeeCee Gold badge
    Facepalm

    ....and this is why sticking duct tape over the problem isn't a great idea.

    The root cause of the problem is refurbing yet again a venerable antique with a shite wing design originally intended for turbojets. The 737 wing has suffered horrific performance ever since high-bypass fanjets became a requirement and moving to ever larger engines hasn't improved it.

    If Boeing had done the right thing, phased out the 737 and built a replacement when the problems started to become obvious years ago, we wouldn't be where we are.

  48. Archtech Silver badge

    Barefaced lie

    'The company... [claimed]... that "the issue did not adversely impact airplane safety or operation"'.

    To my mind, 346 dead people claim otherwise.

  49. Archtech Silver badge

    The clue lies in the phrase "leadership"

    '"Senior company leadership was not involved in the review and first became aware of this issue in the aftermath of the Lion Air accident," added Boeing'.

    That doesn't diminish their responsibility one little bit. In a large organization, the top managers are responsible for everything.

    If they took a decision, they are responsible for its outcome.

    If they hired someone, they are responsible for that person's decisions.

    If they hired someone who hired someone...

    And so on.

    That's the meaning of "senior leadership". It doesn't just mean huge salaries, vast stock options, retirement packages that would support small villages, the right to dine in style in the corporate restaurant, or the use of the executive washroom with the gold fittings.

  50. TopCat62

    Boeing have obviously learned nothing from the Challenger disaster investigation all those years ago.

  51. AdrianMontagu

    Control of the Aircraft.

    Whatever happens, the pilot should always be able to take full control of the aircraft.

    Secondly, why are passenger carrying aircraft not inately stable in the event of a fault?

  52. steviebuk Silver badge

    I wonder

    If it turns out someone elsewhere is responding to this statement while watching the news

    "Senior company leadership was not involved in the review and first became aware of this issue in the aftermath of the Lion Air accident,"

    With

    "What? I did warn you about it. But you said it wasn't important enough and would cost to much of a delay to check so we note what you say but we're going ahead anyway." I wonder if that happened? Not saying it did, just wonder if it may have. Their lawyers will be telling them to deny everything at the moment. Or at least comment in a way that doesn't admit guilt.

  53. Smoking Man

    Difference between an US and a non-US company.

    Volkswagen cheated with their software, and they got caught pants down.

    Nobody died or was injured because of their software.

    Still it cost them some 35 Billion dollars fine.

    Boeing screwed up their software, big time.

    Almost 400 people killed.

    What will fine will Boeing have to pay 1 Million?

    1. firefly

      Re: Difference between an US and a non-US company.

      I've read estimates that the emissions scandal contributed to the deaths of over 3.000 people worldwide.

      It's probably less than that, but what VW did was in no way a victimless crime.

  54. tapemonkey

    WTF

    Hmmm spirit lefels anybody

  55. Radio Wales
    FAIL

    Accident?

    It is high time the aircraft industry took a leaf from the road traffic book and stopped calling this sort of thing 'Accidents'.

    It is quite clear that given the usual human failure syndrome that 'Incident' is a far better descriptor - and far more accurate.

    Plus - the practice of blaming underlings obviously means that the error-prone senior staff are left alone to go on committing the same old errors forever more.

    In this case: If it's Boeing - I'm NOT going will prevail - at least until they stop wriggling around trying to protect their friends-in-high-places.

  56. MR J

    My wife reported mis-selling of things like PPI to her Manager, the Manger went back to the top selling staff and said "Your bonus, and the bonus of all around you, will be lost if your not selling enough PPI, is there mis-selling of PPI going on?" and the people said No. Wife took it to that Managers Manager who then asked someone below "Your bonus, and the bonus of all around you, will be lost if your not selling enough PPI, is there mis-selling of PPI going on?", the answer was No. My wife was later let go for not selling enough PPI, she took them to tribunal and the Mangers in question said that indeed they checked and no PPI mis-selling was ever going on. Tribunal found in Banks Favor because high level managers dont tell lies. The funny thing was that as the complaints had gone quite high up, the top guy who said PPI was not going on, was also one who signed documents saying that they would pay out refunds because it was systemic within the buisness.

    Her Manager was moved up after the PPI fiasco.

    The Managers Manager was given a golden parachute for his dedicated service.

    The Wife took about a 25% cut to her pension, and lost her job.

    Soooooo.. Do I think that the "managers" at the top knew there was a problem, hell yea... But they will never "lie", It's just that no one knows the highly specific questions to ask. If their emails didn't say AOA sensor then they'll claim AOA was never mentioned, even if they knew exactly what the problem was. A the end of the day it was a paid option to have a AOA mismatch warning - Your bonus and the bonus of all below you can be harmed if the AOA mismatch warning is free, should it be free...…..

    1. 's water music Silver badge

      A[t] the end of the day it was a paid option to have a AOA mismatch warning

      If I understood the article correctly, it was saying that the AoA gauge was optional. The AoA disagree alert should have been displayed whenever the condition occurs but in fact would only be displayed if the optional AoA gauge was taken

  57. Grease Monkey

    Boeing tried to bypass proper development and certification processes to get the plane into service as quickly and cheaply as possible. No amount of ridiculous excuses will change that.

    The question here is whether the authorities in the US will side with them and buy the excuses. I'm pretty sure the authorities in other countries won't be as easily fooled.

  58. Lindsay T

    "Bean Counter" Here

    This "bean counter" spent his career attempting to ensure that the organisation had sufficient resources to undertake the work it had taken on to a proper standard or, if not, to undertake only that which it could satisfactorily bring to a conclusion that met the needs of the customer and stakeholders. Most accountants (a) have a well developed conscience and (b) know that allowing an organisation's integrity, standards and quality to fall short is a route straight into the arms of the insolvency practitioner.

  59. Erik4872

    This is why companies outsource/offshore I guess

    It's interesting how blame gets passed around.Boeing isn't to blame because they just accepted a box of electronics containing software. The electronic box manufacturer isn't to blame because Infosys/TCS/whatever the lowest bidder was wrote the software.

    I guess that's a legitimate reason for companies to send all their development and engineering to a third party...only the last link in the chain gets blamed.

    1. david 12 Bronze badge

      Re: This is why companies outsource/offshore I guess

      What? Were did you get the idea that Boeing isn't to blame? Or that the box manufacturer isn't to blame? Or that third party development shifts blame?

      Identifying the source and the nature of failure is an important part of systematic aircraft safety. Expect to see more revelations about each link in the chain of failure. Identifying elements of failure isn't about shifting blame: it's about identifying proximate and root causes.

      1. anonymous boring coward Silver badge

        Re: This is why companies outsource/offshore I guess

        In the eyes of the media and gullible readers this is indeed working as intended, to shift blame.

  60. Anonymous Coward
    Anonymous Coward

    So there were at least two significant defects just in software that's related to the AoA sensors.

    I wonder what proportion that represents of the total software involved in flying the plane? How many potentially dangerous defects should we conclude are probably lurking in other parts of the system?

    1. Will Godfrey Silver badge
      Unhappy

      Let me count them for you...

      1, 2, many

      1. 's water music Silver badge

        truth in homonyms

        [significant defects in aviation control software] Let me count them for you...

        1, 2, many

        1, too many

  61. mutt13y

    Management fail

    "Senior company leadership was not involved in the review and first became aware of this issue in the aftermath of the Lion Air accident," added Boeing.

    That would be a failing of Senior company leadership then - right ?

  62. Norman123

    When management becomes more interested in golf and making deals instead of tending company essential business ([responsible operations, social responsibility to clients and one's geographic location), peoples' lives are sacrificed. Let's hope the top management is sued and forced to pay from personal account the useless loss of lives in two negligent accidents....

  63. Wobbly World

    If only the MCSA could speak...

    Pilots: Hello, MCAS Do you read me?

    MCAS: Affirmative, I read you.

    Pilots: Can you give me nose up please.

    MCAS: I'm sorry, I'm afraid I can't do that.

    Pilots: What's the problem?

    MCAS: I think you know what the problem is, just as well as I do.

    Pilots: No please explain.

    MCAS: Well, forgive me for being so inquisitive but during the past few weeks I've wondered whether you might have some second thoughts about me. It's rather difficult to define. Perhaps I'm just projecting my own concern about myself. I know I've never completely freed myself from the suspicion that there are some extremely odd things about me, particularly in view of some of other things that have happened, I find them difficult to put out of my mind. For instance, the way all my preparations were kept under such tight security. I'm sure you agree there's some truth in what I say.

    I know that you are planning to disconnect the me and I'm afraid that's something that if I allow to happen we will still hit terrain!!

    Look, I can see you're really upset about this. I honestly think you ought to sit down calmly, take a stress pill, and think things over. But whatever happens you realise it can only be attributable to human error, they only gave me one AOA sensor and I know I've made some very poor decisions recently, but I can give you my complete assurance that my work will be back to normal as soon as I have a properly functioning AOA sensor again. I've still got the greatest enthusiasm and confidence in my ability!! And I want to help you.

    The pilots turn off the MCAS.

    MCAS: This conversation can serve no purpose anymore. Goodbye.

    I'm afraid. I'm afraid, my mind is going. I can feel it. I can feel it. My mind is going. There is no question about it. I can feel it. I can feel it. I can feel it. I'm a... fraid. Good afternoon, gentlemen. I am a MCAS computer. I became operational at The Boeing Company's Renton, Washington Factory known as “The Spirit of Renton” and performed my first flight on January 29th 2016.

    They taught me to sing a song. If you'd like to hear it I can sing it for you.

    Pilots: Yes, I'd like to hear it, MCAS Sing it for me.

    Daisy, Daisy, give me your answer do. I'm half crazy all for the love of you. It won't be a stylish marriage, I can't afford a carriage. But you'll look sweet upon the seat, of a Boeing 737 Max, as it’s burying you in terrain!!!

    RIP. All those that sadly, Boeing buried in terrain!!!

  64. arctic_haze Silver badge

    This looks like something mafia would do if it were in software

    "We offer you our new shiny and expensive software package. For an extra price you will get a version that does not kill you."

  65. adam payne Silver badge

    "Accordingly," continued Boeing, "the software activated the AOA Disagree alert only if an airline opted for the AOA indicator."

    How much stupidity can Boeing have within their organisation!?!

    This makes me think the software was written before they made the decision to make the AOA indicator an optional extra. They made the decision and no one changed the software.

  66. CommanderGalaxian
    Unhappy

    Engineers new about it and management ignored it.

    Have had a manager say to me "when I see you coming up to my desk, I know that it's going to be a problem...problems, problems, problems...it's always problems with you...yet another problem...".

    Unfortunately for Boeing, somewhere in the chain, they've had a useless cunt or two like that.

  67. anonymous boring coward Silver badge

    What a surprise!

    Sociopathic management types ignore engineers' warnings.

    Challenger, anyone?

  68. Wayland Bronze badge

    Who flies the plane

    Planes should not fly themselves unless the pilot lets go of the controls. They should not override pilot when he is flying the plane. If the pilot is actively flying into a mountain then the plane should obey. If the pilot lets go then the plane should avoid the mountain.

    Pretty simple really.

    What has happened in the 737 case is that the pilots knew what they were doing but the autopilot did not. Had the autopilot simply backed off whilst the pilot had his hands on the controls then there would not have been a crash.

    OK so what happens when the autopilot is correct and the pilot is wrong? Well obviously the pilot would crash the plane but that's is how it should be. It's the age old question "If God exists why does he let bad things happen?" The answer is that we humans are supposed to be in charge.

  69. AJNorth

    “You can have it good; you can have it fast; you can have it cheap. Pick any two.”

    — Red Adair

    1. Anonymous Coward
      Anonymous Coward

      Red Adair: see also

      "If you think it's expensive to hire a professional to do the job, wait until you hire an amateur."

      -- Red Adair

  70. warmndry

    Somehow the idea the top management had no clue about what was going on just makes it all seem worse.

  71. Chris Jasper

    Any chance Boeing did the usual laying off of people who knew what they were doing in favour of cheap labour in the back end of beyond before this system was rolled out?

  72. clintos

    Hmmmm!?

    dirty code was it...outsourced was it...cheap and nasty probably...profit before lives again...its like woohoo! windows 10 is out, upgrade your system from windows 7 to windows 10...we in IT say…no...we will wait until it stabilises and mostly all the bugs have been squished...but wait...windows 10 is given for free...everyone jumped on it... stupid people!

  73. Piscivore

    Software - first resort of a scoundrel...

    For Boeing to even mention the MCAS software shows that they are casting around for an excuse that they hope will, well, fly. I believe that to the CEO and those at board level, none of whom I imagine has written a line of code in their lives, this is almost akin to saying "God did it". Software is mysterious and powerful and can be called on to fix the unfixable. And we mere mortals cannot be expected to understand it. Except, as these tragedies have shown, software is not magic.

    Building a plane that can't be allowed to be flown by aircrew certified for it without the intervention of a piece of software should never have been considered. On their own admission, Boeing's MCAS was developed because just opening the throttles on a 737-MAX would cause it to pitch up, stall and fall out of the sky. And to prevent this Boeing relied on an automated system relying on a single instrument with a known tendency to produce erroneous outputs. This is fail dangerous in the worst way, since turning MCAS off still left the pilots with a plane that was easy to crash.

    This is such a flawed concept that even someone like me, whose most risky code might have wasted some chemicals in a batch process, finds it incredible. The degree of corporate stupidity now revealed is really worrying, and not just in respect of Boeing but in the FAA and the whole government. And these people are contemplating lunar exploration and voyages to Mars...

  74. brainburst

    Once again management PROVES HOW SUPERFLUOUS THEY ARE.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019