back to article Boeing 737 pilots battled confused safety system that plunged aircraft to their deaths – black box

With investigations continuing into the cause of Lion Air flight 610's deadly crash into the Java Sea, experts are scrutinizing the plane's automated control system. Early data from the doomed plane's flight recorder suggests that for much of its 11-minute flight on October 29, the pilots were struggling with the craft's …

  1. Alan Bourke

    So anyway, tell me again how great an idea self-driving cars are ...

    ...or would be if they were remotely feasible today.

  2. Someone Else Silver badge
    Big Brother

    Oh, and...

    ...yes, we are very sure that driverless cars will never suffer this same fate...same fate...same fate...same fa...............

    Big Brother, because he wants full control, too

  3. naive

    Mathematical proof of correctness or a big RED "707 mode" button

    Even if the technicians made a mistake in installing the sensors, Boeing could be in a lot of trouble.

    Either Human pilots have the last word, or the computer. Since with this flight, the computer crashed the plane, Boeing has some explaining to do, for instance why does the plane take off in the first place if those sensors are faulty, leaving the plane in a situation where it can not be controlled.

    Boeing just needs to make clear that a) the pilot is always responsible and b) make a big red "707 mode" button disabling all nanny systems interfering with actions performed by the pilot. This button allows pilots to take full control, the computer should resort to writing advisory messages on a screen.

    In the end, if people could fly four engined 707's in the 60's without computers, they should be able to fly a 737 without them in 2018.

  4. kurios

    need an off switch

    There are too many circumstances in which automation can be in control of some aspect of aircraft operation. If one of these instances goes wrong, it takes too long to identify which one has gone bad and run the appropriate procedure while under high crew task load. And that's if the aircraft builder has actually kept documentation up to date.

    There should be a single switch that unconditionally inhibits all automation, returning full manual control to a pilot. Not a panacea, but it might have prevented this accident.

  5. Anonymous Coward
    Anonymous Coward

    Re: need an off switch

    A Big Red Switch in the middle of the cockpit should do it.

  6. SkippyBing Silver badge

    Re: need an off switch

    'There should be a single switch that unconditionally inhibits all automation, returning full manual control to a pilot. Not a panacea, but it might have prevented this accident.'

    But what if one of the a pieces of automation stopping you from having an accident? These aircraft aren't designed to be flown in full manual control, there's always some form of automation at work keeping it in limits.

  7. Anonymous Coward
    Anonymous Coward

    Re: need an off switch

    "There should be a single switch that unconditionally inhibits all automation, returning full manual control to a pilot. Not a panacea, but it might have prevented this accident."

    And caused, or failed to prevent, other accidents.

    There is no universal solution, and human pilots have racked up a fairly poor track record, over the years.

  8. Anonymous Coward
    Anonymous Coward

    planes/cars

    This is worryingly similar to my experiences with all car mechanics.

    Take my car in. The EML light is on. Debug codes. MAP sensor is giving implausible reading. Assume sensor is faulty. Replace map sensor. Give car back to customer minus £100.

    Driving out of garage EML light comes on again.

    Fault was not with MAP sensor at all.

    Thank god my car doesn't fly.

  9. Anonymous Coward
    Anonymous Coward

    Re: planes/cars

    Indeed. Multiple failures reported on previous flights and some of the "repairs" consisted of cleaning the electrical contacts.

    Yes, surely it's those rusty, dirty contacts that were the problem, and just needed a good cleaning. On a 3-month-old plane.

  10. Anonymous Coward
    Anonymous Coward

    Re: planes/cars

    "On a 3-month-old plane"

    Years ago I flew Edinburgh to Heathrow on a British Midland 737. During flight the cabin became uncomforatbly warm for a bit but when it goit a bit cooler the captain came on the intercom to apologize for the temperature fluctauation adding that "this is a new plane and we've been having a number of teething problems with it". 2 or 3 weeks later one of British Midland's new 737's crashed at Kegworth.

  11. SkippyBing Silver badge

    Re: planes/cars

    'Yes, surely it's those rusty, dirty contacts that were the problem, and just needed a good cleaning. On a 3-month-old plane.'

    You don't work on a lot of aircraft do you?

    Compared to the production run of the average car even the 737 is a niche product, meanwhile all the parts are subjected to a drop in pressure and temperature that would stop the average automotive engine running every time they fly a sector. So after three months a contact may well just need a clean because it wasn't installed correctly in the first place or even just because.

  12. Alan Brown Silver badge

    Re: planes/cars

    " MAP sensor is giving implausible reading. Assume sensor is faulty. Replace map sensor"

    The first thing my mechanic assumes is a bad plug somewhere.

  13. Alan Brown Silver badge

    Re: planes/cars

    "surely it's those rusty, dirty contacts that were the problem, and just needed a good cleaning. On a 3-month-old plane."

    Don't be surprised.

    They may be on an assembly line, but these things are _hand built_. The looms are hand made and the electrics quite frankly make Lucas look good. Look at the report into the wiring of TWA800.

    "Aircraft grade" electrical connectors are about the mechanicals side. I've seen some pretty shitty contacts on brand new "out of the bag" components and the degree of attention to long-term reliability is significantly lower than that given to automotive connectors because the assumption is that they're going to be opened/closed/inspected every N hours, not left in service for years and expected to provide 100% reliability without being touched.

    (Automotive spec is significantly tougher all around than aviation or military spec)

  14. Anonymous Coward
    Anonymous Coward

    Agile Software Development.

    Would you trust your life or those of your loved ones to this?

  15. Jason Bloomberg Silver badge
    Joke

    Re: Agile Software Development.

    "Sure I would. My team are so crap it would never get to the end of the runaway, let alone ever take off".

    The oldies are the goldies.

  16. sean.fr

    more acceptable to be killed by a human

    Pilot error is a major cause in crashes. Automation basically kills less people. Therefore it is a reasonable design decision to to automate as much as possible. In the same way air bags can hurt you, but we require them to be fitted in new cars because overall the benefits outway the risks by a large margin.

    The route cause is flying with bad instrumentation. Even if you fly manually, bad sensors can kill you. In daylight you can judge you angle of attack, height and speed. At night, or weather probably not. You have to trust the instruments. You can tweek the software, and pilot training, but that is larglying ignoring the real issue. The aircraft was not in a fit state. If you crash a car because your breaks fail, no-one asks why driving without breaks is not taught. ABS can help, but if you loose break oil or the pads are worn, or the tyres are worn, its a maintenance issue.

  17. Dave 32
    Pint

    Re: more acceptable to be killed by a human

    Ah, but driving without brakes is taught. And, I've put it to use many times, when I've blown either a master cylinder, or a brake line. You tend to remember that emergency brake pretty quickly, as you're aiming for something soft and cheap.

    The first time it happened, was as a teenager, as I was pulling into the family driveway, with my dad's new car 30 feet in front of me. I hit the brakes, and the pedal went all the way to the floor, when the master cylinder catastrophically failed (They didn't have redundant cylinders in 1966 models.). Talk about high pucker factor. I did a hard left into the soft dirt of the front yard, and jammed on the emergency brake. It made a royal mess of the lawn, but I didn't crash.

    I've subsequently blown the brake lines on my pickup truck, twice, and on the wife's SUV once.

    Perhaps the most spectacular incident was when my brother blew the brakes on a 5 ton dump-truck, with an overload of 7 tons of crushed stone in it. Not only did he not wreck it, but he actually made it back to his farm, and dumped the crushed stone where he wanted it, via use of the emergency brake and down shifting the transmission.

    Dave

    P.S. Beer, because, after incidents like those, you need a beer or two.

  18. david 12 Bronze badge

    Small Indonesian airline. That suggests that the pilot was an experienced foreign national, and that the co-pilot was not. Was that the case here?

  19. SkippyBing Silver badge

    Lion Air is not a small airline, they're the Asian equivalent of RyanAir or EasyJet and the largest in Indonesia, bigger than the national airline Garuda. They were also the launch customer for the 737 MAX. The pilots were Indian and Indonesian with 6028 and 5174 flying hours respectively, does that answer your slightly loaded question?

  20. david 12 Bronze badge

    Yes. the question was slightly loaded: I was trying to be as polite as I could be while still asking the substantive question.

    I wondered if that had changed: evidently not. Racial politics still plays a part in pilot selection in Indonesia.

    Having said that, it's just as well that everybody reading this should realize that locally, some people are going to be blaming one pilot because he is foreign, and other people will be blaming the co-pilot because he is not foreign.

  21. Borg.King

    AoA sensor replaced prior to fatal flight

    Wasn't the Angle of Attack sensor replaced prior to this last fatal flight of this aircraft?

    So far I have heard reports that there were issues on the 3 previous flights to this one, and the sensor had been replaced. Had one faulty sensor been replaced with another one? Had the wrong sensor been replaced? Did the new sensor pass all ground tests on the system before departure? Had the ground tests been properly executed? Has the cockpit been found to determine the state of the MCAS switches?

    If the MCAS / AoA system had issues on the previous flights, were those pilots aware of the new equipment operating procedures on the 737 MAX? How did those previous pilots overcome the documented system failures?

    The final AAIB report on this will make extremely interesting reading. I hope they find the cockpit voice recorder, though I expect it will just have the two pilots extremely baffled at why their aircraft is repeatedly countermanding their control inputs.

  22. SkippyBing Silver badge

    Re: AoA sensor replaced prior to fatal flight

    It appears, based on reading the FDR data by people more knowledgeable than me that the previous flight's pilots had overcome the MCAS issue by diagnosing a failure that required isolation of the trim system. As they didn't know about MCAS they wouldn't have known that's what the problem system was but the effect was to remove it from the equation anyway. However this may have meant the next crew were starting from square one when it came to diagnosing what was going on with the various issues that they were presented with or have been primed to expect a reversal of the usual trim system rather than starting from scratch. Neither of which are great situations to be in.

    'Has the cockpit been found to determine the state of the MCAS switches?'

    There aren't any as such, and the pilots weren't told it existed previous to the accident. The trim cutout switch position can be determined from the FDR and I believe they hadn't been activated at the time of the accident.

  23. Alan Brown Silver badge

    FAA grounding imminent?

    I'm surprised the FAA hasn't issued an emergency airworthiness directive grounding the aircraft already to be honest.

    This system is necessary thanks to the engines - over the years they've become so big that they've moved out from under the wings to in front of them and also been lifted up to achieve ground clearance - to the point that they can flip the aircraft on its back if the pilots aren't careful. (They've had to do this because it's impossible to give the 737 design longer landing gear)

    This is what happens when you take a "good proven stable design" and keep incrementally slapping shit on it until it is intrinsically unsafe to fly without massive amounts of augmentation. The 737 is essentially a hotrod "funny car" at this point in its career.

  24. imanidiot Silver badge

    Re: FAA grounding imminent?

    " (They've had to do this because it's impossible to give the 737 design longer landing gear)"

    Not impossible, just undesirable for the market the 737 operates in and for certification purposes. The advantage of the 737 design (and why it had originally been designed that way) is that the low ground clearance allows easy ground service to almost all relevant parts of the aircraft with easy methods (stepladders and low moveable scaffolds) where competitors require more safety equipment because the worker needs to be higher up. Redesigning the landing gear to put the plane higher off the ground would be easy but would necessitate a huge amount of new equipment be bought by customers all over the world. Customers that might be more likely to go to the competitor in that case.

    On top of that the landing gear is a very important bit of the aircraft and so structurally intertwined with the entire design of the main fuselage and inner wing spar section that a change in main gear leg length would necessitate a substantial redesign of the main wing spar, which would then require certification as an entirely new aircraft design instead of being allowed to be certified as a modification of the original 737 type air worthiness certification.

  25. Anonymous Coward
    Anonymous Coward

    Hidden (secret) functionality elsewhere ... nowhere should be assumed "safe"....

    .....anything "smart" needs to be treated with caution, not just new aircraft. For example:

    - https://www.theregister.co.uk/2018/07/27/jaguar_land_rover_connected_car_privacy/

    A Morris Minor is a pretty safe bet!

  26. Anonymous Coward
    Anonymous Coward

    Re: Hidden (secret) functionality elsewhere ... nowhere should be assumed "safe"....

    Sorry...truncated link.

    https://www.theregister.co.uk/2018/07/27/jaguar_land_rover_connected_car_privacy/

  27. IJD

    In the end, the fault comes back to Boeing. Adding the MCAS system onto the 737MAX was a bodge to get it through qualification after the engine/airframe changes made it unsafe in a stall, but the crash would not have happened if the system had been properly designed against sensor failure (redundancy, voting) and/or the pilots/airlines had been told about MCAS.

    The first one would have probably meant design changes to add the extra AOA sensors and cost Boeing more at manufacture (bad), the second would have meant recertification and retraining of pilots which could have meant them selling fewer aircraft (bad). In the end somebody decided that money mattered more than safety -- probably not deliberately, but this is the kind of sloppy "it'll probably be just fine" thinking which sooner or later kills people.

    Of course if it comes out that the possible problem was pointed out by engineering but stomped on by management (Ford Pinto, anyone?) and this ever comes out, Boeing will be in deep doo-doo...

  28. disk iops

    > In the end somebody decided that money mattered more than safety -- probably not deliberately

    > but this is the kind of sloppy "it'll probably be just fine" thinking which sooner or later kills people.

    Indeed. Companies need to learn how to say NO. When someone came up with the idea of putting ever bigger engines on the thing and it's no longer stable, the idea should have been buried, not sustained with "hey johnny is systems can write us up a software solution". NO god-damn NO! And in it's soul-less pursuit of sales and profits the aircraft co. decided to do something stupid the regulator should have jumped all over them and stopped it dead in its tracks.

    From https://www.pprune.org/tech-log/615709-737max-stab-trim-architecture-2.html I really like the bit about " And then we had STS, which trimmed the stabilizer without pilot input. Huh???? ... But rational was to tell the pilot ( I use pilot to assert whoever was in charge of moving controls), he needed to trim for the new speed/AoA."

    This is meddling on the part of the software by do-gooders. By interfering in the natural operation of the aircraft the pilot has now mentally checked-out. Any pilot paying attention would feel and recognize right quick he need to re-trim without being "helped", just like it had been for decades prior. If not, then by god he's not fit to sit in that seat! Then they layered on yet another nanny function because Boeing in this case made a DELIBERATE choice to say 'Yes' to some retard at the airline or in marketing. The customer, as a rule does NOT know what they are talking about and I'll bet airline execs don't have a clue or care about physics, they just want to cram more seats into the same space and have it fly farther and faster or negligible fuel. At some point an adult needs to stand up and say, "No, we're not doing that, this 50 year old design can not be modified further." The bane of modern technology is that the software programmers always pipe up with "we can write some code to 'fix' that". And as we've found out they did a typical CRAP job of it and didn't bother to follow the RULES that had long since been established.

    Revised engine nacelle and blade design for better thrust and fuel efficiency is again, fine. Decreasing drag with those winglets - brilliant. Upsizing, rotating and shaping the engine so the plane is no longer stable - STOP right there and do not execute! Or go hire yourself out to Lockheed and work on fighter jets.

    We do NOT need to fly at the ragged edge of performance. We do NOT need to carry ever more ridiculous numbers of helpless/hapless souls at one time. We do NOT need razor-edge efficiency in lift or engine performance that *require* ever more complex software solutions to try to bash it back into some flyable shape. We do not need more fancy software to make up for ever less skilled and mentally not-engaged pilots to pretend they know what they're doing. Progress does NOT have an infinite endpoint. Every activity has a cost and human beings are LIMITED. Apparently modern man has decided that all costs can be papered over and with ever increasing amounts of software.

    Same shit in motorcycles - not to jack the thread. First it was ECU and FI. Ok, reasonable and simple improvements that didn't overwhelm the meat or fundamentally change the relationship between rider and machine. Now we have cornering ABS, corner-by-corner brake and throttle maps, launch control, and gd fly-by-wire etc. All of it completely pointless and unnecessary to the task at hand - riding the damn thing from point A to B. You now have world-class racers, the best in the world who can literally get away with being as clumsy as a 2-bit street hack; pinning it and not getting their ass thrown over the moon. Worse, you have said street hacks with but 5% of the talent and skill riding machines that without electronic nannies would have found themselves quickly in the ER or morgue. "electronics this, electronics that" you hear incessantly in interviews. NO, god damn it! If you can't *directly* control the hydraulics of your brakes and regulate the engine with the throttle (again with no electronic, "here I'm detecting some slip, I'll take over") then the whole thing is a farce. We want to see skilled individuals doing their craft, not who has the best software developer and smartest algorithm and sensors all but riding the bike for him.

    Back to planes - "here, hold my heading and altitude for a couple minutes while I root around in my flight bag for the PB&J and a cup of coffee, so long as sensors appear nominal, otherwise warn me and let go" is/was a proper and acceptable degree of improvement. Though properly this should be and has been solved for decades via "yo, co-pilot, you have the stick". When someone else is doing the flying (eg. the computer) the natural tendency is for the human brain to check out.

    How many millions of hours were logged by mere teenagers in WW2 and wars since in transport planes, flying on partial panel, in lousy weather and getting shot at? Yeah, yeah the big bomber losses were atrocious but it wasn't because the pilot didn't know how to fly or the damn computer was second guessing them based on a shot-out sensors.

    Chasing unreasonable efficiency and lower costs is now taking lives and as the chorus for "more AI because it's better than people" is only going to make the failures bigger and costlier and more importantly the pilots increasingly helpless to diagnose and recover within the limited (by physics) window of opportunity. If you're going to have a pilot in command then the plane must fundamentally comport with human limitations, not spew thousands of messages and alerts at him to the point that their ability to cope is overwhelmed - the damn programmers again (I don't mean just the guy writing the code, but the whole foodchain). The computer must by definition be no more than an advisor or really dumb help. Otherwise toss the pilot out on his ass and have the computer run the entire show.

    If the introduction of computers are making a significant improvement in safety, then the conclusion is some combination of:

    1) the damn things are too complicated for humans to fly which by definition means the trajectory of design is WRONG.

    2) the skill level of the pilots it highly uneven and probably insufficient

    The answer isn't more computer, it's smaller, properly designed planes, fewer, simpler planes and more expensive seats. That or just go to drones and be done with it. If PiC screwup kills only 150 people at a time that's better than killing 800 because the gd computer was interfering and worse could NOT be removed or sufficiently sidelined because the airplane requires the computer to even fly at all, and some programmer decided the software (and it's suppodedly non-dodgy failure detection logic) knew better than the supposedly trained people with hands on the yoke.

  29. dmacleo

    in US (under FAR part 121 carriers) AOA sensors req a FCF (Functional Check Flight) after RAR (Remove And Replace) so odd the FCF (if reqd there, often reqs are mirrored) did not catch anything.

  30. JohnnyS777

    The real reason these people died:

    Money.

    This was a case where a significant FLIGHT CONTROL PROBLEM was encountered on the previous flight. Significant enough that the pilot called "pan pan pan". Only a "mayday" is more serious than that. So what happened? Ground staff followed a checklist and swapped/tested some components and they looked fine on the ground. It's clear the pilots and ground staff did NOT know what was the root cause of the problem. But the airline got to make money, so screw due diligence: Over a hundred people were loaded on and took off to die.They had a FLIGHT CONTROL PROBLEM and didn't test fly the aircraft? Why not? That aircraft should have been grounded until it had been flown without passengers by a competent test pilot who knew ALL the aircraft systems including the MCAS.

    Propose a new rule: If a commercial aircraft experiences a flight control problem on a flight and survives, it remains grounded after landing until (1) the flight control problem has been fully understood and remediated, and (2) a test flight (no passengers!) to normal cruising altitude and speed has been completed by a competent test pilot certified by the manufacturer. Any persons who are airline staff, management or engineer personnel who release the aircraft back to service without these 2 steps completed and documented shall lose all their professional accreditations and shall immediately be banned from working in the airline industry. Should the subject aircraft crash with loss of life, those persons should be charged with manslaughter.

  31. steward
    WTF?

    Dave, I'm really the pilot. You're redundant.

    Remy Redert writes:

    "But this wasn't the software's fault. It never is. It was the designer who allowed a single faulty sensor to put the software..."

    Sensors don't "put" software. They only feed data to software. Software functions to interpret and compare data fed to it.

    It was the software's - or, more particularly, the software designers' - fault.

    As for the software designers' responsibility... they're designing software for planes. The article says that "the pilots fought with the MCAS system, pulling the plane's nose up 26 times before finally losing control." It would seem to me that if pilots send a command more than five times contradicting a computer program, that software designers in the aircraft industry should be designing computer programs to just SHUT OFF at that point (with appropriate warnings) instead of saying "Nyah nyah I'm really the pilot, sod off humans!"

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018