back to article Fondleslab deaths grounded ALL of American Airlines' 737s

Dozens of American Airlines flights were delayed this morning when pilots' iPads abruptly crashed, leaving the entire AA fleet without access to vital flight plans and, resultantly, grounded. American Airlines uses specific software on its pilots' iPads to distribute flight plans and relevant information to the crew. The …

Page:

  1. Neil Barnes Silver badge
    FAIL

    Now an intelligent design

    Would have used pads from two different makers, different OSs, and with the software written by two independent companies. Otherwise they're a single point of failure, as demonstrated.

    1. tony72

      Re: Now an intelligent design

      "Would have used pads from two different makers, different OSs, and with the software written by two independent companies. Otherwise they're a single point of failure, as demonstrated."

      Hmm, that would more than double the cost of getting the software written and of maintaining the two systems, and you'd still have got half your flights grounded today. Unless you're suggesting that every pilot should have permanent acces to two tablets, rather than deploying a mix, in which case even more cost. Hard to justify for a non-safety-critical system. Maybe better to have a web-based fall-back, so in a pinch pilots could use their own smartphones or tablets to sign in and grab the data?

      1. Paul Crawford Silver badge

        Re: Now an intelligent design

        They may not be "safety critical" but they sure are business-critical as shown today.

        Also I doubt the cost of having software for two OS is anywhere near double to cost of one, but we will have to wait and see if it was an OS problem killing the connections or an app problem. Either way, it is a timely reminder of just how much companies depend on IT systems working.

        1. tony72

          Re: Now an intelligent design

          The person I replied to said specifically "[...] software written by two independent companies", so if it's bespoke software, you'd pay twice for that; bespoke software is typically more a development cost than anything else. Granted, if it's off-the-shelf, then that might not be the case.

          1. R 11

            Re: Now an intelligent design

            Nonsense. All the work before commissioning a vendor to write the software is portable to the second vendor.

            Also, the first is likely to have asked most the questions you didn't think about, making it easier for the second.

            If you have two suppliers work simultaneously you may get questions from both that make both products better.

            As for the comment above about buying twice as much hardware, the article mentions the pilot and copilot's iPads both going blank. So you already have two devices, this is just about enduring there's redundancy beyond the physical device.

        2. AdamT

          Re: Now an intelligent design

          well they are not "safety critical" in the sense that the plane falls out of the sky if the iPads fail but I am curious as to what the procedure is if you lose access to your flight plan (and all the "reference manuals") mid flight ...

      2. Kristian Walsh Silver badge

        Re: Now an intelligent design

        The software is already written: Delta Airlines gives its pilots Surface tablets running the same software from Jeppesen. Airlines would one hardware supplier because they can get better discounts by buying 10,000+ tablets from one maker than by splitting it into 4000 and 7000...

        1. BillG
          Holmes

          Re: Now an intelligent design

          Replacing the bulky 35lbs flight bags also allowed airlines to save fuel.

          I don't see how 35lbs results in a significant fuel savings. Rather, I think a smart pilot would want to keep the 35lbs paper flight manuals on-board anyway as a backup. As a passenger, that's O.K by me.

          1. Anonymous Coward
            Anonymous Coward

            Re: Now an intelligent design

            >>I don't see how 35lbs results in a significant fuel savings. Rather, I think a smart pilot would want to keep the 35lbs paper flight manuals on-board anyway as a backup. As a passenger, that's O.K by me.

            You don't see it because you're looking at the problem the wrong way. You're looking at 35 lbs. as a fraction of the plane's total weight, in which case it's a small fraction of 1%, which seems fairly insignificant.

            Instead, imagine how much energy it must take to lift a 35 lb dumbbell several miles into the air and then move it hundreds of miles per hour to a destination that's hundreds of miles away. That's a non-trivial amount of energy/fuel. Multiply that by hundreds or thousands of flights per day and you're quickly talking about a huge amount of fuel and money.

            http://www.nytimes.com/2008/06/11/business/11air.html

            Consider that reducing drink tray weight by 17 lbs. is saving American 1.9M gallons of fuel annually. So cutting the flight paperwork by 34 lbs. is probably saving hundreds of thousands of gallons.

      3. Voland's right hand Silver badge

        Re: Now an intelligent design

        Hmm, that would more than double the cost of getting the software written and of maintaining the two systems

        The cost of a single outage is in the millions. I doubt that the cost of maintaining different makes and versions for the Captain and co-pilot is anywhere near that.

        1. BillG
          Alert

          Re: Now an intelligent design

          Would have used pads from two different makers, different OSs, and with the software written by two independent companies. Otherwise they're a single point of failure, as demonstrated.

          Smart. As I recall, the U.S. Space Shuttle has/had three independent & redundant onboard computers with three different hardware configurations and software written with three different compiler manufacturers.

      4. Sir Sham Cad

        Re: every pilot should have permanent acces to two tablets

        No, he's suggesting that, instead of Pilot + Co pilot having 1x Tablet-X each, the Pilot has 1x tablet-X and Co pilot has 1x Tablet-Y running the same software but on different platforms for system resilience.

        Yes there's an initial cost of porting your app to a different platform but given the business impact of not getting your birds in the air at all for want of a second device type is huge.

        In that case scenario all the flights would have taken off as each of the 737s would have at least one functioning flight plan tablet.

        Proper resilience and business continuity is only expensive until you don't have it.

        1. Neil Barnes Silver badge

          Re: every pilot should have permanent acces to two tablets

          Quite. Two people driving, two different systems.

          1. Anonymous Coward
            Anonymous Coward

            Re: every pilot should have permanent acces to two tablets

            "Replacing the bulky 35lbs flight bags also allowed airlines to save fuel."

            That made me laugh. What's that, 16kg? I can't see that saving a lot of fuel on a 747... or many other planes.

            Besides which any savings were more than wiped out by today's problems.

            Seems they'd have been better off sticking with the "old" solution which worked.

            1. Anonymous Coward
              Anonymous Coward

              Re: every pilot should have permanent acces to two tablets

              Seems they'd have been better off sticking with the "old" solution which worked

              Problem with the old solution is cross referencing every minor update and error addendum every time you looked at the book, that and making sure 50000 pilots placed worldwide got the bit of paper with the "don't push the red button mid flight" update.

              if Boeing issued an addendum for all aircraft that's a few million flight crew that needed be aware of it.

            2. dave 93

              Re: every pilot should have permanent acces to two tablets

              American Airlines spokesperson on the BBC said $1,000,000 saving a year on fuel from the new system. They do have about 1,000 planes in their fleet though.

          2. Eddy Ito
            Facepalm

            Re: every pilot should have permanent acces to two tablets

            Since most of these birds have glass cockpits is there a reason why the data couldn't be uploaded directly to the planes? Perhaps something complicated like a USB stick would work.

            1. Yag

              Re: every pilot should have permanent acces to two tablets

              Since most of these birds have glass cockpits is there a reason why the data couldn't be uploaded directly to the planes? Perhaps something complicated like a USB stick would work.

              In order to include additionnal functionalities to a glass cockpit, you have to ensure that this software won't have an adverse effect on the overall system.

              This mean developping the software following the good ol' DO178b (or the c if you're up to date), and would probably increase the costs of the developpement by a 4-5 factor, if not more...

    2. David Webb

      Re: Now an intelligent design

      From what I recall, the pilot and co-pilot are not allowed to eat the same food as each other, just in case one set of foods a bit off and makes one of them ill, the other pilot who ate different food won't be ill.

      So there is history in the airline industry having systems in place in case something happens to one pilot, this should have been thought of the tablets, what happens if both tablets goes down at the same time, what's the worst case scenario, how can we prevent that.

      1. Tom 38
        Joke

        We had a choice of steak or fish

        From what I recall, the pilot and co-pilot are not allowed to eat the same food as each other, just in case one set of foods a bit off and makes one of them ill, the other pilot who ate different food won't be ill.

        I had the lasagne.

        1. Mark 85

          Re: We had a choice of steak or fish

          Good choice. Don't eat the fish.....

        2. Diamandi Lucas

          Re: We had a choice of steak or fish

          Someone downvoted your post! Surely, they can't be serious?

          1. Tom 38

            Re: We had a choice of steak or fish

            Surely, they can't be serious?

            It's a rorschach test to determine the humourless amongst us. And stop calling me Shirley.

    3. Anonymous Coward
      Anonymous Coward

      Re: Now an intelligent design

      LOL, should have used the Surface Pro like most other airlines that use tablets...Then at least you can control the config totally.

      Failing that, I suspect paper and pencils are still fully operational!

      1. Stoneshop
        Facepalm

        Re: Now an intelligent design

        Given that it affected only part of AA's fleet, it's apparently a problem triggered by the app choking on data specific to that type of plane in combination with the date. So please explain how it could have been caused by the choice of hardware.

      2. Mage Silver badge
        Alert

        Re: Now an intelligent design

        No, NO ONE should be using iPad, Surface Pro, Kindle Fire or Domestic Android tablet for this application.

        OS too fragile

        HW too fragile.

        At the very worse something like a Panasonic ToughPad, with Linux rather than consumer Android or "Windows" and locked down to prevent users adding applications.

        Edit:

        While QNX was great on embedded systems and disk controllers, I'd not use a Blackberry either.

        1. Cliff

          Re: Now an intelligent design

          This happened on take-off and that's a great time to fail. If this happened mid-flight, it's kinda fucking terrifying if you don't have access to an airport chart for the destination.

          Let's say there's a comms problem driving the failure, radios and landing instrument tracking - without a paper backup how do you know the runway elevation and length? How about the frequency of the local tower? Loads of good reasons to have this (redundantly) on paper.

      3. Anonymous Coward
        Anonymous Coward

        Re: Surface Pro

        "should have used the Surface Pro like most other airlines that use tablets.."

        Got a source for that?

        When Delta announced they were going with Surface rather than iPad, it was so unusual that there was a joint MS/Delta press release [1], and a flurry of articles e.g. [2] saying that the drivers didn't want Surface, they wanted what they were used to, ie iPads.

        Shades of MS desperation and a CEO-level deal, methinks.

        [1] http://news.microsoft.com/2013/09/30/delta-to-equip-11000-pilots-with-microsoft-surface-2-electronic-flight-bags/

        "Delta to equip 11,000 pilots with Microsoft Surface 2 electronic flight bags

        Posted September 30, 2013 By barrettevans

        ATLANTA and REDMOND, Wash. — Sept. 30, 2013 — Delta Air Lines (NYSE:DAL) is equipping its 11,000 pilots with electronic flight bags using the Microsoft Surface 2 tablet. Device rollout to pilots flying the Boeing 757 and Boeing 767 fleets will start later this year and all Delta cockpits are projected to be paperless by the end of 2014.

        (continues)"

        [2] http://appleinsider.com/articles/13/09/30/delta-pilots-fought-against-deal-to-replace-ipad-flight-bags-with-microsoft-surface

        1. Anonymous Coward
          Anonymous Coward

          Re: Surface Pro

          "Got a source for that?"

          Pretty much every single airline that went for tablets since the FAA certified Windows based solutions last year seems to have gone for a Windows based option:

          http://www.lovemysurface.net/surface-pro-3-airlines-equip-their-pilots/

          South West Airlines, Lufthansa, Austria for example. Also Air Asia and Delta pilots also use Surface tablets. British Airways City Flyer also went Windows and supplied their pilots with the Panasonic Toughpad

    4. James Haley 2

      Re: Now an intelligent design

      "Would have used pads from two different makers..."

      That would tricky and expensive, as others have noted.

      The simple solution is to publish two versions of the same app, a "latest" version and a "stable" version that's always known to work.

      1. Laura Kerr

        Yer 'avin' a tin, intcha?

        'Latest' versions have bugs; it's not until they've had a damn good kicking in service that they can be deemed stable. If I thought the people flying my plane were using a version that wasn't stable, I wouldn't even get on board.

        Nope, the right answer is platform redundancy, as others have noted.

    5. x 7

      Re: Now an intelligent design

      The only realistic subsidiary to Jeppeson is a Lufthansa subsidiary and as far as I'm aware neither support Android or Windows. You're stuck with the iPads. And remember this stuff has to integrate with the airlines back-end flight planning software, so pointless duplication would be a major fundamental cost

      1. Anonymous Coward
        Anonymous Coward

        Re: Now an intelligent design

        "The only realistic subsidiary to Jeppeson is a Lufthansa subsidiary and as far as I'm aware neither support Android or Windows"

        Jeppeson already supports Windows - and Lufthansa have gone for Windows on the Surface Pro for their pilots...

  2. RIBrsiq
    Trollface

    "Once is happenstance, twice is coincidence and three times is enemy action".

    ...So... what is it when the entire fleet experiences the same event?

    But really: is it any surprise when Apple's patented Magical Thinking™ security does not work?

    Disclaimer: I am assuming that American Airlines have more than 3 airplanes. It seems a fair assumption, but I have no data to back it up.

    1. theblackhand

      Malice?

      From other reports, the issue appears to have been a problem with the application (not the hardware or OS) hence multiple affected aircraft.

      One thing I am unclear on is the need to return to the terminal for Wifi access - I would hazard a guess that they had backup hardware but needed to re-sync the information but that is speculation rather than inside knowledge.

      Hopefully there will be a public report for what went wrong.

      1. silent_count

        Backup hardware

        This could have been prevented with 80's era tech - a 286 hooked up to a dot matrix printer* and a fax to send out flight plans as required. As in, "put down the shiny toy and look at this white stuff with printing on it. It's called paper. It doesn't require batteries or wifi. It just works™."

        * See BOFH 2013, #5. As an added bonus, when they work out who is responsible for this stuff up, they too can be fed to the dot matrix printer.

        1. Preston Munchensonton
          Coat

          Re: Backup hardware

          "As an added bonus, when they work out who is responsible for this stuff up, they too can be fed to the dot matrix printer."

          Now, now. That would be far too messy. This is precisely the purpose for the tape vault.

        2. Mark 85

          Re: Backup hardware

          Until recently, commercial aircraft did have printer on board. Not sure if a computer was involved or some version of fax, though. I suspect it was two way version of fax since the crew could make queries to the company and get back a hard copy printout.

      2. Ralph B

        Re: Malice?

        Could it have been a passenger running a mobile hotspot with No iOS Zone? The symptoms are eerily similar.

        1. Stoneshop
          FAIL

          Re: Malice?

          Could it have been a passenger running a mobile hotspot with No iOS Zone?

          “The pilot came on and said that his first mate’s iPad powered down unexpectedly, and his had too, and that the entire 737 fleet on American had experienced the same behavior,”

          That's one hell of a No iOS Zone created if it affected planes all over the US, or there's some magical synchronicity happening between people running that software.

    2. Mark 85

      You missed "BOFH"... one that was irritated at his boss enough to send the "shutdown iPad" command.

      For the whole fleet of 737's to be shutdown sounds like a single point of failure or malice.

  3. casaloco

    Apple...

    ...what's the betting Apple pulled the app from the iStore as unapproved, and deleted it from the iPads?

    1. Test Man

      Re: Apple...

      We know what you mean but technically speaking it wouldn't have been possible as 1. the apps wouldn't have been delivered from the App Store and 2. they didn't have wifi onboard so any deliberate "pulling" of apps would never have been possible.

      1. The_H

        Re: Apple...

        Maybe it was the "time limited" trial version. Which expired.

        Or maybe the screens went dark in silent tribute to Orville the Duck, who will now never fly?

  4. Vladimir Plouzhnikov

    Stoopid flyboys

    They are holding 'em 'rong! Duh!

    1. smudge
      Facepalm

      Re: Stoopid flyboys

      Knowing American Airlines, as I did slightly many years ago, they could well be doing that.

      About 11-12 years ago, my work took me to Dallas TX, and my customer had a deal with American Airlines so I had to fly with them. Their in-flght corporate videos - eg "Welcome to American Airlines", "Finding your way around Dallas Airport", etc - commenced with an animated rotating Earth.

      But the Earth was rotating in the wrong direction...

      1. Anonymous Coward
        Anonymous Coward

        Re: Stoopid flyboys

        But shouldn't the Earth rotate in the opposite direction in the video because you're facing the screen, not looking at the ground, and you're flying in a different direction to how you arrived and because of the timezone differences, the plane, clouds and sunspots and stuff?

        - AA Marketing and Media Team.

  5. AceRimmer

    It wasn't a fault

    It was the software optimizing itself for maximum fuel savings

  6. graeme leggett Silver badge

    costs

    35 lbs of weight (and hence 'x' amount of fuel) saved every flight against the costs of all aircraft being grounded once - an interesting one for the accountants to evaluate

    1. Anonymous Coward
      Anonymous Coward

      Re: costs per lb

      If saving weight is so critical, why aren't tickets priced based on passenger weight?

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like