back to article Router cockup grounds United Airlines flights WORLDWIDE

A single dodgy router managed to crash the entire United Airlines computer network, leading to the cancellation of flights across the globe. See that long line, that's @united in Boise. see any other lines at other airlines? Nope, don't fly #UnitedAirlines pic.twitter.com/PnWRcGtPJM — Jeralyn Novak (@AgCommunicator) July 8, …

  1. NP-Hardass
    FAIL

    Redundancy?

    'Nuff said.

    1. asdf Silver badge

      Re: Redundancy?

      Redundancy (without the payment) of some of the IT management sounds like the first step actually.

      1. Mark 85 Silver badge

        Re: Redundancy?

        The staff might be at fault, but the blame probably goes up the food chain. Management has the responsibility to ensure that backups systems are there even if it eats into profits.

    2. Anonymous Coward
      Anonymous Coward

      Re: Redundancy?

      Based on inside knowledge, the issue wasn't a SPOF router, but a configuration change to a redundant pair. Also, note that the equipment in question was part of the holdover from the Continental takeover, so it was managed in-house rather than by one of their suppliers.

  2. Derpity
    Facepalm

    Sensational

    A couple of the more "creative" US news sites are linking this with the problems at the NYSE and Wall Street Journal. Obviously the only answer is they all shared systems and terrorists are attacking America! The source country is about to get a home delivery of freedom.

    1. asdf Silver badge

      Re: Sensational

      The only real news story that matters right now (other than waiting for those lovely Pluto pics) is largely being glossed over. China's stock market is losing more money some days than Greece's entire economy for a year. Their volatility curve looks like bitcoins and that ain't good.

      1. Spaceman Spiff

        Re: Sensational

        I am wondering which Wall Street banks are behind this... Heaven help them if the Chinese government finds out!

        1. I ain't Spartacus Gold badge

          Re: Sensational

          The Chinese stock market is down to China not having read their history of the Wall Street Crash in 1929. Apparently 90% of their stocks are owned by individual investors - and loads of them have borrowed money to buy the stocks, which is why the Shanghai index has doubled in price over the last 18 months. So of course when the price drops, you have to sell before the value of the shares you own goes below the value of the debt you took out to pay for them. When half the other people on the market are in the same boat, things go wrong very quickly.

          Margin trading is scary. Don't do it kids!

          One big difference is that China's government can do whatever the hell they like. So they've suspended trading in half the market, they can make the banks write-off the loans, or just abolish the stock market.

          The Chinese banking sector is in some ways quite well regulated. They have reasonably high interest rates and very high capital requirements. A lot of their bad loans are also to local government, so the national government will probably have to bail them out. On the other hand the shadow banking industry is huge, and barely regulated. Possibly they were too cautious with the banks, so people went elsewhere? And that mess will be painful to clean up.

  3. ma1010 Silver badge
    Windows

    Days of glory past

    United used to be a great airline which I enjoyed traveling on - about 30 years ago. Nowadays, not so much. If you want to see more of the horror stories, go to untied.com (no spelling error).

    1. Spaceman Spiff

      Re: Days of glory past

      Yeah. Agree. I liked to fly on Untied Airlines until sometime about 15 years ago. Now, I much prefer Southwest.

  4. Anonymous Coward
    Anonymous Coward

    No surprise here

    I work in the industry (hence AC), and have seen a number of routers borked by power loss because the morons in charge of them don't always remember to save running config to startup...!

    Additionally, at many airports, many airlines only run in a single circuit - they just don't want to pay the fees for redundant circuits...

    Granted this sounds like a much bigger SNAFU, but I'd be surprised if anyone loses their job unless they've been truly negligent. Chances are a techie suggested a resilient solution, and a beancounter vetoed it.

    1. Anonymous Coward
      Anonymous Coward

      Re: No surprise here

      It's probably nothing as simple as a basic lack of redundancy. In the words of the immortal devops borat

      "To make error is human. To propagate error to all server in automatic way is #devops."

  5. Will Godfrey Silver badge
    Happy

    So..

    A router closed all routes.

    1. Captain DaFt

      Re: So..

      In flight announcement:

      "Ladies and Gentlemen; We apologise, but due to technical problems, this flight has been rerouted to... /dev/null? WTF?"

  6. Spaceman Spiff

    All senior networking staff at United (or Untied Airlines as I like to call it) should be shown the door. Using advanced tools (that I developed) at Nokia, we were able to predict when networking or other server gear was about to fail - it is called predictive analytics. We captured 10 billion data points per day from 5000 servers world-wide, stuffed that data into a Hadoop cluster in the Amazon cloud, and then would look for out-of-band behavior, such as failing routers, switches, etc. When Microsoft took our division over (and fired/laid-off about 13000 of us) I was about to implement a series of Kalman filters to refine this process so we could be more precise in our predictions.

    Since United doesn't have to support 100-200 million active users, this should NOT be difficult for them! Shame!

  7. Sgt_Oddball Silver badge

    Now now...

    How many ADSL filters can you find on short notice?

    Or if you've unplugged the wrong network cable in a rats nest comms cabinet?

    Either that or a guitar finally had its revenge....

  8. Ian Tunnacliffe

    Anyone for some facts?

    It was not the "entire computer system" that was out. It was the Weight and Balance application that in US carriers is typically part of the Operations/Dispatch suite. This is different to airlines in other parts of the world that typically do W&B in the Departure Control System (DCS) that also handles seat allocation, passenger check-in, baggage acceptance, boarding control etc etc.

    Weight and Balance is a critical application that produces the load sheet. Valid load sheets are a legal (as well as common sense) requirement and without one a flight may not operate. It is possible to calculate load sheets manually but this has not been done as a routine process for many years and probably the skills required to do it are in very short supply. Hence the FAA ground stop.

    United's major IT systems are outsourced to H-P Enterprise Services. This is a legacy from when EDS (acquired by H-P) ran IT for Continental Airlines (which acquired United along with its name five years ago). I suspect that there may be some tense meetings between United and H-P execs in the days to come.

    1. I ain't Spartacus Gold badge
      Happy

      Can't they just do it like Das Boot? When you want to take off you make all the fattest passengers run to the back of the aircraft, then when you get to crusing altitude you make them lie in a big heap in the aisle over the wings.

      Obviously it's bad for passenger morale when the pilot comes on the PA and screams, "ALLLLAAAARRRRMMMMMM!!!!!!!!". But it's a small price to pay for getting off on time...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019