back to article Power cut crashes Delta's worldwide flight update systems

A computer outage has caused worldwide delays for thousands of passengers using Delta Airlines. The US carrier tweeted about the issues on Monday morning, blaming delayed and cancelled flights on a “computer outage." Delta, based in Atlanta, Georgia, subsequently blamed the crash on a massive power cut at 2.38am ET (7.38am …

  1. wolfetone Silver badge
    Joke

    *Delta knocks on the door of another airline*

    "Hi, I was just going to check if there was a power cut, but I saw one of your planes fly overhead with it's lights on"

    1. Destroy All Monsters Silver badge
      Headmaster

      It's....

      its

  2. tony2heads
    Boffin

    Leap Seconds

    Next positive leap second on 31st December (see IERS bulletin)

    Will people be ready for that one?

    1. John Sager

      Re: Leap Seconds

      This is why there is so much pressure to kill off leap seconds. The ITU recently kicked that can down the road for another few years, but personally, I don't see why this is still such a problem. We've had leap seconds for decades and computer time protocols have been designed to signal future leap seconds for a *long* time. It does involve the strange concept of a specific minute at the end of June or December should have 61 secs. The specs also allow for 59 secs too but that is unlikely now ever to happen.

      You could write the software to deal with a step, but Google decided to just slow down the computer notion of the time in a controlled fashion for several hours so that after the leap second actually occurs the clocks are exactly back in sync. That is probably much more friendly to existing applications.

      1. Bill M

        Re: Leap Seconds

        Novell implemented this quite some decades ago, way before Google even existed. It is called Synthetic Time

        1. Paul Crawford Silver badge

          Re: @Novell time

          And long before that we had ephemeris time (1952), and then TDT (1976), and then GPS from 1980 using continuous time with a leap-second offset rather like a time-zone.

          As I keep saying IT IS A KNOWN FEATURE and if your code can't handle it gracefully you are incompetent due to either:

          1) Not using tested system libraries to handle time, delays, etc.

          2) Writing or modifying said libraries without knowing what you are doing.

          And most of all NOT TESTING YOUR DAMN CODE! Really, just set up a fake NTP time server and have it generate leap seconds regularly backwards and forwards and see if your code works.

      2. choleric

        Re: Leap Seconds

        Because programmers are so lazy they would prefer to get other people to change the whole calendar than fix their code...

        1. Brewster's Angle Grinder Silver badge

          @choleric

          No, I think we should create a kickstarter to adjust the rotation of the Earth so it is always exactly 86400 seconds +/- a small Gaussian error.

          1. John Sager

            Re: @choleric

            Bravo sir! However it'll take more than a few bob to do that. Perhaps if you could persuade the moneybags that it would cure Gerbil Worming then you'll have far more money than you know what to do with.

            1. Brewster's Angle Grinder Silver badge
              Boffin

              Re: @choleric

              Actually, atmospheric temperatures affect the earth's rotation rate: the warmer the atmosphere, the fewer leap seconds needed. And since the year 2000 there have been many fewer leap seconds than predicted in the 90s; the graph shows a dramatic corner. Nobody is sure why.

              1. Anonymous Coward
                Anonymous Coward

                Re: @choleric

                Uh, wouldn't that be because a warmer atmosphere expands, moving mass away from the centre of the Earth, so conservation of angular momentum demands that the Earth's rate of spin slows to compensate. Same thing with the seas, which although they wouldn't move as much, are considerably denser. I'm surprised if that's sufficient to affect the rotation by as much as several seconds in just a few years, but I can't be bothered to do the maths.

                1. Brewster's Angle Grinder Silver badge

                  @Esme

                  IIRC it's not the expansion of the atmosphere, but a reduction in viscosity that allows atmospheric tides to counterbalance lunar torque. However I'm being disingenuous: it was coming out an ice age and at a time when the moon was a lot closer. (600Myr ago, a 21 hour day.) And while the corner in the delta-T is striking, the earth's mass distribution is changing all the time and it's far more likely that's the cause.

          2. Stoneshop
            Boffin

            Re: @choleric

            create a kickstarter to adjust the rotation of the Earth

            Just a swift kick in the right direction would remedy the problem already.

          3. David Harper 1

            Re: @choleric

            The length of the mean solar day *was* 86400 SI seconds around the middle of the 19th century. That rotation rate was embodied in the astronomical observations that were used to define Universal Time in the late 19th century. When UT was replaced as the best measure of time by Ephemeris Time and then by International Atomic Time in the 20th century, both ET and TAI were defined to have the same length of second as UT. And that's why we have a problem with leap seconds: the SI second reflects the rotation speed of the Earth almost 200 years ago, not today.

            1. Brewster's Angle Grinder Silver badge

              @David Harper

              To the nearest second, the length of day remains 86400 SI seconds and will likely remain so for many kiloyears.

    2. Paul Crawford Silver badge

      Re: Leap Seconds

      "Will people be ready for that one?"

      Well the one that followed the aircraft-bothering incident went with practically no issues at all. Simply because folk had woken up and tested things for the inevitable occurrence of another leap-second.

      In fact the Linux bug mentioned had been created by somebody modifying already-working time related code and not testing the damn thing for this situation. As others have already said, leap seconds and means to deal with them have been with us for decades already so its not new stuff. But every new generation of code monkeys seems to be able to break things...

      1. Mark 85

        Re: Leap Seconds

        But.. but...but... the users are the testers.

      2. Anonymous Coward
        Anonymous Coward

        Re: Leap Seconds

        "In fact the Linux bug mentioned had been created by somebody modifying already-working time related code and not testing the damn thing for this situation."

        Was it actually a Linux bug? I find that hard to believe since given the 10s of millions of installations of linux in backend server systems not to mention embedded systems around the world. I think an OS timing bug it would have caused more problems than just an airlines reservation system going down. Far more likely an application bug which was conveniently blamed on the OS. Also, what application crashes just because of a 1 second difference even if the OS was at fault??

    3. Anonymous Coward
      Anonymous Coward

      Re: Leap Seconds

      We've had another leap second since the one that caused the grief, and this time Amadeus were ready for it - we had no problems.

    4. Locky

      Re: Leap Seconds

      Never mind leap seconds, in 2020 they are going to move Australia

      1. Roland6 Silver badge

        Re: Leap Seconds

        I found the whole idea of moving Australia quite funny - namely that GPS hadn't allowed for continental drift... From other investigations, GPS also has problems determining elevation, which also is subject to change...

        1. Alan Brown Silver badge

          Re: Leap Seconds

          "namely that GPS hadn't allowed for continental drift"

          GPS doesn't NEED to adjust fro continental drift. It's a set of global coordinates.

          The problem is that mapmakers hadn't made allowances.

          then again, 7cm/year is absolutely sprinting in geological terms.

  3. Dabooka

    An hour and a half delay?

    That's not bad for Heathrow, what are they complaining about?

    Or is that just T5 and BA?

  4. GavinC

    "Quantas"

    Who? Oh, you mean Queensland and Northern Territory Aerial Services, a.k.a Qantas.

    1. yoganmahew

      While we're at it, it's Sabre, not "Sebre", and Virgin Australia are on the Sabre system, not the Altea system.

      Oh and Delta bought the code rights to the mainframe they are on (Deltamatic). It is still managed by Travelport (for infrastructure). http://www.travelmarketreport.com/articles/Delta-Reacquires-Res-Operations-Systems-From-Travelport

      It's not really clear what data centre was hit and whether it was the one that houses the mainframe. Delta reservations appeared to be okay.

      Grumpy old mainframer...

      1. Roland6 Silver badge
        Pint

        @Grumpy old mainframer...

        Suspect it wasn't the data centre hosting the mainframe, but the data centre hosting all the layers of screen scrapping and Web2.0 stuff :)

        1. yoganmahew

          @Roland6

          I suspect you're right!

  5. Jon 37

    Single point of failure

    Really? They have their main business-critical system at a single datacenter, without geographical redundancy, so a power cut at that datacenter can bring down the whole thing?

    I would have hoped such an essential system would be spread over 2 or 3 sites, so that losing one site has no impact on operations.

    1. Unicornpiss
      Alert

      Re: Single point of failure

      UPS+backup generator=no problem? Unless of course the whole datacenter is wiped out by some unforeseen catastrophe. However a power failure isn't an unforeseen catastrophe, and something fairly easily prepared for. (with a little $$)

      1. RIBrsiq
        Coat

        Re: Single point of failure

        There should never be an unforeseen catastrophe. There should only ever be foreseen catastrophes that you decided not to protect against after crunching the numbers and judging the investment didn't make sense.

        1. Pascal Monett Silver badge

          Re: the investment didn't make sense

          Reasoning which holds up well until the catastrophe occurs and you see the bill for repairs. More often than not, you will then reevaluate your opinion of what "makes sense" as far as investments are concerned.

          True story : at an important government-level organization I will not name further, there was a kerfluffle when a senior engineer warned, in writing, all the way up the hierarchy, that the currently-at-the-time PC upgrade process was an open invitation to virii and expensive downtime.

          He was hauled into his managers' office for a right chewing out, which, being a senior engineer in a function from which nobody could oust him, he took with a verbal barrage of his own (likely containing many words such as "idiotic", "moronic", "abysmally stupid" etc - don't know, wasn't there, but I damn well hope so). Still, he was told that the investment "wasn't worth it" and that he should "stop making waves".

          As fate would have it, the tsunami hit later that year. An outdated PC piloted by a nincompoop got infected, the infection spread to the servers, and everything was shut down for at least 3 days. That's over 500 people with no more PCs for 24 work hours. You do the math.

          He did the math, and presented the cleanup bill with a scathing "I told you so" that, curiously, all the managers took quite meekly.

          The PC upgrade schedule was changed after that. Unbelievable, ain't it ?

          1. A Non e-mouse Silver badge

            @Pascal Monett - Re: the investment didn't make sense

            I've been involved in several incidents over the years where I've said to the boss: "We need to spend X to replace an aging/failing system." I'd be asked: "Is it currently broken or about to fail?", and when I replied "No", was told to forget about it.

            Later on, the system in question would die and management would complain about people not being able to do their jobs. A blank cheque was usually swiftly provided to replace said faulty system.

          2. onefastskater

            Re: the investment didn't make sense

            Proving once again wisdom comes to us when it can no longer do any good!

        2. Marc 25

          Re: Single point of failure

          Rumsfeld:

          Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.

          1. Khaptain Silver badge

            Re: Single point of failure

            This is what Nicolas Taleb described as the Black Swan Theory, he has his way of seeing things, which is usually quite rational.

            1. Destroy All Monsters Silver badge

              Re: Single point of failure

              "There are always bigger fish to fry!"

      2. Jon 37

        Re: Single point of failure

        > UPS+backup generator=no problem?

        Then your single-points-of-failure include the UPS, the generator, the power distribution units, and the emergency-power-cut-off buttons. UPSs, generators and PDUs all break. Idiots switch things off for maintenance without thinking of the consequences, or press emergency-power-cut-off buttons by mistake.

        You can have multiple generators or UPSs, although you still have the risk of a design flaw taking them all out when they're needed (e.g. http://www.zdnet.com/article/365-main-details-sf-outage-problems/ ).

        There have been plenty of stories about datacenter power outages on The Register, despite the standard UPS+generator.

      3. tom dial Silver badge

        Re: Single point of failure

        I remember about 15 years back having my terminal screen wink out while working on a system a thousand or so miles distant at a US military data center. Not coincidentally, others nearby working on various other systems there had the same experience at the same time, and ensuing discussions with the SA revealed that all power to the main computer building had dropped because a contractor (WHO HAD BEEN TOLD) severed the cables from the oubuilding containing the substation, the redundant UPSs, and the backup generators. Power was restored around 6 hours later.

      4. Alan Brown Silver badge

        Re: Single point of failure

        It's actually possible to RAID your UPS systems and run everything via the UPS rather than introduce a switchbreak (large systems use a flywheel and this has the effect of supplying conditioned power to the site).

        It's also possible to RAID the generators that back the UPSes.

        As someone else has mentioned, the problem is managers looking to get a bonus for cutting costs who end up ripping resiliance out of the systems. I wonder if they'd be so keen if they were made liable for the costs of system failures if it traces back to their cost-cutting.

    2. alain williams Silver badge

      Re: Single point of failure

      Some twat manager was probably given a bonus for ''introducing efficiencies'' which he did by removing some ''underused'' systems and ''consolidating'' several sites to one place....

      1. Destroy All Monsters Silver badge

        Re: Single point of failure

        Or the americans found somebody to liberate into democracy.

      2. Mark 85

        Re: Single point of failure

        Worked a privately owned company where the owner decided that the backup generator hadn't been needed in years, so he had it moved and connected to his house. I'll leave to the reader to visualize what happened 6 months later when the power failed at the plant/office. Two months later, we were getting new dual UPS's and two diesel generators.

      3. ecofeco Silver badge

        Re: Single point of failure

        That's the way I'd bet, Alain.

    3. Daedalus
      Facepalm

      Re: Single point of failure

      They probably used up all the fuel doing weekly backup generator tests.

      1. Herby

        Re: Single point of failure

        The running out of fuel for the backup generator might be because the manager's vehicle was a diesel, and somehow got depleted every time he went to check the generator setup.

    4. Lars Silver badge
      Happy

      Re: Single point of failure

      The problem tends to be that nobody ever has the rights and the guts to actually force a real time test. I suppose it's as well that they don't do real time nuclear rockets tests, 8 inch floppy disks or not. What I have seen more than once is companies who think their backups are fine until they find out it's never tried and total rubbish for years (typically backing up stuff from the wrong place, that was the right place long ago).

      1. Dwarf

        Re: Single point of failure

        @Lars

        Go and look up the Netflix Chaos Monkey and its parent Simian Army which does just that, it generates different failures in various tiers of the of their applications within Amazon AWS to ensure that the applications and infrastructure can fail over in the correct manner when things fail.

        See Simian Army

    5. wayne 8

      Re: Single point of failure

      Delta management is not the sharpest corporate tool.

      1. Anonymous Coward
        Anonymous Coward

        Re: Single point of failure

        But definitely Corporate Tools?

  6. Ol'Peculier
    FAIL

    Worse. Airline. I. Have. Ever. Flown. With.

    LHR to ATL about 20 years ago. Rude staff, shoddy aircraft, and the code-share they had with Virgin coming back was a joke, not knowing which check-in desk to use at JFK was bonkers. At least we flew back with Virgin, which was one of the best flights I've ever had (although that might be because we got bumped...)

    1. W4YBO

      Re:Worse. Airline. I. Have. Ever. Flown. With.

      Don't

      Ever

      Leave

      The

      Airport

      1. James O'Shea

        Re: Re:Worse. Airline. I. Have. Ever. Flown. With.

        I'll see your 'Delta' and raise you BWIA: Britain's Worst Investment Abroad. Also known as Better Wait In Airport. BOAC was Better On A Camel. I remember one flight from Kingston, Jamaica, to Miami on BWIA which took off five hours late and then got later. Something about having to take the long way around Cuba 'cause someone in Trinidad, where BWIA was based, had pissed off the Cubans. Another flight, from Kingston to London, on British Airways not long after it ceased being BOAC, had to divert to Bermuda 'cause someone didn't quite close the cap on one of the wing fuel tanks and we were leaking kerosene over the Atlantic. Yes. Really. That year a lot of British Airways staff went on strike, including engineering support crew, and several aircraft, including the one we flew in on and the one we were supposed to fly back on, had 'cracks' in the wings. Apparently big ones, which could be fixed, except that the guys who were supposed to fix 'em were on strike. So British Airways arranged that everyone on our flight was to fly back to relatively calm and peaceful Jamaica on Air Jamaica... except that Air Jamaica didn't have enough aircraft which could fly the Atlantic Kingston-London, and the one and only one they did have was busy on Air Jamaica's regular service, so the return flight was made with a stop in the truly wonderful vacation spot of Gander, Newfoundland. I haven't flown British Airways since... It'll be 40 years, come next year. I have flown on BWIA. They just were always late, they didn't actively risk your life flying them. They and Air Jamaica have combined into Caribbean Airlines. They're still always late.

    2. Howard Long

      Re: LHR to ATL about 20 years ago

      Interesting, because in those days, Delta didn't fly to or from LHR, it was Gary Gatwick. Just sayin'.

      1. Ol'Peculier

        Re: LHR to ATL about 20 years ago

        Well, time fudges the memory. Must have been out from Gatwick, and back to Heathrow with Virgin.

        1. Anonymous Coward
          Anonymous Coward

          Re: LHR to ATL about 20 years ago

          >>>Well, time fudges the memory. Must have been out from Gatwick, and back to Heathrow with Virgin.

          Hmmm...Back in the day Delta only flew from LHR and Virgin only from LGW.

          Anyway agreed, Delta are not very good, but neither are any other trans-Atlantic airline I have tried any different.

    3. Anonymous Coward
      Anonymous Coward

      >Worse. Airline. I. Have. Ever. Flown. With.

      Funny for me that was the old Northwest flying into Detroit in the middle of a blizzard. Never had to fly Air China or Aeroflot and never would though.

      1. This post has been deleted by its author

        1. Alan Brown Silver badge

          Re: Aeroflot

          Aeroflot has been extremely safe for at least 15 years.

          The problem is there are a pletheora of russian airlines, many having splintered off Aeroflot and still using aircraft painted up with Aeroflot's logo - and it's those ones which are still falling out of the sky with monotonous regularity.

          In other words, be very careful who you fly with.

          In other news, one of the single largest contributors to civil air transport safety was the decision most airlines made in the 1990s to stop recruiting ex-military pilots. Originally this was due to a shortage of them that forced more reliance on flight schools, but it soon became clear that civil-trained pilots were much safer because unlike military pilots they'd assess the situation and take into account the number of people behind them, rather than taking a gung-ho approach and try to land no matter what.

  7. Anonymous Coward
    Anonymous Coward

    Massive sales opportunity today!! Active-active data centers anyone?

    1. Marc 25

      I keep trying to call the CTO but oddly, he's a little busy today...

  8. Anonymous Coward
    Anonymous Coward

    "Power Cut"

    BOFH excuse calendar entry #220.

    Let's wait a day or so to find out what really happened.

  9. Paul Osborne

    Boom!

    I am told by a friend who works in the IT infrastructure team for Delta "that a power board in the main data center went BOOM! Not my doing this time.".

    1. asdf

      Re: Boom!

      And with that BOOM probably went their CIO (or equivalent) after this mess is cleaned up. Or maybe he really wants to spend more time with his family like the press release will say.

    2. Dwarf
      Joke

      Re: Boom!

      Doesn't that make it Delta-I then ?

      As in Change in Current (or Amps for our American friends)?

  10. Herby
    Joke

    Maybe the reason was....

    The power connections for the three phase power was "WYE", and this was "Delta".

    explanation

    1. Cynic_999

      Re: Maybe the reason was....

      @Herby - the article appears to be inaccurate (and US-centric). Quote, "Most homes are wired with single-phase that uses one ac voltage delivered over two hot wires and one neutral wire. The voltage across the two hot wires measures 240VAC (for your oven or dryer) and across any hot to neutral measures 120VAC (for everything else)."

      That describes a home fed by a 2-phase supply, not a single phase supply. Single phase has one live and one neutral and only one voltage.

  11. Anonymous Coward
    Trollface

    Confused?

    UPS? Don't they do deliveries?

  12. Anonymous Coward
    Anonymous Coward

    Delta? You must be joking

    My company has its regional HQ in Atlanta. We are not allowed to fly Delta if there is another choice even if it is more expensive. before the ban was put in place I wanted to fly to Cozimel in Mexico. There was a route via ATL but Delta would let us book a 5 hour connection. WTF. If Lufthansa can do 35 mins inc hold baggage in Munich why?

    Mind you American and United aren't much better. Southwest and Alaskan seem to be the top of a poor pile.

    1. David 132 Silver badge

      Re: Delta? You must be joking

      Not sure why you got downvoted (I upvoted you to compensate) - I agree with your assessment. I used to fly United almost exclusively, but their service and attitude to customers has really deteriorated in the last 5 years or so (probably coinciding with the Jeff Smisek helmsmanship, now that I think of it...). Now I'm just using up my accumulated United miles & not bothering to use them for new paid travel. I had to fly United out of Manchester a couple of weeks ago - there's an unholy combination for you.

      That said, I think Southwest are pretty good - they're cheap, but at least they're also cheerful. But I know opinion is sharply divided on them.

    2. a_yank_lurker

      Re: Delta? You must be joking

      I refuse to fly Don't Ever Leave The Airport after numerous bungles on their part (a very long list). I would rather take a bus or train even with the crappy US passenger rail.

  13. David Roberts
    Mushroom

    One of the only airlines?

    FFS it is either the only airline or it is one of the (very) few airlines.

    Am I very unique in being annoyed by this?

    1. yoganmahew

      Re: One of the only airlines?

      Airlines with their own system (including only old-fashioned mainframe-type ones):

      Turkish

      ANZ

      JAL

      Air India (maybe?)

      Iberia, Lufthansa, AF, KLM, Finnair all distribute through Amadeus, but had back-ends at one time. Not sure how many of them still do.

      No, you're not thing only one Uppsy Daisy ;-)

  14. This post has been deleted by its author

  15. onefastskater

    http://www.ajc.com/news/business/delta-outage-grounds-flights-causing-massive-cance/nsB5W/#signin

    Georgia Power spokesman John Kraft said a failure overnight of switch gear equipment caused the outage. He said other Georgia Power customers were not affected because it was an issue with Delta equipment, and said Georgia Power crews were on site working with Delta to repair the equipment.

    1. Version 1.0 Silver badge

      Someone else's problem

      As in, we just dropped our 26kv line across your 4kv feed for 30 seconds - looks link our transformer handled the overload OK ... so it must be a problem at your end.

  16. patrickstar

    I wonder how many of these datacenter power disasters would have been avoided if everyone had just standardized on -48VDC for servers, like in the telco world?

    It's orders of magnitude simpler to provide backup (batteries! No DC/AC converters needed) and failover (no phase sync needed, among other things) for.

    Hell, apart from enormous DC UPS systems, the only time you see those insanely huge DC/AC converters is in HVDC systems... For a reason.

    1. Anonymous Coward
      Anonymous Coward

      Because...

      Because accountants get all antsy when you tell them that a DC powered server is three times the cost of an AC powered one. It was OK in the days of monopoly telcos because the server vendors could hide behind NEBS but it's a brave engineer who'll suggest a few NEBS boxes in a high-class data center vs a boatload of dells in a bit barn.

    2. Solmyr ibn Wali Barad

      DC has cans of worms too. Higher currents for one. And there's a likelihood of self-sustaining arcs between contacts. High-current DC contactors are a marvel of engineering.

  17. MotionCompensation

    Cloud based stuff kept going

    Their cloud based technology kept flying though, no crashes there.

    1. ecofeco Silver badge

      Re: Cloud based stuff kept going

      I see what you did. Have an up vote.

  18. Anonymous Coward
    Anonymous Coward

    But for the grace of god...

    I'll raise a pint to all the worker bees who are spending their days fixing it while a management storm of fecal matter rages around them.

    If you are a CIO, do you attempt to build your own data center operations, knowing that your limited experience is probably going to miss something, or do you outsource to Amazon, knowing that you are going to be held to ransom if you ever try to leave?

    Sometimes I miss the old days. A VAX cluster would have survived Delta landing a plane on it.

    1. onefastskater

      Re: But for the grace of god...

      Surely you don't mean clusters last stand!

  19. Never10_use_Puppylinux

    IBM offers mainframe in the cloud today

    We have some 360s, 370s, & 390s, available for demo in the cloud. (our backend warehouse storage room Just UPS (united parcel service) overnite your card deck and in 24 hours we will mail you back 2 day USPS your printout and card deck. /sarc off.

    Does not that 390 emulate / run Linux? Is there a IBM 360 emulator for Linux available.

    Interesting what the CIO will do to address this power problem in the next few days.

    1. tom dial Silver badge

      Re: IBM offers mainframe in the cloud today

      zSeries run Linux just fine (native), and while power is available will run with substantial component failures, although with some performance degradation, and many of the failures can be corrected without a reboot.

      They also will run briskly at 100% CPU utilisation as long as the paging rate is kept reasonable.

    2. Anonymous Coward
      Anonymous Coward

      Re: IBM offers mainframe in the cloud today

      Tell them, not us... you realize Delta's an IBM customer and the mainframes are managed by IBM today?

  20. Anonymous Coward
    Joke

    Linux kernel and Lua modules blamed for Delta reservations crash

    See, we managed to not once mention MICROS~1 in relation to computer malware or crashes.

  21. Gary Border

    I'd suggest the techies at Delta look carefully at the possibility their UPSes were shutdown by malware. This was one of the mechanisms used to create the "Great Ukrainian Electric Grid Outage of Dec 24, 2015."

  22. Version 1.0 Silver badge

    How's offshoring the tech staff working?

    It's starting to sound like the Delta data center went down and someone had to fly over from India to reset the breakers. The problem in getting everything back up is due to his flight being delayed.

  23. Melanie Winiger

    Mainframe OK.

    The z/TPF mainframe stayed up - it was unaffected.

    All the decentralised systems (Flight Info displays, kiosks, online check-in etc.) failed.

    Source: TPFers Group on Facebook.

    Note:

    Not scoring points and would never gloat over the failings/disasters of others - all too aware that the next operational failure could be in my own back garden.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like