back to article UK flights CRIPPLED by system outage that shut ALL London airspace

All London airspace was closed to incoming and departing traffic for just under an hour on Friday afternoon due to a computer outage at the National Air Traffic Services – Blighty's air traffic control authority. According to the European Organisation for the Safety of Air Navigation, a machine failure resulted in all airspace …

Page:

  1. flayman Bronze badge

    No uninterruptible power supply?

    1. Zog_but_not_the_first

      Quite.

      Hmmmm... Worrying.

      1. smudge

        http://www.allen-diesels.com/case-studies/civil-aviation-authority.php

        Two 16kV, 5MW diesel engines, it seems.

        Note that this is not a comment on Allen Diesels, nor am I suggesting that they are in any way at fault. I merely came across that page when searching around t'interweb to see what sort of backup power supply Swanwick has.

        1. Peter Mount

          I can't remember the name of the film off hand (it's the one about Parkes radio telescope during the Apollo 11 landing), but there they had a power failure & the Diesel backup failed due to not bleeding the fuel lines.

          Not saying thats the problem, but it could still happen.

          1. smudge

            There was a data centre (in Denmark or the Netherlands, I think), where the power supply failed.

            The diesel generator came up as it should, and powered everything for a couple of days, by which time it needed refuelling.

            The diesel tanker duly arrived - and crashed into the outbuilding, completely trashing the generator inside.

            1. Xpositor

              Quite a few years ago when I worked for a bank that was subsequently taken over by a Spanish bank, we had a UPS test one weekend. Everything worked flawlessly. Unfortunately, mid-way through the day on the Monday, all power in the building failed, knocking out the mainframes. Some herbert had forgotten to switch back over to mains power, and the generator duly ran out of diesel.

              1. Phil O'Sophical Silver badge

                Adding to the diesel stories, I know of a site that had diesel backup which was regularly tested. Once a month, for many years, the generator was started, fuel levels verified, etc., Unfortunately the diesel was run for about 2 minutes each time, which had the same effect as lots of short journeys in a car. When the power failed years later the generator started and took over the load flawlessly...until it warmed up. Once hot the thoroughly coked-up engine misfired, wouldn't keep speed, and spluttered to a halt. It took a head-off engine rebuild to get it back online.

          2. Jim Hague

            'The Dish'.

            Great film.

          3. HelpfulJohn

            "The Dish".

            A brilliant film. Very moving in places and full of understated Ozz/UK style humour. The Oz politicians are wonderfully inept, witless and charming.

            The American from NASA is so USAlien he's just barely credible but he is a perfect foil for the more relaxed Ozlanders, also a great straight man.

            http://en.wikipedia.org/wiki/The_Dish

            And the radio telescope's dish is still in the middle of a sheep meadow.

          4. Anonymous Coward
            Anonymous Coward

            The Dish? (With Sam Neill?)

          5. nematoad

            "I can't remember the name of the film off hand (it's the one about Parkes radio telescope during the Apollo 11 landing),"

            It's called "The Dish"

            I loved it when the band played what they thought was the US national anthem and it turned out to be the theme to Hawaii 5-O.

            Classic.

        2. Anonymous Coward
          Anonymous Coward

          @smudge

          Swanwick puff piece

          Reading the page you linked to, it is the most unfortunate bit of trumpet blowing imaginable, since it is telling us that "nothing can go wrong".

          My own UPS horror story? Power cuts out briefly, battery UPS comes on line, there is a thump and a roar as the Diesel generators cut in. Then after a minute white smoke is noticed emerging from the stack by the Diesel shed, the lights flicker and there is an enormous bang. Lights go out.

          The maintenance department, the last time the standby generators were serviced, cleaned out the old oil and omitted to replace it. Subsequent inspections had apparently failed to notice that absence of oil level.

          Very soon after, the post of facilities manager was vacant.

          1. 142

            @ arnaut

            A similar thing happened, at a massive scale, at the Dublin Amazon AWS data centre a few years back http://www.siliconrepublic.com/enterprise/item/23084-mystery-surrounds-outage-at

            They blamed a lightning strike initially but it appears to have been poorly configured failover gear.

          2. Dave Henderson 1

            Re: @smudge

            Bit of a corner-cut stand-by genset if it didn't have the basics of a low oil / low pressure cut-out before it went clunk. Even 30 years ago a decent panel wouldn't have allowed it to run up and hold without pressure being present. Then again, I used to see the most stupid cost-cutting on vital things that might only be needed once in a blue moon - then the kind of thing above would happen.

    2. Anonymous Coward
      Anonymous Coward

      It would appear to be worse...

      One UPS can fail, as can the distribution, but you would have though something as important would have redundancy. I guess we'll have to assume that failed as well.

      Good job they're not supporting a nuclear power plant :-(

      1. Anonymous Coward
        Anonymous Coward

        Re: It would appear to be worse...

        Redundancy means nothing - I work in engineering, and no matter how you plan, something always goes tits-up from little incidents.

        Have a read here:

        Normal Accidents*

        Logging in on my Slack notebook today was:

        Linicks Law: If Murphy's law can fail, it will.

        *OK wiki crap, but it reflects the paper I read on redundant redundancy once.

      2. Bartholomew

        Re: It would appear to be worse...

        I've seen where they have slowly added more load, over years, than the system was designed to handle. The zero crossover switches, switch flawlessly to they synchronised backup generators and, then fail because they are not rated to carry the larger currents. Then you need to call the sparks in to jury rig a temporary workaround solution, which takes time.

      3. Alan Bourke

        Re: It would appear to be worse...

        Nuclear power plants are designed to shut down safely with this in mind.

        1. Anonymous Coward
          Anonymous Coward

          Nuclear power plants are designed to shut down safely

          Sure, but they still need power to do that. Dungeness managed perfectly back in 2013 when they lost all (external) power. Also helped by AGRs being very good at getting rid of decay heat with fairly modest power needs ;-)

      4. werdsmith Silver badge

        Re: It would appear to be worse...

        But even at our unimportant organisation we test the diesel generator once per month, fire it up, put it under load and record the results. The only thing important thing we need uptime for is a customer service call centre.

    3. tirk

      UPS's sometimes fail, vital bits of kit sometime end up "temporarily" being plugged into the regular power supply, etc, etc.

      Just having a plan for when things go wrong isn't the same as it working.

    4. Hans Neeson-Bumpsadese Silver badge

      Generator testing

      I remember one site where they tested the backup generator religiously for a few minutes every week. When a major power outage did occur and they needed to run of the genny, it failed within the hour...all that testing had run the fuel down, and nobody had thought about topping it up.

    5. ideapete
      Facepalm

      Plenty of interruptible

      UPS becomes IPS

  2. Jim Willsher

    Even my home PC is protected from a power failure. It's called a UPS. WTF?

    1. Anonymous Coward
      Anonymous Coward

      And when

      did you last test it?

    2. Anonymous Coward
      Anonymous Coward

      Big UPS

      Each of the backup generators is rated for 5MW. And what is the holdup life on your home PC UPS? Typically they hold up long enough for a controlled shutdown.

      Let's be optimistic and specify a 4 hour hold up time. At 5MW that's 20 000kWH. For comparison, a typical 110AH leisure battery rated at 12V can supply C/10 for about 4 hours before suffering serious damage. That's 44AH * 12V = approx 500WH.

      20 000kWH = 40 000 standard leisure batteries. That's nearly half a million times more than the 10AH unit in a typical PC UPS.

      tl;dr: storing large amounts of electrical power is very expensive, that's why we still drive fossil fuel cars.

    3. Anonymous Coward
      Anonymous Coward

      IN HARTFORD, HEREFORD AND HAMPSHIRE.....

      ...No one seems to have a 13 amp fuse.

      1. Test Man

        Re: IN HARTFORD, HEREFORD AND HAMPSHIRE.....

        "IN HARTFORD, HEREFORD AND HAMPSHIRE....."

        *HERTFORD - get it right!

        1. John Brown (no body) Silver badge

          Re: IN HARTFORD, HEREFORD AND HAMPSHIRE.....

          ""IN HARTFORD, HEREFORD AND HAMPSHIRE....."

          *HERTFORD - get it right!

          Oooh, I bet that hert!

          1. Fruit and Nutcase Silver badge
            Thumb Up

            Re: IN HARTFORD, HEREFORD AND HAMPSHIRE.....

            Oooh, I bet that hert!

            you mean, hertz!

    4. I ain't Spartacus Gold badge

      Jim Willsher,

      What's the battery life on your UPS, to keep your PC alive? Oh, plus the phones, radios, lights, power to the other hundred PCs, links to all the radar data you need for ATC, links to airports etc? That probably takes a tad more battery than just to give 1000W at 240V to a single desktop.

      Presumably the UPS has to keep everything up long enough for the generator to fire up to keep providing power. That's assuming something hasn't gone wrong with the internal power wiring, in which case there's external power coming in, it just can't be distributed (and neither can the power from the genny).

  3. Bert 1
    Mushroom

    Get them to hold at the outer marker

    That'll sort it...

    1. Rob
      Coat

      Re: Get them to hold at the outer marker

      Bruce Willis is on his way as a backup solution, just as soon as his flight arrives.... oh wait....

      1. alexmcm

        Re: Get them to hold at the outer marker

        Picked the wrong week to stop sniffing glue

        http://www.youtube.com/watch?v=hd1ciPnTGKg

  4. SE

    "Luton airport would not comment when we contacted them."

    What a strange thing for them to get cagey about. Unless they've got a couple of shady Sheikhs and rendition flights coming in.

    Maybe somebody in the Luton area can look out of the window for a moment and check?

    1. Cripes Chief!

      Lets hope

      It goes on for a while as I live under the Luton Flight path and it would be nice to have a peaceful nights sleep for a change

    2. Inventor of the Marmite Laser Silver badge

      At least some stuff seems to be arriving/leadving Luton:

      http://www.london-luton.co.uk/en/flights/

      Interestingly NATS says:

      NATS can confirm that a technical problem has been reported at Swanwick air traffic control centre.

      UK airspace has not been closed, but airspace capacity has been restricted in order to manage the situation. We apologise for any delays and our incident response team has been mobilised.

      Every possible action is being taken to assist in resolving the situation and to confirm the details.

      Further information will be released as it becomes available.

      1. Test Man

        Yeah, I didn't think it was possible to close UK (or even London) airspace either, because clearly they CAN fly airplanes without radar, etc., but obviously they'd have to resort to pen and paper, and fly airplanes into the area one at a time.

  5. IglooDude

    I'm glad the air traffic control isn't considered a critical system, or it'd be a bit embarassing not having this kit wired up with emergency generator backup power.

  6. ukgnome

    Yippee-ki-yay

    Someone call John McClane

  7. Spacedman

    Lots of birds on the ground: http://www.flightradar24.com although I'm now fascinated with this glider soaring north of the Thames at 2,500ft and its getting dark... http://www.flightradar24.com/GDCFA/5066834

    1. Anonymous Coward
      Anonymous Coward

      I like this

      absolutely fascinating to watch.....................Must be over now though as I can see things taking off

    2. Martin Gregorie

      Its an ASK-13 dual seat training glider flying out of Booker, and now on the ground there. They usually aero-tow, so that was probably somebody taking a high tow for the last flight of the day. BTW, did you notice that you were seeing a FLARM trace rather than radar?

    3. Mrspudulike

      Thanks for the link. That has quite nicely wasted 20 minutes of a Friday afternoon that would have been spent wondering what to do....

      1. Primus Secundus Tertius

        Friday pm

        @Mrspudulike

        " ... Friday afternoon..."

        You have probably hit on the cause of the problems.

  8. J. R. Hartley

    One does not simply..

    'Have a power failure'

  9. David Pollard

    If memory serves

    Wasn't there a problem with the power supply a few years ago? As I dimly recall, there was a component like a smoothing capacitor in the common feed and it was this that failed. The power supply went down and the UPS came up and was connected to a short; or something along those lines. Everything worked perfectly apart from an unlikely fault that no one had foreseen.

  10. Anonymous Coward
    Mushroom

    Shit happens

    We had a UPS literally blow up, chucking battery acid all over the place and catching fire. When that happens, you have to kill ALL power to the site.

    1. Destroy All Monsters Silver badge

      Re: Shit happens

      A time for colonial marines and nuking sites from orbit.

  11. Anonymous Custard

    12 more sleeps 'til Santa?

    Well all I can say is they've got 12 days to fix it, or my kids won't be happy if Rudolph and co don't get clearance to land.

    That said one of my kids won't be happy anyway, as the storm and power cut we had last night took out the micro-SD card in my Pi, so about 3 weeks worth of her Minecraft tweaking (since the last back-up) looks like it maybe a gonner.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon