back to article London's Gatwick Airport flies back to the future as screens fail

London Gatwick Airport’s shiny new cloud-based flight information display system had a hard landing this morning as its vision of the future was brought down to earth with a bump. While collecting the Cloud Project of the Year award at the Real IT awards in May, the airport proclaimed its new screens were "an innovative, cost …

Page:

  1. Ochib

    So why was there no redundancy in the internet link?

    1. Dan 55 Silver badge

      Because oddly enough it doubles the cabling costs and that wouldn't do.

      1. Mike 'H'

        But...VODAFONE...

        Where the bloody hell was the LTE backup - 3 megabits on their Network must be child's play.

        Point a directional YAGI or dish antenna towards an off-site tower to bypass on-site DAS/macros, likely afflicted with the same fibre cut, and bobs yer uncle.

        Or maybe the airport should spend a bit more for a competing LTE provider for 3-megabit backup services...

      2. Lotaresco Silver badge

        "Because oddly enough it doubles the cabling costs and that wouldn't do."

        It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution and if it is combined with the appropriate type of switch failover to an alternative pair is seamless. The switch will even notify that a pair has failed so that action can be taken by the SOA. The only difference in price is the cost of cable + switches which is minimal because labour is the big spend.

        This is, quite frankly, poor practice on Vodafone's part.

        1. Wayland Bronze badge

          Of course it doubles the cable cost. Just because some of the fibres are dark does not mean you get to use them for free.

        2. ButlerInstitute

          That wouldn't really be redundancy.

          "Pulling a multi-pair cable"

          But then you've got two cores in a single cable, thus failing to provide any resilience when a digger goes through the cable. Ok for a fault with the cable/core itself maybe, but not for a physical break.

          Your alternative core needs to come into the site via a different route, So wouldn't be cut by the same digger. See for example major BBC facilities where there are redundant power and signal cables coming in from opposite sides of the site.

          1. Anonymous Coward
            Anonymous Coward

            Re: That wouldn't really be redundancy.

            A very big site I worked at had multiple connections, unfortunately someone emptied a truck of waist tar down the manhole cover out the front of the building where all the connections went through, business stopped for a couple of days.

        3. qwertyuiop
          FAIL

          Two failures?

          "...It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution..."

          Ah, I see. We increase resilience by using a multi-pair cable but immediately reduce it by using... a multi-pair cable. When the digger goes through the cable it doesn't matter how many pairs are in it! (and yes, I know that in this case it wasn't a digger)

          The only way to truly increase resilience is to have two cables which arrive by completely different routes - and that will increase the cost.

          1. jmch Silver badge

            Re: Two failures?

            "The only way to truly increase resilience is to have two cables which arrive by completely different routes"

            Or have one cable connection and one wireless connection

        4. Anonymous Coward
          Anonymous Coward

          Ummmm.. yeah, it does. Multi-pair cables don't help if the cable's cut (since all the pairs are cut too). If you want to be truly fault-tolerant, you need a second cable, geographically isolated, going preferable to a second ISP but in any case at least to a different access point for your ISP.

        5. jimbo60

          > Pulling a multi-pair cable is a sensible precaution

          Sorry, multi-pair cables don't offer any protection against diggers slicing the entire thing. You need to have different physical routes to avoid that problem.

        6. Mark Dempster

          >It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution and if it is combined with the appropriate type of switch failover to an alternative pair is seamless. The switch will even notify that a pair has failed so that action can be taken by the SOA. The only difference in price is the cost of cable + switches which is minimal because labour is the big spend.<

          That probably wouldn't have prevented the issue, though, as pulling 2 cables from the same location to the same destination means they're separated by millimeters at best - so both would have been cut through at the same time.

          If you're going to this trouble (and for something so important why wouldn't you?) then you need 2 completely different runs of cable, preferably from 2 different providers, coming into the building at different locations.

      3. G_Man

        3G/4G failover

        Forgive my ignorance, but given they are saying that normal operations only requires a 3Mbps link, couldn't they save on truly diverse, resilient fibre by using 3G/4G in a failover event?

    2. macjules Silver badge

      The only redundancy they now have is the moron who failed to QA before pressing the big green 'go' button.

      1. Doctor Syntax Silver badge

        "The only redundancy they now have is the moron who failed to QA before pressing the big green 'go' button."

        No way. Have you any idea how many senior managers would insist on getting their names on the sign-off of a new big shiny? They're going nowhere.

        1. Anonymous Coward
          Anonymous Coward

          @Doctor Syntax

          A few were spotted checking the Departures whiteboards, with a grimace

    3. Pascal Monett Silver badge
      FAIL

      Re: "no redundancy in the internet link"

      Exactly that. Over 1000 monitors tied to a mission-critical component and nobody ever asked "what if the cable was cut" ?

      Complete, utter and total project specification failure.

      1. Version 1.0 Silver badge

        Re: "no redundancy in the internet link"

        "Complete, utter and total project management failure." - but this is exactly how business works these days. Expect a big bonuses all round at the end of the year.

        1. yoganmahew

          Re: "no redundancy in the internet link"

          Outages happen. The question for me is why there was no local cache? It would have grown stale over time, but a well installed local cache with a GUI for updates could put everything into manual mode with zero impact.

          Is this the Internet of Tripe future? One failing link and your IoT belt unbuckles and your trousers fall down exposing your single-point-of-failure-arse?

          1. Captain Scarlet Silver badge
            Trollface

            Re: "no redundancy in the internet link"

            As they are on Vodaphone why don't they use their mobile network instead in the event of a fibre cut.

            (Ok below the belt as its likely nearby equipment may also be affected by the fibre cut)

          2. TkH11

            Re: "no redundancy in the internet link"

            Caching is not the answer to a fundamental failure of resilient network design.

            1. Flakk Silver badge

              Re: "no redundancy in the internet link"

              This tiny whiteboard is the only departures information in Gatwick Airport right now; every screen is offline. Utter chaos. This is a signature flourish at the end of a short trip that’s been full of reminders of how badly the UK’s infrastructure is crumbling. pic.twitter.com/6r7CDVheLf

              — Rob Fahey (@robfahey) August 20, 2018

              To be sure, that the info boards went down due to a network outage is pretty bad. But utter chaos? Really? It seems to me that any traveler who was inconvenienced by this outage for more than a minute has failed to understand the potential of the smartphone they are holding. You can absolutely use it to whine and wail on Twitter, or... I don't know... maybe look up the gate information on Gatwick's website? Failing that, maybe the airline's website?

              1. Anonymous Coward
                Anonymous Coward

                Re: "no redundancy in the internet link"

                Sadly at 17:10 all departure gate data is missing from the website.

                The point and beauty of cloud based computing is you never have to worry about a site going down.

                Just monitored my site and it's now beer o'clock.

                Good luck.

              2. Efer Brick

                Re: "no redundancy in the internet link"

                was it not the same system?

              3. JohnG Silver badge

                Re: "no redundancy in the internet link"

                "...the smartphone they are holding. You can absolutely use it to whine and wail on Twitter, or... I don't know... maybe look up the gate information on Gatwick's website?"

                Reports elsewhere indicated that updates to Gatwick's mobile app were affected by the same fibre outage. It seems that the design was engineered to fail comprehensively.

              4. David 45

                Re: "no redundancy in the internet link"

                Potential of a smart phone? No earthly good if the information is not there to start with. I was there today, dropping off passengers, then attempting to pick up others. Website arrivals flight information was sketchy, with some flights missing or no information against the flight numbers (so no use for one's shiny smart phone there!). Gatwick's auto. phone information system just went dead the second I entered the flight number (so no use for one's shiny smart phone there!). I had no idea of the status of my incoming passengers' flight and felt that Gatwick's main number would probably be inundated, so I didn't bother trying that (so no use for one's shiny smart phone there!). Arrivals concourse info. screens were also all over the place, with my flight number also not showing on there at all. My passengers also said that the baggage reclaim section was also not working. The main complaint I have is that there was no information whatsoever in arrivals that there was a problem - not even any public address announcements and certainly no whiteboards. There were lots of baffled-looking folk with furrowed brows looking at useless screens, obviously wondering what the devil was going on. In fact, as far as I could see, the only information that something was amiss was a rider on the flight information website saying that information screens were not working properly. A bit more communication (in more ways than one!) wouldn't have gone far amiss.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: "no redundancy in the internet link"

                  use a proper system like flightradar24 for any inbound flight they have real time up until the plane gets to the gate. For outbound the schedule will show that minus any delays.

                  There ar millions of ways to fins out whats happening without the airport on screen system if you travel enough you will soon get the hang of it

                  1. WallMeerkat Bronze badge

                    Re: "no redundancy in the internet link"

                    Flightradar24 is nice for viewing the route your flight took, but for arrival data it can be hit or miss, for example it doesn't deal with diverted flights nicely.

              5. tin 2

                Re: "no redundancy in the internet link"

                "It seems to me that any traveler who was inconvenienced by this outage for more than a minute has failed to understand the potential of the smartphone they are holding."

                Seems that information was not making it out of the airport to whatever cloudy goodness was updating the website and app either.

              6. Lotaresco Silver badge

                Re: "no redundancy in the internet link"

                "maybe look up the gate information on Gatwick's website? "

                Have you tried to do that? Good luck trying it. If you're lucky you'll get departure gate information in time to watch your flight depart. If it's working as usual you will get the information 24 hours later.

                I have the Gatwick app. It's never told me a gate number before the flight has departed.

              7. Anonymous Coward
                Anonymous Coward

                Re: "no redundancy in the internet link"

                International travellers without data roaming perhaps?

            2. Hans 1 Silver badge
              Happy

              Re: "no redundancy in the internet link"

              It is pure and utter design failure.

              Why have 100's of systems download flight information from the cloud ? <---- that is already brain-dead, somebody in the design team has never heard of multiplications!

              No resilient internet connection ? <---- that is brain-dead

              Caching is not the answer to a fundamental failure of resilient network design.

              It reduces internet traffic significantly and buys you time for the connection fail-over..

              1. Anonymous Coward
                Anonymous Coward

                Re: "no redundancy in the internet link"

                Welcome to the future, at least the future if the current "Cloud" fad continues.

                Single points of failures all round.

                Someone took a digger to your fibre? Bad luck. Maybe your backup link will take up the slack, but as these are usually lesser specification it will soon grind to a halt.

                Someone took the cloud server down? No local caching or redundancy? Bad luck. Maybe someone will turn it off and on again soon. Maybe the script kiddies will get bored and stop DDOSing it.

                Ironically the internet was developed to survive a nuclear war.

                1. jmch Silver badge

                  Re: "no redundancy in the internet link"

                  "Welcome to the future, at least the future if the current "Cloud" fad continues."

                  the thing is, a properly implemented cloud solution gives many benefits one of which is redundancy. Except management only got the memo where cloud = cost savings and don't realise that you still need to spend money on redundant network links. Every piece of the puzzle has to have redundancy, not only the cloudy bit

                2. Martin Gregorie Silver badge

                  Re: "no redundancy in the internet link"

                  One thing nobody seems to have forgotten - BT and other wonderful network providers currently operating in the UK have been known to engineer their own single point of failure. It happens this way:

                  • The system design team specifies a disaster recovery site and a high speed connection to it
                  • Their network design requires separate dual redundant links from the operations centre (LGW in this case) to the main ops site and to the disaster recovery site via at least two paths which are required to leave the building via separate ducts and then follow different routes.
                  • These specs get handed to the network provider, whose contractors promptly ignore all the fancy separate routing details and put all the cables through a single duct so they can trouser all the money they saved by skipping all that costly separate routing nonsense.
                  • The local council puts a digger through the cable duct....

          3. Lotaresco Silver badge

            Re: "no redundancy in the internet link"

            "The question for me is why there was no local cache? It would have grown stale over time"

            It's an Arrivals and Departures system. The data grows stale in no more than a couple of minutes. A local cache doesn't really help. What is needed is resilient comms and that is standard provision for systems like this. There should be no SPOFs in a real time system.

            1. yoganmahew

              Re: "no redundancy in the internet link"

              @Lotaresco

              "It's an Arrivals and Departures system. The data grows stale in no more than a couple of minutes. A local cache doesn't really help. "

              Not really, the scheduled departure and arrival times are well known days in advance. The gates are usually well known, but at lest could be manually updated (so at least people stand a chance of finding their gate). The amount of data that tranmits by FLIFO for FIDS updates is vanishingly small, being essentially designed in the 1960s. Never mind LTE, you could run it on dial-up...

        2. Quentin North

          Re: "no redundancy in the internet link"

          Not a project management failure, possibly a service specification failure due to "value engineering" by the service provider or customer. The PM probably did exactly what they were asked to do.

      2. Anonymous Coward
        Anonymous Coward

        Re: "no redundancy in the internet link"

        Exactly that. Over 1000 monitors tied to a mission-critical component and nobody ever asked "what if the cable was cut" ?

        (Insert Wiley Coyote gag here)

      3. bpfh

        Re: "no redundancy in the internet link"

        So Gatwick generates the raw departures/arrivals data, posts that data to somebody else’s computer, then the screens pull that back to Gatwick. And it only uses 3 megabits...

        So 2 questions:

        0. Did they really need Somebody Else’s Computer to do this (especially if the computers that run this are still there, but now offsite - out of sight out of mind?)

        1. 3 megabits... cut fibre... nobody thought of setting their mobile phone into WiFi hotspot mode - or dare I say it, backup 3/4G router ? Or is mobile internet that crap there?

        1. Anonymous Coward
          Anonymous Coward

          Re: "no redundancy in the internet link"

          No techies on site as all were offshored a long time ago?

        2. yoganmahew

          Re: "no redundancy in the internet link"

          @bpfh

          "DId they reall need someone else's computer"

          No, but, generally you stick the flight information data on a server somewhere so you can access it from multiple sources. Of course, if you build that as PUBSUB and there was a local server serving coax to the airport... (why coax? Coax can be fixed by anyone with pliers and a piece of tape... Bring back coax!).

          It's a bit weird, that they are having problems still suggests it's not just a fibre cut?

          1. defiler Silver badge

            Re: "no redundancy in the internet link"

            I'm just going to lob this on the end here, since everyone's screaming about having a second cable (expense of installation, potential proximity to first), and others are screaming about having a cellular backup (potential proximity to cable because Vodafone).

            Nobody's saying satellite link. If all you're looking for is 3Mb/sec, it's not expensive. Oh noes - 200ms latency? Who cares? If the power stays up, the data stays up. And if the power drops, the airport is closed anyway.

    4. Anonymous Coward
      Anonymous Coward

      Situation Normal: Outsourced Data

      When LGW was run and managed by BAA, BAA IT provided the information screens in co-operation with CAA/NATS in the tower.

      Now, with GIP owning the Airport, VodaFone providing the data and the tower outsourced to a German company with an unfinished website - hardly surprising really....

      Too many layers of outsourcing....too cheaply....

    5. Multivac

      They didn't need it, everyone has a smart phone right? Just go to https://www.gatwickairport.com/flights/?type=departures .

      If everyone is carrying their own screens around then you you have an airport full of redundancy, different phones, different OS's using different network providers.

    6. GrumpyOldGeek

      Because : time to market, at all costs. Blockers be damned.

    7. jamesdagger

      Diverse routing

      Fully diverse routing can cost a fortune in civils.

      Almost certainly why it wasn't signed off, or was signed off to a pinch point where the break occurred south of that.

  2. Anonymous Coward
    Anonymous Coward

    According to http://www.mediacentre.gatwickairport.com/press-releases/2018/18_05_18_it_awards.aspx

    "We have also just future proofed our entire IT network for the next decade so that we can take advantage of the latest technologies, while also making the network more resilient and tolerant to disruption."

    Bullsh*t...

    1. Hans Neeson-Bumpsadese Silver badge

      take advantage of the latest technologies

      That's no bullsh*t. I've seen some photos, and let me tell you that's one modern-looking whiteboard that they're using.

      1. m0rt Silver badge

        Wouldn't it be funny if they used permanent markers by mistake...?

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019