back to article Telecoms fail in UK takes down passport scanners in Australia

International airline passengers in Australia, New Zealand and other nations have been told that problems processing passports at airports today were caused by a telecoms failure in the United Kingdom. Passengers started to complain of slow check-ins and long queues early Monday, Australian time. Australia's Department of …

  1. Pen-y-gors

    Timezones?

    which ignored us for hours ...

    Hey guys, it was sunday night here - you don't expect 24/7 IT and PR support on international mission-critical systems, surely? That would cost money, and we know how governments feel about paying for support...

    1. monty75

      Re: Timezones?

      They probably pissed the support budget away on "agile consultants"

      1. Rafael #872397

        Re: "agile consultants"

        They probably pissed the support budget away on "agile consultants"

        ...and "devops conferences" :-)

      2. Anonymous Coward
        Anonymous Coward

        Re: Timezones?

        As a former employee of SITA, it wasn't agile consultants that did the damage but all the IT seemingly being outsourced to NIIT of India. A whole load more people will be getting the heave-ho next month.

        anon for obvious reasons.

    2. Anonymous Coward
      Anonymous Coward

      Re: Timezones?

      SITA has an office in Sidney, hope it follows the local timezone.

      SITA also offers a border control management software (http://www.sita.aero/solutions-and-services/solutions/iborders-border-management) - I'm not surprised that passport data today are shared among airlines, airports, governments.... and SITA does that. Just it looks it needs to be more resilient to some communication disruptions.

      1. JimboSmith Silver badge

        Re: Timezones?

        SITA has an office in Sidney, hope it follows the local timezone.

        Ah but do they have one in Sydney?

    3. Mark 65

      Re: Timezones?

      which ignored us for hours ...

      That's just because they're French.

  2. Pen-y-gors

    International data transfer?

    Interesting point about data going international though, were they trying to connect to GCHQ? Or, less likely, they wanted to check an update server for software or AV updates and wouldn't carry on until they'd found it! Either way, big fail.

    1. Dan 55 Silver badge
      Holmes

      Re: International data transfer?

      Maybe some kind of off-the-shelf software that worked with cloud storage in the UK?

    2. smudge
      Big Brother

      Re: International data transfer?

      Interesting point about data going international though, were they trying to connect to GCHQ?

      I suspect they route them all to Assange for him to check, since he's an Aussie with time on his hands. My guess is that he had a visit from Pamela Anderson last night.

    3. Adam 1

      Re: International data transfer?

      > Interesting point about data going international though, were they trying to connect to GCHQ

      Still. It's going to fall over less frequently than Telstra.

    4. Korev Silver badge
      Big Brother

      Re: International data transfer?

      Interesting point about data going international though, were they trying to connect to GCHQ?

      As members of the "Five Eyes" group, the Aussies and Kiwis probably are perfectly able to do those shenanigans themselves

  3. A Non e-mouse Silver badge

    The real issue

    The real issue that El Reg hasn't mentioned, is why did the company only have comms with one telco? Surely if the system is that critical, it would have had resilient/diverse connectivity via several telcos?

    1. Mark 110

      Re: The real issue

      Indeed - sounds like they did have a back-up link but it needed manual failover (guessing DNS change but might be wrong). Rather clunky for a critical service.

    2. M7S
      Black Helicopters

      Re: The real issue

      " if the system is that critical"

      Think about who will be inconvenienced.

      Then think about who will decide how critical that really is.

      I saw a documentary/long advert for Heathrow some time ago which showed their VVIP terminal. I don't expect you get much in the way of queues there even if the computers fail

      1. frank ly

        Re: The real issue

        "... In response, we implemented an alternative communications link and resumed services."

        After what length of delay?

        1. Anonymous Coward
          Anonymous Coward

          Re: The real issue

          The time it takes for a US Robotics modem to dial in and download a copy of a Lotus 123 spreadsheet.

        2. Anonymous Coward
          Anonymous Coward

          Re: The real issue

          "After what length of delay?"

          According to the statement, about 3 hours downtime. Which is not too bad. Unless, of course, you're missing your flight because of it... Stuff like this needs multiple paths with multiple providers.

    3. Anonymous Coward
      Anonymous Coward

      Re: The real issue

      From memory of postings on comp.risks from years ago its not unkown for companies to build multiply redundant comms setups only to discvover when it goes down that somewhere all of the telcos are routing through the same cable or on cables that go through the same duct so a single digger can take them all out at the same time.

      1. Jc (the real one)

        Re: The real issue

        Around 20 years ago , I was working with a Telco in the UK talking about DR. They were proud of their diverse routing. The Data Centre manager took me along the route of the fibre , while my colleague followed the other one. You should have seen the DC Managers face when we met the other guy where both sets of fibre went through the same conduit out of the building!

        I doubt if much changes!

        Jc

        1. Anonymous Coward
          Devil

          Re: The real issue

          @JC (the real one)

          You should have had a 3rd colleague outside with a backhoe. Juicy DR consulting contracts for all!!

        2. Kernel

          Re: The real issue

          "You should have seen the DC Managers face when we met the other guy where both sets of fibre went through the same conduit out of the building!"

          The best I've encountered in my 40-odd years telco career is full diversity - via different wavelengths on the same fibre pair.

          The "geographically diverse servers" for another project were not quite what we had in mind when the request was made either - different racks, side by side, in the same data centre. For a later iteration of the system that was changed - the racks were at least no longer in the same row!

      2. Anonymous Coward
        Anonymous Coward

        We had this...

        I worked for a UK national newspaper and we wanted wholly fault tolerant comms as we would scan and send our broadsheets to the printers. Basically it was a large fax machine but cost rather more than your average office Fax (think upwards of £500K each 20 years ago).

        Anyway, we asked a couple of telecoms providers to come in, Cable and Wireless were one from memory (yes its that long ago), might have been BT for the other, we asked them to confirm they were independent, yes, we were told, one cable would go out the back of the building, one out the front (we weren't stupid, we'd worked that single point of failure out ourselves), We then asked them to physically walk where the cables then went. They got their maps and engineers in and we walked down City Road in London. One company went from the front of the building, the ones at the back walked round the perimeter of the office and lo and behold, they met at a junction box (or whatever you call these things in the ground) about 50M from the office.

        Great amounts of embarrassment from each of them, we then had a more sensible discussion about where there cables were now going to go. I suppose eventually they met up somewhere but we couldn't find out where.

        You just need to hold their feet to the fire and assume that the sales droids in the telecoms company's are idiots and will tell you absolutely anything for a sale. Once you know how they work, its fine.

        1. tony trolle
          FAIL

          Re: We had this...

          We had a car take out a green junction box, still wonder about the speed needed to jump a 10 inch curb, any way it took out 4 ADSL lines and some T1's. We were down for over a day.

          Anyway we had a new fibre cable run in from the other end of the road and on our first real weekend DR test found out the backup batteries needed to be replaced, but the big and do mean big generators worked every time.

          The second weekend test failed as starter motor system was run off the batteries and someone left unplugged when news ones got swapped in; the second "second" test worked after a few hours, and double checked a few more times.

          The third weekend test failed as no one filled up the diesel for the donkey starter motors.

          The off site DR test is another story.

  4. Winkypop Silver badge
    Devil

    Not sharing the data

    Just caring for the data....

  5. Alister

    a major telecom failure in the UK

    I can't help thinking that if there were a "major telecoms failure" it might have impacted other things, which were noticeable. I'm not aware of anything else that fell over last night?

    1. PNGuinn
      Trollface

      Re: a major telecom failure in the UK

      GCHQ took 3 hours to pay the bitcion ransom?

  6. monty75

    TITSUP

    Bit disappointed not to have a Reg "Total Inability To Scan Usual Passports" sub-head

  7. steelpillow Silver badge
    FAIL

    Single point of failure

    One lonesome data centre storing the world's passports?

    Repeat after me:

    "In a critical system there should be no single point of failure.

    "In a critical system there should be no single point of failure.

    "In a critical system there should be no single point of failure.

    "In a critical system....

    Oh, and by the way, if the data centre had burned down, is there even an offline backup in another location? Have they exercised the recovery scenario?

  8. adam payne

    A critical system with a single point of failure, wow well done.

    1. Anonymous Coward
      Anonymous Coward

      Who remembers what NSPoF means?

      "A critical system with a single point of failure, wow well done."

      Dinosaur-style thinking. You need to get with the program. It's the 21st century, shiny outweighs robust, and by far the most important consideration is that someone other than the IT provider has to pick up the costs (financial and otherwise) when the system enters TITSUP phase.

    2. Cynic_999

      "

      A critical system with a single point of failure, wow well done.

      "

      The most important thing is to ensure that you always have someone else to blame.

  9. Roger Kynaston
    Coat

    So are they running XP?

    Curious minds would like to know

  10. Rhubarb

    The data

    Was just resting in my account

  11. TrumpSlurp the Troll

    Other places?

    Wondering where else wouldn't be clearing passengers between 21.00 and midnight GMT.

    US would on west coast. Certainly Hawaii.

    Perhaps a Commonwealth thing?

    Still, who relies on a data centre on the opposite side of the world for critical functions?

    Oh, hang on, someone is rabbiting in my ear about a cloud something.......

    1. Anonymous Coward
      Anonymous Coward

      Re: Other places?

      The USA would be handled by the SITA Data Centre in Atlanta. Apparently everything else comes to London, until all the kit is relocated to India that is.

  12. John Smith 19 Gold badge
    Unhappy

    Obvious question. Did any UK readers spot some kind of phone trouble around then?

    It seems very hard to believe that something that can clobber links from Aus to UK would not have had some effects on local traffic (I'm not sure El Reg is hosted in the UK for example).

  13. Anonymous Coward
    Anonymous Coward

    "Which raises the questions: why on Earth are users in Australia and New Zealand dependent on telecoms in the UK"

    Because that's where GCHQ is based.

  14. Anonymous Coward
    Big Brother

    Why does antipodean data travel pass outside local jurisdictions

    "why on Earth are users in Australia and New Zealand dependent on telecoms in the UK, and does antipodean passport and travel data pass outside local jurisdictions, which sounds a bit naughty!"

    Five Eyes, the pasports have to be passed back to GCHQ who passes them back to the NSA in real time. Which doesn't seem to have any adverse impact on the illicit drug trade, the arms trade or the money laundering trade.

    1. Anonymous Coward
      Anonymous Coward

      Re: Why does antipodean data travel pass outside local jurisdictions

      Former Customs worker and SmartGate project team member here.

      The actual reason Australia and New Zealand are dependent on SITA is for Advanced Passenger Information (API), which tells Immigration who is on the aeroplane for pre-screening purposes. API is used before the passenger gets on the plane, too. It helps make sure the passenger has a valid visa to enter the country.

      see https://en.wikipedia.org/wiki/Advance_Passenger_Information_System

  15. Anonymous Coward
    Anonymous Coward

    Aussie data centre in the UK

    More than likely due to their (.au and .nz passport/visa) hosting partner being located in the UK just north of Milton Keynes, no - I kid you not. This provider hasnt got the best of reliability when my company hosted there, which is one of the many reasons why we moved out a couple of years ago, and it sounds like it's not got much better.

  16. Potemkine Silver badge

    What about redundancy?

    If the service is so critical, why one connection down break everything?

    1. Toni the terrible Bronze badge
      Facepalm

      Re: What about redundancy?

      Because it is not critical to those who count, and isn't a bunch of non-VIP passengers.

  17. Herby
    Joke

    Perhaps the problem was...

    They wanted to scan the Queen's passport.

    Alas, she wasn't traveling.

  18. Kev99 Silver badge

    Let me guess. The idiots down under put everything on the bunch holes connected by vapor instead of dedicated lines as used to be norm. And they had no fall back service or plan.

    Smart.

    Really smart.

  19. JaitcH
    Unhappy

    You want privacy? You want anonymity? NEVER use AIR And ALWAYS pay in CASH!

    As Nick Leeson of Barings Bank, Singapore and Singapore PUB Deputy CEO, Choy Hon Tim, discovered, if you are trying to travel incognito, NEVER fly. Nick was scooped up in Germany and Hon-Tim in Malaysia!

    Passenger travel data is available as an 'option' on many law enforcement screens - even your local Plod. The better ways to avoid these technical trackers is either have multiple passports or use ground transportation. And book segments separately and switch travel times - as close as practicable to departure.

    The US Government is very generous with this technology, giving away free hardware to countries 'of interest' such as drug growing areas, animal part smuggling and terrorists.

    Thailand bought their own equipment but Burma (Myanmar) and Kampuchea (Cambodia) had free gifts from the USA so they could even equip their border gates. These 'gifts' often include cameras. The cameras on the borders of Cambodia, as the immigration staffs will tell you, are "connected directly to Washington, USA" - makes the border police think they are a somebody. Even remote border gates, my favourite, have cameras.

    But they all have a common weakness. Power and communications. My favourite Cambodian gateway has frequent daily power outages, being at the end of the power line, and if my timing is good I can cross the border bearing electronic goodies from Thailand - with a token 'duty' charge of $100 - because they are out of touch!

    Also, remember, C-A-S-H is KING! SITA data includes method of payment with credit card numbers and associated travellers on the same flight. Buy your tickets separately from your friends, and from different travel agents.

    Hotel res systems are just as intrusive - there is always hotel space at most destinations - don't book ahead! The FBI even transmits Wanted notices on several hotel networks. Stay at small, cheap, hotels for anonymity!

    1. allthecoolshortnamesweretaken

      Re: You want privacy? You want anonymity? NEVER use AIR And ALWAYS pay in CASH!

      Also, change hotels every night, and cars a couple of times each day. Keep one eye open for vans that are unmarked or have generic markings. Always a tell: old-looking van, old-looking numberplates, plates fixed with shiny new screws. Keep the other eye open for mail vans, telco vans and, these days, UPS/FedEx/Hermes/etc vans.

  20. Andrew Jones 2

    ......so a terrorist attack on the actual data centre - will disrupt the ability for people to fly in multiple countries around the world?

    That seems like something that might need to be looked into pretty quickly.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like