back to article OK Google, why was your web traffic hijacked and routed through China, Russia today?

People's connections in the US to Google – including its cloud, YouTube, and other websites – were suddenly rerouted through Russia and into China in a textbook Border Gateway Protocol (BGP) hijack. That means folks in Texas, California, Ohio, and so on, firing up their browsers and software to connect to Google and its …

  1. The Man Who Fell To Earth Silver badge
    FAIL

    So much for the original intent of the ARPANET

    Seems it might not be so robust in a time of war.

    1. 404 Silver badge

      Re: So much for the original intent of the ARPANET

      This was a test and only a test...

      shitdamnfuck

      1. bombastic bob Silver badge
        Devil

        Re: So much for the original intent of the ARPANET

        well it was only a test, and apparently a SUCCESS! [just not for Google and people in the U.S. trying to access their services]

        If I'd have know, I would have polluted the snooping by making bizarre requests on google for things that would be extremely embarrassing to anyone looking at the data... [wait, was THAT a NAKED PICTURE of Henry Kissinger?]

        /me laughs because in the 1970's, there was a parody Cosmo edition done by Harvard Lampoon, and the centerfold was, in fact, Henry Kissinger.

        1. The Nazz Silver badge

          Re: So much for the original intent of the ARPANET

          Re Henry Kissinger as done by Monty Python

          https://www.youtube.com/watch?v=T5vo7jLGOb8

          "Nicer legs than Hitler and bigger tits than Cher" always makes me chuckle.

    2. Peter Gathercole Silver badge

      Re: So much for the original intent of the ARPANET

      The original thinking for ARPANET did not include BGP. I believe that the alternative routing strategies were provided by static routing with routes preferences and hopcounts providing alternate pathing.

      For some history, look up RIP, which was deployed sometime around 1969.

      But RIP would never cope in today's massively complicated Internet. Since class-based routing broke down to allow re-use of the previously reserved network ranges that have been freed up to keep IP4 going, the routing tables that the core routers have to know are HUGE.

      But considering how BGP hijacking has been known about for a long time, I'm surprised that it has taken this long for a key based trust system to be introduced.

  2. Anonymous Coward
    Anonymous Coward

    So...is there a solution? Some sort of key exchange to confirm the identity of core routers?

    1. Ole Juul Silver badge

      where are the logs?

      I notice the article doesn't mention who did it. That suggests to me that there is no effective access control to these routers. Perhaps a partial solution would involve verifying and logging access.

      1. Anonymous Coward
        Anonymous Coward

        How would The Register or anyone else outside of said dodgy provider know who did it? This isn't someone at Google that has done this.

      2. rmason Silver badge

        Re: where are the logs?

        @Ole Juul

        The access control for these routers is as follows:

        Do you work for (relevant ISP)?

        Do you have the credentials?

        That's it. Of course things are logged, but by the hooky ISP(s) in question.

        The logs will be where they always are, somewhere google/law enforcement can't look, within a Chinese/Russian/African ISP.

    2. Anonymous Coward
      Anonymous Coward

      Key exchange to confirm the identity of core routers?

      > So...is there a solution? Some sort of key exchange to confirm the identity of core routers?

      The main article links to this:

      “Perhaps the most promising improvement to BGP comes from the Internet Engineering Task Force (IETF) in the form of BGPsec. Like DNSsec, BGPsec is an extension to BGP that introduces several new protections. Among them is Resource Public Key Infrastructure (RPKI), which will provide a way to associate Autonomous Systems with cryptographic certificates to maintain integrity.”

      1. Andy The Hat Silver badge

        Re: Key exchange to confirm the identity of core routers?

        How does this work if the invalid routes are advertised by a valid ISP with valid keys? It appears to me that China Telecom is valid (?) just being naughty for whatever reason but *is* a valid carrier. Of course RPKI would prevent external actors fiddling with configurations without keys but is there any evidence that what is happening is external to these organisations?

    3. Anonymous Coward
      Anonymous Coward

      So...is there a solution?

      Yes. Filter YOU PEER'S BLOODY ROUTE ANNOUNCEMENTS!!!

      Widely deployed in Europe. In fact, some Internet exchanges do not allow you to connect if you do not. I used to help maintain the software that generated the actual ACLs in a SP in one of my past lives.

      USA - nobody does that despite repeated recurring and near identical incidents going as far back as the late 1990es. In fact, the first incident I remember was in 1997 (or was that 1996) when some mom-and-pop ISP playing with gated codebase in a shed in Florida brought most of the USA internet down for a couple of hours.

      The incidents goes to show that a USA telco like ATT, VZ, etc (the ones which Google "peers" with) will accept anything China telecom feeds them and say "thank you, with pleasure".

      1. Pascal Monett Silver badge

        So that explains why I had no trouble with accessing either Google, Youtube or GMail yesterday. Since I live in France, the BGP failed here on account of reroute request denied.

        Damn, that sounds so simple. I wonder why US telcos don't give a damn like that ?

        1. Michael Wojcik Silver badge

          Damn, that sounds so simple. I wonder why US telcos don't give a damn like that ?

          Because it's not that simple.

          As I mentioned just the other day, AS routing is a big, complicated problem, which many experts have been examining for many years. (Bellovin's original paper on the subject was published in 1989.) "Drop all BGP announcements from your peers" isn't a good strategy when you may need to adopt changes published by other ASes.

          There are a bunch of mechanisms (prefix lists, communities, etc) for filtering BGP, and they're widely used. They can't solve the general problem. In fact, the 2008 Pilosolv & Kapela attack (which introduced BGP interception to the public) uses filtering as a critical component - they construct prefixes so that the victim AS will forward traffic to their AS, while some other ASes retain the original, valid route, so they can forward it on.

          Now, it's true that Kapela claimed at the time that "aggressive filtering" by ISPs could prevent BGP hijacking. But he was talking specifically about certain classes of attacks; the filtering would be expensive and require frequent maintenance; and all ASes on the path (for a given packet) would have to implement it for it to be secure.

          If there were an easy, inexpensive fix for BGP hijacking, it would already have been implemented.

          1. seskin

            wait I know... use artificial intelligence to maintain the filters and then.. oh.. wait. Artificial intelligence. Yeah.. never mind.

        2. Fungus Bob Silver badge

          "I wonder why US telcos don't give a damn like that ?"

          They don't care. They don't have to care - they're the phone company...

      2. Doctor Huh?

        Random Kevin Bacon Reference

        The incidents goes to show that a USA telco like ATT, VZ, etc (the ones which Google "peers" with) will accept anything China telecom feeds them and say "thank you, with pleasure".

        I believe the phrase you are looking for is "Thank you sir, may I have another?"

        https://youtu.be/bIZoVO8ZyyQ

    4. P. Lee Silver badge
      Holmes

      >Is there a solution?

      A solution to what?

      This is traffic to Google. Are you're concerned that someone in a foreign country is going to find out what you're doing online and pass that information on to someone you don't know?

  3. onefang Silver badge

    I wonder if this has anything to do with Google thinking for several months at least that my Holland server is in Russia? Other GeoIP providers correctly place it.

    1. Chris Harries

      Unlikely but you can contact them about this. I've done it before

      1. onefang Silver badge

        "Unlikely but you can contact them about this. I've done it before"

        You are assuming that I want to actually be able to understand the Russian adverts that Google shows to me, when I'm not using ad blockers, and VPNing through my Dutch server. If I'm forced to send more information about me to Google than I normally do, then I'm more than happy for them to get incorrect information.

    2. Anonymous Coward
      Anonymous Coward

      I'm all over the place. North Island, South Island.. Sometimes even West Island..

      Dunno why I don't trust Google Maps... P'raps coz an IP with a linked physical address that they know of moves towns and even countries at random?

  4. Winkypop Silver badge
    Coat

    The road less travelled

    Red iCloud at night, hackers delight....

  5. Allan George Dyer Silver badge
    Trollface

    Change it back quickly -

    The NSA wants their feed back.

    1. Mark 85 Silver badge

      Re: Change it back quickly -

      Ah..... someone misread NSA as KGB (or if you prefer the new names: FSS or FIS or anyother name they now go by).

      1. chivo243 Silver badge
        Headmaster

        Re: Change it back quickly -

        @Mark 85

        or FFS?

    2. Velv Silver badge
      Big Brother

      Re: Change it back quickly -

      The NSA wants their feed back.

      The cynical side of me thinks this is how the change was detected.

      1. Dal90

        Re: Change it back quickly -

        The really cynical side of me thinks the NSA wanted something domestically which they're not allowed to do. Now if it's passing the country's border it's fair game as foreign surveillance :/

  6. Andrew Commons

    The last paragraph says it all

    "The internet is built on such an open chain of trust that it is not hard for anybody to inject fake information," Naik said. "It really is that simple."

    And the great digital transformation drive has turned this into critical infrastructure.

    1. Anonymous Coward
      Anonymous Coward

      Re: The last paragraph says it all

      And the great digital transformation drive has turned this into critical infrastructure.

      Because it was easier to do that than to build on the secure trusted protocols built into OSI. Remember those arguments, why use closed OSI protocols dictated by telcos, when we can all use the open, public IP stack that's freely available to the whole world?

      This is why. Too late.

  7. Rufus McDufus

    Cui bono?

    I suspect we'll never really know who did it.

    1. Steve Button

      Re: Cue Bono?

      I suspect even the amazing Bono would not be able to solve this one.

      And how does one get hold of him anyway, shine a light in the sky with the Bonosignal?

      1. Far out man
        Megaphone

        Re: Cue Bono?

        He seems to be ahead of the game.

        With or without you

        I still have not found what I am looking for

        I will follow

        If he has a few spare minutes, maybe we can ask him to get in touch with Google to assist with forecasting Just hope that this does not lead to another freebie

        1. Korev Silver badge
          Coat

          Re: Cue Bono?

          U2 over there, stop the puns now!

    2. Locky

      Re: Cui bono?

      Until next weeks "On Call" submission

      So, doing some slow work at an ISP, I was messing around with vi masterDNS one Monday night....

      1. Nick Kew Silver badge

        @Locky Re: Cui bono?

        Is that an On Call?

        Or might it be more a Who, Me?

    3. Fred West

      Re: Cui bono?

      An ISP in Nigeria did it.

      Its also laughably lame that the usual Google hating plebs don't understand how it's not anything to to with Google (other than Google actually spotting the issue)

      1. Anonymous Coward
        Anonymous Coward

        Re: Cui bono?

        don't understand how it's not anything to to with Google

        duh art ick ile wuz a bout goo gle.

        duh first sent ince wuz "People's connections in the US to Google..."

        iz won sil la bill to much for you?

  8. Wolfclaw Silver badge

    Maybe time for anybody who messes with BGP, to be isolated from the rest of the network and force all their traffic through a dedicated gateway. Yes hard luck on for arguments sake China/USA/Russia/UK/Iran/India citizens, as it would enable even easier snooping, but the rest of the world would be more secure.

    1. Kevin Johnston Silver badge

      Since China is implicated in this, it may be that part of is their thinking...Force every connection through a single gateway and suddenly it is so much easier to ensure you can track what everyone is doing

      1. Anonymous Coward
        Anonymous Coward

        You improve security by range banning the whole of Russia and China. The server I use was under constant attack from these two countries, 24/7. I realise Google is open for business with quite a lot but for this particular protocol the time to speak softly is long gone.

  9. WonkoTheSane
  10. Anonymous Coward
    Anonymous Coward

    What about the UK

    When this happens there? Some Councils have stupidly gone with GSuite. So all your Council Tax info, Names, Address', DOBS and more could all end up on Google Drive but in Russia, or China.

    1. charlie-charlie-tango-alpha
      Facepalm

      Re: What about the UK

      It's much worse than that. Gsuite is used by UK Central Governent departments as well. I have never understood why. It's bad enough that Google knows all about your private email, it now also has full access to some HMG mail, documents, Hangouts discussions etc. FFS why give that kind of advantage to a US commercial company?

      Back in the day when I ran Gov IT systems we insisted the data was all on local boxes we could actually touch. GSI (version 1) changed some of that by moving mail through a commercial (but UK based) system. Later versions further watered down the local storage and processing paradigm. We now seem to be so enamoured of all the "cloud" bollocks that we are prepared to give away most of the crown jewels.

      Face palm for obvious reasons.

      1. Anonymous Coward
        Anonymous Coward

        Re: What about the UK

        And to add to my last comment. If the docs end up in Russia or China, that council is technically in breach of GDPR. Fun.

        Not to mention that to manage GSuite at the commandline you need to use a tool called GAM. It is one big security hole.

        GAM creates a custom key for the admin that set it up on their device. If someone then was able to steal the folder they have "installed" GAM in, they can then run all the admin commands as that admin account with no further authenticated required, even if those commands are now being run from a totally different IP range that they were being run from 5 minutes ago.

        1. Anonymous Coward
          Anonymous Coward

          Re: What about the UK

          they can then run all the admin commands as that admin account with no further authenticated required, even if those commands are now being run from a totally different IP range that they were being run from 5 minutes ago.

          Wait, google can do that?

          Then why the hell do I get 'security warnings' when I sign in from the same machine with the same IP and the same hardware etc etc as I did 2 minutes ago???????

          1. onefang Silver badge

            Re: What about the UK

            "Then why the hell do I get 'security warnings' when I sign in from the same machine with the same IP and the same hardware etc etc as I did 2 minutes ago???????"

            Perhaps for the same reason I do? Coz I'm using insecure protocols, with non trusted programs. Like fetchmail using SSL and POP3 to fetch my email. Waaay less secure than HTTPS, and much less trusted than Google Chrome. Though apparently it's only insecure once every few months, rather than the once a minute it actually polls at.

  11. Anonymous Coward
    Anonymous Coward

    Filtering today and issues with RPKI

    Today, if you try to buy transit, you will be asked to ensure you have route objects and / or AS-SET's registered with the likes of RIPE/ARIN or RADB or manually submit prefixes. This is generally quite good. The issue comes where someone more than one AS hop away starts slipping in routes which contain AS4134 (China Telecom) in the path, which the filtering may not catch.

    The issue with RPKI signing all routes is that apart from the circularity of depending on a network to authorise your acceptance of network routes, is that it transforms the internet into a strictly hierarchical structure. Someone could decide to lean on RIPE or whomever and get you knocked off the net: temporarily of course until you got a new block, but it's something to bear in mind.

  12. Anonymous Coward
    Anonymous Coward

    More please.

    ""We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence."

    I'd actually like to see it happen more often please. So often Google ceases to exist would be nice.

    Google, want to improve your systems in a way that is helpful to people? Turn your servers off, and some public meetings with your higher ups would be cool - I know a great venue in that dark alley over there.

    Failing that, at least stop being evil. Stop the spying, stop the privacy invasions. No, scratch that. Just stop.

    1. Anonymous Coward
      Anonymous Coward

      Re: More please.

      Duh google fanboies iz out in 4rze tonite!

  13. Anonymous Coward
    Anonymous Coward

    Good luck Russia and China sifting that massive pile of selfies and cat videos to find something useful.

  14. Archtech Silver badge

    Long on indignation, shortish on facts

    The fine article does not - as far as I could see - mention a couple of rather important facts.

    1. Were users actually prevented from connecting to Google, etc.? Or did they manage to connect and use the sites normally - just by a circuitous route?

    2. The article speaks of "theft" and suggests that packets were "stolen". Again, that implies that the packets were never delivered to the intended destinations. True or false?

    1. GremlinUK

      Re: Long on indignation, shortish on facts

      1. Yes. Users were prevented from contacting Google's servers. (at least, I was)

      2. if you read the rest of the article, it says that the Chinese server black-holed the packets at least for some of the time, and so, Yes, the other packets were arguably stolen, but who knows if anything was able to inspect them.

    2. David Nash Silver badge

      Re: Long on indignation, shortish on facts

      Actually it does in fact say that connections were failing to be established and consequently services were down.

  15. HxBro
    Facepalm

    Great idea in principle

    1. Advertise new routes for google

    2. Start monitoring packets

    3 ???

    4. Oh F*ck how do we handle this much data, turn it off quick!

    I can imagine the amount of bandwidth required would be something special, no wonder it was blackholed quickly.

    1. Anonymous Coward
      Anonymous Coward

      Re: Great idea in principle

      I can imagine the amount of bandwidth required would be something special, no wonder it was blackholed quickly.

      Maybe more a case of 'swamped' than deliberately black-holed. Assuming malicious rather than accidental event of course.

  16. msknight Silver badge
    Joke

    According to the BBC...

    ...it was MainOne in Africa - https://www.bbc.co.uk/news/technology-46194279

    The following is a joke... of course... to the tune of, "Blame Canada"

    ----

    Time's have changed

    The Internet's getting worse

    They won't obey the IETF

    ...and go to V6 instead

    Should we blame the government?

    Or blame society?

    Or blame the traffic of Internet TV?

    No, blame Africa, blame Africa

    Their update was a surprise.

    They re-routed all our files.

    Blame Africa, blame Africa

    We need to form a full assault

    It's Africa's fault

    ----

    Don't blame my poor old router

    It saw the wrongest route

    And now it's off to China and Japan.

    And Russia's on the path

    My files have gone "Прощай"

    And buggered off to the East instead of West

    Well, blame Afria, blame Africa

    Something technical went wrong

    When Africa came along

    Blame Africa, blame Africa

    They're not even between me and L.A.

    My data could have been a movie, or a best selling book.

    Now it's down a black hole, come and look.

    Should we blame the fibre?

    Should we blame the light?

    Or the technicans who buggered it up last night?

    Heck no

    ---

    Blame Africa, blame Africa

    'Cause of MainOne's hullabaloo

    They lost all your traffic too

    Blame Africa, shame on Africa

    All the smut got lost, the traffic all got crossed

    All this routing mess must be undone

    We must blame them and cause a fuss

    Before someone thinks of blaming us

    1. cambsukguy

      Re: According to the BBC...

      At least someone thinks IPv6 might be the solution.

      I suppose there must still be attack vectors but it does have some security-conscious thought in the design so perhaps would considerably more effort to hijack such large amounts of traffic.

      1. Rupert Fiennes Bronze badge

        Re: According to the BBC...

        Really not going to make the slightest difference. The issue here is how wrong routes get propagated. BGP for v6 is basically the same as v4 here.

  17. Anonymous Coward
    Anonymous Coward

    BGP

    Hi

    To put some perspective on this, BGP configurations are complex and a small error can have a wide effect. Like throwing a pebble into a pond.

    If you put hard filtering on BGP broadcasts from adjacent ISP, then you will loose the flexibility of BGP/dynamic routing to define the best routes through the Internet and larger outages could occur.

    I know ow easy it is to make an error, as I accidentally put a /16 mask on what should have been a /28 network and took out all that ISP's monitoring systems. That was noticed straight away and the outage was only a few minutes for the operations centre.

    I would consider human error first, before looking at melovance........

    A network engineer

    1. Steve Aubrey
      Joke

      Re: BGP

      I can't resist.

      "I know ow easy it is to make an error"

      Cue a Groucho Marx smirk, an eyebrow raised, a cigar pointing. "I just bet that 'ow' was a loud one!"

  18. 89724102172714182892114I7551670349743096734346773478647892349863592355648544996312855148587659264921

    Russia has figured out the pattern in large Prime Numbers:

    https://motherboard.vice.com/en_us/article/pa8dw8/prime-number-pattern-mimics-crystal-patterns

    ...and they have all your passwords is belong to us now.

    1. Throatwarbler Mangrove Silver badge
      Coat

      Username checks out.

  19. Cynic_999 Silver badge

    Why the outrage?

    Both UK and US have been re-routing Internet traffic for decades in order to spy on their citizens, so it's a bit hypocritical to get all upset when someone else follows suit. (There is no technical reason for your Internet data to get routed to Menwith Hill).

    In any case, if this was really deliberate (and it may well not have been), then it was simply a denial of service attack rather than a spying mission because it stopped any TCP connection being made, and no data is sent unless and until the TCP handshake has completed. Had there been servers in Russia, China or Nigeria accepting the connection requests and pretending to be Google, then it would have been far more sinister.

  20. Sgt_Oddball Silver badge

    I've got the answer!

    Why don't we try using blockcha....

    At least let me get my coat, easy now, stop pushing at the back!

    1. Stevie Silver badge

      Re: I've got the answer!

      And I've been saying for years that we should reboot the internet to flush out all the warez and 4chans. I get downvoted every time but now look what's happened!

  21. Stevie Silver badge

    Bah!

    World War Three will not be televised. Or browsed, apparently.

    As I read the article, for some reason that Monty Python/Terry Gilliam "bit" with the German fish being eaten by the Japanese fish, which in turn was eaten by the British fish was running in Mr Brain.

    I may have the order of fish wrong, because the nationalities don't matter in the analogy.

    All our cat pix are belong to bad actors at the state level t'would seem. Then again, given the three-letter agency oversight it turns out *isn't* just a paranoid fantasy, that's been the case since, well, forever.

  22. Anonymous Coward
    Anonymous Coward

    ... with extreme prejudice ...

    As reported: Both hostnames have since stopped resolving to IP addresses....

  23. 10forcash

    So basically people are complaining that traffic destined for Google may be stored, analysed and used for nefarious purposes by someone other than Google?

    Google REALLY doesn't like competition does it?

    1. Cynic_999 Silver badge

      "

      So basically people are complaining that traffic destined for Google may be stored, analysed and used for nefarious purposes by someone other than Google?

      "

      Ignorant people may be complaining about that, but that was not what happened. The only significant traffic that was mis-routed would have been SYN (connect) attempts, which went unanswered. Whatever request or data the person trying to contact Google may have intended to send would thus have not been sent. To anywhere. So the only data that would have been obtained was the fact that a connection request was made to Google from a particular IP address at a particular time for an unknown purpose - hardly something that is likely to be of any concern.

      Possibly there was a bit of UDP data that got through (UDP does not require a connection be established before sending the payload - often used for live streaming video or audio for example), but in almost all cases there would be some sort of 2-way handshake before streaming the data over a UDP connection, so really no risk that Internet calls were being intercepted.

  24. Ken Mitchell

    Harden the Infrastructure

    Sounds like the major players need to harden their IT infrastructure and prevent such malicious actions.

  25. MaxK
    Alert

    Adventure

    A little known fact, but the creator of colosal cave - adventure, also worked on the distributed distance vector routing system used in RGP and later BGP.

    You are in a twisty little maze of routings, all different.

    https://en.wikipedia.org/wiki/William_Crowther_(programmer)

    http://rickadams.org/adventure/

    Max K.

    1. The Oncoming Scorn
      Joke

      Re: Adventure

      No relation to Leslie?

  26. jelabarre59 Silver badge

    Better perf...

    People's connections in the US to Google – including its cloud, YouTube, and other websites – were suddenly rerouted through Russia and into China in a textbook Border Gateway Protocol (BGP) hijack.

    I was wondering why Google's system performance seemed much better on Monday...

  27. Nifty

    Doesn't China already own half of Africa's infrastructure?

  28. jaypatelani

    OpenBGPD ftw. OpenBSD has nailed security very well :)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019