back to article You won't guess where European mobile data was rerouted for two hours. Oh. You can. Yes, it was China Telecom

Yet another large interweb routing blunder has prompted internet engineers to stress the need for additional security at the network's foundational layer, and again raised eyebrows at the behavior of China Telecom. On June 6, more than 70,000 BGP routes were leaked from Swiss colocation company Safe Host to China Telecom in …

  1. Rupert Fiennes Silver badge

    Come off it

    I've dealt with China Telecom "mistakes" since 2006. It happens too often to be coincidence or incompetence.

    1. NoneSuch Silver badge
      Joke

      Re: Come off it

      Intern at the NSA patched the wrong panel.

    2. Alan Brown Silver badge

      Re: Come off it

      I've dealt with their mistakes for longer than that.

      Given the stuff I've encountered and the reactions to being told about it (ranging from "nothing" to changing the ports/ip on the boxes concerned, or simply blocking the emails) I'm more than willing to believe widespread incompetence (and unfireable management) rather than malice.

      One of the problems with entrenched nationalism is a kneejerk hostile response to foreigners saying "you have a problem with this" - something not unique in any way to China but more pronounced in quasi-military and large-scale bureaucratic structures.

      Low level US military network admins didn't take kindly to being told XYZ system was spewing all over the net either - but in that case there was an established path to go up the food chain, get things fixed and "deal with" the admin concerned. The same path was able to be used to deal with federal/state/local gov employees who preferred to dickwave than fix things.

      China (and a bunch of other countries) needs the same kind of escalation ability. The same things keep happening because the same people are in the same places in organisations, getting no heat for screwing up and generally not implementing changes - for the most part because it's "not invented here"

      1. Anonymous Coward
        Anonymous Coward

        Re: Come off it

        In my experience of dealing with, and working for, Chinese companies, escalating problems is basically impossible due to the culture. Telling your boss something is wrong is tantamount to an admission of failure or incompetence, even if it isn't your fault. So nobody does it.

        At least that's how it looked to me as a westerner looking in.

        1. Alan Brown Silver badge

          Re: Come off it

          "Telling your boss something is wrong is tantamount to an admission of failure or incompetence, even if it isn't your fault. So nobody does it."

          Yup. Exactly this - and we had _exactly_ the same problem and the same responses in both Japan and Korea.

          It turned out the solution in both countries was to find ways of politely bringing it to senior management attention in a way that couldn't be ignored and then passing it to one of the local media outlets if it was - because media in both countries took great delight in showing up such failings - guess who took the heat for THAT? A few such incidents and notifications from JP-CERT tended not to be ignored, although I suspect a few admins ended up looking for new jobs.

    3. Anonymous Coward
      Anonymous Coward

      Re: Come off it

      A real thief does not get caught. A clever con is out in the open.

      I'm not sure this matches either.

      1. Alan Brown Silver badge

        Re: Come off it

        "A real thief does not get caught. A clever con is out in the open."

        The best cons are pulled off by offering payoffs in the next life. They have people flocking to give the scammer money and violently attacking anyone who questions or points out the scam.

        Some of the biggest such cons have managed to pull off being tax exempt.

    4. Anonymous Coward
      Anonymous Coward

      Re: Come off it

      "I've dealt with China Telecom "mistakes" since 2006. It happens too often to be coincidence or incompetence."

      I'm pretty sure there error rate is still well below SingTel though...or are we just "unlucky"?

    5. sabroni Silver badge

      Re: It happens too often to be coincidence or incompetence.

      I love how you qualified that with examples to show that China Telecom is worse than others, rather than just making a vauge accusation.

  2. LDS Silver badge
    Joke

    That's what happens when you use a Huawei router....

    ... any mistake routes everything to China Telecom.

    No need for backdoors, just wait for a fat-fingered admin.

    1. Headley_Grange Silver badge

      Re: That's what happens when you use a Huawei router....

      Is China Telecom better and cheaper than British Telecom? I'm thinking of changing ISP.

      1. Joseba4242

        Re: That's what happens when you use a Huawei router....

        China Telecom is much better. British Telecom will do silly things disrupting your internet connection such as filtering your prefixes.

      2. Roj Blake Silver badge

        Re: That's what happens when you use a Huawei router....

        China Telecom is a lot better - you'll never be bothered by Facebook ever again.

        1. Flywheel Silver badge

          Re: That's what happens when you use a Huawei router....

          Yeah, and Google won't be spying on you either!

      3. trisul

        Re: That's what happens when you use a Huawei router....

        China Telecom is the best, because you know the Chinese Army has a full backup of all your files.

      4. TimMaher Bronze badge
        Happy

        Re: That's what happens when you use a Huawei router....

        You already have.

      5. Anonymous Coward
        Anonymous Coward

        Re: That's what happens when you use a Huawei router....

        "Is China Telecom better and cheaper than British Telecom?"

        Almost everything is cheaper than BT.

        As for better - it depends. It's possible for BT to be OK in that their infrastructure is pretty solid and their cable routes tend to be less likely to be disrupted than other vendors in my experience.

        But, I find that BT support and the majority of their change teams are likely to result in a reduction in your willingness to consider all men and women as equals. And likely result in work not being completed on schedule when they discover the service they documented isn't correctly documented or the "engineers" struggle to comprehend differences between numbers and letters or that taking out three independent services simultaneously in spite of their being no shared equipment/lines breaks resilience...

        As such, unless your requirement is for no changes ever and you have so much tested resilience that any minor operational issue that you are likely to encounter avoids the need for you to interact with BT, then you may find them acceptable. Other than knowing you can probably get the same service for less else where.

        That's the most positive recommendation for British Telecom that I can manage...

        (this assumes business level services - xDSL/cable/other home services tend to either work or result in pain regardless of provider. It's only the level of pain that varies and whether it is on the account management/billing side or service side or both)

        1. Anonymous Coward
          Anonymous Coward

          Re: That's what happens when you use a Huawei router....

          Ah, you've met them too. I remember an RFS I put out - 80 page response from 1 vendor, 40 page from another and, copied from the back of an envelope no doubt, an email with a price (lower than other 2), a timeline, (shorter than the other 2) and no breakdown or work product description

      6. MOV r0,r0

        Re: That's what happens when you use a Huawei router....

        China Telecom is delicate and fragile, BT is oven-safe and dishwasher-proof.

        1. Anonymous Coward
          Anonymous Coward

          Re: That's what happens when you use a Huawei router....

          That's because BT only provide you with mugs

      7. 080

        Re: That's what happens when you use a Huawei router....

        At least you won't get idiot scammers pretending to be from China Telecom

  3. Down not across Silver badge

    Peering

    The BGP leak this month was likely a simple mistake but China Telecom appears to have made the most of it. And that has sparked internet engineers to again press their colleagues to adopt better security measures on this critical underlying internet infrastructure.

    Maybe it is time to start requiring adoption of MANRS or equivalent as a pre-requisite for peering. Lax handling of BGP, no peering for you.

    1. Alan Brown Silver badge

      Re: Peering

      "Maybe it is time to start requiring adoption of MANRS or equivalent"

      It's well overdue for that - and to start threatening automatic _de_peering of networks who spew regularly or for prolonged periods until they implement it.

      FWIW: If you think BGP is bad, the world's telephone routing protcols are similar to BGP and have even LESS concept of network security.

      The assumption is that anyone who can plug into the phone networks at that level is trustable - which has led to some "interesting" phone prefix hijacking over the years (such as blocks of Niue and Chile unallocated area codes being used for porn lines answered in London whilst charged the full international termination rates to clients)

    2. Roland6 Silver badge

      Re: Peering

      Was there really a leak?

      If China Telecom were Safe House's peers then surely there is an expectation that some traffic "destined for European netizens " will be routed over China Telecom's european network. The only question is whether China Telecom rerouted that traffic and tromboned it (over their network) to China...

  4. Anonymous Coward
    Anonymous Coward

    I'll say it again....

    I believe that China has been testing out their cybersecurity options in case they are pushed into a corner.

    The US is already compromised by millions of low-budget Android devices that at a push of a button could be rooted remotely by pushing malicious ads and then installing an app similar to the open source "cSploit" app that could cause chaos by manipulating DNS, arp spoofing, man-in-the-middle attacks and more.

    https://github.com/cSploit

    https://arstechnica.com/information-technology/2017/03/preinstalled-malware-targets-android-users-of-two-companies/

    China already has their Great Firewall and just recently created a law that any computer network in China can be pen-tested by the government.

    Russia just passed a measure to create their own Great Firewall as well.

    https://www.theguardian.com/world/2016/nov/29/putin-china-internet-great-firewall-russia-cybersecurity-pact

    And both Russia and China have been buying up any gold reserves they can get their hands on.

    https://www.marketwatch.com/story/why-china-and-russia-are-buying-so-much-gold-2016-08-01

    The writing is on the wall.

    1. Wellyboot Silver badge

      Re: I'll say it again....

      Pen-testing is a good way to improve netywork security.

      Is the pen-testing free? Do they give you all of the results? Do they offer to fix?

      Do you trust the Chinese Goverment?

      Do you trust your Goverment? :0)

      1. Flywheel Silver badge
        Black Helicopters

        Re: I'll say it again....

        UK reader here ...

        Do you trust the Chinese Goverment? Do you trust your Goverment?

        I trust the Chinese Government in that they say they will explicitly hack any machine they can gain access to.

        I don't trust what passes for a Government in the UK when they say that "work hard to keep us safe from online threats and harms". My firewall logs [allegedly] state that the opposite is true.

    2. Louis Schreurs

      Re: I'll say it again....

      Any trump / wall jokes ?

    3. Chris G Silver badge

      Re: I'll say it again....

      The Firewalls are to a large degree in response to US belligerence and hardly surprising that countries the US would like to exclude from the rest of the world would want their own self contained internets if the worst happened.

      The potentially sinister side of that is obvious and some of the directions control and censorship in the 'Free world™' are also potentially sinister but maybe less obvious.

      As for buying gold, all or most of the central banks are in the market as a hedge against unlimited QE and money printing by the States.

      1. Anonymous Coward
        Anonymous Coward

        Re: I'll say it again....

        "The Firewalls are to a large degree in response to US belligerence and hardly surprising that countries the US would like to exclude from the rest of the world would want their own self contained internets if the worst happened."

        For the Russian firewalls, are they really down to US belligerence or is it Russia's attitude towards the rest of the world? Russia has realised the importance of large scale hosting/cloud solutions without really developing it's own significant player in the market. To prevent US/Chinese companies filling this role and making Russia vulnerable in a future international crisis, the firewalls are a step to force Russian companies to avoid an over-dependence on the Internet. There will be areas where over-dependence can't be helped (outsourced IT as an obvious example) but at lease it allows Russia to choose/push companies away from services that may cause significant issues. Whether this stance is justified based on history/politics/the Russian political mindset etc is left as an exercise for the reader but I believe, driven by externalities.

        The Chinese (and similarly the Middle Eastern countries using similar solutions) are largely around controlling information and driven by internal policies around controlling their respective populations and their populations access to information rather than being driven by concerns about the activities of other countries. For China, I'd point to Winnie the Pooh as an example of this type of policy.

        1. Alan Brown Silver badge

          Re: I'll say it again....

          "For the Russian firewalls, are they really down to US belligerence or is it Russia's attitude towards the rest of the world? "

          How long do you think any of these terrestrial firewalls are going to be effective for when you have not just Elon's Skynet in operation but several more providing resliience?

          You can already send limited quantities of data via Iridium and other non-dish-based satellite services - it's how a lot of the stuff showing the Rohingya massacres got out of Burma in areas where the army would have pounced in minutes had they seen dishes pointing anywhere or anything resembling the usual videocomms kit. As it was, they triangulated on and killed several of the journalists by matching locations in the actual video footage.

          It works the other way too - the USA's answer to the "great firewalls" has been to allow legislated monopolies which get away with both choking the living daylights out of connections, making them too expensive for most consumers and making it harder for middle america to reach neutral news sources (If you've ever spent time there you'll know that the average middle-american newspaper might have at best 3-4 pages of out-of-state news, with half a page of international news. Parochial is somehow not quite enough to describe it)

    4. Anonymous Coward
      Anonymous Coward

      Re: I'll say it again....

      repeatedly, to my own face in the mirror, every night.

      the foreigners are coming

      I'll say it again

      the foreigners are coming

      They'll rape your wife and steal your job and break your internet

      BUILD A WALL!!!!!!

      1. Anonymous Coward
        Anonymous Coward

        don't bother with a wall

        just turn your country into such a hellhole that only the really desperate would even try to get in

        #HostileEnvironment

    5. Anonymous Coward
      Anonymous Coward

      Re: I'll say it again....

      Jusding by the dates on those articles, this is less insight/foresight and more poorly understood explanations of historical events.

      For cSploit, do you understand what it is used for? Are you really suggesting that millions of Android phones will be used to run pen testing software at some point in the future rather than just utilising existing systems (compromised or otherwise) to do the same task now rather than waiting for the magic button to be pressed?

      For mobile devices with pre-installed malware, we've known about targetted attacks by years and the NSA has been caught with the tools to do it. While I'm not suggesting China can't do it, the article you list doesn't point fingers at who did do it or the victims, making it difficult to work out the likely attacker.

      For gold - while it's generally a safe harbour in uncertain times, their are other reasons for two of the worlds three biggest gold producers to be buying. Based on this chart for prices during 2016, I would suggest they were supporting the price between June and November for their own benefit: https://www.bullionbypost.co.uk/gold-price/gold-price-2016/

      For the Russian firewalls, it's largely down to Russian fears (potentially justified) of being cutoff from the rest of the Internet and wanting to ensure that Russia's internal infrastructure works where possible. Russia haven't got a Baidu/Alibaba scale company yet, so have had a tendency to rely on AWS/Azure/Google and hence the concern and firewalls.

      For you primary point:

      "I believe that China has been testing out their cybersecurity options in case they are pushed into a corner."

      I don't believe this point is any more or less valid than it has been over the last 15 or so years since the move to mobile devices. Bad actors (both state-sponsored and independent) actively look for vulnerabilities or misconfigurations and exploit them. Even 15 years ago, it wasn't new but advances in mobile devices and our reliance on them has meant the value of those targets has risen if you can pinpoint your attacks.

      The big difference between attacks 15 years ago and attacks now is that rerouting traffic via a BGP leak would result in >70% of the traffic being encrypted (Fortinet says >75%, Google says >90%) while in 2007 I suspect less than 20% was encrypted and the encryption was relatively weak (3DES/MD5 vs AES128/SHA2 or higher).

  5. swm Bronze badge

    In the early days of the ARPANET (redacted) was doing traffic analysis and discovered an undocumented NSA node on the network. These things are not new.

    1. LDS Silver badge
      Devil

      Well, ARPANET was funded by the Department of Defense.... why they shouldn't play with it too?

  6. sanmigueelbeer Silver badge
    Pint

    China to US: You "divert" our FedEx package, we divert the net. Kapish?

  7. naive

    What are the risks of these BGP errors

    Trust and cooperation was a design principle of the Internet.

    As far as I understand things, bad routing just introduces extra latency.

    Traffic that should be confidential is encrypted, and can not be decrypted (yet).. right ?.

    So except from collecting meta information, like insight in traffic streams between ip adresses, there shouldn't be much to gain from broadcasting erroneous routing information by countries interested in analyzing the internet traffic of other countries.

    1. Anonymous Coward
      Anonymous Coward

      Re: What are the risks of these BGP errors

      So except from collecting meta information .... there shouldn't be much to gain

      Oh dear. Oh very dear.

      You do know just how much intelligence can be collected from "just" metadata ? (a factoid that, incidentally, is one of the reasons that the constant Western govt screaming for full decrypt ability is so suspect)

      1. Roland6 Silver badge

        Re: What are the risks of these BGP errors

        So except from collecting meta information .... there shouldn't be much to gain

        Oh dear. Oh very dear.

        Clearly 'naive' would benenfit from reading Gordon Welchman's book "The Hut Six Story"...

      2. Anonymous Coward
        Anonymous Coward

        Re: What are the risks of these BGP errors

        The challenge with getting metadata is that you need time to see patterns to services - 2 hours isn't a great deal of additional metadata.

        Looking at our netflow data that we use for determining long term service usage patterns, it takes a few weeks to get a good idea of what a user is doing from metadata if they use a service everyday. ie. when they start and finish work, take lunch etc. for a service that is used daily. For data that you have little idea about before it is redirected, I suspect you would want longer than a BGP attack would allow.

    2. Claptrap314 Silver badge

      Re: What are the risks of these BGP errors

      And this is why things are so f*cked up in at least the first four levels.

      We CANNOT trust 7 billion people. We CANNOT choose which of those 7 billion people are attempting to get on the net with us.

      Just no.

  8. Korev Silver badge
    Joke

    Culprit

    Was it Ping who did it?

    1. Arthur the cat Silver badge

      Re: Culprit

      Was it Ping who did it?

      I'm not sure ducks know BGP.

    2. Rupert Fiennes Silver badge

      Re: Culprit

      I used to know a very nice Chinese girl called Ping: I can't believe she was responsible :-)

    3. Louis Schreurs

      Re: Culprit

      I’m pretty sure it was Pong.

  9. Nick Kew Silver badge

    Hmmm...

    "It should be noted that the United States remains the number one source of BGP errors ... but when BGP leaks have been flagged as potentially suspicious there has been a persistent connection to Chinese and Russian operators,"

    Chicken and egg? If a leak having a Chinese or Russian connection tends flags it as "potentially suspicious", the above will naturally follow.

    Mine's a Welsh leek, please. They make a nice soup.

    1. Rameses Niblick the Third Kerplunk Kerplunk Whoops Where's My Thribble? Silver badge

      Re: Hmmm...

      I came to post basically this comment, albeit without the leek observation. Given the weather we've had recently, the only leaks I'm worried about are the ones around the sunroof of my old Ford.

  10. imanidiot Silver badge
    Black Helicopters

    A fuckup is a fuckup, a hack's a hack

    I have no doubt many of these incidents are just fuckups. "Never attribute to malice that which can be adequately ascribed to stupidity". However, a fuckup this large requires either a special kind of stupid or malice and the latter starts being the more believable explanation. Generally though I have a hard time believing the problems caused by China or Russia are all (or partly) caused by malicious effort, and all the incidents coming from the US are entirely and purely accidental. I don't believe for a second the TLA's wouldn't use this sort of attack. --> Is that a Blackhawk I hear approaching?

    1. steelpillow Silver badge
      Holmes

      Re: A fuckup is a fuckup, a hack's a hack

      Except, most exploits involve social engineering - a fuckup (whether spontaneous or phished for) smartly exploited by a hack.

    2. Alan Brown Silver badge

      Re: A fuckup is a fuckup, a hack's a hack

      "However, a fuckup this large requires either a special kind of stupid"

      Never underestimate the stupidity of people in sufficiently large groups - particularly where there are rigidish social structures.

      There are more societies where copilots will sit and watch the captain totally screw up and fly a large passenger aircraft into the ground and be afraid to intervene than ones where the crew will scream bloody murder and take over the controls - in fact such cultures have repeatedly happened in corporate america too (including at least one US airline!)

  11. Anonymous Coward
    Anonymous Coward

    Physical bandwidth restrictions

    In a project, we wished to enforce a bandwidth limitation to ensure that some streams didn't go down a certain route. We simple wired-up only four of eight Ethernet wires on those paths, thus forcing the chips to negotiate a lower bandwidth connection at the hardware level. Guaranteed that the high bandwidth streams couldn't fit.

    Another approach is to set TTL to be just enough for the expected route.

    1. Anonymous Coward
      Anonymous Coward

      Re: Physical bandwidth restrictions

      This system is needed to ensure that the data gets from point A to point B. It was actually working correctly- even with your implementation there'd need to be some slack to allow it to change routes if, say, a cable was cut (which is not exactly a rare event). It was just misconfigured.

    2. Anonymous Coward
      Anonymous Coward

      Re: Physical bandwidth restrictions

      And would you be able to do this if the majority of the switches/routers/cables involved were not managed by you?

      I am guessing CT are connected to DE-CIX at Frankfurt via a handful of ports. There are >850 other users connected to DE-CIX (according to Wikipedia), the majority of which will be utilising a great deal more bandwidth than CT has total capacity.

      For the TTL approach, how many hops should you count on to reach your destination? Should you allow for multiple paths and redundancy? Are you just sending traffic to a single destination?

  12. Anonymous Coward
    Anonymous Coward

    from where I'm sitting

    Swiss company owes china telecoms an apology.

    1. Roland6 Silver badge

      Re: from where I'm sitting

      Does China Telecom get paid by the Mbyte and hence this accidential rerouting of traffic had financial benefits...

      1. json

        Re: from where I'm sitting

        No that's not how peering works.. peering is reciprocal traffic either party can max out the peering bandwidth without thinking about "charges".

  13. Rich 2

    Simple fix

    if (route update pointing to China Telecom) {

    If (existing route is currently pointing to somewhere in (say) Europe) {

    igore route update

    }}

    Jesus! It's not rocket science (and yes, of course it's a tad more complicated that this, but even so)

    1. Anonymous Coward
      Anonymous Coward

      Re: Simple fix

      You should realise that China Telecoms network isn't just a conduit into China.

      It's a peer 1 provider just like many other multinational companies.

      Jesus, it's not rocket science.

      https://www.chinatelecomeurope.com/global-network/

    2. Anonymous Coward
      Anonymous Coward

      Re: Simple fix

      "and yes, of course it's a tad more complicated that this, but even so"

      If by "a tad more complicated" you mean you are completely wrong and your pseudocode is meaningless as a representation of a regional tier-1 provider that doesn't cover transit, resiliance, load balancing or multihoming.

      There are two likely causes:

      a) customer-based: a China Telecom customer who had purchased transit capabilities accidentally fed other providers routing information into its own announcements and China Telecoms filtering failed to detect and block the issue

      b) CT-based: i.e. China Telecoms transit policy incorrectly readvertised a peers routes for transit to all of Europe and Cina when they should have only been used for transit to China.

      Based on https://blog.apnic.net/2019/06/07/large-european-routing-leak-sends-traffic-through-china-telecom/ it was option (a)

      1. Rich 2

        Re: Simple fix

        "...a tad more complicated that this"

        Oh dear. I didn't add the sarcasm icon did I?

        1. Anonymous Coward
          Anonymous Coward

          Re: Simple fix

          Sarcasm? Your suggested fix shows you don't understand the issue or how BGP works for inter-AS traffic..

          But sure, let's call it sarcasm.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019