back to article Router glitch causes widespread net outages

Internet services throughout North America and Europe saw widespread outages and slowdowns on Monday after backbone provider Level 3 Communications suffered a global failure, network providers said. Time Warner Cable in the US, Research in Motion services for BlackBerry subscribers, and UK ISPs Eclipse Internet, Easynet, and …

COMMENTS

This topic is closed for new posts.
  1. Anonymous Coward
    Anonymous Coward

    Chuffing annoying

    ... when you're in the middle of a conf call with important clients over a SIP connection. :-(

  2. Anonymous Coward
    Anonymous Coward

    So don't buy Juniper, easy.

    1. Sam Liddicott

      the thing is...

      The thing is knowing which brand it is you shouldn't buy BEFORE it goes wrong.

    2. Peter Rathlev
      Facepalm

      @AC 20:54

      Yeah, don't buy Juniper. And make sure that every network that you would traverse to reach any place you want also would not buy Juniper. And then hope that this only ever happens to Juniper. :-)

      E.g. http://www.cisco.com/warp/public/707/cisco-sa-20100827-bgp.shtml

      1. Peter Gathercole Silver badge

        Don't buy....

        ...all your core products from a single supplier.

        Makes it difficult to put together a coherent service, but at least when it comes to networking gear, prevents you from having the same flaw take out an entire layer of your infrastructure.

  3. JeffyPooh Silver badge
    Pint

    No problem here...

    Our Internet is working jus

    1. Peter Rathlev
      Joke

      Candlejack?

      You didn't even say Candlejack... That's supposed to

      1. Anonymous Coward 15
        Joke

        If you mention

        Candlejack and Hypnotoad in the same senALL GLORY TO THE HYPN-

    2. Anonymous Coward
      Anonymous Coward

      Serious here

      No problems here at all that day!

  4. Ian McNee
    Facepalm

    Virgin Media Affected

    Plenty of time-outs and go-slows with domestic VM cable broadband this afternoon. Naturally VM's status page showed everything was fine and dandy so I must have just imagined it all!

    1. Paw Bokenfohr
      Meh

      +1 here...

      ...on Virgin Media cable, though it started on Sunday for me, so not sure that it's the same problem. As with you, the VM service status site is telling me all is fine in my area. As usual.

  5. Gene Cash Silver badge
    Facepalm

    Finally! Good to see 'em go completely casters-up

    I've been having problems for 4 months with a Level 3 router in DC. Traceroute shows packets leave Orlando, hit the router in DC, and never make it back down to my power company in North Carolina. I have to VPN to work so I can pay my bill.

    I tell Brighthouse and they ask if rebooting my computer fixes it.

  6. ian 22

    As always....

    The internet is a little broken...

  7. Microphage
    IT Angle

    I wonder what really happened ?

    "Internet services throughout the North America and Europe saw widespread outages and slowdowns on Monday after backbone provider Level 3 Communications suffered a global failure, network providers said".

    |

    How about a software upgrade failed to .. upgrade :) Else, I do believe this is a case of market forces at work as in they supply the minium connectivity with no redundencacy, so as when one node fails there is no backup. Or also the people running things now-a-days, don't know what they are doing. Level 3 bla bla blaaaa ... black sheep ....

    |

    "A core dump is when a serious error"

    |

    Do they not have procedure or fall-back for the system to by-pass such an unsheduled eventually, if not what the f**k are they earning their salaries for ?

    1. itzman

      what really happened..

      ..is probably what has been stated.

      The whole 'net runs on BGP. If that gets corrupted, no amount of redundancy is going to fix it.

      Especially if that redundancy involves the same core routers that already caused the problem.

    2. Anonymous Coward
      Anonymous Coward

      That's the trouble: building redundancy in case of hardware failure is easy. Building redundancy against software bugs is extremely difficult, because (a) it's much harder to recognise "bad behaviour" compared to "device has failed completely", and (b) it's the software which implements the redundancy anyway.

      Maybe if you'd built two parallel backbones, one entirely out of Juniper and one entirely out of Cisco, you might have fared better. But even then, the two networks would have to be interconnected. If they are all in the same AS then they are going to have to trust each others' OSPF/ISIS/iBGP announcements. Having all those rebooting Juniper routers rebooting and injecting junk is not going to help the stability of the Cisco part.

      And then which device do you connect your end-customers onto anyway?

  8. catsrevenge

    catsrevenge

    For crying out loud "Ladies", please !!! .. Get over it already, life goes on, business as usual now. One small, minor hiccup and everyone`s ready to blow a gasket? It`s done,over with, and behind us already. Move forward, Life is good !

  9. Version 1.0 Silver badge

    It happens

    So get over it - stuff breaks, code breaks, and even though we try to expect the unexpected, sometimes the unexpected arrives a little early. In general everything worked the way it's supposed to and the packets all arrived eventually - well, all the packets that really mattered anyway.

    If you want to live in a world where everything always works and it's never your fault then go and join Steve.

  10. Amonynous
    Boffin

    Effect != Cause

    "A core dump happens when a device experiences a serious error that causes it to lose the contents in its main memory."

    No, this is confusing cause and effect. A core dump is what the device creates when it encounters a serious error from which it is unable to recover. It 'dumps' the contents of its memory (the 'core', from the days of magnetic core memory) to non-volatile storage so that the developer can analyse it later in an attempt to identify and fix the cause of said error.

    In the really old days, core dumps used to be helpful when we all tinkered with our operating systems and other low level software.

    In the inbetween days, core dumps were mostly useful for mailbombing people who had annoyed you so that you filled up their disk quota. This has been frowned upon for some time, and indeed made illegal in many jurisdictions.

    These days a core dump is a largely useless and symbolic gesture designed to make you feel less bad about your network going titsup by making you think someone, somewhere actually might care and try to fix their shonky code. They don't and they won't.**

    If you're really lucky, you might be running some stack of O/S and applications configured in such a ways as to fill your non-volatile storage with core dumps if you ask it nicely. Having no free disk will then cause your system to encounter a serious error and then encounter an even more serious error when it tries to core dump and cannot do so.

    ** Not quite true, you can pay horrific support fees for high-end storage and network kit, and support people will quite happily take receipt of your core dumps, send them to some bloke in California or the Far East 'in the dev team', who will then ignore them and fail to fix their shonky code. You feel even more hopeful that someone will fix your problem eventually by virtue of having paid through the nose for the service, but it is still a lie.

  11. Anonymous Coward
    WTF?

    Reliable......

    We should put all our data in the cloud including backups .......... My A*****

  12. Watashi

    Head in the cloud

    Bodes well for cloud compu

  13. Anonymous Coward
    Anonymous Coward

    Juniper advised of this issue months ago

    Juniper advised of this issue months ago. All the major vendors have had BGP bugs like this - the fix is to improve the BGP standard and make it more resilient to glitches like this.

  14. Chris 211

    title

    live with it, nothing is bug free. The internet is only a best endevers provider anyway, you want control, build your own.

  15. Field Marshal Von Krakenfart
    Black Helicopters

    Router Glitch?????????????

    "MPCs (Modular Port Concentrators) installed in an MX Series router may crash upon receipt of very specific and unlikely route prefix install/delete actions"

    Just the Chinese testing their secret hardware commands....

  16. BoxedSet

    T800 prophecy

    Skynet is becoming self-aware . . . .

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019