Feeds

back to article Router glitch causes widespread net outages

Internet services throughout North America and Europe saw widespread outages and slowdowns on Monday after backbone provider Level 3 Communications suffered a global failure, network providers said. Time Warner Cable in the US, Research in Motion services for BlackBerry subscribers, and UK ISPs Eclipse Internet, Easynet, and …

COMMENTS

This topic is closed for new posts.
Anonymous Coward

Chuffing annoying

... when you're in the middle of a conf call with important clients over a SIP connection. :-(

2
0
Anonymous Coward

So don't buy Juniper, easy.

0
3

the thing is...

The thing is knowing which brand it is you shouldn't buy BEFORE it goes wrong.

2
0
Facepalm

@AC 20:54

Yeah, don't buy Juniper. And make sure that every network that you would traverse to reach any place you want also would not buy Juniper. And then hope that this only ever happens to Juniper. :-)

E.g. http://www.cisco.com/warp/public/707/cisco-sa-20100827-bgp.shtml

6
0
Silver badge

Don't buy....

...all your core products from a single supplier.

Makes it difficult to put together a coherent service, but at least when it comes to networking gear, prevents you from having the same flaw take out an entire layer of your infrastructure.

0
0
Pint

No problem here...

Our Internet is working jus

3
0
Joke

Candlejack?

You didn't even say Candlejack... That's supposed to

1
0
Bronze badge
Joke

If you mention

Candlejack and Hypnotoad in the same senALL GLORY TO THE HYPN-

0
0
Anonymous Coward

Serious here

No problems here at all that day!

0
0
Facepalm

Virgin Media Affected

Plenty of time-outs and go-slows with domestic VM cable broadband this afternoon. Naturally VM's status page showed everything was fine and dandy so I must have just imagined it all!

1
1
Meh

+1 here...

...on Virgin Media cable, though it started on Sunday for me, so not sure that it's the same problem. As with you, the VM service status site is telling me all is fine in my area. As usual.

0
0
Silver badge
Facepalm

Finally! Good to see 'em go completely casters-up

I've been having problems for 4 months with a Level 3 router in DC. Traceroute shows packets leave Orlando, hit the router in DC, and never make it back down to my power company in North Carolina. I have to VPN to work so I can pay my bill.

I tell Brighthouse and they ask if rebooting my computer fixes it.

1
0
Bronze badge

As always....

The internet is a little broken...

2
0
IT Angle

I wonder what really happened ?

"Internet services throughout the North America and Europe saw widespread outages and slowdowns on Monday after backbone provider Level 3 Communications suffered a global failure, network providers said".

|

How about a software upgrade failed to .. upgrade :) Else, I do believe this is a case of market forces at work as in they supply the minium connectivity with no redundencacy, so as when one node fails there is no backup. Or also the people running things now-a-days, don't know what they are doing. Level 3 bla bla blaaaa ... black sheep ....

|

"A core dump is when a serious error"

|

Do they not have procedure or fall-back for the system to by-pass such an unsheduled eventually, if not what the f**k are they earning their salaries for ?

1
3
Silver badge

what really happened..

..is probably what has been stated.

The whole 'net runs on BGP. If that gets corrupted, no amount of redundancy is going to fix it.

Especially if that redundancy involves the same core routers that already caused the problem.

0
0
Anonymous Coward

That's the trouble: building redundancy in case of hardware failure is easy. Building redundancy against software bugs is extremely difficult, because (a) it's much harder to recognise "bad behaviour" compared to "device has failed completely", and (b) it's the software which implements the redundancy anyway.

Maybe if you'd built two parallel backbones, one entirely out of Juniper and one entirely out of Cisco, you might have fared better. But even then, the two networks would have to be interconnected. If they are all in the same AS then they are going to have to trust each others' OSPF/ISIS/iBGP announcements. Having all those rebooting Juniper routers rebooting and injecting junk is not going to help the stability of the Cisco part.

And then which device do you connect your end-customers onto anyway?

0
0

catsrevenge

For crying out loud "Ladies", please !!! .. Get over it already, life goes on, business as usual now. One small, minor hiccup and everyone`s ready to blow a gasket? It`s done,over with, and behind us already. Move forward, Life is good !

0
1
Silver badge

It happens

So get over it - stuff breaks, code breaks, and even though we try to expect the unexpected, sometimes the unexpected arrives a little early. In general everything worked the way it's supposed to and the packets all arrived eventually - well, all the packets that really mattered anyway.

If you want to live in a world where everything always works and it's never your fault then go and join Steve.

0
2
Boffin

Effect != Cause

"A core dump happens when a device experiences a serious error that causes it to lose the contents in its main memory."

No, this is confusing cause and effect. A core dump is what the device creates when it encounters a serious error from which it is unable to recover. It 'dumps' the contents of its memory (the 'core', from the days of magnetic core memory) to non-volatile storage so that the developer can analyse it later in an attempt to identify and fix the cause of said error.

In the really old days, core dumps used to be helpful when we all tinkered with our operating systems and other low level software.

In the inbetween days, core dumps were mostly useful for mailbombing people who had annoyed you so that you filled up their disk quota. This has been frowned upon for some time, and indeed made illegal in many jurisdictions.

These days a core dump is a largely useless and symbolic gesture designed to make you feel less bad about your network going titsup by making you think someone, somewhere actually might care and try to fix their shonky code. They don't and they won't.**

If you're really lucky, you might be running some stack of O/S and applications configured in such a ways as to fill your non-volatile storage with core dumps if you ask it nicely. Having no free disk will then cause your system to encounter a serious error and then encounter an even more serious error when it tries to core dump and cannot do so.

** Not quite true, you can pay horrific support fees for high-end storage and network kit, and support people will quite happily take receipt of your core dumps, send them to some bloke in California or the Far East 'in the dev team', who will then ignore them and fail to fix their shonky code. You feel even more hopeful that someone will fix your problem eventually by virtue of having paid through the nose for the service, but it is still a lie.

0
0
WTF?

Reliable......

We should put all our data in the cloud including backups .......... My A*****

1
0

Head in the cloud

Bodes well for cloud compu

0
0
Anonymous Coward

Juniper advised of this issue months ago

Juniper advised of this issue months ago. All the major vendors have had BGP bugs like this - the fix is to improve the BGP standard and make it more resilient to glitches like this.

0
0

title

live with it, nothing is bug free. The internet is only a best endevers provider anyway, you want control, build your own.

0
0
Black Helicopters

Router Glitch?????????????

"MPCs (Modular Port Concentrators) installed in an MX Series router may crash upon receipt of very specific and unlikely route prefix install/delete actions"

Just the Chinese testing their secret hardware commands....

0
0

T800 prophecy

Skynet is becoming self-aware . . . .

0
0
This topic is closed for new posts.