still happening @ 11:30
up & down like a whore's knickers this morning
mainly authentication errors
couldn't talk to my ISP as they are clearly using ip telephony :(
Brit internet and communications provider Entanet is slowing bringing its systems back to life today after they metaphorically keeled over last night. The Shropshire-based supplier of broadband, leased lines, telecoms and more has offered scant detail about its network outage, much to the chagrin of some of its biz and ISP …
Could you tell me which brand adsl modem do you have that works _properly_ with Enta's ipv6 allocation? It's manual (gave me a /56 and a gateway) and not DHCPv6, and my self-described "IPv6 capable" Zoom ADSL router/modem unfortunately (and after several chats & phone calls with Zoom) only supports DHCPv6 properly (allegedly). Any pointer - any - would be appreciated! my surname at gmail if you could :)
The router will need to support IPv6 Prefix delegation to work with Enta. Unfortunately, there are a few companies, including Cisco, that advertise some of their products as being IPv6 compatible, when they are not fully compatible. IOS routers are fully compatible, however their non-IOS devices are a bit of a crapshoot. As far as the modem itself goes, the draytek vigor 120 should work with IPv6. I was unable to test IPv6 with it, as the router I had didn't support IPv6 PD.
From what we saw last night the network was giving over 90% packet loss but didn't actually flatline. So auto failover to another provider didn't kick in and had to be done manually. All of which takes time to sort out.
Oh and they aren't "just an ISP" these guys sell data bandwidth in many forms as well as other "telco" services including colo. I can't see any of their corporates being too happy if they lost their private networks or access to their colo.
As for twitter, I would guess that people were probably taking to the 3G networks to complain.
Here in fancy Edinburgh, their graph (http://noc.enta.net/21cn-interconnect-status/) was at 0 flat.
In other places it had a bit of connectivity, just that tiny bit... here, nada. Zero. Zilch.
Had to try phoning their support more than 30 times to even get to receive the pre-recorded message saying there was a known fault and they were investigating. All the other calls either mysteriously disconnected or gave me the weird "There's been a fault" message.
Luckily it all came back ~8:20pm.
"I can't see any of their corporates being too happy if they lost their private networks or access to their colo."
If they want to bet their company on a small ISP thats up to them. If it was me I'd stick to one of the major backbone operators for my access. As for co-location, frankly, if you can't be bothered to maintain your own systems then you deserve what you get.
Leased lines and colos - not just a consumer ISP, thus the whinging.
And, also, they didn't just fall over - by the sound of it they were advertising routes they had no reason to advertise and the whole internet tried to send them traffic. Similar to the India YouTube thing that happened earlier - basically your traffic may have been routed through Entanet even if you've got nothing to do with them - not only while they were down, but actually being the CAUSE of them being down too.
It's like a guy with an Ethernet cable telling the world "Hey, you can all pass all your traffic through me!", except that they are big enough and trusted enough to not do stupid things like that.
This post has been deleted by its author
This post has been deleted by its author
Nice to be quoted in a Reg article there*! From their later updates, it seems the original problem was excess route entries overflowing the TCAM (fast lookup memory in big routers), which makes them fall back to a much much slower routing approach - which can't keep up with the level of traffic you see on an ISP backbone.
That was problem #1: the lab stress-test leaked out and flooded the live network, bringing it to its knees. (IPv6 itself was unaffected - it's routed separately anyway - but since the line is authenticated over IPv4, as soon as you disconnect, your authentication packets get lost, stopping the PPP link coming back up.) Like their autopsy says, filtering out the routes then re-booting the routers cleared the problem - then they discovered a Cisco line card was still misbehaving.
The wonky line card seems to have been problem #2: it seems to be working again now, but they're keeping most traffic away from that card until Cisco can figure out why it misbehaved in the first place.
* Apparently the router problem took out their VoIP lines and access to their own status system - downside to "eat your own dogfood" as an ISP, when your own services are down, you can't use them to communicate about the problem - normally, I've found Entanet to be very helpful and responsive. In this case, the live traffic graphs on noc.enta.net were enough to tell me it wasn't just my line that was dead - how many other ISPs give you that kind of detail?