Virgin Media struck dumb by NATIONWIDE DNS outage Virgin Media customers were kicked offline last night, after a DNS blunder from the cable company disrupted its network on Tuesday. It comes after rival BT suffered a similar, widespread technical cockup over the weekend. Virgin Media, which is owned by US telecoms giant Liberty Global, coughed to a "nationwide DNS outage" on …
"All your data are belong to us"
Sorry to be finicky (ha, as if, this is El Reg), to follow the meme correctly, shouldn't that be "All your datum are belong to us."
Ahem. Aaaand back to the story... DNS availability does seem to be Virgin Broadband's major weakness in my experience. Throughput and latency over cable is generally good, but DNS response times and/or availability seems to vary from just about acceptable to utter crap on a day to day basis.
Were they activating their new link to GCHQ this time?
Actually, no, data is actually going to Maryland, and has been doing so for a while. That it breaks UK Data Protection with it apparently doesn't matter to its US owner:
;; QUESTION SECTION:
;virginmedia.com. IN MX
;; ANSWER SECTION:
virginmedia.com. 3220 IN MX 1 aspmx.l.google.com.
virginmedia.com. 3220 IN MX 5 alt1.aspmx.l.google.com.
virginmedia.com. 3220 IN MX 5 alt2.aspmx.l.google.com.
virginmedia.com. 3220 IN MX 10 aspmx2.googlemail.com.
virginmedia.com. 3220 IN MX 10 aspmx3.googlemail.com.
Even when equipped with barge pole, do not touch.
My VM service continued to work last night. No DNS problems here. However, I did get problems last Thursday that were very similar to what was reported for BT at the weekend.
So yes, I'm suspicious. There does seem to be a systematic outbreak of this problem and a failure to state the cause doesn't help.
"...Virgin Media's status pages claim everything is fine..."
That's pretty par for the course with VM.
Sometimes even the call handler will be saying there's nothing wrong, and have you tried switching it off and on again, etc. And later you find out that the whole area has a problem.
Yeah - those pages are about as much use as paraffin wax teapot* - a static page showing Cnut** holding back the waves with an "Everything's fine" speech bubble would be equally accurate.
*It's been pointed out to me often that you could eat a chocolate teapot, thereby providing some good use.
**Preferably mispelled
during a long (>48 hours outage) and after several calls over two days with the 'help' desk, who could only say there was no known issue or resolution time (and kept trying to get me to check my network settings, grrrr), i was finally passed to second level support, who instantly replied that "oh, it's a planned outage for network changes, service will resume at xx:yy" which it did
i) planned, so why didn't they email affected customers in advance?
ii) planned, so why didn't they put it on the status page, phone status etc.?
iii) planned, so why didn't they tell their own help desk people?
iv) planned, in what way?
Yep their status page is awful.
Whenever I have an internet outage, I check the status page from my phone and it usually says something like "Phone - OK, Internet - OK, TV - There are reported problems with TV service in your area". Despite the fact that I don't have TV with them and it's my Internet that's down I can only assume "Internet" and "TV" are linked in some way, and only bother contacting the helpdesk if all three services are listed as OK.
I just love the status pages.
I just love that they offer to send you an email when an issue is resolved.
I just love that it was email that was down, and they offered to send me an email when it was fixed... (I leave the reader to do the facepalm)
(yes, yes, I know, I could have used a different email address if I had one, or if I had push email it should have come through to my phone or I could of chosen to receive an SMS. But where would the Virgin bashing fun be in that)
Here's how I didn't notice the outage, in case anyone's interested.
I run a small Linux server in my loft (I use a Raspberry Pi for its low power consumption, but anything will do).
It runs a DHCP server and a DNS server. Neither is difficult to configure. Superhub is in normal mode, with DHCP switched off.
In the DNS server, I have a config file which redirects all spam, ads or anything else I don't like, such as social media bollocks. My LG TV no longer tries to advertise things to me (still annoyed about that - never buy LG) and nor does my phone, at least when I'm in the house.
Anything in the house has a DNS entry. Anything else gets forwarded to opendns.
Still have to use UKbay to get round the pathetic attempt at blocking piratebay though.
It does vary from distro to distro. You need to set up dhcpd and bind. I use slackware and that basically means installing dhcpd and bind packages, changing /etc/rc.d/rc.dhcpd and /etc/rc.d/rc.bind to be executable, editing the config files, and starting them.
There's probably a bit too much to post here, but this post: http://topchan.info/your-own-adblocking-dns will give you some pointers.
On debian (and therefore probably ubuntu) at least, a local caching DNS server is a simple matter of installing bind9 and editing /etc/bind/named.conf.options. You can put whatever DNS servers you want in the forwarders section like so:
forwarders {
// ISP DNS servers go here
2.3.4.5;
6.7.8.9;
10.11.12.13;
};
Restart bind and you're done. I use OpenNIC DNS servers with Level 3 as a backup.
If you want a simple single subnet DHCP server, just install udhcpd and you're done. ISC dhcpd is a little more involved.
You can combine both in a single piece of software by using the excellent dnsmasq (which has the added bonus of providing DNS names for you local network automatically) although it's harder to set up than the above bind example - but simpler overall.
That would be why I never noticed any DNS problems then.
Definite fail by VM for locking the Superhub DNS away from the users.
I have heard that the reason is so that they don't have to support any user change to the DNS but since the third thing* they ask you to do is factory reset the router I don't see the point.
* First thing - reboot your computer, second thing reboot the router.
Annoyingly there is no way to change the Superhub's DNS settings, so I could only fix the PCs and not my wife's phone or the set-top box.
I spent 40 minutes on hold to find that out nugget of information.
At least "Cable Modem mode" is easy to set up, just not something I felt like doing during an outage.
I've been with VM for BB for years, but over the last couple of years the DNS provided by them has been getting worse and worse.
When I switched to public DNS provides like google or OpenDNS, the broadband now works really fast.
So why cripple you fast broadband with pathetic DNS? Most users will not ever change their DNS, and will just say "My Broadband is shit"
We've been on Virgin for 4+ years and it took 15 seconds to realize their DNS is #epicfail fsck'd.
We use a variety of open DNS, plus known good DNS servers and never ever rely on Virgin's DNS. The fact their router does is immaterial as all of our home hosts have hard coded DNS entries so we can avoid Virgin's DNS fsckery.
It went down on Friday.
http://community.virginmedia.com/t5/Up-to-152Mb/DNS-Issues/td-p/2361435
Also the service is getting worse in general, my area is waiting now almost 2 years for a resegmentation and all I get is usual bullshit about how they are sorry and the work takes a long time to plan.
If the Virgin provided router you're talking about is the SuperHub and it's in modem mode, and you use a decent router of your own, the option is there.
I didn't notice the problem for the obvious reason. I just wish my supposedly 'up to 152Mbs' connection showed me a bit more oomph than the 40-50Mbs I usually get1 - consistently from speed test sites.
When the Virgin Media guy came out to do the installation and pointed out I wouldn't need my existing router, I said I'd be putting the SuperHub straight into modem mode so I would need my router. His reply was a small laugh and the comment "Each to their own" as if what I said was crazy talk.
1. To be fair to them, though, looking at the Superhub's status, I see it says the maximum speed is something around 160Mbs, so maybe I should replace the old/cheap cable running from it to my router before complaining. Just in case it's not up to the job.
Well, quite. If you use a router that has the option then you'll have the option.
What if you're a "normal person"?
Normal people use the equipment provided by the ISP for the purpose. They don't buy a router because they got given one "free".
So if an "average person" got through to Virgin's support, they could only be talked through fixing their connection for their laptop/desktop, but not for their phone, On-Demand TV or other services which they have paid for.
"You're the entity who said it was easy to do."
Please feel free to point out where I said that, because I can't see it.
What I can see is you saying that the Virgin crap and many other devices don't provide the option, and me countering that with an if - if you were referring to the Superhub, and [if] the user has a decent router; if those are true, the option is there.
You then moved the goalposts by referring to normal people - which in the context of what I said is irrelevant; I didn't say "normal people who don't understand this stuff can do it easily,", merely that "the option is there" - which it is in the situation I described.
I therefore made the moving goalposts comment because of those moved goalposts.
I suppose I could give you the benefit of the doubt and assume your claim that I said it's easy is also goalpost relocation, but I suspect the real reason might be more fundamental.
You do remember that GCHQ can legally (according to them) track all calls to overseas servers, like facebook, twitter and ... google.
I don't think they mentioned any exclusions for DNS servers.
http://www.bbc.co.uk/news/technology-27887639
and, of course, the NSA are able to collect that data legally too.
Why do companies in the communication business have such an inability to update a basic service status page? I can appreciate they won't change it if one or two people ring up with issues but when they know there is an issue it should be changed almost instantly.
As mentioned in the article it's infuriating to spend ages on hold to be told there are no problems in the area or worse, to find out there is a problem in the area. If one simple page could be updated it would save a lot of people wasting time on hold.
"Why do companies in the communication business have such an inability to update a basic service status page?"
Because they always downplay the reports and state that "some" user were affected, even if "some" means the entire active userbase (bear in mind that some people were away on holiday, at work, asleep or just enjoying RealLife(tm) and so were not affected.
Why do companies in the communication business have such an inability to update a basic service status page?
Admitting a fault introduces the potential for people to ask for compensation, and then they have to explain to them that screwing up is explicitly permitted under clause 133.b, printed in 6 point light grey on white Arial font - at which point they have lost that customer for good. So, pretending it's not happening is beneficial. At least, that's the only reason I can come up with why such a thing isn't even automated.
However, I would like to observe that a status page will not be terribly visible to clients if there is a DNS problem ..
AAISP's issues were self-inflicted, but as is usual for them, they provided a no-bullshit explanation, and apologised. Their MD even publicly apologised to his own staff for messing them about out-of-hours.
The only other ISP that ever came close to that standard of openness and transparency was Nildram, before Pipex absorbed them. BT Wholesale bollixed up my line on a Saturday evening, and I was expecting nothing to happen until Monday morning. On the Sunday afternoon, BT Wholesale rang me directly (and remember, they don't normally acknowledge the existence of end-users) with a grovelling apology and a promise to fix the line ASAP.
(Ironically, the first thing that clued me into Nildram imploding was broken DNS...!)
Walking in to work a few years back, I overheard a couple of students bemoaning the fact that their NTL connections were screwed.
"Nah, you're probably fine, but their DNS is fucked. Just use the Uni's, they're brilliant !"
That's the kind of "appreciation" I (and the DNS servers I run) could do without. When I checked the query logs, I found several thousand NTL IP addresses - not to mention some Uni in the Far East that was using us as a forwarder for all of their queries.
I decided enough was enough, and locked the DNS servers down nice and tight. It was as if a million voices screamed out and were suddenly silenced...
That's the kind of "appreciation" I (and the DNS servers I run) could do without.
I run a recursive nameserver on the quiet - it makes customer-site diagnostics a lot easier when I've got a spare nameserver to hand.
Last year I found what initially appeared to be a DDOS against my server. It turned out to be a *huge* number of DNS queries from a Dutch netblock.
It appears that certain less-scrupulous ISPs hand out other people's DNS services from time to time. So they got firewalled. I expect they got a lot of complaints form that...
Vic.
I work for an ISP and this week we have had any number of calls that start with "Are you having any network problems at all?"
The answer has always been no. In most cases it's been a perfectly normal circuit fault, but there's no doubt that the number of "no fault" calls we've received has gone up this week. I think we can lay this firmly at the door of the media's reporting of the BT outage at the weekend. As soon as somebody has a problem whether it's a webpage not loading, a genuine connection problem, a mistyped URL or anything else the user is assuming it must be an outage on the scale of BTs problems on saturday.
BTW Virgin's DNS has been ropey for as long as I can remember.
I stopped using Virgin's DNS when there was no granularity in Our Dave's Save The Children filtering. My kids use the OpenDNS servers so I can choose what is blocked for them, whilst my wife and I use Google's, but it's no good telling people to switch DNS when probably 95% of their customers have no idea how to do that. I had to disable the hub's dhcp and set my own up.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
