back to article Virgin Media struck dumb by NATIONWIDE DNS outage

Virgin Media customers were kicked offline last night, after a DNS blunder from the cable company disrupted its network on Tuesday. It comes after rival BT suffered a similar, widespread technical cockup over the weekend. Virgin Media, which is owned by US telecoms giant Liberty Global, coughed to a "nationwide DNS outage" on …

  1. Ben Rose
    Big Brother

    Cheltenham again?

    Were they activating their new link to GCHQ this time?

    All your data are belong to us.

    1. Allan 1

      Re: Cheltenham again?

      Yeah, I'm not buying the "failure" excuse either. Happened too soon after BT had an identical "failure".

    2. TitterYeNot
      Headmaster

      Re: Cheltenham again?

      "All your data are belong to us"

      Sorry to be finicky (ha, as if, this is El Reg), to follow the meme correctly, shouldn't that be "All your datum are belong to us."

      Ahem. Aaaand back to the story... DNS availability does seem to be Virgin Broadband's major weakness in my experience. Throughput and latency over cable is generally good, but DNS response times and/or availability seems to vary from just about acceptable to utter crap on a day to day basis.

      1. Danny 14 Silver badge

        Re: Cheltenham again?

        Nah, BT's intern simply went to virgin instead.

      2. Annihilator
        Happy

        Re: Cheltenham again?

        "Sorry to be finicky"

        "finicky" is just a lazy person's word for "accurate"

    3. Anonymous Coward
      Anonymous Coward

      Re: Cheltenham again?

      Were they activating their new link to GCHQ this time?

      Actually, no, data is actually going to Maryland, and has been doing so for a while. That it breaks UK Data Protection with it apparently doesn't matter to its US owner:

      ;; QUESTION SECTION:

      ;virginmedia.com. IN MX

      ;; ANSWER SECTION:

      virginmedia.com. 3220 IN MX 1 aspmx.l.google.com.

      virginmedia.com. 3220 IN MX 5 alt1.aspmx.l.google.com.

      virginmedia.com. 3220 IN MX 5 alt2.aspmx.l.google.com.

      virginmedia.com. 3220 IN MX 10 aspmx2.googlemail.com.

      virginmedia.com. 3220 IN MX 10 aspmx3.googlemail.com.

      Even when equipped with barge pole, do not touch.

      1. Anonymous Coward
        Anonymous Coward

        Re: Cheltenham again?

        Virginmedia email has been hosted by Gmail for many years. Goes back deep into when NTL owned it. This is just the same as Sky using Yahoo (was Gmail). And BT also used Yahoo for many years, only just migrating their users away from it. Nothing secretive here.

        1. Anonymous Coward
          Anonymous Coward

          Re: Cheltenham again?

          Nothing secretive here.

          I wasn't suggesting it was secretive, other than that I suspect it's never been made very clear to VM users that their data is effectively exported to the land of the *cough* "free". This situation has indeed existed for quite a while.

          1. Paw Bokenfohr

            Re: Cheltenham again?

            @AC:

            VM are quite clear with their users (of which I am one) that their email service is provided by Google, and it's very obvious that this is the case when you use webmail anyway, so you really have no need to be concerned about it.

  2. Lionel Baden

    Suspect Much ?

    I cant be the only one, who thinks that this is starting to look a little suspect.

    *where is the black helicopter Icon ?

    1. ukgnome Silver badge
      Black Helicopters

      Re: Suspect Much ?

      helicopter what? You can't see it? And you never will, until it's too late

      Mwah hahahha

    2. Pypes

      Re: Suspect Much ?

      VM's DNS servers have been in the process of falling over for several years now, I'm calling coincidence with regards to the BT outage.

    3. Anonymous Coward
      Anonymous Coward

      Re: Suspect Much ?

      My VM service continued to work last night. No DNS problems here. However, I did get problems last Thursday that were very similar to what was reported for BT at the weekend.

      So yes, I'm suspicious. There does seem to be a systematic outbreak of this problem and a failure to state the cause doesn't help.

  3. Terry 6 Silver badge

    Hmm

    "...Virgin Media's status pages claim everything is fine..."

    That's pretty par for the course with VM.

    Sometimes even the call handler will be saying there's nothing wrong, and have you tried switching it off and on again, etc. And later you find out that the whole area has a problem.

    1. malle-herbert Silver badge
      Joke

      Re: have you tried switching it off and on again ?

      Have you tried sticking it up your arse ?

      1. Rol Silver badge

        Re: have you tried switching it off and on again ?

        "Have you tried sticking it up your arse ?"

        "I'm afraid my arse is too full of excuses for the poor service to fit anything not first pulled from my managers arse already"

    2. EddieD

      Re: Hmm

      Yeah - those pages are about as much use as paraffin wax teapot* - a static page showing Cnut** holding back the waves with an "Everything's fine" speech bubble would be equally accurate.

      *It's been pointed out to me often that you could eat a chocolate teapot, thereby providing some good use.

      **Preferably mispelled

    3. Harry the Bastard

      yep, vm status pages are useless, as is the telephone status

      during a long (>48 hours outage) and after several calls over two days with the 'help' desk, who could only say there was no known issue or resolution time (and kept trying to get me to check my network settings, grrrr), i was finally passed to second level support, who instantly replied that "oh, it's a planned outage for network changes, service will resume at xx:yy" which it did

      i) planned, so why didn't they email affected customers in advance?

      ii) planned, so why didn't they put it on the status page, phone status etc.?

      iii) planned, so why didn't they tell their own help desk people?

      iv) planned, in what way?

      1. Jason 24

        Re: yep, vm status pages are useless, as is the telephone status

        "iv) planned, in what way?"

        Planned as in "bollocks, I rebooted the wrong server". "Ah well, may as well do that maintenance since it's off anyway". "Oh shit, that didn't go to plan, what do we do now??"

        I would suspect.

      2. jubtastic1

        Re: yep, vm status pages are useless, as is the telephone status

        They did try to warn the help desk guy but he's notoriously hard to get hold of, so you can't really blame them for that one.

    4. wikkity
      Joke

      Re: Status Pages

      They couldn't update them as they could resolve the servers host name

    5. dotdavid

      Re: Hmm

      Yep their status page is awful.

      Whenever I have an internet outage, I check the status page from my phone and it usually says something like "Phone - OK, Internet - OK, TV - There are reported problems with TV service in your area". Despite the fact that I don't have TV with them and it's my Internet that's down I can only assume "Internet" and "TV" are linked in some way, and only bother contacting the helpdesk if all three services are listed as OK.

    6. helicoil

      Re: Hmm

      All that is worth it when you ring up tech support and the guy on the other end has a heavy geordie accent. That sort of service is priceless.

    7. Velv Silver badge
      Facepalm

      Re: Hmm

      I just love the status pages.

      I just love that they offer to send you an email when an issue is resolved.

      I just love that it was email that was down, and they offered to send me an email when it was fixed... (I leave the reader to do the facepalm)

      (yes, yes, I know, I could have used a different email address if I had one, or if I had push email it should have come through to my phone or I could of chosen to receive an SMS. But where would the Virgin bashing fun be in that)

  4. Anonymous Coward
    Anonymous Coward

    Two Mass DNS Outages In Less Than A week

    This is looking a little suspicious.

    To lose one DNS is unfortunate, to lose two is a good old fashioned conspiracy.

    1. ma6cb

      Re: Two Mass DNS Outages In Less Than A week

      Was thinking the same myself... Does seem rather odd the 2 of the UKs largest Internet backbone and service providers have DNS issues within 48 hours of each other...

  5. Robert Ramsay

    Heh...

    I'd forgotten I'd switched over to OpenDNS until I read this story...

    1. Anonymous Coward
      Happy

      Re: Heh...

      Yup, best to give to the Yanks to log and store, says our lot doing it...

    2. pPPPP

      Re: Heh...

      Here's how I didn't notice the outage, in case anyone's interested.

      I run a small Linux server in my loft (I use a Raspberry Pi for its low power consumption, but anything will do).

      It runs a DHCP server and a DNS server. Neither is difficult to configure. Superhub is in normal mode, with DHCP switched off.

      In the DNS server, I have a config file which redirects all spam, ads or anything else I don't like, such as social media bollocks. My LG TV no longer tries to advertise things to me (still annoyed about that - never buy LG) and nor does my phone, at least when I'm in the house.

      Anything in the house has a DNS entry. Anything else gets forwarded to opendns.

      Still have to use UKbay to get round the pathetic attempt at blocking piratebay though.

      1. Lionel Baden

        Re: Heh...

        Do you have a copy / tutorial of your setup you would be happy to share ?

        1. pPPPP

          Re: Heh...

          It does vary from distro to distro. You need to set up dhcpd and bind. I use slackware and that basically means installing dhcpd and bind packages, changing /etc/rc.d/rc.dhcpd and /etc/rc.d/rc.bind to be executable, editing the config files, and starting them.

          There's probably a bit too much to post here, but this post: http://topchan.info/your-own-adblocking-dns will give you some pointers.

        2. BlartVersenwaldIII

          Re: Heh...

          On debian (and therefore probably ubuntu) at least, a local caching DNS server is a simple matter of installing bind9 and editing /etc/bind/named.conf.options. You can put whatever DNS servers you want in the forwarders section like so:

          forwarders {

          // ISP DNS servers go here

          2.3.4.5;

          6.7.8.9;

          10.11.12.13;

          };

          Restart bind and you're done. I use OpenNIC DNS servers with Level 3 as a backup.

          If you want a simple single subnet DHCP server, just install udhcpd and you're done. ISC dhcpd is a little more involved.

          You can combine both in a single piece of software by using the excellent dnsmasq (which has the added bonus of providing DNS names for you local network automatically) although it's harder to set up than the above bind example - but simpler overall.

  6. Anonymous Coward
    Anonymous Coward

    Hmmmm DNS ...

    for some reason, I never trust ISP provided DNS. Google and/or OpenDNS for me. Which is why I didn't notice the BT outage (I used BT for my work BB, with OpenDNS) nor this one (my private BB is Virgin, with Google DNS).

    1. 's water music Silver badge

      Re: Hmmmm DNS ...

      for some reason, I never trust ISP provided DNS... Google DNS...

      Google. Sounds legit.

      :-)

    2. Velv Silver badge
      Trollface

      Re: Hmmmm DNS ...

      Yea, because giving google even more information about the websites you visit is such a good thing.

      Run your own DNS server and leave it to look up the root hints

  7. Bob AMG

    Time to 8.8.8.8 and 8.8.4.4 they are googles DNS servers that anyone can use. Not the fastest but better than note.

    1. Anonymous Coward
      Anonymous Coward

      Try 4.2.2.2 - 6 instead.

    2. Omgwtfbbqtime Silver badge

      That would be why I never noticed any DNS problems then.

      Definite fail by VM for locking the Superhub DNS away from the users.

      I have heard that the reason is so that they don't have to support any user change to the DNS but since the third thing* they ask you to do is factory reset the router I don't see the point.

      * First thing - reboot your computer, second thing reboot the router.

  8. Proud Father

    Didn't even notice

    SuperHub in modem mode, Netgear router set to use OpenDNS (208.67.222.222 and 208.67.220.220)

  9. Richard 12 Silver badge
    FAIL

    The Hub's not so "Super"

    Annoyingly there is no way to change the Superhub's DNS settings, so I could only fix the PCs and not my wife's phone or the set-top box.

    I spent 40 minutes on hold to find that out nugget of information.

    At least "Cable Modem mode" is easy to set up, just not something I felt like doing during an outage.

    1. Sir Runcible Spoon Silver badge

      Re: The Hub's not so "Super"

      does this superhub thing allow you to set up the dhcp details that are handed out?

      1. pPPPP

        Re: The Hub's not so "Super"

        All you can do is reserve IP addresses for MAC addresses. That's it.

  10. jca111

    Dont Use VM DNS

    I've been with VM for BB for years, but over the last couple of years the DNS provided by them has been getting worse and worse.

    When I switched to public DNS provides like google or OpenDNS, the broadband now works really fast.

    So why cripple you fast broadband with pathetic DNS? Most users will not ever change their DNS, and will just say "My Broadband is shit"

  11. JimmyPage Silver badge
    Black Helicopters

    HMG ????

    Of course, nobbling DNS is the cheapest way to mask certain areas of the internet from the masses. So any issues with major ISPs DNS provision is suspicious.

    1. Lyndon Hills 1

      Re: HMG ????

      Can't say I noticed, as I don't use their DNS, but as it happens i had an email from Virgin telling me about a new option to switch on safe internet. that was going to be done using DNS, IIRC.

      1. Rol Silver badge

        Re: HMG ????

        Me too, but no thank you.

        They also offered to upgrade me to 50Mb, but, I smelt a con, which I guess would involve the Super Hub being installed and the price going up beyond inflation AGAIN!!!!!!!

  12. s. pam
    Mushroom

    Virgin's DNS had its Cherry popped a LONG time ago

    We've been on Virgin for 4+ years and it took 15 seconds to realize their DNS is #epicfail fsck'd.

    We use a variety of open DNS, plus known good DNS servers and never ever rely on Virgin's DNS. The fact their router does is immaterial as all of our home hosts have hard coded DNS entries so we can avoid Virgin's DNS fsckery.

  13. DrSpudz
    Joke

    Who knew Virgin Media used NO-IP as part of their infrastructure...

    1. Tom 13

      @ DrSpudz

      Darn it! You beat me to it.

  14. darpax

    Yesterday? Bollocks

    It went down on Friday.

    http://community.virginmedia.com/t5/Up-to-152Mb/DNS-Issues/td-p/2361435

    Also the service is getting worse in general, my area is waiting now almost 2 years for a resegmentation and all I get is usual bullshit about how they are sorry and the work takes a long time to plan.

  15. Cthonus
    Coffee/keyboard

    Does your horse require water?*

    "Virgin Media's status pages claim everything is fine"

    The last time I looked at their status page it said for my area (Central Scotland!) people were reporting problems logging on to London Underground wifi. Ya don't say?

    *Cowboys, the lot of them.

    1. Anonymous Coward
      Anonymous Coward

      Re: Does your horse require water?*

      > Cowboys, the lot of them.

      Have a look at the financial reports for Liberty Global.

      They aren't short of cash.

  16. phuzz Silver badge

    I've been meaning to pick up a new router and only use the notsoSuperHub in modem mode. Any recommendations?

    I'm looking for GB ports, 802.11ac and ideally support from openWRT or similar.

    1. Fuzz

      Try the Trendnet TEW-818DRU

      Or if you want something cheaper then some of the TP-Link stuff is fairly decent once you install dd-wrt

    2. Fred Flintstone Gold badge

      We switched to Draytek a long time ago, and I also use it at home. Also has the option of jacking in a 3G/4G USB stick in it with a SIM card so it can use that as backup circuit.

      Simply good kit.

  17. Anonymous John

    "It was quickly fixed although changing DNS settings in the interim would have solved the problem,"

    And what percentage of your users knows how to do that?

    1. Richard 12 Silver badge

      You can't do it anyway.

      The Virgin-provided routers do not expose options to do that anyway, neither client-side (internal DHCP) nor router-side.

      Many devices don't expose this setting, either expecting the DHCP server to properly work or even assuming the Gateway is the DNS.

      1. VinceH Silver badge

        Re: You can't do it anyway.

        If the Virgin provided router you're talking about is the SuperHub and it's in modem mode, and you use a decent router of your own, the option is there.

        I didn't notice the problem for the obvious reason. I just wish my supposedly 'up to 152Mbs' connection showed me a bit more oomph than the 40-50Mbs I usually get1 - consistently from speed test sites.

        When the Virgin Media guy came out to do the installation and pointed out I wouldn't need my existing router, I said I'd be putting the SuperHub straight into modem mode so I would need my router. His reply was a small laugh and the comment "Each to their own" as if what I said was crazy talk.

        1. To be fair to them, though, looking at the Superhub's status, I see it says the maximum speed is something around 160Mbs, so maybe I should replace the old/cheap cable running from it to my router before complaining. Just in case it's not up to the job.

        1. Richard 12 Silver badge
          Holmes

          Re: You can't do it anyway.

          Well, quite. If you use a router that has the option then you'll have the option.

          What if you're a "normal person"?

          Normal people use the equipment provided by the ISP for the purpose. They don't buy a router because they got given one "free".

          So if an "average person" got through to Virgin's support, they could only be talked through fixing their connection for their laptop/desktop, but not for their phone, On-Demand TV or other services which they have paid for.

          1. VinceH Silver badge

            Re: You can't do it anyway.

            Ah, you're trying to induct me into the moving goalpost spotters club. I see.

            1. Richard 12 Silver badge

              Re: You can't do it anyway.

              You what?

              You're the entity who said it was easy to do.

              1. VinceH Silver badge
                Facepalm

                Re: You can't do it anyway.

                "You're the entity who said it was easy to do."

                Please feel free to point out where I said that, because I can't see it.

                What I can see is you saying that the Virgin crap and many other devices don't provide the option, and me countering that with an if - if you were referring to the Superhub, and [if] the user has a decent router; if those are true, the option is there.

                You then moved the goalposts by referring to normal people - which in the context of what I said is irrelevant; I didn't say "normal people who don't understand this stuff can do it easily,", merely that "the option is there" - which it is in the situation I described.

                I therefore made the moving goalposts comment because of those moved goalposts.

                I suppose I could give you the benefit of the doubt and assume your claim that I said it's easy is also goalpost relocation, but I suspect the real reason might be more fundamental.

  18. smartypants

    Wasn't just last night

    DNS has been intermittent at peak times for last few days. Last night was particularly bad. The hamster is obviously tiring on its treadmill and needs some more nuts

    1. Chris King Silver badge

      Better get me a new hamster...

      ...this one's broken !

  19. xeroks

    google dns

    You do remember that GCHQ can legally (according to them) track all calls to overseas servers, like facebook, twitter and ... google.

    I don't think they mentioned any exclusions for DNS servers.

    http://www.bbc.co.uk/news/technology-27887639

    and, of course, the NSA are able to collect that data legally too.

  20. PaulR79

    Service status updates

    Why do companies in the communication business have such an inability to update a basic service status page? I can appreciate they won't change it if one or two people ring up with issues but when they know there is an issue it should be changed almost instantly.

    As mentioned in the article it's infuriating to spend ages on hold to be told there are no problems in the area or worse, to find out there is a problem in the area. If one simple page could be updated it would save a lot of people wasting time on hold.

    1. John Brown (no body) Silver badge

      Re: Service status updates

      "Why do companies in the communication business have such an inability to update a basic service status page?"

      Because they always downplay the reports and state that "some" user were affected, even if "some" means the entire active userbase (bear in mind that some people were away on holiday, at work, asleep or just enjoying RealLife(tm) and so were not affected.

    2. Anonymous Coward
      Anonymous Coward

      Re: Service status updates

      Why do companies in the communication business have such an inability to update a basic service status page?

      Admitting a fault introduces the potential for people to ask for compensation, and then they have to explain to them that screwing up is explicitly permitted under clause 133.b, printed in 6 point light grey on white Arial font - at which point they have lost that customer for good. So, pretending it's not happening is beneficial. At least, that's the only reason I can come up with why such a thing isn't even automated.

      However, I would like to observe that a status page will not be terribly visible to clients if there is a DNS problem ..

      1. Richard 12 Silver badge

        Re: Service status updates

        It'd be useful for the poor abused sods on the support phones though.

        Plus smartphones on mobile data - unless it affected Virhin mobile as well! (Don't know)

  21. ElNumbre
    Go

    Other ISPs

    Interestingly, AAISP had DNS issues last week. The difference is, they don't force you to use their servers from a locked-down NTU, and their status pages are kept current and upto date.

    1. Chris King Silver badge

      Re: Other ISPs

      AAISP's issues were self-inflicted, but as is usual for them, they provided a no-bullshit explanation, and apologised. Their MD even publicly apologised to his own staff for messing them about out-of-hours.

      The only other ISP that ever came close to that standard of openness and transparency was Nildram, before Pipex absorbed them. BT Wholesale bollixed up my line on a Saturday evening, and I was expecting nothing to happen until Monday morning. On the Sunday afternoon, BT Wholesale rang me directly (and remember, they don't normally acknowledge the existence of end-users) with a grovelling apology and a promise to fix the line ASAP.

      (Ironically, the first thing that clued me into Nildram imploding was broken DNS...!)

  22. Stuart Halliday
    Facepalm

    If you're sensible, you add a OpenDNS and a Google DNS line to your router...

    Nuff said.

  23. Chris King Silver badge

    VM DNS has been rubbish for as long as I can remember

    Walking in to work a few years back, I overheard a couple of students bemoaning the fact that their NTL connections were screwed.

    "Nah, you're probably fine, but their DNS is fucked. Just use the Uni's, they're brilliant !"

    That's the kind of "appreciation" I (and the DNS servers I run) could do without. When I checked the query logs, I found several thousand NTL IP addresses - not to mention some Uni in the Far East that was using us as a forwarder for all of their queries.

    I decided enough was enough, and locked the DNS servers down nice and tight. It was as if a million voices screamed out and were suddenly silenced...

    1. Vic

      Re: VM DNS has been rubbish for as long as I can remember

      That's the kind of "appreciation" I (and the DNS servers I run) could do without.

      I run a recursive nameserver on the quiet - it makes customer-site diagnostics a lot easier when I've got a spare nameserver to hand.

      Last year I found what initially appeared to be a DDOS against my server. It turned out to be a *huge* number of DNS queries from a Dutch netblock.

      It appears that certain less-scrupulous ISPs hand out other people's DNS services from time to time. So they got firewalled. I expect they got a lot of complaints form that...

      Vic.

  24. Neoc

    Interesting

    Some of Vodafone Australia's mobile customers were suffering from a DNS outage yesterday as well (call it 11am - 3pm AEST). In my case, my mobile was affected but my tablet wasn't.

  25. eJ2095

    Super Dud

    Speaking of which...

    Time for me to get a new router..

    Any suggestions looking a price from 50 to 100, Need ability to lock down transfer speed on certain devices, and auto cut off at certain times.

    Ta

  26. Grease Monkey

    I work for an ISP and this week we have had any number of calls that start with "Are you having any network problems at all?"

    The answer has always been no. In most cases it's been a perfectly normal circuit fault, but there's no doubt that the number of "no fault" calls we've received has gone up this week. I think we can lay this firmly at the door of the media's reporting of the BT outage at the weekend. As soon as somebody has a problem whether it's a webpage not loading, a genuine connection problem, a mistyped URL or anything else the user is assuming it must be an outage on the scale of BTs problems on saturday.

    BTW Virgin's DNS has been ropey for as long as I can remember.

  27. jbuk1

    I've not used an ISP's DNS server for some time so I didn't even notice tbh.

  28. Spuddyspudspud

    Never noticed....

    I stopped using Virgin's DNS when there was no granularity in Our Dave's Save The Children filtering. My kids use the OpenDNS servers so I can choose what is blocked for them, whilst my wife and I use Google's, but it's no good telling people to switch DNS when probably 95% of their customers have no idea how to do that. I had to disable the hub's dhcp and set my own up.

  29. OmgTheyLetMePostInTheUK

    Never ever ask a virgin what went wrong!!!!!!!!!!!!!!!!!!!!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019