back to article BT at last coughs to 'major outage' after broadband went titsup across UK on Sat

BT was hit by a huge DNS outage on Saturday morning but the telecoms giant was very slow to respond to customer complaints, it has been claimed. A DNS flaw downed BT's network across Blighty, according to anecdotal reports on Twitter. But the one-time state monopoly was very sluggish to respond to gripes from subscribers who …

COMMENTS

This topic is closed for new posts.
  1. Anonymous Coward
    Joke

    "I KNOW HOW THE INTERNET WORKS"

    Yeah... the customer is always WRONG !

    1. Destroy All Monsters Silver badge

      Re: "I KNOW HOW THE INTERNET WORKS"

      "I'm sorry. I'm sorry!"

    2. Chad H.

      Re: "I KNOW HOW THE INTERNET WORKS"

      I'm glad there is one person on the internet that knows how it works. Now if only we could get you into BT, Verizon, et al....

      1. Anonymous Coward
        Anonymous Coward

        Re: "I KNOW HOW THE INTERNET WORKS"

        There are plenty of people working for BT that know how the Internet works. Sadly they don't work for, and probably aren't consulted by, the bit of BT that is responsible for the stuff that went titsup.

      2. Down not across

        Re: "I KNOW HOW THE INTERNET WORKS"

        I'm glad there is one person on the internet that knows how it works. Now if only we could get you into BT, Verizon, et al....

        That's a bit harsh.

        I suspect BT and Verizon (since they were named) have quite a few people who know rather well how internet works. After all for example Verizon acquired WorldCom who in turn gobbled up MCI which had purchased UUNET. Quite a bit of knowledge on how internet works there even if some employees were lost along the way. Whether anyone in management, marketing or beancounting listens to the ones in the know is another matter entirely.

  2. Danny 14

    But on the plus side the intern knows that deleting zones can be bad.

    1. Anonymous Coward
      Anonymous Coward

      1ary file

      I remember back in the day when pipex was registrar for .co.uk and someone saved the 1ary file off (in vi) and accidentally performed a 'delete to end of file and save' command :D

      It only took just over 4 hours for the missing half of .co.uk to come back, but in those days there weren't so many.

  3. Anonymous Coward
    Anonymous Coward

    Wasn't a DNS issue...

    or at least wasn't *just* a DNS issue.

    I got called into work as our primary BTNet wasn't routing data, as wasn't the backup BT Infinity.

    We have our own DNS servers, failover is Google and no pages loading.

    Traceroutes to loads of sites were timing out within BT.

    There looked to be a major routing issue, and even the tech support line for leased lines was so busy it was "leave a message".

    1. FlatSpot
      FAIL

      Re: Wasn't a DNS issue...

      Yep wasn't a DNS issue. I tried Google DNS and not BT DNS and it was still not routing correctly, some sites were ok, some were not!?!

      More likely that GCHQ black box failed....

      1. Destroy All Monsters Silver badge

        Re: Wasn't a DNS issue...

        It went all white?

    2. rhydian

      Re: Wasn't a DNS issue...

      Agreed, it wasn't simply DNS falling over. I'd swapped my BT homehub router for a new Netgear on friday night (configed to OpenDNS). Spent most of saturday morning trying to get it to work again before I heard the news.

      Good job I'm leaving them...

    3. Ross R

      Re: Wasn't a DNS issue...

      Same here. Was using google DNS and had problems on some sites. Traceroutes to said sites stopped in BT's core network.

    4. Vimes

      Re: Wasn't a DNS issue...

      Seven years to the day that they tested Phorm (and lied to their own customer service department about what was going on too if memory serves). Their own internal documents referred to the whole business as a 'stealth' trial.

      I wonder what has really been going on here?

      1. Ben Tasker

        Re: Wasn't a DNS issue...

        Yup, though as lots of people on social media were trying to be clever and tell others it was a DNS issue, I fully expect that if BT do cough to the root cause they'll blame DNS.

        Was sat troubleshooting the loss of connectivity, and it looks like a couple of interfaces went down on a box near Telehouse based on comparison of traceroutes before and after. Possibly elsewhere too, but that's what we were seeing.

        BT's DNS servers, inevitably, are the other side of that hop, so whilst there was an issue reaching BT's DNS, it wasn't _the_ issue.

    5. Anonymous Coward
      Anonymous Coward

      Re: Wasn't a DNS issue...

      And perhaps related - VM or Google were having problems at the same time. *.google.co.uk sites were available via VM but *.google.com sites were not.

    6. Grease Monkey Silver badge

      Re: Wasn't a DNS issue...

      "I got called into work as our primary BTNet wasn't routing data, as wasn't the backup BT Infinity"

      You use a home broadband product for work?

      1. rhydian

        Re: Wasn't a DNS issue...

        BT offer business Infinity packages.

    7. Anonymous Coward
      Anonymous Coward

      Re: Wasn't a DNS issue...

      Same issue here: Connectivity failure with BTNet LL.

      Same leave a message on the business help line.

  4. MJI Silver badge

    Hardly noticed

    Email worked, all the forums I use regularly worked, the load of crap my daughter looks at, didn't work.

    So effing what if twatter is dead or something called tumbler was broken - who cares!

    The friendface ripoff can't remember if that worked or not as I don't use it.

    I could check my emails and read posts on fixing cars, what else do you need? (There must have been some porn available)

    1. Matt Bryant Silver badge
      Thumb Up

      Re: MJI Re: Hardly noticed

      "Email worked, all the forums I use regularly worked, the load of crap my daughter looks at, didn't work...." Same her, except the crap wifey wastes bandwidth on (including Twatter and Faecesbook) were working fine, which suggests a localised routing issue and not a BT network-wide routing issue.

    2. Colin Wilson 2

      Re: Hardly noticed

      >> Email worked, all the forums I use regularly worked, the load of crap my daughter looks at, didn't work

      Which is quite interesting in itself. Facebook, Twitter, Apple etc. all failed to work. But BBC News etc. carried on working fine. Very spooky! What are they preparing for?

      1. MJI Silver badge

        Re: Hardly noticed

        I checked BT web site, it was rubbish!

        BBC worked well.

        As everyone said very odd

  5. GreggS

    Funny

    As Virgin Media's DNS was also lousy at the weekend. Switched to using both Google & OpenDNS and they were fine.

    1. Mat

      Re: Funny

      Unfortunately with only a superhub.. I can't change my DNS servers!

      1. Sir Runcible Spoon

        Re: Funny

        "Unfortunately with only a superhub.. I can't change my DNS servers!"

        So you can't change them on your PC?

      2. Uplink

        Re: Funny

        When the router can't be persuaded to give out Google DNS, I go the extra mile and tell my computer to not use DNS information from DHCP and just use Google. Actually, encountering BT DNS (or any other ISP DNS really) on a new computer promptly makes me take a detour to the settings and switch to Google. BT and other ISPs seem to think they're doing you a service when they break DNS hierarchies and ignore that DNSSEC exists.

      3. CommanderGalaxian
        Joke

        Re: Funny

        What's so "super" about it then?

  6. Archivist

    A human right - or no hope for mankind?

    Not being a BT customer my first inkling of the problem was a BBC radio news item. It included an inane interview with a (typical?) BT customer who said he and his wife and a laptop AND a tablet and BOTH wouldn't work. "I mean it could have got me bank details 'n all!"

    Taps run, toilets flush, and the internet works. There you are, that's your life sorted....

    1. Anonymous Coward
      Anonymous Coward

      Re: A human right - or no hope for mankind?

      Every time internet went down at my last job my boss always said that :

      "internet isn't a 'right' it's a privilege..."

      1. Destroy All Monsters Silver badge

        Re: A human right - or no hope for mankind?

        If you are paying for it, it is a service.

        "Positive rights" are an idea that sprung from festering progressive minds.

        There aren't any in a world not based on pony lore.

    2. Anonymous Coward
      Anonymous Coward

      Re: A human right - or no hope for mankind?

      I heared that chap and thought "what....?

      In fact that was the first I'd heard about the problem, which is odd as I'd been on the net most of the morning on my BT infinity thing and hadn't noticed anything untoward at all.

  7. Anonymous Coward
    Anonymous Coward

    Not Just BT

    Our Virgin service at home was crap all day Saturday as well. Now it works, now it doesn't, frequent complete losses of connection for anything up to half an hour. Using Google DNS or Virgin.

    I foolishly tried to phone them to see if there was a message about outages. Spent two and a half minutes entering account number, listening to options, entering characters from password and listening to that bloody irritating recorded woman's voice. Then got to "I'll pass you to a representative." immediately followed by "Sorry we're experiencing technical difficulties. Please call back later." Line then disconnected.

    So at least that part of the Virgin Media 'experience' was as normal!

    Maybe the cleaners unplugged the wrong thing at GCHQ.

  8. CommanderGalaxian
    Linux

    Filter trouble then.

    That'll be them fiddling with their new DNS based censorship filters then. The good news (for the moment) is that they haven't (yet) blocked things like openDNS.

    1. Loyal Commenter Silver badge

      Re: Filter trouble then.

      Your singular downvote, like mine, probably signifies that this is closer to the truth than we'd like to think.

  9. MrXavia

    Problems started on Friday, I had to switch to alternate DNS servers.

    But even with alternate DNS there were still problems, I am sure that there were routing issues as well that may or may not have been BT related, my mobile broadband has some issues also.

  10. Loyal Commenter Silver badge
    Joke

    BT DNS servers not working?

    Just point your DNS directly at GCHQ and cut out the middle man.

  11. 2460 Something

    Not just DNS

    Tried Google's DNS and openDNS and still had intermittent problems getting pages loaded. Google surprisingly was fine (on their dedicated fast line??) but click through to most other pages and it just dropped out.

    Tried ringing BT a few times (since my area wasn't on the outage list when I finally did get it loaded via ssh tunnel to work), after a few minutes in the queue each time call either got disconnected or dropped (but still left connected) from queue. First class service as always. Thanks BT!

  12. bdj

    Resolution or Revolution?

    Certainly wasn't just specific to BT's DNS server (or however they resolve it via the homehub 5). I was unable to connect to many public DNS servers for the duration, but googles services did work which would suggest routing is at fault.

    C:\Windows\system32>nslookup

    Default Server: BThomehub.home

    Address: 192.168.1.1

    > bt.com

    Server: BThomehub.home

    Address: 192.168.1.1

    Non-authoritative answer:

    Name: bt.com

    Addresses: 2a00:2381:ffff::1

    193.113.9.164

    > server 208.67.222.222

    Default Server: resolver1.opendns.com

    Address: 208.67.222.222

    > bt.com

    Server: resolver1.opendns.com

    Address: 208.67.222.222

    DNS request timed out.

    timeout was 2 seconds.

    DNS request timed out.

    timeout was 2 seconds.

    DNS request timed out.

    timeout was 2 seconds.

    DNS request timed out.

    timeout was 2 seconds.

    *** R

  13. Loyal Commenter Silver badge

    Got a problem with your ducts?

    Call Central Services.

  14. Nick Everitt

    Most frustrating thing about the outage was if you looked at BT's status page all it said was that some customaers in a few areas of the country were having issues with their connection when it was very obvious from looking at twitter or downdetetctor that it was very widespread and effecting a lot of people!

    I have used BT as an FTTC supplier for a few years now and they have been very reliable, couple of outages that have been short and very quickly rectifified so no real complaints about the odd outage but their status page really should be properly updated so people dont waste time cheking/changing their own stuff when its something BT need to resolve.

    1. Anonymous Coward
      Anonymous Coward

      Yeah. When I looked 0121 was not mentioned, but I had the same issues.

      my mobile wouldn't work over wifi. But it worked fine over the 3G sure signal connection that goes out via the same broadband connection! VPN tunnel I guess!

    2. Guus Leeuw

      Few Areas

      Dear Sir,

      not knowing the full extend of these problems, 2 semi-nationwide areas being affected is the same as few areas being affected, seeing that 2 really is only a few... There were some people in those few areas of which BT knew that they had problems. All the rest (of the people) couldn't shouldn't wouldn't report the issue... So all in all, "some people in a few areas"...

      No, I'm a not a spin-doctor, but I've seen so much crap coming out of Gov / Big Orgs that I kinda know what they're saying when they're saying stuff... Also when they're not saying stuff :D

      Regards,

      Guus

  15. Mrg9999

    Internet down at the weekend?

    Have a backup internet provider

    or

    Go for walk

    drink beer

    do something productive

  16. Brianc

    This seems to have been the biggest outage in a long time and yet the media have hardly mentioned it at all. Why not?

  17. Brianc

    Biggest outage in a long time but no media interest

    This has to have been the biggest outage in the UK for ages and there were ripples in other countries too. Why no media interest?

  18. Lost in Cyberspace

    Meanwhile..

    2 of my customers have borked Windows 8 by hitting refresh... All because their Internet didn't work.

    Gah, why is it so easy for clients to do a refresh anyway?!

  19. Ken Moorhouse Silver badge

    Access to Port 80 revoked

    One of my clients rang up to tell me he had sporadic internet: Google worked, but none of the links beyond that would.

    I've had this before when there is a billing problem: BT cut access to Port 80 until the bill is paid, so you can't access the web but your email still comes in and goes out. Google worked because it uses https.

    Didn't get chance to rule out DNS completely. but google's DNS seemed to work fine for my client.

  20. Grease Monkey Silver badge

    Eggs and Baskets

    I have never relied solely on my ISP for DNS. Using your ISP as primary DNS is probably best from a performance POV, but why have the same provider for secondary (or even tertiary) DNS?

    What amazed me about so many commentards on saturday morning was how many of them thought they were smart because they knew how to change DNS servers, but were still dumb enough only to point themselves at one DNS server.

    1. Ben Tasker

      Re: Eggs and Baskets

      What amazed me about so many commentards on saturday morning was how many of them thought they were smart because they knew how to change DNS servers, but were still dumb enough only to point themselves at one DNS server.

      And were 'smart' enough to change DNS servers, and tell everyone else it was a DNS issue, whilst completely failing to take note of the fact that changing DNS didn't help with accessing quite a lot of sites. There was a hell of a lot of the blind leading the blind on the net on Saturday

  21. Anonymous Coward
    Anonymous Coward

    Heathrow baggage chaos

    http://www.hindustantimes.com/entertainment/music/heathrow-baggage-chaos-it-could-be-several-days-before-ustad-gets-his-sarod/article1-1235370.aspx

  22. herman

    Awww come-on, that picture with the story headline was titsdown...

  23. Peter 26

    When looking into it on Saturday I saw that there was two major routers with issues. One was a BT router in telehouse, the other was a Demon router (I don't recall the location, probably telehouse too). That might explain the virgin media issues some were having? I'm not sure who owns Demon now, but I thought they had some links to Cable & Wireless...

    Anyway after getting a banned from CS:GO due to the network issues making me abandon a game, and figured out who to blame, I decided to go out and mow the lawn.

  24. JMB

    A couple of years ago when we had a major network fault in this area, BT Internet did not seem to have system to recognise that they were getting lots of complaints and the complaints did not seem to be passed on by the Indian call centre. Whatever you say to the call centre they just put you through the standard series of tests to prove your router is faulty which can take an hour so many people do not bother calling them. In that case the network monitoring did not detect the fault and they seemed to have nothing to detect a drop in activity on the network.

  25. Anonymous Coward
    Anonymous Coward

    DNS or routing?

    I run a ping test to internal and external servers and site, both from within work and without. Pings work -> internal (incuding vpn to datacentre) and external sites using URL and IP addresses all worked throughout. Pinging from home via Infinity to the externally available equivalents showed a mix of sites being found and not found, using both URLs and IP addresses (which absolves DNS issues?). reference sites like BBC were ok. This kicked off at 9:21, giving me the impression we had a problem in the office as I could not ping our London-based sites, but could ping the datacentre ones (Milton Keynes). I eventually remoted onto a PC in Swansea that uses TalkTalk - that had no issues with the sites I could not reach. My mobile phone connected fine as well with 3G (not so when using my WiFi connection).

    I guess I changed my local PC to Google DNS just as it was fixed, so I'm not convinced as to what the real issue was.

This topic is closed for new posts.

Other stories you might like