back to article Yesterday: Openreach boss quits. Today: BT network goes TITSUP

Less than 24 hours after BT Openreach chief Joe Garner quit the telco's troubled infrastructure division, BT customers all over the UK are saying they can't get online – with the apparent network outage possibly taking up to three days to fix. The outage manifested as a packet routing problem, with some unfortunate souls …

  1. teebie

    "There has been absolutely no suggestion he took the router config data with him when he left the building"

    Shouldn't this be accompanied by "at the time of writing the start of the start of this sentence"

  2. Blane Bramble

    "The Register has contacted BT for comment on the outage. We'll update this story if we hear back from them."

    You might have to wait 3 days, they can't get online at the moment.

  3. Tsung

    More like 3 hours than 3 days, solved already by BT Service page

    1. Blane Bramble

      Considering mine had been down for more than 3 hours when I left the house this morning, that seems unlikely...

    2. Zola

      More like 3 hours than 3 days, solved already by BT Service page

      The problem actually started around 9pm Monday night, so took over 15 hours to fix, which is hardly impressive. Though better than 3 days, perhaps they're under promising and then over delivering.

      1. James 139

        Depends how you count the "days".

        15 hours is almost 2 "man days" based on an 8 hour day and 1 person.

        Maybe with 1 trained person and 1 trainee that might make it 3 "man days" in 15 hours.

        1. anonymous boring coward Silver badge

          You have to correct for "BT man days", so that would be 3 hours per day. The other 5 are spent in a van parked in the countryside, eating sandwiches, texting and listening to the radio

  4. Anonymous Coward
    Anonymous Coward

    All Over The UK?

    So the south east, East Anglia and the East Midlands is "all over the UK" is it?

    1. gazthejourno (Written by Reg staff)

      Re: All Over The UK?

      Given that I've seen other angry Twitterererererers shouting at BT with locations given from many other places, no. Try reading the story, eh?

    2. Blane Bramble

      Re: All Over The UK?

      From BT's service status:

      "Glencarse - 01738 (estimated clear date 20/11/2015)

      Belford - 01668 (estimated clear date 20/11/2015)

      North Weald - 01992 (estimated clear date 20/11/2015)

      Husband Bosworth - 01858 (estimated clear date 20/11/2015)

      Buckland Newton - 01300 (estimated clear date 20/11/2015)

      Glenwherry - 02825 (estimated clear date 20/11/2015)

      Brenchley- 01892 (estimated clear date 20/11/2015)

      Ferndown- 01202 (estimated clear date 20/11/2015)

      Treforest - 01443 (estimated clear date 20/11/2015)

      Rothley - 01162 (estimated clear date 20/11/2015)

      Dinas Mawddwy - 01650 (estimated clear date 20/11/2015)

      Ashreigney - 01769 (estimated clear date 20/11/2015)

      Watton - 01377 (estimated clear date 19/11/15)

      Chichester - 01243 (estimated clear date 19/11/15)

      Swansea - 01792 (estimated clear date 19/11/15)

      New Cumnock - 01290 (estimated clear date 19/11/15)

      Churston - 01803 (estimated clear date 19/11/15)

      Colwall - 01684 (estimated clear date 19/11/15)

      Painswick - 01452 (estimated clear date 19/11/15)

      Lydbrook - 01594 (estimated clear date 19/11/15)

      New Cross - 0203 0207 0208 (estimated clear date 19/11/15)

      Broadstairs - 01843 (estimated clear date 19/11/2015)

      Bayston Hill - 01743 (estimated clear date 19/11/2015)

      Burton Bradstock - 01308 (estimated clear date 19/11/2015)

      Magherafelt - 02879 (estimated clear date 19/11/2015)

      Bunbury - 01829 (estimated clear date 19/11/2015)

      Yockleton - 01743 (estimated clear date 19/11/2015)

      Lisbellaw - 02866 (estimated clear date 18/11/15)

      Alyth - 01828 (estimated clear date 18/11/15)

      Epping - 01992 (estimated clear date 18/11/15)

      St Boswells - 01835 (estimated clear date 18/11/15)

      Midhurst - 01730 (estimated clear date 18/11/15)

      Sedgley - 01902 (estimated clear date 18/11/15)

      Carrickmore - 02880 (estimated clear date 18/11/15)

      Salisbury - 01722 (estimated clear date 18/11/2015)

      Walthamstow - 0203 0208 (estimated clear date 18/11/2015)"

      Feel free to Google some of the locations to see how all over the UK this is.

      1. Roland6 Silver badge

        Re: All Over The UK?

        Re: "From BT's service status:"

        An impressive list, which raises a question as just how their network is structured and hence whether sufficient consideration has been given to redundancy and avoidance of single points of failure.

        Unfortunately, for those who like to throw stones, I expect all of the other major ISP's to have similar problems in their infrastructures.

      2. Warm Braw

        Re: All Over The UK?

        Presumably the routing tables had confused Glencarse with Glencelbow.

    3. Anonymous Coward
      Anonymous Coward

      Re: All Over The UK?

      Well - the bits that matter anyway...

    4. I. Aproveofitspendingonspecificprojects

      Re: All Over The UK?

      > So the south east, East Anglia and the East Midlands is "all over the UK" is it

      Of couse it is stoooopid!!!! Don't you ever watch the weather forecasts?!!!!

  5. Stevie

    Bah!

    Your packets are important to us. Please stay online for the next available router.

  6. Peter 26

    Fixed for me

    It went down for me at about 9pm last night for certain routes. It started working again at about 1pm today.

    The worst thing was there was no easy way to report it to BT, you can only report a BT broadband issue by going to their website and running the broadband live test, which we couldn't access!

    1. DJV Silver badge
      FAIL

      "no easy way to report it to BT"

      Obviously another one from the "No keyboard detected. Press F1 to continue" school of thought stupidity.

    2. Cynical Observer

      Admittedly - they don't make it easy to find the information...

      24 hours a day, 365 days a year.

      The best time to call is

      between 12 noon and 4pm.

      Please make sure you're at your computer when you call.

      From a landline: 0800 111 4567

      From a mobile: 0330 123 4567*

      From abroad: +44 179 359 6931**

  7. sysconfig
    Coat

    Thieves helping Openreach to go all superfast everywhere

    If thieves are going for copper, it may be a good time to roll out that promised superfast fibre.

    Mine's the one with empty pockets, promised.

    1. Corinne

      Re: Thieves helping Openreach to go all superfast everywhere

      That can't be the reason for my outage this morning - no copper here, bloody aluminium :(

  8. jason 7

    Back up and running.

    As per Peter 26 said, some parts were still running (Netflix thankfully) last night and I found switching my router DNS to Googles helped get a bit more back this morning.

    All speedy again now.

    I guess it was their DNS servers down...again.

    1. Len

      Re: Back up and running.

      Seriously, people should stop voluntarily sending ALL their DNS traffic to Google. All the sites you visit, all the mail servers you check, all the sync services you use, all the FTP connections you make etc. all sent to Google.

      Just use a OpenNIC Project server near you: https://www.opennicproject.org

      1. James O'Shea

        Re: Back up and running.

        "Just use a OpenNIC Project server near you: https://www.opennicproject.org"

        Oh, really? Let's see those servers... From their page:

        81.2.237.32 (ns1.cz) -- 95.79% uptime

        31.14.133.188 (ns5.it) -- 94.30% uptime

        212.47.233.10 (ns1.idf.fr) -- 84.65% uptime

        89.18.27.34 (ns1.buh.ro) -- 89.33% uptime

        Most interesting. So, because you're paranoid about Google and its 99.999% uptime, and its speed, and its not giving a damn where you connect to, you're willing to go to a server in the Czech Republic or Romania. Or, God help us, Italy or France. And put up with uptimes of as low as 85%. I _do_ wonder what the Czech, Romanian, Italian, or French governments might want to have to say about stuff flying past their borders. I'm pretty sure that the only diff between them and the US and UK governments is the level of power and competence. Sadly, I do not usually think of 'competence' and 'Italian Government', or 'Romanian Government', or even 'French Government' in the same sentence. Insufficient data about 'Czech Government', but a quick Google (yes...) indicates that they aren't much, if any, better. (I'm not sure if there's been a competent government in Italy since Ferderick II was Holy Roman Emperor...) (Yes, I know, the Holy Roman Empire was not Holy, was not Roman, and was not an Empire. Now go away.)

        And, of course, I'm on the wrong side of the Atlantic. The site doesn't list any servers over here. What, no Canadians? Not even any Mexicans? Colombians? Anyone?

        1. Anonymous Coward
          Anonymous Coward

          Re: Back up and running.

          I'll admit to using OpenNIC for a while, it was after reading a post here on El Reg I tried it.

          Having done so I think their 80% + figures are a bit over the top.

          My experience was so bad I was looking at local problems before I realised OpenNIC were so crap.

          I really didn't believe they could be that bad, another life lesson learned.

          I like life lessons, they remind me I am both stupid and clever at the same time.

      2. Unubtanium

        Re: Back up and running.

        Or just use Root DNS servers FFS!! Like 4.2.2.1 or 4.2.2.2!!

      3. jason 7

        Re: Back up and running.

        Yawn...

  9. AndrueC Silver badge
    Headmaster

    Um, I know it makes for a slightly amusing headline but so far it seems to be a BT Retail issue so it has nothing to do with Joe's departure.

    If Joe had tripped over any of his cables on his way out there'd be howls of protest from more than just BT customers ;)

  10. TonyJ

    Anyone else have DNS issues full stop with them?

    Now I know BT (Infinity for Business) do oddball things with DNS lookups but I've found for the last god-knows-how-many-months that if I use the router as the DNS server, I get no lookups.

    If I change the clients to use, say 8.8.8.8, then everything is peachy.

    It's all a bit odd but I've been too lazy to do any real digging.

    1. Cynical Observer
      Black Helicopters

      Re: Anyone else have DNS issues full stop with them?

      Got rid of the BT HomeHub in favour of a Draytek and have tended to pick from this list of DNS servers for the last year

      http://public-dns.tk/nameserver/gb.html.

      Did have to change servers on one occasion as the DNS server was unreachable but otherwise, it tends to be fairly reliable.

      If nothing else, my thoughts are that using the DNS servers of one ISP when the broadband is provided by a second should confuse mix things up nicely. :-)

      1. Known Hero

        Re: Anyone else have DNS issues full stop with them?

        .tk ????

        Is that extension not reserved for phishers ;)

        *god yes i used to have a .tk address !!! a long time ago.

    2. Roland6 Silver badge

      Re: Anyone else have DNS issues full stop with them?

      Re: Infinity for Business

      That does raise a question as to whether this fault was wholly within the BT residential network or also included their (BT's) business customers. Certainly as a user of the OpenReach Wholesale network, I've not noticed any issues with my service.

  11. chivo243 Silver badge
    Headmaster

    make the screen shot a bit smaller please

    any smaller and it won't be there...

  12. Eclectic Man Silver badge

    Not entirely true

    As far as I know, JG has not actually left BT yet, it was merely announced yesterday that he is to leave BT to take over at Nationwide Building Society in the first half of next year.

    1. Steve Davies 3 Silver badge

      Re: Not entirely true

      Gardening Leave perhaps?

  13. Gert Leboski

    Just use your own.

    I'm on Virgin Media, but for years have run a DNS server at home. No DNS forwarders, just go straight to the root servers. You could run that on a Raspberry Pi with very little cost. Haven't had a DNS related snafu hit me in all that time.

    1. John Sager

      Re: Just use your own.

      I used to run a DNS server like that, but I found that the NS servers for some domains didn't like queries from ISP end user address ranges. It might be better now but I still have my local DNS server forwarding to my ISP's servers to avoid that.

  14. Anonymous Coward
    Anonymous Coward

    Just Thinking Out Loud

    [Anyone stupid enough to use BT as their ISP deserves what they get]

    1. 7-zark-7

      Re: Just Thinking Out Loud

      I must say i've had excellent service from BT. Went to Vispa for a while and then returned to BT. Even managed to get BT to replace phone line from pole to house as it was an old 2wire thing with the shielding hanging off. Solid FTTC (70/18 down/up) and reliable uptime. I work from home 100% too.

      Just giving another point of view.

      1. jason 7

        Re: Just Thinking Out Loud

        Yeah, I hate BT but at the end of the day...they own the infrastructure and for 363 days of the year its pretty darn smooth. I have seen so many customers struggle with this alternative ISP and that 'boutique' ISP, the latest cool ISP etc. etc.

        They all start off with blazing speeds but they all fail or slow to a crawl in the end.

        I tell them "just go back to BT!" They moan, they grumble but the "Hey my internet is really slow!" moans dry up.

        Just how it is. I do make sure they dump the Homehubs though.

        1. Cynic_999

          Re: Just Thinking Out Loud

          I switched to BT as they were the only ISP able to provide FTTC at the time - and have been very pleasantly surprised. I've had one fault (earth leak in the copper from the cabinet causing intermittent slow speed), which received a visit and repair the same day I reported it. I had a partial outage this morning - some routes were unreachable but that now appears to be fixed. Two partial outages lasting a few hours each over a period of 3 years is not something I would consider to be bad service - and the rest of the time my real download speed (not line speed) has never fallen below 50Mbps (which is as likely to be due to a bottleneck at the server end rather than my end), and is usually sitting just above 70Mbps. I usually download between 100GB - 250GB per week.

          1. Anonymous Coward
            Anonymous Coward

            Re: Just Thinking Out Loud

            My experience of BT is - going on holiday for NINE days; then coming back to find my phone bill, a red letter dated the day after the phone bills date, a disconnection notice dated 2 days after that, and no phone.

            I spent nearly a YEAR arguing with them over that, they point blank refused to do anything or to waive the "reconnection" fee, so I ditched my phone line for several years, and only got it replaced when LLU came in.

            The joke was firmly on them, because despite cutting off the phone service, my internet connection worked right up to the line being physically disconnected.

          2. Anonymous Coward
            Anonymous Coward

            Re: Just Thinking Out Loud

            agreed I was the first in our Village to have Infinity several years back now, and get around 75mb download can count the issues I've had on a few fingers within the 4 years

      2. Dave Harvey

        Re: Just Thinking Out Loud

        Of course, OpenReach and BT are SUPPOSED to be 100% operationally independent - so which do you think is the case:

        1) You would have been able to get Openreach to do the same work via another ISP

        OR

        2) There is a "special arrangement" between BT Retail and Openreach.

        If (2), then perhaps someone should tell OFCOM (as if they didn't already know!)

        1. Anonymous Coward
          Anonymous Coward

          Re: Just Thinking Out Loud

          Strangely, independent ISPs seem to have a far better handle on getting OR to do repair work, I have heard BT people moan that the BT system is so poor, it is nearly impossible to track a job request and make sure OR are on the ball.

          Certainly my experience is that the small boys get BTOR out fast, whereas BT drag their feet forever (meaning years in one case!!!)

      3. Anonymous Coward
        Anonymous Coward

        Re: Just Thinking Out Loud

        When were you with Vispa? They used to be a really great ISP, and got problems sorted out very quickly, but a few years ago the co-founder got forced out and an arse took over, since then it have been dire. I quit, and followed the old owner to his new ISP Aquiss.

    2. TonyJ

      Re: Just Thinking Out Loud

      "...[Anyone stupid enough to use BT as their ISP deserves what they get]..."

      Let's see what I get:

      Excellent UK based service on the only occasions (twice in almost 4 years) that I've called;

      Next business day engineer on site;

      Uptime that has been so close to 100% it may as well be;

      Supposedly a 20:1 contention ratio (not that I've noticed one way or the other)

      Apparently, no blocking or forced traffic management based slowdown.

      Superb up/down connection and throughput speeds, even over wifi.

      No closed ports - can (and have) hosted Exchange, websites etc.

      And no, I don't work for or have any affiliation with them. I'm just a happy customer that pays a decent amount for a decent service rather than being one of the many cocks out there that wants to pay feck all for my broadband and expects the same levels of service.

      OK I have weird DNS issues but I just use different ones on each client I have. I can live with that.

  15. lewisgreaves
    Facepalm

    RE: All over the UK

    I've just had a quick look at the list and spotted Wales, Scotland and the south coast of England. Working from that all the others would seem to be somewhere in between those 3 points so "all over" should/would work as a statement.

    1. Ken Moorhouse Silver badge

      Re: RE: All over the UK

      Lundy's internet access was completely unaffected.

      1. Anonymous Coward
        Anonymous Coward

        Re: RE: All over the UK

        This is only a test ;)

        1. Anonymous Coward
          Anonymous Coward

          Re: RE: All over the UK

          The test proved the main packet sniffing hub in Cheltenham is going to need a few beellions of investment if it is to slurp the entire UK data stream in search of Salah Abdeslam or any other bad guys du jour, without interrupting the streaming of cat videos. Otherwise ISPs will have a lot more pissed-off customers to deal with.

          Government announces additional funding for GCHQ. What a coincidence ;)

  16. Blinkered

    Wow - I have my 'as advertised speed back'

    No outage here in Scotland - and for the first time in months (nay years) - I'm actually getting my speeds as advertised. I guess the lack of data on the rest of the network means there is lots of CAPACITY. Something BT to be woefully short of. Even iPlayer isn't buffering hahahaha - off to do some serious downloads (all those ISO images n stuff of course :D

  17. Anonymous Coward
    Anonymous Coward

    24/7 BT

    Depends how hard you complain and how high you get.

    I have the mobile number of the tech I deal with. If he is unavailable he leaves me the contact details of a colleague. Before he goes unavailable.

    Mails are answered usually in seconds.

    Top class, but I had to fight for it.... still, he can't help the fact that a 50 pair underground cable has 49.5 pairs broken (I never knew adsl could still run on a single wire, albeit at a reduced speed). That came after an incident with a neighbour (50 metres away) with a dodgy PSU on an occasionally used computer that caused REIN or SHINE that took weeks to pin down. But we got it sorted and I learned a lot.

    Even when the faults are closed I can contact him anytime.....

    Ironically a friend was with Virgin and the last 3 years suffered huge problems. Virgin blamed OR and vice versa. His support phone bill was immense "but they are cheaper"

    Eventually on my (insane) suggestion he went to BT. 2 days after the same issues. One call, and a day later, the engineer reported there was a fault on a bit of kit in the exchange that would have also affected several other users.....

    Problem solved.

    Regrettably until OR are separated completely, and publicly reachable & accountable, this will continue....

    I still wouldn't generally recommend BT, but we are in the sticks and have little choice.

  18. 2460 Something
    Coffee/keyboard

    "There has been absolutely no suggestion he took the router config data with him when he left the building"

    Love it, Genuinely laughed out loud and then had to explain to rest of office :D

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like