back to article Yesterday: Openreach boss quits. Today: BT network goes TITSUP

Less than 24 hours after BT Openreach chief Joe Garner quit the telco's troubled infrastructure division, BT customers all over the UK are saying they can't get online – with the apparent network outage possibly taking up to three days to fix. The outage manifested as a packet routing problem, with some unfortunate souls …

"There has been absolutely no suggestion he took the router config data with him when he left the building"

Shouldn't this be accompanied by "at the time of writing the start of the start of this sentence"

4
0

"The Register has contacted BT for comment on the outage. We'll update this story if we hear back from them."

You might have to wait 3 days, they can't get online at the moment.

19
0

More like 3 hours than 3 days, solved already by BT Service page

1
1

Considering mine had been down for more than 3 hours when I left the house this morning, that seems unlikely...

4
0

More like 3 hours than 3 days, solved already by BT Service page

The problem actually started around 9pm Monday night, so took over 15 hours to fix, which is hardly impressive. Though better than 3 days, perhaps they're under promising and then over delivering.

3
0

Depends how you count the "days".

15 hours is almost 2 "man days" based on an 8 hour day and 1 person.

Maybe with 1 trained person and 1 trainee that might make it 3 "man days" in 15 hours.

0
0
Silver badge

You have to correct for "BT man days", so that would be 3 hours per day. The other 5 are spent in a van parked in the countryside, eating sandwiches, texting and listening to the radio

0
0
Anonymous Coward

All Over The UK?

So the south east, East Anglia and the East Midlands is "all over the UK" is it?

1
8
(Written by Reg staff)

Re: All Over The UK?

Given that I've seen other angry Twitterererererers shouting at BT with locations given from many other places, no. Try reading the story, eh?

13
0

Re: All Over The UK?

From BT's service status:

"Glencarse - 01738 (estimated clear date 20/11/2015)

Belford - 01668 (estimated clear date 20/11/2015)

North Weald - 01992 (estimated clear date 20/11/2015)

Husband Bosworth - 01858 (estimated clear date 20/11/2015)

Buckland Newton - 01300 (estimated clear date 20/11/2015)

Glenwherry - 02825 (estimated clear date 20/11/2015)

Brenchley- 01892 (estimated clear date 20/11/2015)

Ferndown- 01202 (estimated clear date 20/11/2015)

Treforest - 01443 (estimated clear date 20/11/2015)

Rothley - 01162 (estimated clear date 20/11/2015)

Dinas Mawddwy - 01650 (estimated clear date 20/11/2015)

Ashreigney - 01769 (estimated clear date 20/11/2015)

Watton - 01377 (estimated clear date 19/11/15)

Chichester - 01243 (estimated clear date 19/11/15)

Swansea - 01792 (estimated clear date 19/11/15)

New Cumnock - 01290 (estimated clear date 19/11/15)

Churston - 01803 (estimated clear date 19/11/15)

Colwall - 01684 (estimated clear date 19/11/15)

Painswick - 01452 (estimated clear date 19/11/15)

Lydbrook - 01594 (estimated clear date 19/11/15)

New Cross - 0203 0207 0208 (estimated clear date 19/11/15)

Broadstairs - 01843 (estimated clear date 19/11/2015)

Bayston Hill - 01743 (estimated clear date 19/11/2015)

Burton Bradstock - 01308 (estimated clear date 19/11/2015)

Magherafelt - 02879 (estimated clear date 19/11/2015)

Bunbury - 01829 (estimated clear date 19/11/2015)

Yockleton - 01743 (estimated clear date 19/11/2015)

Lisbellaw - 02866 (estimated clear date 18/11/15)

Alyth - 01828 (estimated clear date 18/11/15)

Epping - 01992 (estimated clear date 18/11/15)

St Boswells - 01835 (estimated clear date 18/11/15)

Midhurst - 01730 (estimated clear date 18/11/15)

Sedgley - 01902 (estimated clear date 18/11/15)

Carrickmore - 02880 (estimated clear date 18/11/15)

Salisbury - 01722 (estimated clear date 18/11/2015)

Walthamstow - 0203 0208 (estimated clear date 18/11/2015)"

Feel free to Google some of the locations to see how all over the UK this is.

5
0
Anonymous Coward

Re: All Over The UK?

Well - the bits that matter anyway...

5
0
Silver badge

Re: All Over The UK?

Re: "From BT's service status:"

An impressive list, which raises a question as just how their network is structured and hence whether sufficient consideration has been given to redundancy and avoidance of single points of failure.

Unfortunately, for those who like to throw stones, I expect all of the other major ISP's to have similar problems in their infrastructures.

2
0
Silver badge

Re: All Over The UK?

Presumably the routing tables had confused Glencarse with Glencelbow.

9
0

Re: All Over The UK?

> So the south east, East Anglia and the East Midlands is "all over the UK" is it

Of couse it is stoooopid!!!! Don't you ever watch the weather forecasts?!!!!

3
1
Silver badge

Bah!

Your packets are important to us. Please stay online for the next available router.

7
0

Fixed for me

It went down for me at about 9pm last night for certain routes. It started working again at about 1pm today.

The worst thing was there was no easy way to report it to BT, you can only report a BT broadband issue by going to their website and running the broadband live test, which we couldn't access!

6
0
DJV
Silver badge
FAIL

"no easy way to report it to BT"

Obviously another one from the "No keyboard detected. Press F1 to continue" school of thought stupidity.

11
0

Admittedly - they don't make it easy to find the information...

24 hours a day, 365 days a year.

The best time to call is

between 12 noon and 4pm.

Please make sure you're at your computer when you call.

From a landline: 0800 111 4567

From a mobile: 0330 123 4567*

From abroad: +44 179 359 6931**

0
0
Coat

Thieves helping Openreach to go all superfast everywhere

If thieves are going for copper, it may be a good time to roll out that promised superfast fibre.

Mine's the one with empty pockets, promised.

0
0

Re: Thieves helping Openreach to go all superfast everywhere

That can't be the reason for my outage this morning - no copper here, bloody aluminium :(

3
0

Back up and running.

As per Peter 26 said, some parts were still running (Netflix thankfully) last night and I found switching my router DNS to Googles helped get a bit more back this morning.

All speedy again now.

I guess it was their DNS servers down...again.

0
0
Len
Silver badge

Re: Back up and running.

Seriously, people should stop voluntarily sending ALL their DNS traffic to Google. All the sites you visit, all the mail servers you check, all the sync services you use, all the FTP connections you make etc. all sent to Google.

Just use a OpenNIC Project server near you: https://www.opennicproject.org

5
5
Silver badge

Re: Back up and running.

"Just use a OpenNIC Project server near you: https://www.opennicproject.org"

Oh, really? Let's see those servers... From their page:

81.2.237.32 (ns1.cz) -- 95.79% uptime

31.14.133.188 (ns5.it) -- 94.30% uptime

212.47.233.10 (ns1.idf.fr) -- 84.65% uptime

89.18.27.34 (ns1.buh.ro) -- 89.33% uptime

Most interesting. So, because you're paranoid about Google and its 99.999% uptime, and its speed, and its not giving a damn where you connect to, you're willing to go to a server in the Czech Republic or Romania. Or, God help us, Italy or France. And put up with uptimes of as low as 85%. I _do_ wonder what the Czech, Romanian, Italian, or French governments might want to have to say about stuff flying past their borders. I'm pretty sure that the only diff between them and the US and UK governments is the level of power and competence. Sadly, I do not usually think of 'competence' and 'Italian Government', or 'Romanian Government', or even 'French Government' in the same sentence. Insufficient data about 'Czech Government', but a quick Google (yes...) indicates that they aren't much, if any, better. (I'm not sure if there's been a competent government in Italy since Ferderick II was Holy Roman Emperor...) (Yes, I know, the Holy Roman Empire was not Holy, was not Roman, and was not an Empire. Now go away.)

And, of course, I'm on the wrong side of the Atlantic. The site doesn't list any servers over here. What, no Canadians? Not even any Mexicans? Colombians? Anyone?

10
0

Re: Back up and running.

Or just use Root DNS servers FFS!! Like 4.2.2.1 or 4.2.2.2!!

0
0

Re: Back up and running.

Yawn...

0
0
Silver badge

Re: Back up and running.

I'll admit to using OpenNIC for a while, it was after reading a post here on El Reg I tried it.

Having done so I think their 80% + figures are a bit over the top.

My experience was so bad I was looking at local problems before I realised OpenNIC were so crap.

I really didn't believe they could be that bad, another life lesson learned.

I like life lessons, they remind me I am both stupid and clever at the same time.

0
0
Silver badge
Headmaster

Um, I know it makes for a slightly amusing headline but so far it seems to be a BT Retail issue so it has nothing to do with Joe's departure.

If Joe had tripped over any of his cables on his way out there'd be howls of protest from more than just BT customers ;)

0
0
Silver badge

Anyone else have DNS issues full stop with them?

Now I know BT (Infinity for Business) do oddball things with DNS lookups but I've found for the last god-knows-how-many-months that if I use the router as the DNS server, I get no lookups.

If I change the clients to use, say 8.8.8.8, then everything is peachy.

It's all a bit odd but I've been too lazy to do any real digging.

1
0
Black Helicopters

Re: Anyone else have DNS issues full stop with them?

Got rid of the BT HomeHub in favour of a Draytek and have tended to pick from this list of DNS servers for the last year

http://public-dns.tk/nameserver/gb.html.

Did have to change servers on one occasion as the DNS server was unreachable but otherwise, it tends to be fairly reliable.

If nothing else, my thoughts are that using the DNS servers of one ISP when the broadband is provided by a second should confuse mix things up nicely. :-)

0
0
Silver badge

Re: Anyone else have DNS issues full stop with them?

Re: Infinity for Business

That does raise a question as to whether this fault was wholly within the BT residential network or also included their (BT's) business customers. Certainly as a user of the OpenReach Wholesale network, I've not noticed any issues with my service.

0
0

Re: Anyone else have DNS issues full stop with them?

.tk ????

Is that extension not reserved for phishers ;)

*god yes i used to have a .tk address !!! a long time ago.

0
0
Silver badge
Headmaster

make the screen shot a bit smaller please

any smaller and it won't be there...

0
0

Not entirely true

As far as I know, JG has not actually left BT yet, it was merely announced yesterday that he is to leave BT to take over at Nationwide Building Society in the first half of next year.

0
0
Silver badge

Re: Not entirely true

Gardening Leave perhaps?

0
0

Just use your own.

I'm on Virgin Media, but for years have run a DNS server at home. No DNS forwarders, just go straight to the root servers. You could run that on a Raspberry Pi with very little cost. Haven't had a DNS related snafu hit me in all that time.

0
0

Re: Just use your own.

I used to run a DNS server like that, but I found that the NS servers for some domains didn't like queries from ISP end user address ranges. It might be better now but I still have my local DNS server forwarding to my ISP's servers to avoid that.

0
0
Silver badge

Just Thinking Out Loud

[Anyone stupid enough to use BT as their ISP deserves what they get]

1
7

Re: Just Thinking Out Loud

I must say i've had excellent service from BT. Went to Vispa for a while and then returned to BT. Even managed to get BT to replace phone line from pole to house as it was an old 2wire thing with the shielding hanging off. Solid FTTC (70/18 down/up) and reliable uptime. I work from home 100% too.

Just giving another point of view.

3
0

Re: Just Thinking Out Loud

Yeah, I hate BT but at the end of the day...they own the infrastructure and for 363 days of the year its pretty darn smooth. I have seen so many customers struggle with this alternative ISP and that 'boutique' ISP, the latest cool ISP etc. etc.

They all start off with blazing speeds but they all fail or slow to a crawl in the end.

I tell them "just go back to BT!" They moan, they grumble but the "Hey my internet is really slow!" moans dry up.

Just how it is. I do make sure they dump the Homehubs though.

0
0

Re: Just Thinking Out Loud

Of course, OpenReach and BT are SUPPOSED to be 100% operationally independent - so which do you think is the case:

1) You would have been able to get Openreach to do the same work via another ISP

OR

2) There is a "special arrangement" between BT Retail and Openreach.

If (2), then perhaps someone should tell OFCOM (as if they didn't already know!)

0
0
Silver badge

Re: Just Thinking Out Loud

I switched to BT as they were the only ISP able to provide FTTC at the time - and have been very pleasantly surprised. I've had one fault (earth leak in the copper from the cabinet causing intermittent slow speed), which received a visit and repair the same day I reported it. I had a partial outage this morning - some routes were unreachable but that now appears to be fixed. Two partial outages lasting a few hours each over a period of 3 years is not something I would consider to be bad service - and the rest of the time my real download speed (not line speed) has never fallen below 50Mbps (which is as likely to be due to a bottleneck at the server end rather than my end), and is usually sitting just above 70Mbps. I usually download between 100GB - 250GB per week.

2
0
Silver badge

Re: Just Thinking Out Loud

When were you with Vispa? They used to be a really great ISP, and got problems sorted out very quickly, but a few years ago the co-founder got forced out and an arse took over, since then it have been dire. I quit, and followed the old owner to his new ISP Aquiss.

0
0
Silver badge

Re: Just Thinking Out Loud

Strangely, independent ISPs seem to have a far better handle on getting OR to do repair work, I have heard BT people moan that the BT system is so poor, it is nearly impossible to track a job request and make sure OR are on the ball.

Certainly my experience is that the small boys get BTOR out fast, whereas BT drag their feet forever (meaning years in one case!!!)

0
0
Silver badge

Re: Just Thinking Out Loud

My experience of BT is - going on holiday for NINE days; then coming back to find my phone bill, a red letter dated the day after the phone bills date, a disconnection notice dated 2 days after that, and no phone.

I spent nearly a YEAR arguing with them over that, they point blank refused to do anything or to waive the "reconnection" fee, so I ditched my phone line for several years, and only got it replaced when LLU came in.

The joke was firmly on them, because despite cutting off the phone service, my internet connection worked right up to the line being physically disconnected.

0
0
Anonymous Coward

Re: Just Thinking Out Loud

agreed I was the first in our Village to have Infinity several years back now, and get around 75mb download can count the issues I've had on a few fingers within the 4 years

0
0
Silver badge

Re: Just Thinking Out Loud

"...[Anyone stupid enough to use BT as their ISP deserves what they get]..."

Let's see what I get:

Excellent UK based service on the only occasions (twice in almost 4 years) that I've called;

Next business day engineer on site;

Uptime that has been so close to 100% it may as well be;

Supposedly a 20:1 contention ratio (not that I've noticed one way or the other)

Apparently, no blocking or forced traffic management based slowdown.

Superb up/down connection and throughput speeds, even over wifi.

No closed ports - can (and have) hosted Exchange, websites etc.

And no, I don't work for or have any affiliation with them. I'm just a happy customer that pays a decent amount for a decent service rather than being one of the many cocks out there that wants to pay feck all for my broadband and expects the same levels of service.

OK I have weird DNS issues but I just use different ones on each client I have. I can live with that.

1
0
Facepalm

RE: All over the UK

I've just had a quick look at the list and spotted Wales, Scotland and the south coast of England. Working from that all the others would seem to be somewhere in between those 3 points so "all over" should/would work as a statement.

1
0
Silver badge

Re: RE: All over the UK

Lundy's internet access was completely unaffected.

3
0
Anonymous Coward

Re: RE: All over the UK

This is only a test ;)

0
0
Anonymous Coward

Re: RE: All over the UK

The test proved the main packet sniffing hub in Cheltenham is going to need a few beellions of investment if it is to slurp the entire UK data stream in search of Salah Abdeslam or any other bad guys du jour, without interrupting the streaming of cat videos. Otherwise ISPs will have a lot more pissed-off customers to deal with.

Government announces additional funding for GCHQ. What a coincidence ;)

0
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2018