The BT network has failed for the second time in a few days, again cutting off users in and Scotland and northern England. The ongoing outage is also affecting third party ISPs who use resell access via the national telco. According to ADSL24, BT is blaming the latest problems on an unnamed hardware vendor. Andrews and Arnold …
I thought the internet was conceived to deal with exactly this kind of problem.
Maybe it's that the internet was, but BT wasn't. What a piss poor show.
It wasn't actually designed to survive disruptions
The internet wasn't actually designed to survive disruptions, whether it's technical failures or the commonly held belief of surviving nuclear war. It was created as a US Department of Defence testbed, though not to create a resilient way to communicate during a war, but simply as a way to connect up scarce (at the time) large-computer resources.
The further development was carried out by educations institutions, and while the distributed layout and multiple paths allowed for some degree of resilience, it was never designed for that purpose, and still isn't.
Either way, what inherent resilience does exist, is on a much larger scale than a bunch of users using a single ISP, in one region within a relatively small country. The internet is still up and running, if I'm not mistaken, and 99.9999% of internet users probably aren't even aware anything went wrong.
And as for knocking BT, yes, I suppose they should be running duplicated and redundant switches, hubs, routers, and every other piece of networking equipment they across their entire network, just so b166er can't accuse them of not following the non-existent myth that the internet is completely resilient.
Every single ISP in the land suffers from failures, from time to time - but if 100% resilience is that important to you, use multiple ISPs.
"I thought the internet was conceived to deal with exactly this kind of problem."
No you didn't. You didn't think this, you were spoon fed it by some idiot* and decided it sounded good. I bet you tell it to other people all the time without any actual understanding of what it means in real life. Hint : "the internet" didn't break.
*screw you, slashdot, seriously.
Due to expense there's probably very little or no redundancy in the infrastructure.
If there's only one route the traffic can go then you're stuffed if that route fails.
Edinburgh 'super exchange'
Edinburgh is much more important than the green box at the end of the street - it's one of only 10 places in the system where your connection is authenticated and the IP address dished out to your router.
http://www.kitz.co.uk/adsl/equip2.htm is a pretty comprehensive explanation - the diagram at the top is clear (and the site is still one of the best places to go for real information on all things ADSL). Skip down to the section called "RAS / B-RAS (Broadband Remote Access Server)"
"Due to the way BTw perform their routing, you may find that the RAS that you connect via is not necessarily the nearest to you location wise - its also quite common for different DSLAMs at your exchange to be routed to several different RAS's." This would explain why some services were unaffected (see later posts on this discussion).
Single Point Of Failure. . .
You get what you pay for
BT is like any company. It provides what the market is prepared to pay for.
No-one pays for resiliency so they don't get it.
Few people pay for a decent service so they don't get that either.
...or maybe you don't
BT IPstream = all down
BTnet Premium = all up
BT IPclear = all down
Yet IPclear is the most expensive. So maybe you don't.
AndrueC sums it up neatly. 100% availability comes at a price that would make the service unaffordable for many. Telecoms is all about trade-offs, get used to it :)
i thought it was made of cats! why are bt abusing teh kitteh?
Wasn't A&As fault...
No, it wasn't our fault! :-)
Edinburgh, Stirling, Bannockburn ADSL All Dead
I work in Edinburgh and have machines always-on via Stirling and Bannockburn exchanges. All died at about 10AM, have come back on intermitantly, but far from reliable at the moment.
MPLS site-2-site lines have survived though - and these run through local exchange so its only the internet connected stuff that appears to have been effected.
Three's 3G service also working fine.
Orkney just came back on-line, maybe BT's equipment isn't waterproof!
... or futureproof!
So BT have got most of Scotland with a single point of failure. Thats what I like about the company: their predictable in their incompetence!!
I mean, how much profit do they make in a year? Are they too busy paying the board bonuses to invest in decent redundancy?
Status phone line
Not only did the internet connection die, but the service status phone line melted too. All I was getting was engaged tone.
For a telecommunication company, BT are bloody difficult to communicated with!
The title is required, and must contain letters and/or digits.
And how exactly are you supposed to get the service status update if your ADSL is down. Cant even get to the BT webpage to check the phone number. and since my phone is on o2, I have to walk into the garden to get a data feed and pull the web page up in circa 5 minutes.
The title is required, and must contain letters and/or digits. → #
Also very irritating is that when you phone their 0800h to report that you have lost your broadband, there is a recorded message suggesting you report the fault online.
When you get through to the operator it takes five minutes or so to tell them that your broadband is not working because of all the questions they ask, they then test your line. Why not a simple automated process where you confirm your telephone number then press a number to indicate complete broadband failure. That way they very quickly get a picture of what areas affected.
down here too...
Broadband has been up and down since about 9am here.....
Started with loss of connection for roughly 5-10 seconds at a time - moving to several minutes at a time - then a full hour or so outage and now back to intermittent up and down......
Thank god for Three MiFi devices!
advised to reboot their equipment
what kind of shit do they think people have at their end?
I live a good 10k from the exchange and my telephone wires get lost in woods and always have a high noise level.
Even my totally shit 5 year old router can spot when the service comes back without needing a reboot.
Advising people to reboot their equipment=fuck off I'm crap at my job and having a bad day.
It's the old
It's the old "Have you tried turning it off and on again" excuse!
Paris, cos she likes to be turned on again after an outage.
The title is required, and must contain letters and/or digits.
People who call for help before rebooting equipment=fuck off, I'm crap with hardware and having a bad day.
advised to reboot their equipment #
When I eventually got through to the call centre in India I was told to do a reset to defaults. I didn't
BT its not Rocket Science
They always have to blaime someone else, or if there is a fault with there stuff .. they fix it and
turn around and say ... no fault was found. Personally I blaime Dotty the Goat.
I just spent an hour this morning trying to work out why our dual redundant network connections, across two ISPs and two exchanges, had both died.
It seems we are dual redundant, but BT arent! Idiots!
You're clearly not truly redundant then...
"It seems we are dual redundant, but BT arent! Idiots!"
You're not dual redundant if both your ISPs are just punting BTWholesale. It's amazing how many idiots make this same fundamental mistake. They buy "redundant" services from two different providers without some fairly basic checks that would reveal the two ISPs to be going through exactly the same equipment. So it's not just BT who are idiots, you're the one paying fora redundant service that isn't.
If your local exchange doesn't have any ISPs that aren't on BT Wholesale and you're not covered by the likes of Virgin then there is very little point in getting ADSL from two providers. You could try 3G as a fallback, always assuming you can get a good 3G signal.
Way to go in using two ISPs who rely on the same backbone.
What moron set that up for you? Two isps that use BT wholesale, brilliant.
Even if you have no cable or LLU there's 3G for emergencies.
Take a guess....
I wonder if they outsource their IT to BT? LOL
And who IS your second provider on a Market 1 exchange then?
And for rural Scotland, which was affected by these two snafus, and where there's no 3G coverage, and your exchange is Market 1, who exactly WOULD you use for your second choice of provider?
Don't go accusing people of having a poor setup unless you're sure they have a choice....
Satellite's always an option. Or dialup. Or VHF link.
Expensive? Slow? Of course, but how much is availability worth to you? What figures did your Business Impact Analysis give? you. You did do one, didn't you?
"Don't go accusing people of having a poor setup unless you're sure they have a choice...."
Of course you have a choice, even if it's expensive or slow it's still a choice. And even without expensive or slow backup there is still a choice - don't bother with the second line. Otherwise you are wasting quite a lot of money.
I remember a client of mine who had an E1 leased circuit, with ISDN backup *and* dial up behind that. Sounds good doesn't it? Except that all the lines entered both buildings through the same service ducts and one day somebody went through a duct when digging a hole for a replacement telegraph pole. After that incident they took my original advice and kept the E1 line with a two hop microwave link as backup. No point paying for a cheap redundancy option if it doesn't work.
is this related to the Orange Mobile Data Network failing too???
Went down Monday (1st) afternoon and this morning (3rd)...
Orange's data network has been crap for going on 6 months.
O2's data network is down today.....
...and when I asked them earlier they had no idea when it would be back on line again,
You're clearly not truly redundant then. ???
No one is truley redundant .. unless of course you work for one of the dying Quango's - then you will be in due course.
You pay for money and take your chances. You could go for leased lines .. with diferrent end points and along come a Pikey and dig up your precious cable .. and does like Gollum. Then both leased lines go down.
So go easy on the poor bloke and get back to work you freeloaders.
The post is required, and must contain letters.
Not sure why that's a joke, because BT's in-house security team is referred to as 'Hadrian' :)
You get what you pay for - oh yeah?
There have now been THREE major failures in six days at BT Edinburgh. Friday, then Saturday afternoon when no one was looking or publishing papers, then again on Wednesday.
BT India seemed completely unaware of the problems..... so what's new?
Support lines might be made of the same chocolate as the Edinburgh facility.
What hope of BT either giving details of the issue, getting a longer-lasting fix, or simply going back to tins and string?
Bad management, bad equipment, bad maintenance? Is it time for tartan dongles?
Meanwhile, someone has wired up Mount Everest with 3G. I bet they get better service than the one BT provides.
Getting sick of this
Thats the 5th outage in so many months for me. Sadly I only have BT kit in my exchange and moving to another ISP will still mean the same crappy speeds due to BT's shit aluminum telephone cables they have in my area so I get less than a 1mb speeds on the main exchange. How about I pretend to pay you seeing as your only pretending to provide me a Internet service ffs.
I'll get my coat ready as I will be heading to the pub straight from work.
I ought to be sympathetic, but I ain't. The way your rant is worded it seems that you are one of those numpties who think that fast internet access is some sort of a right. It isn't.
If you're that keen on fast internet access maybe you should have thought of that when buying your house. Even BT have managed to work that into their advertising.
When I bought my house....
...there was no fast internet access, only dial up.
I'd be prepared to pay more for FTTC or even FTTP, but at present, on a Market 1 exchange with 3,200 odd lines, there is no date for that and not really a lot of chance of it happening. I used to have 5 Mbps ADSL, but cable damage has reduced that below 2Mbps and BT don't class that as a fault, it has to go somewhat lower. How a factor of 3 speed reduction is not a fault is beyond me, but that's how it works apparently.
I don't really want the speed, although kids gaming and watching Youtube makes me want it more, I just want something reliable that isn't dependent on dodgy copper connections lying in flooded chambers.
Maybe this winter the whole bloody thing will stop working and at least then BT can be made to fix it.
All is still not well here in NI.
SDSL has been up and down like a yo yo since before 12 today. Perversely my cheapie consumer grade ADSL backup line is still running strong. Fingers and toes crossed...
Thats because your SDSL connection will be provided over IPStream which whats suffereing at the moment.
As compensation.. i demand fibre be installed into my area.. without the race to infinity!
BT - totally incompetent
Don't they have duplicated fibre systems routed separately backing each other up?
Basic engineering - it's called redundancy.
You're presuming this is a transport problem - it could be authentication or a host of other issues. Those too should be backed up but sometimes shit happens.
I think my point is that full resilience is possible - but it's expensive, and £15 a month DSL doesn't really pay for that. I've bought fully resillient circuits before and for DSL equivalent speed - say 8Mbps it's going to be a hundred times £15 a month, at least. Not just here, but anywhere in Europe.