Olympic cock-up knocks East London off Internet
Businesses and customers in East London were still without internet and phone service this morning after Olympic contractors dug through a fibre optic cable. Contractors working on an Olympic venue near Stratford, east London, dug through the fibre optic cable on Saturday knocking a BT exchange at Old Street offline. One of the …
is some sort of communication protocol whereby the loss of a node doesn't mean the whole thing stops working...
Problems in the South East as well.
Thank you Reg! - More informative than 02 Broadband!
Ah finally some information! I'm with Be, and my broadband is down because of this. Though Be havent said anything about it - and keep posting rediculous recovery times. From the amount of mess this has created, looks like I'll be on the datacard for days...
Someone should tell the management at BT and at TI that diversity means more than just ensuring a decent ethnic balance. It also means running in two cables through separate routes into a building and having them travel in different directions. Reroute as required the traffic using extra capacity that you should have!
BT: F- must do better
Well maybe if it was more interesting they would have paid more attention and not hit the cables!
Taxi!
When BT say the damaged stuff is in a deep tunnel, the guy who was doing the digging must have been really enthusiastic with the digging machinery.
The not-very-public rationale for putting BT cables in deep tunnels was that they were hard to break with bombs. Obviously, Mr. Bamford hadn't come to their attention.
who decided to host that apology on the domain that's unavailable?
"if you have a problem with your email services, please email us on dumb@ass.com"
d'oh!
BT state :-
"A large thrust borer being used by a third party construction contractor on work not related to BT‚s business penetrated one of our deep level tunnels in that area. The thrust borer has completely penetrated the tunnel and has caused material damage to both the copper and fibre cables in the tunnel.
BT engineers are working around the clock to restore service as soon as possible.
More than 70,000 PSTN customers were initially affected, though almost half of these have now been restored. Several thousand broadband lines, a large number of private circuits and Featurenet services have been impacted. Emergency and Police services were also affected but these services have been restored.
The tunnel is so badly damaged that it is not possible to work there pending structural support work and it is likely to be three days or more before that work can be completed and damage repair work undertaken. "
"Workmen were using BORING equipment some 34 feet below the surface when they hit the cable at about 2.00pm on Saturday"
Perhaps they should use more interesting equipment next time so they pay more attention.
As result of this particular piece of contractual shafting, customers of the ISP Namesco and others enjoyed a lost weekend from around 1pm on Saturday until 3am this morning. Luckily, a recorded message on its telephone support line directed customers to a helpful page of updates.
Alot of people i keep in contact with got their connection back after a good 24hours without being fixed, unfortunate for the people who are still at a loss but for once the loss of internet isn't from BT themselfs.
I go past Stratford train station every day and see the drilling equipment that they use - you are talking a hundred-foot-high, several feet diameter vertical boring drill, not some idiot with a Black & Decker.
Which makes it WORSE in my opinion, because you can't exactly NOT know where you're putting that thing. It looks like they are drilling to fill foundation columns for an extension of the station, in that particular case, but there's a lot of works around Stratford at the moment for the Olympics. However, my prime candidate would be that particular one ... maybe I'll shout out from the station later today and see if they point the finger.
have BE updated option 2 on 08081013430 yet?
is this related? My phone still works, just no interwebs.
Certainly having a huge boring machine chomp into your deep level tunnel is rare. But this seems to have taken out much of an exchange in the UK's capital city.
Are BT so cheap or poor as to not have fibre ring technology serving their major exchanges? The technology is decades old!
We're nowhere near London (thank the gods) but there's some of those "large thrust borers" (sounds like a proper tool!) operating near a site into which I have an awful lot of fibre.
The BT report I saw indicated the tunnel was 32 *metres* below street level, but even 32 feet is well below the reach of most JCBs. There were 70,000 copper phone lines running through that tunnel as well as a lot of fibre links serving that exchange, some mobile phone masts and ISPs. For the copper wires, diversity isn't available (well, you can sometimes get a second line differently routed as a backup from the next nearest exchange, but obviously that's a very rare configuration) and for most of the others, it would be an extra cost option; it seems most of the companies involved chose not to pay the extra for full resilience. How many of us are actually willing to pay an extra 30% or 50% for our Internet access to average one hour of downtime instead of eight each year?
With an iPhone, if ADSL goes out I can still access e-mail and the web - and do so on the move as well; I did have a 3G datacard, but haven't used it for a while. This kind of outage might take both out for a while, but at least getting the mobile network back up will be quite a high priority.
@mothballs...
Absolutely! What is life without internet?
Two days is the worst duration for downtime. I find that on the third, I start to dimly recall the things I used to do without it.
Bob
If it's in a different cable it is diverse. If it goes by a totally different route it is separated. What you want is separation not diversity.
You really have to be into your job to remember that. <G>
Mine's the one with the pair of 81s in the pocket.
didn't they THINK to see what was under the ground they were digging up?!!! Luck there wasn't a main gas line eh?!!! How thick can you get.
From the photo I've seen, it looks like a pile-driver or something similar has gone straight through a tunnel perhaps 5-6 feet across. The tunnel is completely blocked by something painted yellow.
As this was 32 metres, or well over 100 feet below the surface, I really hope these contractors know where all the tube lines are around wherever they may be working or something much more tragic could happen. For sure there is going to be a big bill to be paid for by somebody. Quite apart from the damage itself, I imagine some outfits will want compensation.
They are doing massive amounts of work round there. They know perfectly well that there are all sorts of things underground. The line they hit was a major trunk, not an odd bit of cable.
How can they have not known where that line was, long before they started digging? Did they not do any checking, or did they manage to miss something that size during the check?
Maybe all the people and companies who lost service can get together and sue the diggers for incompetence.
not sure if anyone also noticed but a number of banks were also taken out when the cable was cut ... settlement processing for sunday is a major issue now
I live in Dorset, but still lost my net access because of this. I'm with namesco/NDO who have 2 support lines. The premium rate one said that all systems were operating normally. The local rate one said that there was a problem with BT, and directed me to the website support page for updates.
Not a lot of point telling me to look up something on the web if I'm complaining that I can't see the web. Yes, I know that namesco have a dial-up 'backup' for their broadband customers, but all the details about it are on the website, which I can't see. Besides, who has a modem in their machine these days?
Thank you Reg for finally letting me know what's going on.
Strange, but I'm on BT out of the Stratford Exchange and not 150 metres from the Olympic Park, and have experienced absolutely no loss of service at all...
Diverse routing is a customer option and some customers choose not to pay the extra. As with all things, you get what you pay for so don't blame the carrier for "all" of the issues.
...the builders dug exactly where the planners told them to dig to avoid the documented location of all underground obstacles, but the documents were wrong? It's easy to point the finger of blame at the workmen on site now, but it wouldn't be the first time people have gone digging in an area that was supposedly clear of obstructions only to discover a tunnel/pipeline/etc that either was shown as being in a different location, or wasn't even shown in the first place.
presumably because no-one could ring them to let them know??
What happened to Disaster recovery? As an (ex) IT Manager I ensured DR procedures were in place to limit effect on users. I can't get a job as I don't have the current NVQs, MCSE, ITIL etc,. I always felt doing the job was worth more to the company than attending training courses where a newbie tries to teach you how to do what you have been doing for 20 years.
So I am in Dorset; A fault occurs in East London and I lose my internet connection for over 48 hours?!
My son is around the corner, same ISP and he was unaffected. Probably the best quote was from Tiscali BB support when I was informed that I had a local BT fault. I suppose 140 miles is quite local to someone!
What happened to Tiscali/BT back up procedures that should have rerouted to alternative servers or routes.
Lets hope the UK ID DB is located in the same area and uses the same management teams.
Die hard 3 when they nicked the gold reserves..
They will have a shock if they try and Rob a bank theres bugger all in em they will end up oweing the bank cash lol
It's quite a coincidence how the three of us whose webs went missing for an entire weekend all live in Dorset. I firmly believe this warrants investigation at the highest levels, or something.
Sign up, sign up for The Register's weekly mobile & wireless newsletter - click here
