The Register® — Biting the hand that feeds IT

Blundering workmen cut major Docklands fibre

Alex Walsh

Ouch 

Unhappy

I had a tour of Telehouse in Docklands a few years back. The amount of stuff in there was mindboggling and I'm sure has increased dramatically since. It seems daft to have all this stuff in one place, even if there is theoretical redundancy with other sites.

Dick Lovewell

Blundering workmen? 

Paris Hilton

Or blundering planners? The workmen might have been digging exactly where they were told as far as I can tell. Not that I'm a shovels & bum-cracks fan, but it seemed a tad presumptious to blame them just yet.

Paris, because when her hole's being dug she's happy to show everybody the evidence.

dervheid

That's the weakness with fibre optics... 

No electromagnetic leakage to detect and betray it's presence.

Of course, if the fuckwits had done proper checks before they started...

Anonymous Coward

erm 

I'm a Be customer and could download at 1meg/sec most of yesterday. Good enough for me :)

Loki

Isnt the point... 

Paris Hilton

... that the internet is a distributed network where packets will route via whatever routes are available?

Doesn't this mean that either:

a) 100% of traffic for these providers can ONLY route via the link on the docklands (ie: no other routes physically exist... which surely would be strange for large providers to have no backup links)

or

b) something is really screwed up with IP in the UK (IP as in TCP/IP ... not intellectual property)

Paris - because dockland workers are a big fan of hers.

Anonymous Coward

Hurr 

Thumb Down

GX and O2 are based in Global Switch, not telehouse.

Joe

12 Hours to Fix..... 

Alert

Some of our Branches up north were affected by this. Birmingham, Leeds and Manchester all went down at 1pm. Spoke to our provider (who is provided by GC) several times and was told they had to repair 200m of fibre cable!!! From the email I got this morning from our provider it took 12 hours and came up at 3.30am!!

Anonymous Coward

C4L 

Alert

Surprised they knew their arse from their elbows. They are the biggest group of incompetent idiots I have ever had the mis-fortune of dealing with.

Peter Flint-murray

Ahhhh 

So thats what happened, my normally 14Mbps connection was down to about 300kbps!

Dave Edmondston

Typical UK 

Thumb Down

Everything routed through the centre of London. Why?

Internet - distributed network.

Britain = London = Me, me, me

Everything in this country seems to depend on the arse end of the bottom corner of England.

Simon

Heheh I can hear the conversation now... 

Coat

"Hey Steve, I gotta bloody tough tree root here"

"Then get the pick axe on it Jo"

*thwack*thunk*Bbzzzzttt*

"Oops...."

(yeah ok, it may not have made a short circuit sound but for comic strip effect....)

Mines the one with the detailed map of cable, pipes and anything else important in the pocket.

Pete James

I'm with Dick! 

Got to agree with Dickie Lovey here. I had a teen student holiday job on a building site in a City where a JCB went stright through a whole truckload of telecomms and power cables which looked as though they had been just thrown in . The plans we had got from the council showed them to be about 30 yards down the road.

Every cloud has a silver lining though - we cut the power to the nearby Police station. All the woodentops were forced to come outside and do some work seeing as their kettles weren't working!

Andrew Kelly

Centre of London? 

Do you know where Docklands is?

Having had first hand of these hairy arsed navvies I would put the blame firmly with them. Dig now, worry about the consequenxces later is their motto. And then try to a pull a fast one so you don't have to pay for the damage.

Anonymous Coward

@ Loki 

Happy

HA Ha Ha, You dont work in Telecomms do you?

"a) 100% of traffic for these providers can ONLY route via the link on the docklands (ie: no other routes physically exist... which surely would be strange for large providers to have no backup links)"

Ha Ha Ha Ha, Now back to planning that new 60 Gig flattened "ring" (where the main and protection fibres arent just in the same duct but in the same cable!

AC, because

jeremy

wasn't me guv 

Recently working opposite a hotel being built, in the 8 months it took to build the workmen went through our electric and phone cables at least 4 times.

Very first thing that happened when a workman was asked, was to fetch the foreman who's standard answer before even knowing the problem was "It can't have been us, we are not even digging in that area" ... which sounded a little empty after the 3rd or 4th time.

Stu Reeves

@Loki 

Flame

That's right the magic pixie dust helps those packets get to there destinations all on there own, you click a link to the register and all you packets just magically fnd there way there, just from your pc.

Can we have a bang head against desk icon...

PS read the article, it didn't go down, it slowed down.....

Anonymous Coward

Ahhh... 

That explains why streaming HD video was an issue last night then!

Echowitch

@Dave Edmondston 

Paris Hilton

Hmm I wonder why.....maybe everything goes through London as...

1. Its the Capital of the UK

2. Its the Business Heart of the UK

3. Its where all the Major Businesses are based

4. The South Coast has the UK's Silicon Valley (London to Newbury to Guildford and everything in between.

5. Its the capital and heart of everything as its closest to Europe

Yes Manchester, Edinburgh, Newcastle, etc, are all big cities but frankly none of them are as vital to the economy as London. The same argument can be said of any nations capital.

Paris......as she loves mens cables, and deep dark holes ;)

Anonymous Coward

Backup links 

Thumb Up

its makes me laugh when i hear these isp's are affected, my company use GX which was fibrenet and had our link knocked out, but like many of our other suppliers who also were on this fibre we just moved into a at risk state, running on the other part of our ring... you have to question how tight these isp's are being as the cost of cross london darkfibre is cheep as chips..

Anonymous Coward

Cable 

This was a 288 core fibre however, so its not as though it could have been missed...

James O'Brien

@Dick Lovewell 

Happy

Dont know what is funnier, your comment about the cracks or the Paris one. Good one.

Steve

Diggers, doing it for fun 

Happy

I have two digger stories, one strangely in docklands. The g/f's sister is a geo technical engineer, and was working on a site not far away from Yeo Street, and told the driller (ok, not a digger, but close) right *there* cos it's safe, look at the council supplied plan, there's only London Clay there. Next thing *kzzzzzzzzrt!* and one driller thirty foot away. They'd found a very large electricity main that no-one, not even London 'leccy/EDF knew about.

The second story concerns a council in the Midlands, who had their server room in the basement of the building. Not only did the diggers manage to sever all the comms and electricity lines, they also when through a water main and flooded out the server room. Oooh how we laughed when we took that support call!

Steve

digging 

No amount of maps and planning will help. It's always cheaper to dig wherever they need to and pay someone to clean up the mess than it is to dig carefully with a map.

kain preacher

@AC 

his was a 288 core fibre however, so its not as though it could have been missed...

Um a trencher thats 2 inches (5 cm )wide 5 fee(1.5m) t long can snap that before they know its there .

Maverick

@ Pete James 

Thumb Down

you mean the the police _actually_ started work?

now c'mon we try to be realistic on here mate - there is NO way the useless t*ssers moved away from checking speed camera films!

Sir Runcible Spoon

why docklands? 

Boffin

It's all about power, that's why.

Anonymous Coward

Reminds me of a joke... 

Coat

(That I think I saw here.)

How does a network engineer lost in the desert get rescued?

They bury 5 metres of CAT5 and hitch a ride home with the JCB driver.

Here all week, tip waitress, veal etc.

Mines the one with the 5 metres of CAT5

Anonymous Coward

hurr 

Thumb Down

Any telco/network provider worth their salt *does* have resiliency though. We certainly do, just diverted traffic around the other side of our ring and customers were reconnected within minutes.

Anonymous Coward

The maps are shite 

Stop

Anyone who has any experience in these matters will know that all these cries of "stupid hairy arsed navvies should have looked at the plans" are just bollocks. No doubt, workmen do occasionally mess up, but you should see just how fucking useless these plans are - both the councils and the utilities very often don't have the slightest fucking clue what's where under the road, their plans are totally inaccurate and hence next to useless.

Just take a moment to think about how often your own wiring plans in server rooms and elsewhere are totally unrepresentative of reality - then bear in mind that these guys are dealing with all that wiring buried under layers of tarmac, concrete and muck.

Not that any of you are going to listen to this.

James

Strategicaly speaking ... 

Happy

.. it seems fairly stupid to have major data systems based near a large river, liable to flooding, in the centre of a city that could be a prime target for terrorist attacks.

It would be better to have our main data systems somewhere deep underground in a geologically stable part of the UK. There are plenty of old mines that could be used, as well as all those "not so" secret Nuclear bunkers....

david bates

@Strategicaly speaking ... 

We did,

Kingsway, Anchor and Guardian (London, Birmingham and Manchester exchanges respectively). As far as Im aware Anchor is no longer mainatained (just used for ducting) and is heavily flooded. Dunno about Kingsway and Guardian.

John Dougald McCallum

@James strategicaly speaking 

Emmmm there are not very many of these not so secret nuclear bunkers probibly only seven or eight as they were meant as regional command centers IE a seat of local & military goverment also they are for the most part rather pockey wee places not the kind of environment that I'd want to work in.Also most of the large ones have already been sold of or demolished and as for London being the center of the ISP universe as has been said it is where most if not all the major Goverment and business HQ's are,and I aggree with the posters that said that utility maps are U.S.theyare actualy worse than useless.

Matthew

not found 

> Global Crossing's press representatives couldn't be reached

I wonder why not? where their IP phone, email, etc. not working?

Chris Collins

over reliance on london docklands 

Unhappy

I agree with the concerns on the over reliance of that location for out internet transit, something like 90% of isps are based there and pretty much all the mainstream uk websites, just about all uk internet transit is routed around that area. To some it may seem fine as london is the capital but if that location was to be bombed tommorow we would be up s*** creek connectivity wise.

If you look at germany as a comparison they have major POP's in all their major cities, frankfurt, berlin, hamburg etc. with datacentres scattered around the country. Planned datacentres in sheffield and peterbrough cant come soon enough but we also need some serious major transit laid down so we have other major POP's outside london and manchester.

How many tier 1's have fibre going through a POP outside london in the uk?

Harry

"No electromagnetic leakage to detect and betray it's presence." 

Thumb Up

Well, there's a thought. It probably wouldn't cost much (relatively) to include a bit of conductor in the cable or trunking just for that purpose.

It wouldn't even have to be permanently energised, just turned on when work was needed in the area.

Quick, lets get an anti-patent on the idea, otherwise some country with a backward patent office will fail to spot the obviousness of the idea and the idea will be priced out of the market.

Hmmm, does publication in El-Reg constitute an anti-patent? If the page is indexed by archive.org then surely the idea becomes public domain and ensures that such a patent could be disputed as "already in the public knowledge" before the patent was applied for ???

Paul

Its still SLOOOOOW 

Unhappy

Has been for the past couple of days and doesnt seem to be getting much better, I usually get 14 megabits down, service is now intermittent at best.

Gareth Jones

@Dick Lovewell 

Damn right. I've come across several instances where major cables and fibres have been cut by workmen because their presence simply wasn't documented. You can't blame the workmen for digging through a fibre if nobody told them it was there.

Inachu

The old Q diagnostic center would have been nicd to use on this 

Flame

But too bad they were bought out and the lead programmer for the network tool still owes me $7.

Predicting this kind of network downtime is impossible.

Anonymous Coward

Plans? We don't need no plans! 

Stop

Plans say "dig over there, that's where you are supposed to put the cable."

Workmen say "That ground's too rocky but there's clay this side of the street. We'll stick it in over here, nobody will ever know the difference."

Proof? Bundles of cables and pipes in the right place, just a few in the wrong place... not necessarily in this instance (was it?) but just look down practically any street in our sceptic isle.

Danger! Fsckwits at work!