back to article Hapless Network Rail contractors KO broadband in Uxbridge

Hapless Network Rail contractors drove a pile through 30 cable ducts, cutting phone, broadband services and many cell towers in the Uxbridge, Middlesex - potentially knocking services out until next week. Services were cut on Wednesday, with Openreach customers reportedly being told it could take more than a week to fix. Up to …

  1. Anonymous Coward
    Anonymous Coward

    Incidents like this show why putting everything in the cloud is not a good idea.

    1. wyatt

      Nothing to do with the cloud, where ever it may be data needs to be accessed to be processed. If you can't access it it doesn't matter where you process it.

      What does matter is your ability to deal with an outage of this kind. Having said that there are sometimes that you can only do so much before you come to a grinding halt and go to the pub.

      1. OliP

        except when your internet gateway, domain controllers, storage and applications all live in the cloud - as many do now.

        Redundancy - its a beautiful thing

      2. Anonymous Coward
        Anonymous Coward

        ooh oooh oooh....

        Ever heard of local networks and processing using systems local to the users and no internet or external connection required?

      3. a_yank_lurker

        "Nothing to do with the cloud, where ever it may be data needs to be accessed to be processed. If you can't access it it doesn't matter where you process it." - If the data is local to you then you can process it. The issue noted is users may not be able to access remote, cloudy data whether it is on an internal company network in the "cloud".

      4. Anonymous Coward
        Facepalm

        Nothing to do with the cloud

        @ wyatt: "Nothing to do with the cloud, where ever it may be data needs to be accessed to be processed. If you can't access it it doesn't matter where you process it."

        The solution being to not put all thirty cables through the one cable conduit.

      5. John Brown (no body) Silver badge

        "Nothing to do with the cloud, where ever it may be data needs to be accessed to be processed. If you can't access it it doesn't matter where you process it."

        Like all those small businesses conned into "upgrading" to Office365 instead of a locally installed MSOffice2013 and can't even type a letter now?

  2. chivo243 Silver badge
    Devil

    Reinforces our saying in the office...

    Ouch! My fibre!

  3. Pen-y-gors

    Redundancy?

    What is needed is redundancy, no single point of failure (at least on a large scale - someone taking down a single phone poll or cell mast is different) But the Uxbridge 'exchange' should have multiple connections, going in different directions, possibly even a nicrowave backup connection. Of course this costs serious money, and the customers have to decide how much they're willing to pay to prevent a 1 in 10000 chance of losing connection for a week.

    1. sandwich

      Re: Redundancy?

      Someone will be once it's all repaired.

  4. Anonymous Coward
    Anonymous Coward

    I regret to announce

    Services between here and the cloud been disrupted due to piles on the line.

    1. GitMeMyShootinIrons

      Re: I regret to announce

      You need more fibre in your diet.

  5. lglethal Silver badge
    Facepalm

    Enquiring minds and all that?

    So were the cable positions not properly makred on council plans? Or was the Network rail check something along the lines of - "What are the chances of there being cables laid this close to the Train lines?"?

    1. Anonymous Coward
      Anonymous Coward

      Re: Enquiring minds and all that?

      Why would they be on council plans on Network Rail property?

      1. Martin Summers Silver badge

        Re: Enquiring minds and all that?

        Oh believe me, some people think councils are responsible for absolutely everything.

        1. Doctor Syntax Silver badge

          Re: Enquiring minds and all that?

          "Oh believe me, some people think councils are responsible for absolutely everything."

          Our local council seems to avoiding responsibility for as much as it can except for the PC bits or those which get column inches for the leaders, even if they're not part of the council's remit.

        2. Anonymous Coward
          Anonymous Coward

          Re: Enquiring minds and all that?

          Unfortunately, in effect, they are.

    2. katrinab Silver badge

      Re: Enquiring minds and all that?

      Well, the first commercial telegraph line in the world was laid along that very railway track by Great Western Railway in 1839, this eventually became British Rail Telecommunications, then Racal Telecom, then the wireless division became Vodafone, and the cable division became Global Crossing, then Level 3.

      So, yes, the chances of cables being laid this close to the train lines is pretty high. They were laid there to connect the signalling boxes, and they realised they could make money by selling surplus capacity to other customers.

      1. phuzz Silver badge

        Re: Enquiring minds and all that?

        "the cable division became Global Crossing, then Level 3"

        Are those both supposed to be subtle train/rail references?

  6. Anonymous Coward
    Anonymous Coward

    Shame they didnt do this on the lines that Southern use - theyd have had all day Weds/Thurs to fix it with practically no trains to make it complicated!

  7. Ana Cronym

    She said the cables were "not highlighted in the thorough surveys" it carried out prior to the work.

    You Keep Using That Word, I Do Not Think It Means What You Think It Means.

  8. Anonymous Coward
    Facepalm

    "not highlighted in the thorough surveys"

    Oddly metal detectors don't work on fibre in plastic pipes.

    1. cd

      Re: "not highlighted in the thorough surveys"

      The special detectable foil tape they put in the ditch like a benediction mostly seems to work. That said, having watched the survey and excavation parts a number of times, there is often a discrepancy; the really good ops go slowly and with handheld shovels helping spot until they find the all the runs, then they go to town.

      Sounds more like nepotism or a cheap outsource on the dig, anyone find out?

      Meanwhile some lucky electricians are making tres tray money right now.

      Far as trains running, that's what flagmen/watchmen are for.

      1. Skoorb

        Re: "not highlighted in the thorough surveys"

        It doesn't matter about having watchmen/flagmen. Network Rail now won't let anyone work within 3 metres of the tracks without a sentinel card, which requires mandatory training (with basic tests) and a medical.

        Getting OR technicians on or about the railway is going to be a pain in the bum.

    2. Ktwo

      Re: "not highlighted in the thorough surveys"

      Fibre may well have been in ducts too, but there was significant copper running to distro cabs etc.

      Also, this was the only cable route from the west drayton telephone exchange across the tracks towards Yiewsley and Uxbridge.

    3. H in The Hague

      Re: "not highlighted in the thorough surveys"

      "Oddly metal detectors don't work on fibre in plastic pipes."

      That's one of the reasons for putting a detectable marker tape over the ducts. E.g.

      http://www.cablejoints.co.uk/sub-product-details/centriforce-cable-covers-centritile-stokboard/underground-warning-tape

      "Underground detection warning tape is manufactured using one or more stainless steel wires laid in a sinusoidal pattern .." So metal detectors do pick it up.

      Here in NL there is actually a central register of all underground infrastructure, which you have to consult before undertaking any mechanical digging operations. Still doesn't always stop excavators ripping cables out :)

      1. TitterYeNot

        Re: "not highlighted in the thorough surveys"

        "Here in NL there is actually a central register of all underground infrastructure, which you have to consult before undertaking any mechanical digging operations. Still doesn't always stop excavators ripping cables out"

        Yes, here in the UK there are similar digital maps of water, gas , electricity and telecoms infrastructure kept by the utilities companies that dig up the countryside. However, if an engineer lays a new pipe or whatever and doesn't update the relevant map correctly, its no use to someone else who subsequently looks at the map, thinks the area is clear, and so drills a nice big hole through said pipe.

        Another issue is that while upated maps like GasPlans are released on a regular basis (every few months) there's no guarantee that the guy digging that hole has the latest version, so could well be digging up cables that were laid a year ago but aren't on the version of the plans he's just looked at on his laptop.

        1. Steven Jones

          Re: "not highlighted in the thorough surveys"

          I think you can 100% guarantee those BT ducts have been there for a very long time. No latest version required. In any event, it only needs the duct to be a few metres from where the piling operator thinks it is, map or not, and chaos will ensue.

          I don't know what the procedure for this "thorough" survey, but if it's just consulting maps and assuming their accurate, that will surely not be enough. I would hope that the actual surveying involves using detection equipment and positively locating what are known services. The Network Rail contractors surely must have known there were major infrastructure ducting (and other infrastructure services) and positively located them before they started the work. I know that area quite well and it is a densely populated urban area.

        2. Diogenes

          Re: "not highlighted in the thorough surveys"

          We have a similar thing here, called dial before you dig. BIL is an experienced excavator operator and rang himself despite assurances from the engineers they did. I can only image the look on BILs face whenhe was thrown 30meters when he hit an underground power cable carrying many thousands of volts. DB4UD said it was actually one street over. Lucky bil did ring, power company sent him a billfor 250k , with recording from db4ud , power co withdrew invoice.

  9. This Side Up

    Uxbridge?

    Obviously it was near West Drayton on the Great Western Main Line and not at Uxbridge. This isn't the first time this has happened. Usually it's the signalling that gets disrupted. The problem is that they don't know exactly where the cables are even if they think they do. The cable ducts are buried under the balast to make it harder for thieves to get at them (most are copper) and every time the track gets tamped and new ballast is tipped on top the cables get pushed a little further into the cess or down the embankment. Even if they are shown on plans they may not be where they were when the plans were drawn.

    Btw what's the new measurement train got to do with it?

    1. Anonymous Coward
      Anonymous Coward

      Re: Uxbridge?

      Btw what's the new measurement train got to do with it?

      Nothing. But it looks cool. In fact I passed it slowly the other night, and you could see all the HUUGE monitors they've got, wiggly graphs that tell clever people important things, replays of the cameras filming turds on the track, flashing lights, some particularly impressive glass fronted wiring cabinets. A sort of mobile mission control by looks.

      If Dr No had a train, it'd look like the NMT. In fact, if I were rich enough to have my own train, it'd look like the NMT (although I'm not sure I'd bother with sensors that actually measure anything).

      But they really ought to have it repainted a nice colour scheme, rather than that AWFUL yellow. My favourite is the now gone East Coast "silver with maroon stripe" livery. And if they must have them, they need a fairing around the buffers like the original 252. But then again, I'm a closet trainspotter.

      1. ~chrisw

        Re: Uxbridge?

        Whereas in Japan, one regional Shinkansen operator puts track measurement equipment on the underside of every train that runs - yielding a constant stream of valuable, relevant data every day. NR insist on running the bananamobile up and down the country every other day at great cost when they could just improve how they capture the data. Always a few steps ahead...

    2. Fruit and Nutcase Silver badge
      Coat

      Re: Uxbridge?

      @This Side Up

      Btw what's the new measurement train got to do with it?

      NMT - probably no connection whatsoever..

      I guess it's quicker and cheaper to use a picture that has already been used on the site than acquire the rights for a picture that has not.

    3. Alistair
      Joke

      Re: Uxbridge?

      @ TSU

      Better question, why is that train winking at me?

  10. Commswonk

    Interesting point...

    Openreach has said its engineers are on site working 24/7 to get the cables fixed.

    Let us assume (in the absence of information to the contrary) that the above statement is true.

    While Openreach is a part of BT it is able to divert jointers from anywhere it wishes to address major failures of this sort. What might happen if Openreach was forcibly detached from BT? BT would lose the ability to draft in effort from "wherever" and would be reliant on the new company being prepared to shuffle manpower around, which it might be disinclined to do in the absence of a closely specified and enforced SLA, which of course would have to be paid for.

    It also raises the question about exactly which cables Openreach manages and which ones it doesn't. It isn't entirely clear from the article if the cables concerned were all "subscribers lines" (of one type or another) or if they were trunk circuits between BT premises, or a mix of both. If Openreach looks after the trunk network as well as "local" circuits (which the Wikipedia article suggests they don't) then splitting off Openreach as it currently exists could have unintended consequences. And if Openreach doesn't look after trunk routes then who does?

    1. AndrueC Silver badge
      Boffin

      Re: Interesting point...

      And if Openreach doesn't look after trunk routes then who does?

      A good question. According to the Wikipedia article on BT Group it seems that it might be the little known 'BT Technology':

      "Internal service unit:

      BT Technology, Service & Operations – responsible for the innovation, design, test, build and running of BT’s global networks and systems"

      Also from the relevant BT website:

      "This year we upgraded our core IP network to handle record volumes of data traffic in the UK. Our IT reliability has improved for the fourth consecutive year and the operational reliability of our core voice and broadband network platforms has hit a five-year high."

      1. Commswonk

        Re: Interesting point...

        @AndrueC: A good question. According to the Wikipedia article on BT Group it seems that it might be the little known 'BT Technology':

        The linked page also hints that it could be BT Wholesale - "operates BT's Networks". The words "responsible for" (as used to describe BT Technology) does not actual mean that anyone gets down and dirty in cable chambers, merely that they are responsible for it, and that they get "contracted effort", which could of course be Openreach.

        A labyrinthine muddle of which Philip Green could be proud, which could of course be the intention.

    2. Anonymous Coward
      Anonymous Coward

      Re: Interesting point...

      which it (Openreach) might be disinclined to do in the absence of a closely specified and enforced SLA, which of course would have to be paid for.

      The incompetence of Ofcom is a worry in enforcing standards of service, but having worked for a range of dedicated infrastructure businesses I can assure you that they are generally very good at this sort of thing, and would have far more focus as a standalone business compared to being a vertically integrated operation. As a bundled infrastructure monopoly, Openreach are a weakly regulated cash cow that BT group merely want to cut costs at.

    3. Steve Davies 3 Silver badge

      Re: Interesting point...

      As the problem is on or close to the permanent way (the railway track) and dozens of trains pass the point every day then all sorts of Health and Safety issues come into play.

      Only BT staff who are properly qualified and safety trained will be allowed to fix the problem.

      The passing of trains may also delay work from time to time.

      BT will be liable for compensation but they'll get it from Network Rail. BT didn't cause the problem (for a change)

      Fixing it is going to be a pig.

      1. AndrueC Silver badge
        Boffin

        Re: Interesting point...

        I asked the question on Thinkbroadband and amongst others got this reply.

        Which sadly doesn't really help add clarity :)

        1. Commswonk

          Re: Interesting point...

          Which sadly doesn't really help add clarity

          Not total clarity perhaps, but helpful nonetheless, and interesting reading. For those agitating for the shotgun divorce of BT & OR it really could be a question of "be careful what you wish for".

          1. AndrueC Silver badge
            Boffin

            Re: Interesting point...

            Ah ha! Got even more detail here concentrating on the technical side. It does sound to me like the answer to your original question is 'BT TSO' but also that it's a tad complicated all round. Colour me-not-surprised :)

            1. Commswonk

              Re: Interesting point...

              Um. From the post you referenced I am still not sure. Not sure that "core network" adequately describes the concept of "trunk routes" against "local circuits".

              Consider this question and I think the distinction becomes more obvious (or confused!):

              Who is responsible for ensuring that my car has a valid MOT certificate?

              I am.

              Do I carry out the test myself?

              Definitely not; I must find an authorised tester to do it for me.

              If nothing else this shows the distinction between "responsible for" and "who does it".

  11. DNTP

    Conspiracy to sell train tickets

    After all, when all the internet is out… you're going to have to take the train in to work.

  12. Robert Carnegie Silver badge

    I expect

    (when they're back online)

    a few new words in the "Uxbridge English Dictionary".

    For instance:

    "Network Rail - to complain loudly at length about the network."

    "Line break - what some moronic railway worker does with a telegraph pole."

    "O-pen-reach - to be able only to contact the outside world by writing letters."

    If you got here by searching for articles mentioning "line break" - hello!

  13. Boris the Cockroach Silver badge

    6 days to fix?

    Virgin (I hate them) only took 2 days after some numpty with a JCB pulled out all the Portsmouth feeds a few years ago.....

    Wheres the "your data is safe in the cloud, but you cant actually reach it " Icon

    1. Anonymous Coward
      Anonymous Coward

      Re: 6 days to fix?

      Virgin (I hate them)

      Deserving of a hundred upvotes on its own.

      Memo to world: No matter how shite LLU servicers are, never, ever sign up for Virginmedia, who are useless, price gouging turds.

      1. Martin-73 Silver badge

        Re: 6 days to fix?

        And fcking incompetent too. Had a broken line at a customer's house (VM's side of the NTE). Caused by gate installers nicking the cable with a crowbar. Said cable was ONE INCH below the ground, not in conduit. VM refused to accept responsibility and said there would be a charge. So I fixed it. Took me AGES to find out why I kept getting a belt off the phoneline but could not get dialtone. Some nitwit had split the pairs, and was using the blue and orange wires for the line, leaving the white/blue and white/orange spare... the crosstalk will be fun if the customer ever wants a 2nd line...

    2. Doctor Syntax Silver badge

      Re: 6 days to fix?

      "Virgin (I hate them) only took 2 days after some numpty with a JCB pulled out all the Portsmouth feeds a few years ago....."

      Was that adjacent to a railway line?

      1. Evilgoat_2

        Re: 6 days to fix?

        Over it actually. However the contractors had already lifted all the ducts out of the ground and carefully marked them. Said numpty then ripped out a significant bit of fibre with a bibcat. Nothing had to be dug up to make the repair and the road was already closed with bus replacement service running for the trains at night.

  14. Anonymous Coward
    Anonymous Coward

    This is a utility company

    An my experiences of sub'ing for utility companies is that the REAL position of cable ducts, pipes and remote stations is usually 100m+ away from where the "plans" show them.

    I once spent two days looking for a remote station - I never found it, but DID find another, unlisted one that hadnt had an entry in its inspection book since it was commissioned 20 odd years previously.

  15. John Brown (no body) Silver badge

    Up to 15,000 customers affected.

    Up to? So does this mean it's actually only about 7,000 in real world terms?

    And will Network Rail or the contractors be paying out compensation to those affected or will they just have to pay a one off "fine" to each of the cable owners and the customers have to individually claim compo from their provider?

  16. PeteS46

    OOPS! West Drayton is? (used to be) where the Lord of the UK Skies lives (aka Distress and Diversion - who you call if you REALLY have a problem). If they are OOS then both civil and military aviation around UK are at severe risk. Hopefully they have enough redundant circuits and diverse routing to cope. Anyone know different?

  17. We're all in it together

    Who'd have thunk?

    Just as well it was 'thoroughly surveyed' as had it been 'not thoroughly surveyed whatsoever' most of the UK would've crashed.

    Six days for Openreach to respond and fix break down relates to -

    Zero hour receives phone call from panicky Network Rail supervisor. Operator in BT Mumbai replies with 'have you powered off your router?' Eventually call is passed back to the UK.

    Zero hour plus one 'What's your account number again and address? Sorry, by the trackside with smoke emerging from a hole isn't on our database''

    Day two - 'you've come through to global services, you need residential, putting you through. Sorry, we are experiencing a high volume of calls. Please hold. Please enter the number of the line you are calling about. Sorry we don't recognise that number goodbye'

    Day three - 'why don't you ring the local exchange direct - none of the engineers have jobs today as the back end system is **** and never allocates anything'

    Day four engineer on site - 'I've run out of crimps. Need to order some. Bye'

    Day five - 'don't suppose you know where these cables run to? I haven't brought any documentation with me'

    Day six - 'local engineer who ran the cables in the first place finds out and repairs them all. The end.

    '

  18. ICPurvis47
    Flame

    "Engineers" - Again!

    "Openreach has said its engineers are on site working 24/7 to get the cables fixed"

    NO, I very much doubt that any Engineers were involved, more likely Linemen or Wiremen. Engineers are the ones that have studied for four to eight years to get at least a BSc, and are more likely to be developing new tech rather than fixing old tech. Would you refer to your local butcher as a Surgeon? He cuts flesh and gets blood on his hands, but that does not qualify him to rub shoulders with the other person. In some countries (Germany, for one) it is illegal to use the term Engineer unless the correct qualifications are held. Apparently, anyone who gets their hands dirty is allowed to do so in the UK.

  19. Anonymous Coward
    Anonymous Coward

    Mostly down still but apparently coming back up at last.

    I am an unlucky still down as opposed to lucky restored person.

  20. Anonymous Coward
    Anonymous Coward

    Jeez...

    The Uxbridge area always looked third world. Now the internet is out it genuinely is.

  21. thewelshexile

    If they have cut 1500 fibres then it's going to take a long while, they will need to run new fibre pairs in. Then splice and test each pair as they are spliced back into the network. I have worked a few major cable breaks and they take time to get back up and working. It's just one of those jobs you cannot rush, Network Rail should have maps of all cables on their work site and these should have been marked prior to work starting.

    Looks like a case of sloppy work marking the work site by Network Rail.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like