back to article Virgin Media biz service goes TITSUP* across London

A London-wide Virgin Media outage caused by a "fibre break" has left business customers across the UK capital without broadband - including a number of borough councils. Bromley Council apologised for its phones and online systems being out of action, blaming the "London wide technical issue" caused by the outage. Virgin …

  1. Alister

    right! own up!

    Who was driving the JCB?

    1. Aladdin Sane

      Re: right! own up!

      Bloody Nizlopi.

      1. m0rt

        Re: right! own up!

        Not his fault. It is his anti HSE dad.

    2. big_D Silver badge

      Re: right! own up!

      We had that here, in Germany, a while back. New building site, JCB dug through the cable BUNDLE and we were without telephone and internet for 2 days, then a re-routed cable with limited bandwidth (2mbps instead of 100mbps) for a few days, until they could lay a new fibre bundle.

      1. Anonymous Coward
        Anonymous Coward

        Re: right! own up!

        Remember something like this happening in the SF Bay area in late 90s where a construction worker took out a major fibre cable ... at the time PacBell said (with I assume 24 hour working) it was going to take about 2 weeks to resplice all the fibres

        1. kain preacher

          Re: right! own up!

          Are you sure that was not regular copper ? Regular fiber is much easier to splice. I remember when they were doing a BART extension they hit a bundle of 500 pair copper lines from pac bell. The contractor and BART had to pay. Pac bell handed out free cell phones till it was fixed.

    3. Boris the Cockroach Silver badge

      Re: right! own up!

      Bet it was the same 16 yr old that dug up Portsmouths cable connection a few years ago.....

      What we really need to do is route 33Kv underground power feeds next to the critical internet connections

      Then when the moron in the JCB gets into action, we're one moron less.....

      1. Anonymous Coward
        Anonymous Coward

        Re: right! own up!

        That has happened. I had an outage many years ago, road works half a mile away took out the whole town and the driver.

        1. big_D Silver badge

          Re: right! own up!

          When I was working for Plessey in Fareham, we had a new Megastream line laid, to replace the exisitng Kilostream line from the data centre in Addelstone.

          I think the record was BT digging through the old cable 4 times in 1 day!

    4. TheVogon

      Re: right! own up!

      A large business without a backup connection?!

  2. Carl Thomas
    FAIL

    One fibre break

    One single fibre break takes down a bunch of councils. Normal broadband at the access layer I can understand, the entire London Grid having a single fibre point of failure not so much.

    So, so bad. I hope that at very least this was a case of redundancy not working as expected rather than that it wasn't built with any.

    1. Angry IT Monkey

      Re: One fibre break

      This would depend on what that fibre is linked to. if it was, say, a datacentre hosting multiple councils' infrastructure...

      1. Anonymous Coward
        Anonymous Coward

        Re: One fibre break

        Any worthwhile DC should have diverse connections to avoid this. But then it is a council owned site so perhaps this is to be expected.

    2. Warm Braw

      Re: One fibre break

      When one of VM's predecessors sold us a fibre connection, the story was that the fibre formed a ring so that a single break would not result in an outage. It quickly transpired that the ring had some rather significant single-fibre branches and that the local youths were entertaining themselves by pouring petrol into the ducts, igniting it and watching the pavements erupt...

    3. kain preacher

      Re: One fibre break

      In Morgin Hill California they had a break of a fiber ring(every thing was in one duct). THat break took out the net, cell towers and 911(999). Yes you had redundancies but every thing ran in one duct.

      1. Alan Brown Silver badge

        Re: One fibre break

        "Yes you had redundancies but every thing ran in one duct."

        This happens depressingly often.

        When you specify redundant pathing to a telco it's more common that they interpret this as "2 links, one duct" than not.

  3. silks
    FAIL

    Surely councils would have a backup link(s) with another provider that would seamlessly fail over...

    1. wolfetone Silver badge

      "Surely councils would have a backup link(s) with another provider that would seamlessly fail over..."

      That would cost money. And unless you are the Surrey council with a golden handshake deal, you haven't got a fat lot of cash to spend on backups like that.

      1. Anonymous Coward
        Anonymous Coward

        I've got a feeling that you missed sarcasm. The hint was the word "Surely", methinks

        1. silks

          Indeed, and well spotted :)

    2. Hans Neeson-Bumpsadese Silver badge

      Surely councils would have a backup link(s) with another provider that would seamlessly fail over...

      That level of resilience costs money - money which councils have precious little of. The majority of people aren't familiar with IT infrastructure redundancy models and the like, but they are familiar with council services.

      That leaves the council with two options...

      Option The First - close a few libraries and buy some redundant connections from another ISP. If something goes wrong (and it's still an *if*) the public don't directly see any benefit. Regardless of what happens, the public *do* see a reduction in services provided by their council.

      Option The Second - use meager resources to keep existing services running. Cross fingers that you don't have problems with your one connection to the Internet. If you do, try and deflect blame onto the supplier (Virgin in this case, but it could equally well have been BT - both of which the public are fimiliar with, and don;t hold particularly high regard anyway)

      1. Doctor Syntax Silver badge

        Option The Third - close a few libraries.

      2. TheVogon

        "That level of resilience costs money"

        Dual WAN connectivity really isn't very expensive these days and there are cheapskate options like a VPN over internet. - and would likely cost a lot less money in the long term than a few thousand employees sometimes not being able to work...

    3. kain preacher

      When I worked in a data ce3nter by redundant links to mitigated against physicals breaks was a joke. At some point separate physical links end up in the same duct.

    4. Anonymous Coward
      Anonymous Coward

      But even if they did have redundancy its not unknown to discover all the different redundant connections go through the same duct (or worse, all the "independent" network providers are all buying their bandwidth from the same company so they are all using the same cable bundle!)

  4. EddieD

    Won't be the last...

    Today Virgin, Tomorrow BT, there's always some fibres just a sneeze away from being cut.

    We're getting more and more of these frangile fibre thingies that are worming their way through soil, and today it was some borough councils, not too serious, but London City Airport are remoting their ATC via fibre - okay, they're using three redundant connections and they that say "it will be impossible that all three would fail together"

    Triplicated Infrastructure terminated so unexpectedly prematurely.

    Hmm.

    1. Elmer Phud

      Re: Won't be the last...

      Fibre or fat cable -- both tend to have similar effects when faced with a JCB driver on Red Bull and a tight price.

    2. Allonymous Coward

      Re: Won't be the last...

      > Triplicated Infrastructure terminated so unexpectedly prematurely

      Have an upvote.

  5. JakeMS
    Joke

    "A London-wide Virgin Media outage caused by a "fibre break" has left business customers across the capital without broadband - including a number of borough councils."

    So is the fibre having a KitKat?

  6. Tascam Holiday
    Facepalm

    Redundancy

    Let me guess: VM's net admins made a fuss about a lack of redundancy, management took it on board and defenestrated 226 employees.

  7. wyatt

    From memory there use to be a pretty major VM hub on the outskirts of Birmingham that if it failed, took down a significant number of local councils. Don't know if that's still the case.

    I have to say I've been pleasantly surprised by the VM contractors laying a new duct along my street. Not only have they not caused an outage, they've done a pretty good job of making good their work.

    1. Elmer Phud

      Is it because they is contractors?

    2. Anonymous Coward
      Anonymous Coward

      It may look nice now, but over the next year or so you will have other services digging up the place to replace the damaged water/gas.electricity services.

    3. eJ2095

      Yup there still is..

      On the side of a curved main road as well..

      One good crash and yup you guessed it

  8. IHateWearingATie

    Redundant fibre, same duct.

    Client that I worked for previously thought they had redundant fibre connections going out different ends of their main site. Turns out both fibres ended up in the same duct somewhere which unfortunately was wrecked by a JCB.

  9. Chris Hills

    Maybe...

    Maybe there were 2 fibre breaks, but only the second break caused the loss of service. I would expect there to be regular fibre breaks that we never hear about, it is just coincidental that two parts of the same ring were broken at the same time.

    1. Mark 110

      Re: Maybe...

      We used to get all sorts of crap back in my Telewest days (yes I am that old) when big trunk fibres broke - either through classic JCB strikes or something to do with exploding electric grid transformers under roads (whoi new that was a thing?).

      Anyway - even when the majority of routing failed onto redundant circuits for most traffic we'd always find the phones continued to ring because of strange undocumented static route configurations. It was usually quicker for the engineers to finish resplicing 1000 fibres than it was to dig through the routing tables.

    2. Ken Moorhouse Silver badge

      Re: ...broken at the same time.

      Or more likely they were either unaware of the first break, or didn't treat its repair with the necessary urgency.

  10. Anonymous Coward
    Anonymous Coward

    I wonder if they'll fix it faster than the 12 days it took to resolve a SNR issue on my street. The only thing that happened on schedule in that situation was the little bit on the service status website getting changed at the exact minute it should have been fixed.

    "ETA for the fix is the 12th June, 10:00"

    (9:59.... clock ticks over.... status site immediately updates to "ETA for the fix is the 12th June, 16:00". Repeat day after day, if it gets to 18:00 then just go to 10:00 the next day)

    Hilariously, it seems my pokey little terraced house has better resilience than Bromley Council, with a 4G MiFi hanging out of one of my router USB ports providing a surprisingly good backup (the 4G upload speed on O2 was 12x what the Virgin 50Mb service provides)

  11. Stevie

    Bah!

    Gotta love Teh Cloudz.

  12. Commswonk

    Those who fail to learn the lessons of history...

    Bromley Council apologised for its phones and online systems being out of action...

    Doubtless numerous others experienced a similar fate.

    Does anyone know the date on which Do not put all your eggs in one basket was the subject of a Cancellation Notice?

  13. Anonymous Coward
    Facepalm

    Single fibre point of failure ..

    We’re aware of a fibre break which has caused a temporary loss of services for some of our business customers in London .. We’ve identified the problem" .. that being to save money we used a single fibre cable without any redundant backup routes.

    1. Alister

      Re: Single fibre point of failure ..

      that being to save money we used a single fibre cable without any redundant backup routes.

      It's easy to criticise, but as mentioned multiple times in this thread, it is quite common to have multiple redundant routes that at some point will share the same duct, whether that be near the premises or near the exchange.

      For commercial developments and industrial estates (where most datacentres are situated) it is very rare for more than one duct to be provided - wayleaves cost too much to allow otherwise, and again as cables / fibres enter a telephone exchange there is commonly only one large duct space entry. Both these areas are vulnerable to roving JCBs.

  14. Anonymous Coward
    Anonymous Coward

    Perhaps I'm being cynical.

    Is it anything to do with it being Friday and Wimbledon?

  15. swampdog

    Das Kabel Kaputzenist

    Should they have had two cables?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like