back to article UK ISPs crippled by undersea cable snap

A major internet routing outage struck UK telcos over the bank holiday weekend - knackering access to the World of Warcraft website*, the BBC, Amazon, Facebook and other sites for more than 24 hours. It's understood that a submarine cable carrying web traffic snapped between Blighty and the Netherlands, causing headaches for …

COMMENTS

This topic is closed for new posts.
  1. Steve Evans

    Ahhh...

    That explains it... I noticed some weirdness yesterday, but given it was on a friend's Talk-Talk connection I didn't realise it was anything unusual!

    So where's the Sun headline...?

    Undersea internet cable severed, Europe cut off.

    1. Anonymous Coward
      Anonymous Coward

      Re: Ahhh...

      lol me too, I had wireshark out and everything wonder wtf BT were doing with my connection now - I kept getting disconnected from Gtalk/IRC every few minutes and was suffering dropped packets like a mofo.

      I do question why access to the BBC was affected, where are their servers?

    2. Anonymous Coward
      Anonymous Coward

      Re: Ahhh...

      No worries. Everything will be back once the Yanks have finished installing their tap.

  2. ravenviz Silver badge
    Boffin

    This is why we need to go wireless for global SP.

    1. Captain Scarlet Silver badge
      Trollface

      I agree

      Every time a plane flies over the dish we lose connection, horrible pings to servers around the world make them annoying to use.

    2. Anonymous Coward
      Anonymous Coward

      You know of a 400Gbps+ wireless technology?

      1. ravenviz Silver badge
        1. Anonymous Coward
          Anonymous Coward

          I'z can doo reedingz, me!

          400GBps?

          "... Rohm says the chip could in future achieve 30 Gbps speeds..."

          Aaand I can duz hard sumz too!

          1. Comments are attributed to your handle
            Joke

            Re: I'z can doo reedingz, me!

            Nah, the last 398.5 Gbps will be the easy part!

          2. ravenviz Silver badge
            Boffin

            Re: I'z can doo reedingz, me!

            And here.

            1. Anonymous Coward
              Holmes

              Re: I'z can doo reedingz, me!

              It requires LOS, cannot work over a curved surface (which the earth is) and also no obstructions or interferers.

              Throughput(Guided (Fibre) Optics) > Throughput(Unguided Optics)

              at any moment in time.

              Doing optics without the fibre and calling it wireless isn't really a substitute, which is what I think you are trying to say?

    3. Stoneshop
      Boffin

      @ravenviz

      So, what about the backhaul from all those access points, and the links between the backhaul nodes, etcetera? Wireless too? How do you propose crossing a major stretch of water, such as the Atlantic?

      1. ravenviz Silver badge
        Boffin

        Re: @ravenviz

        Satellites.

        1. Richard 12 Silver badge
          WTF?

          @ravenviz: Ever used a satellite link?

          They're the only thing available to mobile stations like ships or "internet to a field/beach in the middle of nowhere". (Or oddly Hyde Park a few years ago, before 3G took off)

          Approximately 500ms ping is normal. In some cases I've had to adjust timeouts to get a connection to stay up - under poor signal quality roundtrip time goes up quite a bit due to retries.

          I rather doubt you'd be happy over that one.

          1. Anonymous Coward
            Anonymous Coward

            Re: I rather doubt you'd be happy over that one.

            dont need low latency to trolololol you

          2. Christian Berger

            Low to medium orbit satellites...

            Those could potentially solve the latency issue. You'd need lots of them and track them. Tracking them probably wouldn't be to hard for regional internet exchanges.

            However that still doesn't get you anywhere near the bandwidth of a single fibre.

    4. Anonymous Coward
      Anonymous Coward

      "Hello, I don't have a clue how the Internet works or any understanding of transmission technologies, bandwidth or spectrum. Strangely though, I feel empowered and emboldened to offer my suggestions for improvement in front of a group of people who are probably networking experts."

      Does that about sum it up?

  3. Jamie Kitson

    neither had responded at time of publication

    > El Reg has asked O2 and Interoute to comment on this story, but neither had responded at time of publication. We'll update the piece if O2 or Interoute do furnish us with statements.

    They're not receiving emails at the moment.

  4. Anonymous Coward
    Anonymous Coward

    well that would explain the oddness, I don't really remember it being a big problem for me and I'm on be. Spent all sunday playing planetside 2.

  5. Lee Dowling Silver badge

    Okay, tell me that it's not just me.

    The point of running huge data centres with failover hardware, and the point of IP routing is basically so that, should something happen, the dead bits die and everyone is automatically shifted onto the non-dead bits.

    Assuming that the capacity of your average ISP isn't being constantly tested 24/7 on all of its lines (which kinda makes the word "redundancy" not applicable), why should any ISP have to take any physical action for this kind of thing to be noticed, monitored, and then failed-over to other routes? I can understand warnings, and notifications, of it happening but why does it require manual intervention? Why are there half-a-dozen routing protocols professing automatic best-route selection if nobody uses them? Why can I set up a simple fail-over over several routes in about two lines of "ip" commands and yet big ISP's, datacenters and network providers can't do the equivalent on their expensive Cisco equipment?

    I completely understand "capacity problems because we're running on one-line-less than normal" but "takes 30 hours for someone to work out what to do and find another hole to plug the cable into?".

    Do we really have an Internet where manual intervention is required to provide the routing that's required when something fails?

    1. John Sturdy

      Yes, that puzzled me too. Perhaps routing changes have to be approved by corporate bean-counters?

      Or maybe "the authorities" need some warning so they can set up wiretaps on new routes, but I would expect that to be automatic too.

    2. Chris007
      Unhappy

      @Lee dowling

      Unfortunately whilst people are only willing to pay peanuts for their broadband, this will happen more and more as less and less investment is made.

      1. Anonymous Coward
        Anonymous Coward

        Re: @Lee dowling

        Presumably capacity is optimised to run on whatever is the cheapest solution per Gb at that moment.

        1. Anonymous Coward
          Anonymous Coward

          Re: @Lee dowling

          Hmm - capacity on this scale isn't "Pay As You Go". It's contracted for 12 months or longer and failover protection will be bought or not based on a simple sum based on the cost of the protection versus the cost of lost business if the primary route fails for the maximum time allowed in the SLA.

    3. Anonymous Coward
      Anonymous Coward

      @Lee Dowling

      Re: Do we really have an Internet where manual intervention is required to provide the routing that's required when something fails?

      The outage affected/affects non-protected pathways - i.e. there was no redundancy, probably to save on costs.

      I'm guessing that all of the affected parties that required manual re-routing did so because using alternative routes costs them more.

    4. Yet Another Anonymous coward Silver badge

      >Do we really have an Internet where manual intervention is required to provide the routing that's required when >something fails?

      No - you can have an internet where the router has the company credit card and will go and buy capacity on whatever link it needs, automatically at whatever the cost.

    5. Anonymous Coward
      Anonymous Coward

      It can happen. If someone else's kit down the chain on your alternate route thinks the best path to your destination is back the way you just came, those packets evaporate. As an individual ISP you don't own all the routers on the way to where you're going, other people's routing decisions determine how effective your back-up is.

      You could build your own private backup, totally under your control, but £15 a month broadband doesn't pay for that.

  6. Anonymous Coward
    Anonymous Coward

    WoW down for maintenance

    I suspect it really is maitenance, as it started working after a couple of hours, and there is a massive patch due.

  7. Crisp

    World of Warcraft

    Do people still play that?

    1. Anonymous Coward
      Anonymous Coward

      Re: World of Warcraft

      only to wait out the clock till death

    2. Colin Wilson 2

      Re: World of Warcraft

      > Do people still play that?

      /wave

  8. Eradicate all BB entrants

    Mind made up then

    Been with Be for a few years and they have always been pretty good but my one longtime niggle with them has been DNS. Sometimes I think they entire infrastructure is running through a speedtouch modem.

    Sky can fibre me for the same cost. Guess its time to migrate.

    1. Tegne
      FAIL

      Re: Mind made up then

      Likewise. I've been procrastinating about moving from BE to 40MB SKY as BE have been so good in the past. But the have lost their way so regrettably it's time to help line Murdochs pockets. The only saving grace I had was I have a VPN option on my Usenet account. I've never used it before but it worked a treat and my WoW latency was at least 10ms lower than BE's standard.

      1. Blitterbug

        Re: moving from BE to 40MB SKY

        Okay then, but personally I'd rather eat my own vomit. Apologies to brekkie / tea-time readers...

    2. Andydude
      Unhappy

      Re: Mind made up then

      Yeah me too - ever since they were bought out they've been a bit arse, and it's been years and still no fibre solution. I'm off elsewhere when my contracts up...

  9. Arctic fox
    Trollface

    "knackering access to......Facebook...........for more than 24 hours"

    "'It is an ill wind indeed that blows no good at all."

  10. Anonymous Coward
    Anonymous Coward

    No Noncebook/Twatter? What did the Jeremy Kyle "guests" do for 24 hours?

    Maybe had a real conversation with a real friend?

    Scabs.

    1. Anonymous Coward
      Anonymous Coward

      not all facebook users support jeremy kyle

      some of them still support jerry springer

    2. John Brown (no body) Silver badge
      Thumb Up

      No Noncebook/Twatter? What did the Jeremy Kyle "guests" do for 24 hours?

      They tweeted the BBC about Essex lions.

  11. Boris the Cockroach Silver badge
    WTF?

    I thought

    the internet was designed to be immune from a section of cable being taken out

    so that your packet gets around a broken fibre or a nuclear weapon burst.

    Or is it that cheap ISPs route everything via a single point of failure......

    1. Colin Miller
      Facepalm

      Re: I thought

      Sigh. And if the other links are running at 80+% of capacity? Traffic will be rerouted, but then the other links are saturated.

      Idiots analagy. Assume you work in London and drive to Hemel Hempsted or Milton Keynes, and both the M1 and M11 are closed. Yes, you can divert. But it's not going to be fun…

  12. despairing citizen
    Big Brother

    Yet another advert for offshore data centres

    There are some advantages to having your data centre "local",

    Amazing these minor technical risks are never mentioned by the magic bean salesmen of the offshoring consultancy firms.

  13. Scott Broukell
    Coat

    urm ....... cable theft ?

    <this part of the post was removed unlawfully by thieves>

    1. Brewster's Angle Grinder Silver badge

      Re: urm ....... cable theft ?

      I take it yours is the waterproof one with the diving helmet?

  14. Anonymous Coward
    Anonymous Coward

    VPN - It's not just for privacy/piracy.

    I'm on Be and noticed Sunday lunchtime that some sites would timeout, luckily I have VPN access to various countries so I had to do my own re-routing, thanks Telefonica...

    1. Stoneshop
      FAIL

      Re: VPN - It's not just for privacy/piracy.

      Err, you can have all the VPN's you want, but when your ISP's connectivity goes titsup you're no better off than the next guy.

      1. Anonymous Coward
        Anonymous Coward

        Re: VPN - It's not just for privacy/piracy.

        In this case my ISP's connectivity was half-fucked, the routes to the VPN(s) and then on to certain destinations were absolutely fine, yet the direct route to the certain destinations was not always fine.

        The only downside of my VPN access is that it bypasses the very helpful ADSL firewall so I have to beef up network security on the PC to stop the casusal 'hackers' (loosely put, probably bots) from trying to connect to certain ports random IPs.

  15. Anonymous John

    No truth in the rumour that the Essex lion had bitten through it, then?

    1. Graham Dawson Silver badge

      Only if it was a sealion.

  16. TeeCee Gold badge
    Facepalm

    "...suggested that a submarine may have hit a line in the Atlantic"

    Yeah right. Submarines, those things that are known for their habit of regularly bashing into things on the bottom of the sea, 'cos it isn't at all risky for them to do so.

    Unlike, say, trawl nets or anchors.........

  17. hahnchen
    Thumb Down

    Spent hours on the phone with O2 on Sunday

    Took ages to convince first line support there was a problem, which involved switching my router on and off again ad infinitum.

    Could tell something was broken by looking at traceroutes and comparing that with my not-broken 3G connection.

    Finally got through to second line support, took my details, and told me they'd take a look at it after the bank holiday. Nice work guys.

    1. AdamW

      Re: Spent hours on the phone with O2 on Sunday

      http://irc.beusergroup.co.uk:8080/?channels=Be you get better info from these guys

  18. Anonymous Coward
    Anonymous Coward

    I blame Apple.

  19. Bradley Hardleigh-Hadderchance
    Pint

    Don't worry it's just a recursive error.

    They just ripped up some cables when they were laying some new ones.

    See recursion.

    http://www.outpost9.com/reference/jargon/jargon_33.html

    Submarine Commander:

    Yes, sorry 'bout that. Slight tits up. Ripped up some cables when laying some new ones. Shame really.

    Should be ok by morning. Yes, yes, yes, we shall relay all the old cables being ripped up. Unless we rip up any of the new ones... when we are relaying the old ones. If you follow me, sir...Or fail to lay any of the new ones whilst replacing the old ones. Could get a bit sticky sir.

    Sorry!

    Anyway, send us a line, if you can get through sir. Yes, yes, old cable's as good as new cable....

    (mutters under breath) if you can find any fucking cable left by the time we have finished (/muttering).

    At least they didn't have the problem the HMS Banana had. I heard they went six times around the world before they finally run out of fuel.

    See: banana problem /n./

    See:http://www.outpost9.com/reference/jargon/jargon_17.html

    Talk about redundancy.

    1. John H Woods Silver badge

      Re: Don't worry it's just a recursive error.

      You guys probably know this already, but I have just discovered what happens if you google 'recursion'

  20. iRadiate

    spent hours on the phone to 02?

    I saw loss of connectivity too. I just went out and enjoyed the rest of my life during the downtime. Next day all was well.

  21. Anonymous Coward
    Anonymous Coward

    The lose of a cross channel is common event. It is common enough to have several specialist ships based on the North coast of France just for doing the repairs. Common route protection schemes will detect and reroute sub-second what ever the tech. Even the slowest of routing protocols can figure it out in a minute or two.

    Interroute fibre is meshed. So it does not re-route you it is a commercial not technical issues. It may not be expressed that way in the contract, but premium clients will get resilience. Less valueable clients get less resilience. You get what you pay for.

    1. Anonymous Coward
      Anonymous Coward

      Bellheads vs Netheads

      "So it does not re-route you it is a commercial not technical issues. It may not be expressed that way in the contract, but premium clients will get resilience. Less valueable clients get less resilience. You get what you pay for."

      Usually it is expressed in no uncertain terms in contracts if a customer is buying protected or unprotected services. Customers may not always understand the implications, or even the technology options if they're unfamiliar with the way wavelength services are delivered, ie wavelengths are generally dumb clocked light. That can be especially critical when buying submarine capacity when fixes often take more than a truck roll. Another common problem is trying to sweat the asset. Two links running <50% good, >50%, not so good in the event of losing one.

    2. Christian Berger

      That's what I was thinking

      I mean the whole point of IP is that the routing is automatic. So damage is automatically routed around.

      Seems like a system designed to cope with nuclear war does not withstand real-life unleashed capitalism.

      1. theblackhand
        Mushroom

        Re: Seems like a system designed to cope with nuclear war...

        The idea for surviving a nuclear war was that there would be some way to communicate - the idea wasn't to preserve every part of the network.

        Just consider yourself nuked :-)

      2. /dev/null
        Facepalm

        Re: That's what I was thinking

        You still believe that myth?

  22. Anonymous Coward
    Anonymous Coward

    What is the problem here?

    Why not just dial into a different BBS?

  23. johnnymotel

    How the heck do they go about re-joining one of these cables?

    1. Anonymous Coward
      Anonymous Coward

      knitting with armour!

      Hibernia Atlantic made some videos showing aspects of submarine cable laying and repairing. Basic fibre splicing is the same as for dry/terrestrial cables. Tricky bits come in dealing with armoured cable and the power. This video shows how jointing is done-

      http://www.youtube.com/watch?v=dIa-kyYndwQ&list=UUoR_9pS8jCcf-eEArrJnI6Q&index=53&feature=plcp

  24. Wild Bill

    My Be connection was fine yesterday. I even successfully completed my download of every episode of Knightmare.

  25. GreyWolf
    Devil

    "current information suggests that the culprit was a ship's anchor"

    Drunken or lazy captain with an anchor...that's the maritime equivalent of a builder with a JCB...and it happens frequently, despite the cable runs being marked on the bloody chart.

  26. Anonymous Coward
    Anonymous Coward

    Interoute?

    amazed they are still in business

This topic is closed for new posts.

Other stories you might like