back to article Is everything OK over there, Britain? Have you tried turning the UK off and on again? ISPs, financial orgs fall over in Freaky Friday of outages

Today was more Friday the 13th than Friday the 31st in the UK, it seems. Not only is it Brexit Day, marking Blighty's withdrawal from Europe, but a bunch of services and internet connectivity broke. Here's a quick summary of what went wrong, according to Register readers writing in. If we've missed anything out, please let us …

  1. Anonymous Coward
    Anonymous Coward

    Your reader who has 500 users but no independent internet backup should get their management to invest in some decent resilience. We have 3 independent fibre providers and routes out of our building, two independent phone systems with two independent providers oh and an actual DR capability. No reason not too with comms being so cheap in 2020.

    1. IGotOut

      Until a major route from a separate carrier knackers, and you find all those independent routes are not as independent as you think.

      I've had problems in different countries knock us offline.

      1. A Non e-mouse Silver badge

        Back in the days of kilostream & megastream circuits, I discovered most north/south lines went through just one exchange in Birmingham.

        1. Chloe Cresswell

          That'd be Anchor?

          1. TechnicalBen Silver badge
            Black Helicopters

            That'd be...

            see icon?

        2. david flacks

          It wasn't just north/south lines. I remember the incident.

          Woking-London routed via Brum as well so we lost the kilostreams to every one of our customers in the city. The joy of non-diverse routing for anywhere except the City Of London in those days.

          1st time we actually used our dialback modems in anger and found there was a bug in the code that meant they negotiated 4800 baud connections rather than 9600.

        3. big_D Silver badge

          Back in the days of kilostream and megastream, I discovered that BT couldn't lay a new megastream cable without digging through the old kilostream cable it was replacing!

          I think the record was 10 outages in 1 day!

    2. A Non e-mouse Silver badge
      Facepalm

      A friend took over the running of a small shop. Their till software & credit card machines all rely on the cloud. I did say that they need to think of what happens when (not if!) their net connection goes down. I did try to explain that the cost of a 2nd net connection would be dwarfed by the loss of income if things went down.

      Then things went down.

      And they paniced.

      And still won't invest in a second 'net conection.

      1. big_D Silver badge
        Facepalm

        Because lightning never strikes twice!

      2. paulf Silver badge
        Pirate

        Back in my Saturday job days (late 1990s) the small retail establishment, where I was gainfully employed, didn't get a proper credit card machine until around 1996. Up until then we had to phone up a meat-bag to get a 6-digit number code to authorise each credit card transaction. Debit cards were still reasonably unusual with people paying by cheque+guarantee card.

        We kept the clack-click mechanical card imprinter as a back up in case the phone line went down or the machine went wrong.

        1. Sherrie Ludwig

          Knuckle-buster to the rescue

          Just last year I was in queue to purchase at a chain bookstore when their cc processing went down. They fiddled, tried for a bit but no hope. The manager on duty unearthed a knuckle-buster and credit card slips from a store room, but had no idea how to use it, like a young person today with a rotary dial phone. I showed them how to employ it, warned that it was only for cards with raised numbers, to check the expiration date visually, and get a signature and the phone number in case of problems. Then said that to process the charges, they would probably have to manually key in all charges off the slips, but check with head office first. Since my card was properly charged, I can only assume it all worked.

    3. Tinslave_the_Barelegged

      > We have 3 independent fibre providers and routes out of our building

      That may be possible in urban areas, but quite a few business are in more rural locations where suppliers and connections are literally thin on the ground.

      1. werdsmith Silver badge

        My backup line is 4G. My main connection is via Virgin, I also have a deal with my neighbours on a different provider, they use my WiFi when they are down, and I can use theirs likewise.

        1. katrinab Silver badge

          My local 4G mast is attached to the same street cabinet as my fttc and landline phone line.

          1. Dazed and Confused Silver badge

            My local 4G mast is attached to the same street cabinet as my fttc and landline phone line.

            My 4G backup goes from greased lightning to utter shite within minutes of the BT cabinet failing as everyone on the estate switches from their FTTC broadband to watching you tube on their phones.

          2. Anonymous Coward
            Anonymous Coward

            4G Masts do not connect via FTTC PCP cabinets, they are fed via a leased line directly back to the nearest exchange/POP. It might be located right next to your green FTTC cabinet because that's where the nearest BT ducting is, or where the nearest T-Node happened to be making life easier for all involved ;-)

            (That, or the cabinet could have come AFTER the 4G mast now BT Openreach had to pull fibre for the mast).

            1. phuzz Silver badge

              Of course, if they're running through the same ducting, then they're equally vulnerable to someone digging through them whilst trying to lay a new driveway (eg).

        2. big_D Silver badge

          4G... I get around 300bps on my "4G" connection most of the time at work. If I'm very lucky, I might get Edge connectivity.

        3. eldakka Silver badge

          Could always set up a microwave link between your site and one or two towns/exchanges over so that local outages due to local exchanges/cell towers going down could be bypassed.

          But that aint cheap, you'd have to do a cost-benefit analysis on that.

      2. Anonymous Coward
        Anonymous Coward

        Re: more rural locations

        15 or years ago I was involved in a project were some nut job had decided to have a datacentre inside a country house hotel (AKA vanity project). Getting a fibre in wasn't a problem. But when we asked for a second one coming on a different route BT sucked in a bit and said "Sure, that'll be £250,000" The management up stream lost interest at that point.

    4. Adam JC

      You can't really be *truly* carrier-resilient on VoIP, certainly not for inbound calls. The CLI has to terminate at a physical POP somewhere and if the carrier is Tier 1, like Gamma, and suffers an outage like that, you can't reroute calls - In-fact, you can't do jack shit except perhaps have a backup trunk with another provider, but even then your incoming calls are still fubar!

    5. Venerable and Fragrant Wind of Change

      The quote just says no SIP/VOIP service. It doesn't say there's no resilience - bearing in mind the range of perfectly valid fallbacks (for example, mobile phones, old-fashioned landlines) likely to be available.

  2. Irongut

    Netflix has been down for several hours.

    1. werdsmith Silver badge

      That’s a blessing.

      1. John Brown (no body) Silver badge

        Why? If you don't like it, you don't pay for it. On the other hand, people who are paying for probably have a different opinion to you.

        1. First Light

          Maybe she/he likes it TOO much.

          1. Anonymous Coward
            Anonymous Coward

            Excellent opportunity to "chill" without Netflix.

            Unless you were using Netflix as the route to 'chilling', in which case I can understand some people getting a bit bothered about it (just not as hot and bothered as they were planning).

            1. Jou (Mxyzptlk) Bronze badge

              I am prepared.

              I've got enough offline movies and series. Music to listen. Books to read. Games to play, including computer games with an offline installer.

              And enough bed to sleep.

              1. Clunking Fist Bronze badge

                Re: I am prepared.

                I have lawns to mow, house to paint, children to raise, exercise to take. Netflicks? That would be nice...

        2. Venerable and Fragrant Wind of Change

          Walking on eggshells

          Um, do you really need to jump down the throat of someone who posts a harmless lighthearted quip?

  3. Doctor Syntax Silver badge

    "We understand this was likely due to the end of the month demands such as payroll runs and the deadline for submitting tax returns."

    There'll be another month end next month. And one the month after. Etc.

    1. Joe W Silver badge
      Happy

      And the transfer period for the German football(*} league also ended. It was likely all them damned krauts trying to pinch the British football players.

      (*) soccer for those leftpondians

    2. A Non e-mouse Silver badge

      This isn't the first time January 31st has happened.

      1. Anonymous Coward
        Anonymous Coward

        Yeah, but this one's speshal, its January 31rd, wen we get are cuntry bak!!!!1!!!

        1. Anonymous Coward
          Anonymous Coward

          @"Yeah, but this one's speshal, its January 31rd, wen we get are cuntry bak!!!!1!!!"

          Those who are actually getting their country back are, in reality, a tiny fraction of the UK population.

          The actual majority in the UK have actually lost their country given that they will have less control of what is done to them than before.

          1. Spanker

            (ex)Remainers are turning into those Japanese soldiers still found fighting WW2 in 1960 on some remote island. What a stupendous waste of your noisy time that was and you still don't plan on letting that tide in, eh?

          2. W.S.Gosset Silver badge

            > given that they will have less control of what is done to them than before.

            Actually, the EU expressly removed democracy, formally.

            To be clear, your EU vote had essentially 0 power. Check out the EU's FAQs on the EU Parliament's/your MEP's legislative power.

    3. Pascal Monett Silver badge

      Yeah. That's a pretty lousy excuse for not wanting to invest in some more processing power.

      I know that, to scale your web functionality to the maximum of your customer base is a ridiculous notion, but you might want to make sure that you have baseline plus a comfortable margin at all times, and emergency supplemental capacity when the demand balloons.

      I'm sure there are plenty of companies that are willing to help, it's just a question of writing a check . . Oh, right.

      1. Stubbly Dude

        There's this thing called auto-scaling that some large cloud companies offer apparently. It's terribly useful. Assuming that Tide is actually cloudy and not in someone's bloody garage...

    4. Anonymous Coward
      Anonymous Coward

      Stress testing and scalability.

      We heard of them...

    5. the Jim bloke Silver badge
      Coat

      There'll be another month end next month. And one the month after. Etc.

      but after Brexit, no one will have jobs...

    6. Andy The Hat Silver badge

      30% increase in traffic causing it to fall over? That's either incredibly short-sighted network management (by the bean-counters) or complete cobblers ...

  4. redpawn

    Just call a plumber

    "And again, the Internet is not something that you just dump something on. It's not a big truck. It's a series of tubes. And if you don't understand, those tubes can be filled and if they are filled, when you put your message in, it gets in line and it's going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material."

    1. Anonymous Coward
      Anonymous Coward

      Re: Just call a plumber

      The terrible thing is that the "tubes" analogy is kinda correct.

      1. cantankerous swineherd Silver badge

        Re: Just call a plumber

        Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. - Tannenbaum

        1. Stuart Castle

          Re: Just call a plumber

          You joke, but Amazon do apparently offer trucks as a method to transfer exabytes of data to AWS.. https://techcrunch.com/2016/11/30/amazon-will-truck-your-massive-piles-of-data-to-the-cloud-with-an-18-wheeler/

          1. TechnicalBen Silver badge

            Re: Just call a plumber

            IIRC a lot of companies even offer to return the tapes/hdds.

    2. Doctor Syntax Silver badge

      Re: Just call a sanitation engineer

      Given the nature of much of that material.

    3. Stubbly Dude

      Re: Just call a plumber

      So a bit like a gangbang where you are last in line because your bus was late...

  5. kschrock

    A historic day in every way.

    God bless England, Her Majesty, and above all, the fine people of England, who finally gave the world what so many folks had been asking for, a second referendum on Brexit.

    Well played Mr. Johnson.

    1. Anonymous Coward
      Anonymous Coward

      Re: A historic day in every way.

      Yeah, but not by this generation.

    2. Intractable Potsherd Silver badge

      Re: A historic day in every way.

      '... finally gave the world what so many folks had been asking for, a second referendum on Brexit."

      Ummm - there has only been *one* referendum on Brexit. If there had been a second one, fewer of us would be upset still (the difference between leave and stay was indistinguishable from noise - not a mandate to fuck the citizens of the country for the next two generations).

      However, read another way, yes, so many folks around the world might have been asking for a the UK to leave the EU, especially EU citizens who were pissed off at the UK's dog-in-a-manger attitude to being a member ("we'll approve these rules, but we won't follow them...") English exceptionalism at its best - anyone thinking of making a trade deal with us, remember how fucking treacherous our decision-makers are.

      1. The Original Steve

        Re: A historic day in every way.

        There's been two referendums on our membership of what we now call the EU. One a couple of years after we joined, and the other in 2016. Basically a generation between them.

        And 1.4 million people in a population (not electorate!) of 65 million is only a rounding error if you didn't like the result.

        Finally, whilst the the British often demanded exceptions and different treatment / opt-outs (Euro, Shengan etc) we're famous for sticking to the rules that were agreed. Unlike certain other countries I can think of... The nation of queuers will stick to all rules no matter now ridiculous they may seem. Not sure why us attempting to negotiate opt-outs and exceptions when the rules are made is a bad thing in your eyes.

        1. Intractable Potsherd Silver badge

          Re: A historic day in every way.

          No, there has only been one referendum on Brexit - 2016.

          1.4 million in a population of 65 million *is* a rounding error, especially when about 3 million UK citizens living overseas were excluded from this very important decision. When is a UK citizen not a UK citizen? When it suits the government, of course.

          Yeah, opt-outs like the Social Chapter which would have benefitted those people who voted to leave because they couldn't see the advantages of being members of the EU. The Euro was a bad idea, yes. Schengen not so much.

          EU rules that were gold-plated by Westminster to be almost unworkable, *then* stuck to rigidly. When complaints were made, the response was "Well, it's the EU wot dun it, innit." Don't blame other countries for sticking to what was intended.

    3. Clunking Fist Bronze badge

      Re: A historic day in every way.

      God bless all those natural labour voters who took one look at JC and said yeah nah. (And thanks to the good folk of Wales, too.)

  6. IGotOut

    I know it's a crazy idea...

    ...but could some of these be linked?

    1. Anonymous Coward
      Anonymous Coward

      Re: I know it's a crazy idea...

      BT definitely had significant SIP issues on Friday - if Gamma use them as an upstream provider (ie full outage) or maybe just connectivity to BT is down because of BT. I don't use Gamma so don't know...

      Anyone have any details of the BT issue? A major issue with BT connectivity could affect Nationwide if Nationwide use them as a supplier.

      Speculation of course - we use BT SIP and while we had some issues, they were unrelated to the larger outage.

    2. Stubbly Dude

      Re: I know it's a crazy idea...

      Well, they all happened on the same day, in the same country... beyond that I'm stumped..

  7. HildyJ Silver badge
    Devil

    Turn the UK on and off again

    Turning the UK on and off again won't work. It needs to be smacked upside the head.

    Then again, the same can be said for my benighted (and knightless) Merica.

    1. Dr Paul Taylor

      Re: Turn the UK on and off again

      Well, after four years of general fiddling with the switch, the UK has now been turned off.

      It needs to be smacked upside the head.

      It has probably needed that for a 100 years. Maybe it will get it.

      As for turning it back on again, we'll have to see about that.

    2. Boris the Cockroach Silver badge
      Trollface

      Re: Turn the UK on and off again

      But dont worry, you've got a king again

      After all... whatever he does , the senate will find him not guilty

      1. Anonymous Coward
        Anonymous Coward

        Re: Turn the UK on and off again

        I think you'll find he's actually an deified Emperor - a Caesar in fact.

        Hence the saying "Render unto Caesar" ;)

      2. Clunking Fist Bronze badge

        Re: Turn the UK on and off again

        "the senate will find him not guilty" Sanction torture and illegally invade other countries on pretexts and turn them into shitholes (Bush II, Obama): okay. Make a phone call: beyond the pale.

        Trump will be judged in the Court of Public Opinion in a few months time. So Democrats who have a proven track record of enriching themselves without delivering anything to the ordinary folk, are having a tanty cos their gravy train driver didn't get the nod.

        First they said he was a Russian agent and that didn't work, now they are trying this. One day they'll pick a decent candidate and win an election! But they hate Bernie as he doesn't really play the gravy train game.

    3. Tom 7 Silver badge

      Re: Turn the UK on and off again

      We turned it off by smashing the power switch!

    4. Stubbly Dude

      Re: Turn the UK on and off again

      I think it's more of a 'turn it upside down and shake it' type of reboot that's needed now.

  8. Anonymous Coward
    Anonymous Coward

    Fibre

    Wasn't there a major outage recently because the main AND backup fibre ran through the same physical junction box though separate everywhere else?

    Said fibres getting cremated due to an underground substation fire.

    1. Anonymous Coward
  9. Anonymous Coward
    Anonymous Coward

    Welcome

    to post-Brexit Little Britain.

    1. TRT Silver badge

      Re: Welcome

      Computer says.... No.

      1. el kabong Silver badge

        Computer knows nothing

        Whatever it says... it does not matter.

        1. Anonymous Coward
          Anonymous Coward

          Re: Computer knows nothing

          Unless you're adversely affected by the computer's decision. The occurrences of which are only going to increase in future.

      2. Jou (Mxyzptlk) Bronze badge

        Re: Welcome

        *cough* *cough*

    2. Anonymous Coward
      Anonymous Coward

      Re: Welcome

      Yeah, good luck to you, guys.

      The UK people will be missed but for sure the country's government won't !

  10. Anonymous Coward
    Anonymous Coward

    Gamma

    My company has two diverse feeds to two separate Gamma SBCs. Both connections were affected yesterday. Not impressed. Once the dust has settled we'll be having a stern word with our account manager.

    1. Anonymous Coward
      Anonymous Coward

      Re: Gamma

      I used to work for a small(ish) UK ISP in the early days, and despite having their own connectivity at their head office, nested in the corner of the room was a Plusnet ADSL circuit for emergency use.

      I worked there for 9 months and we had to use that connection - more than once during several large outages... Go figure. (Anon, for very obvious reasons).

  11. macjules Silver badge

    Online biz-friendly bank Tide.co

    Worth noting that even though they call themselves a 'bank' they are not. They are simply a prepayment solution managed by Prepay Technologies Ltd, so not subject to FCA supervision. It is a Bulgarian operation with an offshore development centre in India but "headquartered" in London for the benefit of the VC investors..

    1. Strahd Ivarius
      Trollface

      Re: Online biz-friendly bank Tide.co

      So the unexpected surge was due to Bulgarians buying .uk domains just before Brexit?

      1. Trollslayer Silver badge
        Thumb Up

        Re: Online biz-friendly bank Tide.co

        You beat me to it!

    2. Snake Silver badge

      Re: Online biz-friendly bank Tide.co

      In the States, "Tide" is a brand of laundry detergents.

      I wonder if the Bulgarian's selection of "Tide" has any laundering associations... O.o

      1. John Brown (no body) Silver badge

        Re: Online biz-friendly bank Tide.co

        "In the States, "Tide" is a brand of laundry detergents."

        Is was (or maybe still is) here in the UK too.

      2. Tom 7 Silver badge

        Re: Online biz-friendly bank Tide.co

        In the UK "Tide" is description of the sewerage we are about to get.

  12. Richard Parkin

    War with Spain now :)

    First day out and it’s WAR with Spain :)

    Captain, art thou sleeping there below?

    https://www.theguardian.com/politic...-talks-eu-to-back-spain-over-gibraltar-claims

    1. druck Silver badge

      Re: War with Spain now :)

      The Spanish will have to wait their turn, it's traditional to have a war with France first.

      1. Anonymous Coward
        Anonymous Coward

        Re: War with Spain now :)

        The Scots are traditionally France's allies...

      2. baud Bronze badge

        Re: War with Spain now :)

        Nah, we've just beaten England in rugby, we're not interested in getting in another fight with Blighty

  13. Anonymous Coward
    Anonymous Coward

    Down for maintenance while they install the post Brexit spying gear or they've started ripping out the Huawei gear.

  14. Anonymous Coward
    Anonymous Coward

    Not just the UK

    Woke up yesterday to an email saying the major US IT supplier I was working with had lost comms to one of their biggest sites States side.

  15. Jou (Mxyzptlk) Bronze badge

    No problem here...

    ...I could access threregister from Germany without problems throughout last Friday. Looks like the vulture has a good connection.

  16. Ken Moorhouse Silver badge

    Traditional Banks...

    Publish their opening hours.

    Looks like Tide need to start publishing moon phases together with your location so that customers can tell if it's in or out.

    https://www.tide-forecast.com/

    1. Tom 7 Silver badge

      Re: Traditional Banks...

      Strangely MagicSeaweed is used to check the tide round here.

  17. CountCadaver Bronze badge

    Ninewells Hospital in Dundee also

    Ninewells hospital phone system crapped the bed also from lunchtime onwards also....message "our telecoms service provider is working on a resolution as a matter of urgency" for over 10 hours and counting...

    1. Anonymous Coward
      Anonymous Coward

      Re: Ninewells Hospital in Dundee also

      Dundee and Angus College in lockdown due to ransomware attack...

      1. Anonymous Coward
        Anonymous Coward

        Re: Ninewells Hospital in Dundee also

        given how crap their IT systems are...not much of a loss tbh, post merger they kept all the PHB from "industry" (read expensive failures with no idea how to run a college dept) and those with a clue left due to the idiotic BS emanating from their new overlords, such as shutting down the welding and fabrication vocational course (popular with people in their 20s and 30s upskilling) in favour of an dumbed down NC aimed at 16 year olds on apprenticeships and with much reduced practical welding / fabrication time (they'll do that at their work, we're here to make them literate and numerate to improve their academic skillbase etc etc etc etc ignoring entirely that the weld/fab course was popular with industry due to it spitting out coded welders at the end who went into decent paying welding jobs and that many local firms don't have the headcount or resources to train apprentices for 4 years and often would prefer someone in their 20s or 30s with a decent work ethic and practical knowedge - less risk for one and better chances of retention due to those beyond school leaving age having committments - partner, kids, mortgage etc etc), worse with talk of replacing those "dangerous" welders with some nice safe "virtual reality" welding simulators (of course costing a fortune, not replicating actual welding in anyway shape or form and likely netting the PHB a sizeable kick back)

  18. Anonymous Coward
    Anonymous Coward

    Vodafone flaky here in Nottingham

    Last 24 hours the high speed broadband is down to dial-up speed.

  19. Version 1.0 Silver badge

    Highly Reliable (marketing) post exbrit profits

    Internet connections, system backups, bank IT system are all highly reliable and trustworthy. This is documented by the highly paid marketing department and the corporate management teams who take a nice bonus every year because the corporate profits have been raised by cutting back the support systems, moving the system backups and IT support to the third world and cutting deals with their friends.

    It used to be that when you set up a backup of any kind the prime question was always, "How many ways are there for this to fail?"

    Nowadays it just, "There's a website that tells you how to do it."

    1. Jou (Mxyzptlk) Bronze badge
      Pint

      Re: Highly Reliable (marketing) post exbrit profits

      "exbrit" - that typo deserves a

    2. ecofeco Silver badge

      Re: Highly Reliable (marketing) post exbrit profits

      Too damn true.

  20. Anonymous Coward
    Anonymous Coward

    How long before we get?:

    The Russians did it.

  21. Anonymous Coward
    Anonymous Coward

    So...

    It's really a case of telco / big IT shops in the UK just being absolutely fucking incompetent....

  22. Celeste Reinard

    Bricks-it

    Fes up, who of you lot broke the internet? Hmmmm? We know it was one of you... And we expect the real story at our desk at Mission Control, monday morning, chez BOFH, or at Who, me?

    1. redpawn

      Re: Bricks-it

      Someone opened the email titled "Join the Crew"

  23. Dolvaran

    Your country has crashed

    Reflect, repent, reboot

    Peace will return.

    (with apologies to the original)

  24. AshOnline
    IT Angle

    Did BT have any issues?

    Apart from downdetector who rely on user input, with users who dont have a clue.

    I see no evidence that BT had an outage.

    In fact, as a BT Wholesale reseller, i can say none of our Tetrabyte customer internet connections were affected, neither was our Tetrabyte hosted voip systems!

    I think the key here is the fact that Gamma went down and many users checked out BT downdetector

  25. AbeSapian

    Is Brexit BrokesIt?

    It's all in the title. Of course, being in Ameirka, I can't talk too much. We're still Trumped.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020