Is everything OK over there, Britain? Have you tried turning the UK off and on again? ISPs, financial orgs fall over in Freaky Friday of outages Today was more Friday the 13th than Friday the 31st in the UK, it seems. Not only is it Brexit Day, marking Blighty's withdrawal from Europe, but a bunch of services and internet connectivity broke. Here's a quick summary of what went wrong, according to Register readers writing in. If we've missed anything out, please let us …
Your reader who has 500 users but no independent internet backup should get their management to invest in some decent resilience. We have 3 independent fibre providers and routes out of our building, two independent phone systems with two independent providers oh and an actual DR capability. No reason not too with comms being so cheap in 2020.
It wasn't just north/south lines. I remember the incident.
Woking-London routed via Brum as well so we lost the kilostreams to every one of our customers in the city. The joy of non-diverse routing for anywhere except the City Of London in those days.
1st time we actually used our dialback modems in anger and found there was a bug in the code that meant they negotiated 4800 baud connections rather than 9600.
A friend took over the running of a small shop. Their till software & credit card machines all rely on the cloud. I did say that they need to think of what happens when (not if!) their net connection goes down. I did try to explain that the cost of a 2nd net connection would be dwarfed by the loss of income if things went down.
Then things went down.
And they paniced.
And still won't invest in a second 'net conection.
Back in my Saturday job days (late 1990s) the small retail establishment, where I was gainfully employed, didn't get a proper credit card machine until around 1996. Up until then we had to phone up a meat-bag to get a 6-digit number code to authorise each credit card transaction. Debit cards were still reasonably unusual with people paying by cheque+guarantee card.
We kept the clack-click mechanical card imprinter as a back up in case the phone line went down or the machine went wrong.
Just last year I was in queue to purchase at a chain bookstore when their cc processing went down. They fiddled, tried for a bit but no hope. The manager on duty unearthed a knuckle-buster and credit card slips from a store room, but had no idea how to use it, like a young person today with a rotary dial phone. I showed them how to employ it, warned that it was only for cards with raised numbers, to check the expiration date visually, and get a signature and the phone number in case of problems. Then said that to process the charges, they would probably have to manually key in all charges off the slips, but check with head office first. Since my card was properly charged, I can only assume it all worked.
4G Masts do not connect via FTTC PCP cabinets, they are fed via a leased line directly back to the nearest exchange/POP. It might be located right next to your green FTTC cabinet because that's where the nearest BT ducting is, or where the nearest T-Node happened to be making life easier for all involved ;-)
(That, or the cabinet could have come AFTER the 4G mast now BT Openreach had to pull fibre for the mast).
15 or years ago I was involved in a project were some nut job had decided to have a datacentre inside a country house hotel (AKA vanity project). Getting a fibre in wasn't a problem. But when we asked for a second one coming on a different route BT sucked in a bit and said "Sure, that'll be £250,000" The management up stream lost interest at that point.
You can't really be *truly* carrier-resilient on VoIP, certainly not for inbound calls. The CLI has to terminate at a physical POP somewhere and if the carrier is Tier 1, like Gamma, and suffers an outage like that, you can't reroute calls - In-fact, you can't do jack shit except perhaps have a backup trunk with another provider, but even then your incoming calls are still fubar!
@"Yeah, but this one's speshal, its January 31rd, wen we get are cuntry bak!!!!1!!!"
Those who are actually getting their country back are, in reality, a tiny fraction of the UK population.
The actual majority in the UK have actually lost their country given that they will have less control of what is done to them than before.
Yeah. That's a pretty lousy excuse for not wanting to invest in some more processing power.
I know that, to scale your web functionality to the maximum of your customer base is a ridiculous notion, but you might want to make sure that you have baseline plus a comfortable margin at all times, and emergency supplemental capacity when the demand balloons.
I'm sure there are plenty of companies that are willing to help, it's just a question of writing a check . . Oh, right.
"And again, the Internet is not something that you just dump something on. It's not a big truck. It's a series of tubes. And if you don't understand, those tubes can be filled and if they are filled, when you put your message in, it gets in line and it's going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material."
'... finally gave the world what so many folks had been asking for, a second referendum on Brexit."
Ummm - there has only been *one* referendum on Brexit. If there had been a second one, fewer of us would be upset still (the difference between leave and stay was indistinguishable from noise - not a mandate to fuck the citizens of the country for the next two generations).
However, read another way, yes, so many folks around the world might have been asking for a the UK to leave the EU, especially EU citizens who were pissed off at the UK's dog-in-a-manger attitude to being a member ("we'll approve these rules, but we won't follow them...") English exceptionalism at its best - anyone thinking of making a trade deal with us, remember how fucking treacherous our decision-makers are.
There's been two referendums on our membership of what we now call the EU. One a couple of years after we joined, and the other in 2016. Basically a generation between them.
And 1.4 million people in a population (not electorate!) of 65 million is only a rounding error if you didn't like the result.
Finally, whilst the the British often demanded exceptions and different treatment / opt-outs (Euro, Shengan etc) we're famous for sticking to the rules that were agreed. Unlike certain other countries I can think of... The nation of queuers will stick to all rules no matter now ridiculous they may seem. Not sure why us attempting to negotiate opt-outs and exceptions when the rules are made is a bad thing in your eyes.
No, there has only been one referendum on Brexit - 2016.
1.4 million in a population of 65 million *is* a rounding error, especially when about 3 million UK citizens living overseas were excluded from this very important decision. When is a UK citizen not a UK citizen? When it suits the government, of course.
Yeah, opt-outs like the Social Chapter which would have benefitted those people who voted to leave because they couldn't see the advantages of being members of the EU. The Euro was a bad idea, yes. Schengen not so much.
EU rules that were gold-plated by Westminster to be almost unworkable, *then* stuck to rigidly. When complaints were made, the response was "Well, it's the EU wot dun it, innit." Don't blame other countries for sticking to what was intended.
BT definitely had significant SIP issues on Friday - if Gamma use them as an upstream provider (ie full outage) or maybe just connectivity to BT is down because of BT. I don't use Gamma so don't know...
Anyone have any details of the BT issue? A major issue with BT connectivity could affect Nationwide if Nationwide use them as a supplier.
Speculation of course - we use BT SIP and while we had some issues, they were unrelated to the larger outage.
Well, after four years of general fiddling with the switch, the UK has now been turned off.
It needs to be smacked upside the head.
It has probably needed that for a 100 years. Maybe it will get it.
As for turning it back on again, we'll have to see about that.
"the senate will find him not guilty" Sanction torture and illegally invade other countries on pretexts and turn them into shitholes (Bush II, Obama): okay. Make a phone call: beyond the pale.
Trump will be judged in the Court of Public Opinion in a few months time. So Democrats who have a proven track record of enriching themselves without delivering anything to the ordinary folk, are having a tanty cos their gravy train driver didn't get the nod.
First they said he was a Russian agent and that didn't work, now they are trying this. One day they'll pick a decent candidate and win an election! But they hate Bernie as he doesn't really play the gravy train game.
Are you thinking of this incident: BT denies cable fire was in [Manchester Guardian] A-bomb exchange?
I used to work for a small(ish) UK ISP in the early days, and despite having their own connectivity at their head office, nested in the corner of the room was a Plusnet ADSL circuit for emergency use.
I worked there for 9 months and we had to use that connection - more than once during several large outages... Go figure. (Anon, for very obvious reasons).
Worth noting that even though they call themselves a 'bank' they are not. They are simply a prepayment solution managed by Prepay Technologies Ltd, so not subject to FCA supervision. It is a Bulgarian operation with an offshore development centre in India but "headquartered" in London for the benefit of the VC investors..
given how crap their IT systems are...not much of a loss tbh, post merger they kept all the PHB from "industry" (read expensive failures with no idea how to run a college dept) and those with a clue left due to the idiotic BS emanating from their new overlords, such as shutting down the welding and fabrication vocational course (popular with people in their 20s and 30s upskilling) in favour of an dumbed down NC aimed at 16 year olds on apprenticeships and with much reduced practical welding / fabrication time (they'll do that at their work, we're here to make them literate and numerate to improve their academic skillbase etc etc etc etc ignoring entirely that the weld/fab course was popular with industry due to it spitting out coded welders at the end who went into decent paying welding jobs and that many local firms don't have the headcount or resources to train apprentices for 4 years and often would prefer someone in their 20s or 30s with a decent work ethic and practical knowedge - less risk for one and better chances of retention due to those beyond school leaving age having committments - partner, kids, mortgage etc etc), worse with talk of replacing those "dangerous" welders with some nice safe "virtual reality" welding simulators (of course costing a fortune, not replicating actual welding in anyway shape or form and likely netting the PHB a sizeable kick back)
Internet connections, system backups, bank IT system are all highly reliable and trustworthy. This is documented by the highly paid marketing department and the corporate management teams who take a nice bonus every year because the corporate profits have been raised by cutting back the support systems, moving the system backups and IT support to the third world and cutting deals with their friends.
It used to be that when you set up a backup of any kind the prime question was always, "How many ways are there for this to fail?"
Nowadays it just, "There's a website that tells you how to do it."
Apart from downdetector who rely on user input, with users who dont have a clue.
I see no evidence that BT had an outage.
In fact, as a BT Wholesale reseller, i can say none of our Tetrabyte customer internet connections were affected, neither was our Tetrabyte hosted voip systems!
I think the key here is the fact that Gamma went down and many users checked out BT downdetector
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
