back to article Why millions of Brits' mobile phones were knackered on Thursday: An expired Ericsson software certificate

Ericsson says an expired software certificate caused the outage that left tens of millions in the UK unable to call or text from their mobile phones, nor use 4G connections, on Thursday. The Swedish equipment maker, which manufactures much of the backend gear in the world's cellular networks, said today the downtime was due to …

Silver badge

Don't feel so bad Ericsson, you probably did us all a favour!

How may "zombies" had to stop staring at phones on public tranport and actually read something in the paper, look around them or worse actually talk to other passengers! Oh the humanity! Oh the number of cat videos and half-naked teenage girls who's Instagram pages didn't get visited today, oh think we need a charity single by Sting and Bono to help them through this terrible time.

Yes, I know there probably was serious fallout for businesses and people urgently trying to arrange personal business but for most of us it's just bloody annoying and for the most part we had to take a break from our screens and actually take in the world around us for a day.

47
20
Silver badge

Re: Don't feel so bad Ericsson, you probably did us all a favour!

or worse actually talk to other passengers!

Talk about expecting miracles.

I remember using public transport more often than I do now, back before mobile phones or even smart phones.

I don't recall random passengers striking up conversations with others on public transport then either*.

But perhaps the technique of sitting frozen frigid in embarrassed silence and trying not to make eye contact with anyone is a lost skill now.

* Well, maybe once, and I'm fairly sure she was a visitor from another dimension.

39
0

Re: Don't feel so bad Ericsson, you probably did us all a favour!

Don't be daft, I just tethered to my other phone and carried on being a zombie.

25
0
Silver badge

Re: Don't feel so bad Ericsson, you probably did us all a favour!

> How may "zombies" had to stop staring at phones on public tranport and actually read something in the paper, look around them or worse actually talk to other passengers

Not me! I have Solitaire on my phone for emergencies such as this!

39
1
Silver badge

Re: Don't feel so bad Ericsson, you probably did us all a favour!

I was on the tube one evening, about 8pm. It wasn't packed, but most seats were taken.

Everyone was doing the usual - reading the adverts, looking at their phones, trying to avoid eye-contact.

Then at one stop, 3 or 4 people got on.. Shall we say "in the party spirit"... They were singing, and talking to the rest of us, and cracking jokes with us, and goading us all into generally joining in.

The whole carriage joined in, and started cracking jokes too. Even when these people got off, everyone else on the carriage continued chatting, and everyone said "bye" when they got off at their stop.

Just needs an ice-breaker...

37
0
Anonymous Coward

Re: Don't feel so bad Ericsson, you probably did us all a favour!

“look around them or worse actually talk to other passengers!”

What sort of monster are you?

20
0
Silver badge

Re: Don't feel so bad Ericsson, you probably did us all a favour!

Damn, I must be a freak. On a long train journey, I more often than not find myself in conversation with one or more actual people, merely by virtue of occupying neighbouring seats.

p.s. my O2 4G returned sometime yesterday evening. When I put the phone on the charger around midnight, it was there.

4
0
Silver badge

Re: Don't feel so bad Ericsson, you probably did us all a favour!

OK, grandpa, you've identified a problem: things are changing in ways that you don't like. You've expressed dismay and contempt, which thousands of others have done before you.

Do you have a solution to propose? I mean, other than simply shutting down the cellular network and/or the Internet, which (aside from the problems doing so and making it stick) has its own negative consequences.

Because if not, you're in danger of looking like Abe Simpson. https://i.kym-cdn.com/photos/images/newsfeed/001/044/247/297.png

1
1
Anonymous Coward

Re: Don't feel so bad Ericsson, you probably did us all a favour!

You realise that “newspapers” (and magazines, like the one we are all reading here) are on the internet now?

The idea of printed newspapers nowadays is terribly retro, so why would I have a bulky inky smelly non-virtual one…?

Rightly or wrongly pretty much the only people who buy them are other journalists and similar PR or media people.

1
2

Re: Don't feel so bad Ericsson, you probably did us all a favour!

Imagine the shortage of digital dopamine from not being able to share, like or post anything. Tragedy lol.

0
0

Reminder

Google Calendar.

December 5 2019

Renew certificate.

74
0
Anonymous Coward

Re: Reminder

And then you leave, and no-one else has access to your calendar/mailbox.

No problem, you say, use a group calendar! And then due to reorgs/scope creep/laziness, "your" groups calendar falls into disuse. Or the mailing list gets retired. Or the recipients filter annoying certificate-provider emails to trash.

Yes, I've seen them all. Though the "best" was a replacement root certificate, replacing a perfectly good root cert, but not published to the thousands of systems that depended on the Certification Authority chain.

Certificates - Great in theory, and they even tell you exactly what's gone wrong, but will bite the unwary.

28
0
Anonymous Coward

Re: Reminder

perhaps it was somebody streamlined out, who "forgot" to mark a date in the calendar :D

5
0
Silver badge
Paris Hilton

Re: Reminder

And your successor thinks: OK, so I need to renew a certificate... but which one? Then proceeds to go off to renew their 50m swimming certificate.

3
0
Silver badge

Re: Reminder

That would be Arnold J Rimmer, BSC SSC.

13
0

Re: Reminder

... reminds me of >

https://www.theregister.co.uk/2003/11/06/microsoft_forgets_to_renew_hotmail/

1
0
LDS
Silver badge

"but will bite the unwary"

I'm surprised about how many applications using certificate, don't have any kind of management and warnings about them. You have to manage everything "out-of-band", and even most CA software more or less think they're done as soon as they issue a certificate, and doesn't make management and especially warning very friendly.

Often, applications certificate features looks "bolt-on" somehow, and nothing is done to tell when a certificate is about to expire. All the telemetry, tracking, big data analysis an nothing warns when a damned certificate is about to "die"???

1
0
Silver badge

Re: Reminder

From the way it's written, this doesn't sound like the security certificates people here seem to be assuming. A lot of software like this uses keys (or certificates) to enable features - when it runs out, the software/feature stops working. Thus you have to keep paying the vendor's support fees for as long as you want to keep using the software/feature.

And typically there is some management function that will a) warn you about impending expiry, and b) allow installation of new keys/certificates.

It sounds a lot like "something went wrong" with this renewal process, so come the expiry time of the key, the software/feature stopped working - and the network stopped working.

0
0
Silver badge

Note to self ..

Next time replace that Symantec certificate ...

8
0

Re: Note to self ..

use LetsEncrypt and Certbot

18
3
Pint

Re: Note to self ..

Bummer. I was going to write <code>sudo certbot-auto</code>.

Have an up vote.

5
2

Re: Note to self ..

That works great until your internet connection goes down, or the server gets firewalled by someone who doesn't understand certbot...

6
4

Re: Note to self ..

These things don't have internet access. They're not a hobbyist website. They're core nodes in a telecom network. It's national infrastructure.

16
2

Re: Note to self ..

“These things don't have internet access. They're not a hobbyist website. They're core nodes in a telecom network. It's national infrastructure.”

Yes....My question is if “older software” means that a fix was available via an existing patch or upgrade that had been “delayed” or whether this was a new and unexpected issue.

I don’t expect that even with Internet access that the certificate could have been renewed automatically.

3
0
Anonymous Coward

"These things don't have internet access"

Well there's the problem straight away - no wonder none of the traffic was able to access any data based web services.

And wow it's going to be a slow process with an engineer visiting every box with their serial cable to update the certs.

3
1
WTF?

Re: These things don't have internet access

@tomalak Call me a dillettante dabbler, but are you telling us that the "core nodes in a telecom network" which provides Internet access to millions ... don't have internet access?! Me not understand X-(

5
2

Re: Note to self ..

If you allow people who don't know what they are doing to have access to your server firewall rules, you have bigger problems than you yet know...

4
0
Silver badge

Re: These things don't have internet access

but are you telling us that the "core nodes in a telecom network" which provides Internet access to millions ... don't have internet access?

No longer having internet access was the problem.

4
0

Re: Note to self ..

Even better, set up a certbot renew cron job

0
3
Anonymous Coward

Re: Note to self ..

tbh it was probably a certificate that had a reeeeally long expiry date. Maybe 10-plus years. Hence why it took so long to sort out?

2
0
Anonymous Coward

"And wow it's going to be a slow process with an engineer visiting every box with their serial cable to update the certs."

Why do think it took so long to restore?

0
0

Re: Note to self ..

Yep, that's exactly the point isn't it? Someone sets something up, assuming that the system will work as infinitum, but it ends up being forgotten by someone else in the system.

It doesn't have to just be server firewall rules. It can be something upstream, eg. a new router, that quietly locks out regular but infrequent network activity. The server admin is not necessarily the network admin. No one notices until it's too late.

The result is a popcorn moment.

0
0
LDS
Silver badge

"which provides Internet access to millions ... don't have internet access?"

I really hope so. I hope they are reachable for management only from an internal management network separated from the Internet traffic they carry. I really do no expect any management access being connected directly to the Internet.

These devices are used by the very companies that build the core network infrastructure, they should not need "the internet" or any other network to be reached by the control rooms...

Still, if the certificate was used for the management network access....

1
0
Silver badge

More detail

Was this software administered by O2 or Ericsson? 'Cause one of them needs a huge slap for missing that deadline.

9
0
Silver badge
FAIL

Re: More detail

More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for <SOFTWARE_COMPONENT>, please renew or lose all packet data connectivity for your subscribers on <EXPIRY DATE>" every time anyone logs in to the management s/w when such a scenario becomes likely (e.g. for the last month). This should be a basic part of any s/w licensing feature.

39
1
Facepalm

why the fsck the s/w doesn't present a big flashing dialog

Opps! Sorry that was me. I must have kicked the reminder machine that is under my desk, and I think I dislodged the network cable.

10
0
Silver badge

Re: More detail

More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for..

.. failing to insert money. Back in the simpler days, you bought kit, it had software, you paid for a service contract that supported it, including access to software updates. Then along came software as a service, and new revenue streams. So instead of buying kit, you pay an annual rent or it can stop working.

Which can (or should) factor into vendor selection given it can work out to a lot of money, especially if vendors want $$$ every time you add a device, or in some cases just add a new virtual circuit. Alcatel's NMS used to work on that model where you bought licence packs of points, and actions cost points. They weren't selected for a large network I worked on mainly for that reason. Nice kit, lousy business model.

I'm a bit suprised that this happened. An expired cert should have been flagged as a critical risk, if that resulted in a network shutdown. Plus given $$$ for new licences, a sales bod should have been chasing for renewals.

5
2
Joke

Re: One of them needs a huge slap

Even better, a "limited number" of slaps. Or maybe that's just the PR department ;-)

3
0

Re: More detail

As long as someone actually logs into the management console... or is looking at the notitication alert emails that its sending out.... thats if someone has actually configured it to send out emails....

7
0
Silver badge

Re: More detail

"More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for ...."

Perhaps this was the responsibility of people amongst the 18,000 laid off by Ericsson in the last year.

12
0

Re: More detail

"I'm a bit suprised that this happened. An expired cert should have been flagged as a critical risk, if that resulted in a network shutdown."

It should be considered almost as important as filing the annual accounts - what do the tax authorities respond with if someone forgets that?

2
0
Silver badge

Re: More detail

"thats if someone has actually configured it to send out emails...."

And if the recipient of the emails is still there.

It's easy enough to set up a warning system. Protecting that warning system against the ravages of management changes is a different matter and almost certainly outside the powers of whoever set it up. If you were the one who was the designated recipient of the email and you've just been booted out of the job are you going to be in a mood to warn whoever did the booting that that particular mail box needs to be monitored? Is the booter even going to listen if you did? And will the booter get booted out in the next bout of changes?

There needs to be personal responsibility on those making such changes to ensure that everything like this gets covered under the new organisation. HMG has woken up to the fact that national infrastructure needs to be protected even when it's in private hands. Maybe that protection should extend to personal sanctions on those involved, even up to CEOs and board members. Make them sweat a little. After a few big personal fines or gaol sentences businesses would become a little less cavalier about reorganisations and outsourcing.

7
0
Anonymous Coward

Re: More detail

was thinking the same, is it Ericsson who install and manage it or O2, you'd kind of think O2 would have noticed via SCCM or some other monitoring tool that it is going to expire

0
4
LDS
Silver badge

"And if the recipient of the emails is still there."

I hope nobody really uses emails for that anymore - but for small networks. What are SNMP and all those expensive network monitoring systems for?? Big red lights should appear besides any device which have certificates about to expire. It's akin having and hardware components about to fail. You get proactive SMART alerts, but nothing about certificates...

0
0
Anonymous Coward

Hey ...

My old-skool Nokia 3310 was working just fine.

O2? Nah, not me! lolol

5
19

Re: Hey ...

Who cares then? You're not relevant!

14
3
Silver badge

Re: Hey ...

Couple on the table next to me at lunch didn't have any service on their phones. Not to be outdone by Sky Mobile not working they switched to their O2 backup sims. Sadly that wasn't working for them either and therefore "Every network must be down at the moment not just Sky!" They were most amazed when I received a call.........

21
0
Happy

Re: Hey ...

So was mine, and yes, it's O2 (Tesco),

0
0

Test successful

At least we know the certificate is respected downstream!

43
0
Silver badge

Re: Test successful

The system is robust. The people, not so much.

22
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018