back to article Expired cert... Really? #O2down meltdown shows we should fear bungles and bugs more than hackers

It's a bit of a cliche that "everything's connected", but O2's stunning outage yesterday – chalked up by Swedish kitmaker Ericsson to an expired software certificate – is a reminder of how true that is. Payment terminals croaked, bus displays went blank. Strangers blinked at each other in the street, like Robinson Crusoe …

Bad news. The fog's getting thicker.

And Leon is getting laaaaarrrrrger.

47
1
Silver badge

Re: Bad news. The fog's getting thicker.

The fog's getting thicker.....And Leon is getting laaaaarrrrrger.

In fog, the time to worry is when the word "Scania" looms into view and is getting rapidly larger.

16
0

Re: Bad news. The fog's getting thicker.

Or ovloV

20
0
Flame

Acronyms

FFS (For F£$k Sake) expand your acronyms the first time you use them!

I've got better things to do on a Friday mid-morning than work out whether M2M means made to measure, machine-to-machine, or some defunct Norwegian pop duo!

Well, slightly better, I mean - reading the Reg ......

71
1

Re: Acronyms

Indeed. '"MVP" mentality' - Model/View/Presenter mentality? Most Valued Professional mentality?

28
1

Re: Acronyms

Beat me to it

10
1
Anonymous Coward

Re: Acronyms

As this is a "co.uk" site they're abbreviations not acronyms

15
10

Re: Acronyms

"As this is a "co.uk" site they're abbreviations not acronyms"

No, these are all TLAs (Three Letter Acronyms).

14
13
Anonymous Coward

Re: Acronyms

Minimum Viable Product

What normal people would call an alpha release

12
2
Headmaster

Re: Acronyms

All of these were, indeed Three Letter Abreviations (TLA's)

12
3
Silver badge

Re: Acronyms

Agreed, in my book an acronym should be pronounceable, as in SNAFU

14
0
Silver badge

Re: Acronyms

"No, these are all TLAs (Three Letter Acronyms)."

Two out of three ain't bad.

Three Letter Abbreviations.

15
1
TRT
Silver badge

Re: Acronyms

MJE.

Miniumum Journalistic Effort

or

Maximum Jargon Enclosure

9
0

Re: <strike>Acronyms</strike> Initialism

Came up on the first Google search so it must be right.

Acronym = Letters that from words

Abbreviation = Shortened word E.G. St, Dr etc

Initialism = First letter of each word and enunciated E.G. VIP

If I'm wrong blame Google, it's not that I'm lazy... honest!

13
2

Re: <strike>Acronyms</strike> Initialism

Wow, on a dot-uk site, no-one seems to have a copy of Fowler? This all falls under 'curtailment', and Britons do not need to keep their vocabulary in the Victorian era. Acronymn is a 20th century invention.

2
0

Re: Acronyms

Most Valuable Player

3
0

Re: <strike>Acronyms</strike> Initialism

Abbreviation = Shortened word E.G. St, Dr etc

I always thought St and Dr were contractions but never bothered with the apostrophe.

6
2
TRT
Silver badge

Re: <strike>Acronyms</strike> Initialism

No. St. and Dr. are abbreviations.

Can't and Don't are contractions - they are made up from two or more words.

5
0
Silver badge
Happy

Re: Acronyms

"No, these are all TLAs (Three Letter Acronyms)."

Ok, so what is M2M then? TLAAN? (Two Letters And A Number)

4
1

Re: Acronyms

what about ETLAs??? :P

0
0

Re: Acronyms

MVP, what counts for any normal techy solution in the current day. Deliver the absolute minimum, promise the earth & walk away, safe in the knowledge that unless the customer is really, really big there is sod all anyone can do about it.

And even if you are really big, this is still probably sod all you can do about it.

0
0
Silver badge

Hanlon's razor strikes again.

11
0
Silver badge

Most definitely - Never attribute to malice that which is adequately explained by stupidity.

5
0
Bronze badge

Hanlon's razor

How about DBN. Don't Be Naive. People can be bad so stop giving them the benefit of the doubt. If they did something wrong don't let them off just because you think they did not mean it.

0
0
Silver badge

Was this

Not the same Ericsson who caused a series of outages on O2 in 2012?

Lessons learned of course .......

19
0
Silver badge

Re: Was this

Lessons learned of course

Not sure which of the lessons from the 2012 outage would be applicable to yesterday's situation?

2
2

Re: Was this

They're funny things, Accidents. You never have them till you're having them.”

― A.A. Milne, The House at Pooh Corner

28
0
Silver badge

Re: Was this

Do not blame Ericsson here.

UK telco operations have a well established and entrenched fear of certificates for anything.

Once upon a time, before I went back to write software, I still did network architecture including security aspects. So while working in a major UK telco I proposed the idea of certificates everywhere for purposes of inventory, identification and security of provisioning. I was freshly out of a vendor where I did most of the design and implementation of a x509 retrofit into everything and they became the foundation of how the system fits together. So I was expecting some questions or a technical discussion.

I got none.

The faces around the table looked like they were a still frame from The Shining. They looked at the idea like I was serving a disemboweled body with maggots and suggesting they eat it. They were horrified at the idea despite having less than 60% accurate inventory and a long standing requirement to secure key aspects of the network management.

This fear has its roots in incidents like the one in O2. It is also the root cause of incidents like O2.

UK telcos (and most telcos in general) fail to understand the most basic principle of using X509 for infrastructure purposes.

It is: YOU RUN YOUR OWN CA. No vendor roots. The root is yours. And so are ALL certs.

Because they do not understand it and fear it, they either use vendor certs (which expire at the most unfortunate moment) or outsource it to an external CA which defeats the purpose of the exercise as you are no longer in control of your network. Either one of these results in an incident like O2 which in turn results in more fear, more vendor use and more outsourcing.

Ad naseum, rinse repeat.

Oh, and by the way, no lessons will be learned from this incident - O2 will NOT start running its own CA as it should.

82
0
Anonymous Coward

Re: Was this

How difficult is it to put the certificate expiry date in the electronic diary with a reminder a fortnight before

7
4
Silver badge

Re: Was this

In what electronic diary? Notifying whom?

Do you know how many certificates large enterprises have to manage now? It would be a full time job for someone - but if you made it that, you'd be screwed when they went on vacation or quit and the reminder from their electronic diary went to /dev/null.

The whole system around certificates is irretrievably broken if you require humans to be in the middle of it. It has to be automated - a subscription service that automatically updates. We will never see the end of such issues so long as humans have to be "reminded", because we are fallible. If the certificate for some weird page hardly anyone visits expires, it might be weeks before the company is notified. If the certificate required for mobile data to work at a large provider expires, it could do a lot of damage in the hours required for the problem to be diagnosed and corrected.

11
0

Re: Was this

Alternatively I have first hand experience of a UK telco that did act as a CA, but then managed to 'lose' the passphrase to their root cert! You couldn't make it up

16
1
Silver badge

Re: Was this

Cheap almost free open source monitoring software can keep an eye on certificates and give you prior warning that the date in one is approaching. You can choose how much warning you want and it will display it on a dashboard in red, ,send you an email or automatically open an ITIL compliant helpdesk ticket for you, with P1 urgency if you want.

Even the most shoddy IT shops I've dealt with have this sorted. It's really simple stuff.

7
2

Re: Was this

Not so simple when you have thousands of certs to look after. However when you have that many certs then all the more reason to have processes in place to manage certs properly

7
1
Silver badge

Re: Was this

Thousands of certs is precisely why they should be electronically tracked.

7
0
Silver badge

Re: Was this

And that still requires a manual process to insure EVERY certificate finds its way into that electronic monitoring system. This is better than a manual process around every renewal since you only need to do it once for a certificate and then you are good for as long as that particular certificate-requiring function remains exactly the same.

Better, but not good enough.

4
0

Re: Was this

werdsmith, your missing a vital point, your assuming O2 (the company) actually give a fook (shareholders will if share price slides longer than 24hours).

Give it a week and nobody will even remember they had an outage, once they can upload fish face pictures to instatwat or pictures of their lunch to twatbook

2
2

Re: Was this

I can see what you're getting at. The certificate system has a different purpose for this situation. It isn't about somebody such as me, downloading software from a myriad of possible suppliers, possibly via intermediaries, where the certificate is about blocking access to possible malware, now with such things as HTTPS. Secure delivery still needs attention, but once a genuine copy of the software is delivered and authorised for use, the supplier's action (or inaction) shouldn't be able to stop it working.

Yeah, I suppose contracts can set up something like software rental, and that's nothing new. But if you shut down your customer I am sure the lawyers would be interested in the procedures you followed.

0
0
Silver badge

Re: Was this

The whole system around certificates is irretrievably broken if you require humans to be in the middle of it. It has to be automated - a subscription service that automatically updates.

Suggest you dust down the risk assessments from the mid-1990's for Single-Sign-On solutions - these worked well whilst everything worked, break something and everything fell into a rather big heap, from which it was easier to reset and start again than trying to recover...

The obvious issue with subscription services is ensuring the bank account(s) from which monies are automatically taken always have sufficient funds (or haven't been closed) and if there is a hiccup in payment processing things get escalated so that action can be taken before certificates expire...

1
0

Move fast, break things

break your neck too.

9
1
Silver badge

What was it that Giffgaff did that they come in for so much stick?

3
3

Appealed to a customer base of the lowest common denominator?

3
12

GiffGaff made a point of not blamesplaining that it wuz O2 wut dun it, they just apologised to their customers as though they were at fault.

Worse than Hitler, really.

34
0
Silver badge

blamesplaining

There are currently 30 google results for that abomination for a word.

If it becomes popular, we're holding you directly responsible. The tar is already being warmed and the chickens are being plucked...

39
0

Why the thumbs down? I'm with giffgaff!

But a look in their forums shows tons of people just screaming at them, who didn't even bother reading the news. Even the Grauniad mentioned it in enough depth to say it wasn't Giffgaff at fault

5
1
Silver badge

You see people in Grauniad comments doing the same.

In fact all over the internet.

2
0
Silver badge

"But a look in their forums shows tons of people just screaming at them, who didn't even bother reading the news."

How were they supposed to read the news when their phone data connection was down? You don't honestly think they would have something old fashioned like a landline based connection or a radio or even a TV, do you? No, of course not. The world had just ended!

4
0

Giff, Gaff, you mean Telefonica aka O2?

Maybe its just me but their adverts really get on my goat, moreso than any other telcos ads (which are bad) every add they spout all i can hear in my head is Liar Liar Bums on fire, your telefonica in disguise you charlatan!

replace Giff Gaff with Tesco, Sky and Lyca......it fits!

0
2
Silver badge

"How were they supposed to read the news when their phone data connection was down?"

How were they able to post in forums if they had no data connection...

I suggest that those able to access forums weren't those truely impacted by this outage, who's smartphone would have been reduced to a games console for Snake and Tetris (aside: showing my age here)

0
0

Painter's 2nd Law of IT

If an IT organisation has to manage something that can expire and must be renewed then it follows that it shall, at some point expire without having been renewed.

48
0
Silver badge
Joke

Re: Painter's 2nd Law of IT - Fixed it for you...

If an IT organisation has to manage something that can expire and must be renewed then it follows that it shall, at some point expire without having been renewed at the worst possible moment.

Certificates always expire at the time when a) the responsible IT bod is on annual leave, or b) there has been a change in management/HR/re-organisation such that no-one is sure who is responsible for the certificates or who can approve paying for their renewal, or c) just after a major IT upgrade, so everyone thinks that the failure is due to the new equipment. Other options are also available...

25
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018