back to article Total Inability To Support User Phones: O2 fries, burning data for 32 million Brits

Customers of O2, GiffGaff and virtual operators who use Telefonica's network in the UK have been hit by a spectacular outage across the country. Transport information services have also been affected. Ericsson, whose central user database caused O2's 2012 25-hour mega-outage, told The Reg: "We are aware of the issue and are …

Silver badge
Facepalm

Don't be nasty, of course they have a Test network.

Some companies have a separate Production network as well.

2
0
Anonymous Coward

Not so simply

It's not a case of not testing, it's a case of you can't test enough. I used to managed core network upgrades for another mobile operator. The vendor would test the software for 100s of hours with 10,000s of test cases, then we would test in our own labs for weeks.

No matter how much testing was performed there was always a bug/service affecting issue found within hours in the real world. The unique combination of users, features, integrations, traffic load etc make it near impossible to find every issue before piloting.

It was always a question of when would we find it, and how career limiting it would be

1
0
Silver badge

Bah!

It's a good job you guys have that better and faster interwebs you were talking about last week, when you downvoted me for mentioning the small matter of provider uptime.

10
0

Nigel from O2 on DownDetector

Cracking...

... and his handle is N-O2

aka Laughing Gas.

27
1
Silver badge

Re: Nigel from O2 on DownDetector - aka Laughing Gas.

Obviously no chemists around.

Laughing gas is N2O. NO2 is extremely toxic. It's what they add to nitric acid to make RFNA, the other half of some rocket fuels.

7
0
Anonymous Coward

Re: Nigel from O2 on DownDetector

Well, it would be funny if NO2 was laughing gas. It isn't. NO2 is nitrogen dioxide an nasty brown gas that dissolves in water forming Nitric Acid, very nasty stuff. Laughing gas is nitrous oxide, N20.

0
1

Whats with all these Spanish owned comapnies in the UK.

Telefonica operate a very reliable service in Spain (albeit with crap customer service).

Sabadell in Spain manage to keep their computer systems up and their customers happy.

AENA run some of the best Airports in the world (Badajas by Madrid is superb example of how to do an airport right) -- but the UK gets Heathrow and Gatwick.

If you gave them Gibraltar back would they leave you alone?

20
1
Silver badge

Wrong solution.

Right solution? INVADE. And show the cheeky buggers who's boss!

"Come cheer up, me lads, 'tis to GLORY we steer, to add something more to this WONDERFUL year".

1
0
Silver badge

"Whats with all these Spanish owned comapnies in the UK."

Revenge for 1588. (Napoleon got the blame for 1805).

0
0

Confidentially as a member of an elete military unit cunninly disguised as Drunken Revellers and/or greey haired pensioners I can say the invasion is well under way from the Island Fortress of Ibiza and the mainland stronghold of Benidorm we have established secure beach heads on most of the Mediteranean coast (inland areas being deemed of no strategic importance).

We are currently instructing the natives on the Joys of British culture -- Pints of Stella, lethally hot curry, shopping at Aldi, etc. -- which they have taken to quite well, although queuing and how to make a left turn at a roundabout needs a lot more work.

0
0
Bronze badge

Re: "Whats with all these Spanish owned comapnies in the UK."

Royalty

0
0

Given that quite a few networks around the world use Ericsson equipment and not every network seems to have fallen over quite this bad, I would guess it's something to do with network size / design that aggravate the issue for some, if that's the case it's probably not that easy to catch in a test environment. :(

As I'm Swedish I'm contractually obliged to try and protect Ericsson...

10
0
Gold badge

Whereas we Brits (*) are contractually obliged to slag off the home team.

(Whingeing Poms, I believe is the preferred expression.)

1
0
Anonymous Coward

My work phone is on O2!!

OMFG the world will collapse!

do I connect it to some wifi? or do I just ignore it... well for once I have a totally legitimate reason for not giving a stuff about email! long may the outage continue!!!

4
0

Re: My work phone is on O2!!

Depends if your local has WiFi. Seems like a legitimate reason so setup camp.

0
0
Anonymous Coward

First Youtube was closing down, now the whole Internet has!

I can't imagine what all those snowflake teenagers threatening suicide because they think youtube is closing down must be thinking right now . . . . . RIP the Internet.

4
5

Re: First Youtube was closing down, now the whole Internet has!

What a surprise, someone having a go at youth, using the alt right term Snowflake, doesn't have the sheer balls to post non-anon!

9
9

It strikes me that if all the mobile operators in the country cooperated and allowed customers to roam each others networks this kind of problem - as well as the sh1te signal issues that plague the country could be pretty much non existent.

16
4

Cascade failure. What do you think the impact of suddenly having 33M devices roam onto your network might be? How much will it cost to size your network to be ready to take millions of extra users at a moment’s notice? Who would ever build our rural cells when you could just let your users roam onto the network of the mug who has?

8
0

Rural cells are being built?? Citation needed!

0
0
Silver badge

Rural cells are being built?? Citation needed!

There's one on the bridge at Bradford on Avon, and there's one in Trowbridge.

-oh you meant mobile telephone cells.

0
0
Anonymous Coward

tales from next month

Will be interesting if dumb comment and downright lies drops over the period of the outage. Then we know which 32 million to blame.

Next feeble economic figures will be blamed by the tory twats on this no doubt like when its either too sunny or too rainy.

1
7
Anonymous Coward

Huawei arrest

Coincidence or not?

7
0
WTF?

Not the US

What? A Chinese citizen arrested in Canada on a US warrant? I think Brexit is a more likely culprit.

3
0

This is awful

I have to talk to people face-to-face! The horror!

5
2

Not really a crash

It's not really a crash. With the tumblr porn filters on the way, it turns out that 4G just isn't needed any more.

9
0

So their Ericsson appliances/applications have somehow failed.

Perhaps they should have avoided this single point of failure for such a critical piece of comms infrastructure (using redundant bits of kit from the same supplier doesn't work if all instances of it are affected by the same issue).

If other operators around the world are affected then it sounds like a botched update, or a very well coordinated cyberattack. Or perhaps they're all using some cloudy service and THAT has gone down?

3
0
Anonymous Coward

I just heard O2 have hired Paul Pester to sort this out

8
0
Facepalm

Oh, the fallability of the BBC!

In their report about the failure, https://www.bbc.co.uk/news/business-46464730, they quote a lady in Norwich

"I'm disabled ... I'm in a wheelchair," she told the BBC. "So having no data but also no calls as well means I can't contact anyone if I have a fall or if I need anything."

It makes you wonder if they used telepathy to contact her.

21
4
WTF?

Re: Oh, the fallability of the BBC!

Your point appears to be that if someone comes and takes out your toilet, you'd be happy shitting out of your windows because, well, we all did in the good old days didn't we?

They might have used this thing called "email" she used while at her home (sat at a device known as a Personal Computer or "PC"...or even a device called a "tablet" connected to a magical device called a "WiFi router").

You can't just scoff and pretend that people don't rely on technology for certain things or suggest that because you're happy without quick access to porn that everyone else is just dandy too.

2
10

Re: Oh, the fallability of the BBC!

Where I work there's no signal at the best of times but data, somehow, usually anyway works. So my relatives are on whatsapp.

You're doing the annoying old thing of assuming your situation is the only one that matters and all others are either the same or you simply don't care

4
0
Silver badge

Re: Oh, the fallability of the BBC!

They might have used this thing called "email" she used while at her home (sat at a device known as a Personal Computer or "PC"...or even a device called a "tablet" connected to a magical device called a "WiFi router").

Do you not see the problem with this reply?

5
0

Re: Private Eye always has

Do you not see the problem with this reply?"...

...not unless the form factor of a laptop or desktop got miniaturised recently...

Internet != Telephony...maybe she doesn't like WhatsApp...

0
0
Thumb Up

Lyca Customers

Time your networked moved off O2 and merged with Virgin Mobile. Then we can finally have a network called LycaVirgin.

23
0
Silver badge

Re: Lyca Customers

The Catholic Church will object, they own that trademark.

3
0
Coat

Re: Lyca Customers

In touch for the very first time?

13
0
Silver badge
Pint

Can't chat on Whatapp and Facebook?

Try the off-line alternative, and go to the pub.

5
2
Silver badge

Re: Can't chat on Whatapp and Facebook?

But there are people there!

10
1

In unrelated news....

https://www.theregister.co.uk/2018/12/06/uk_gchq_bulk_equipment_interference/

3
2

Still out for me at 4pm, Oh well, better get some popcorn popping

2
0

I have no network or data in Nottingham. Emergency calls only... impressive levels of ineptitude...

1
1
Silver badge
Facepalm

Or alternatively...O2 have maintained essential Emergency services even though their nice-to-have services have been down?

2
0

Other mobile operators around the world are also affected?

...according to the O2 status page. Why would that be? Has everyone upgraded/changed/implemented whatever has broken the system at the same time...or is everything being routed through a single point?

Seems a bit odd to me.

1
2

Re: Other mobile operators around the world are also affected?

also not heard anything about any other mobile operator in the world having similar issues. maybe searching would not reveal, but seems odd.

0
0

Re: Other mobile operators around the world are also affected?

SoftBank did. Presumably they were running one of the old software versions too.

0
0

Re: Other mobile operators around the world are also affected?

Would appear to mainly Japanese network Softbank are the 'others'

0
0
Anonymous Coward

2018/12/06 T 16:20 - "Emergency calls only" here in Cambridge

So no nothing.

1
1
Anonymous Coward

Expired certificate!

Apparently the root cause was an expired certificate....

Press release: https://mb.cision.com/Main/15448/2694397/959040.pdf

6
0

This is terrible. I had to get cash out at lunch because the Korean fried chicken stall I frequent in Farringdon was unable to take cards owing to this outage. Oh the indignity of it all.

9
0
Silver badge
WTF?

Ericsson - they may well have found the problem

Ericsson customers affected were using two specific software versions of the node “Serving GPRS Support Node - Mobility Management Entity” (SGSN-MME).

“An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers,” it said.

According to Reuters....

2
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018