back to article Total Inability To Support User Phones: O2 fries, burning data for 32 million Brits

Customers of O2, GiffGaff and virtual operators who use Telefonica's network in the UK have been hit by a spectacular outage across the country. Transport information services have also been affected. Ericsson, whose central user database caused O2's 2012 25-hour mega-outage, told The Reg: "We are aware of the issue and are …

  1. tfewster Silver badge
    Facepalm

    Don't be nasty, of course they have a Test network.

    Some companies have a separate Production network as well.

  2. Anonymous Coward
    Anonymous Coward

    Not so simply

    It's not a case of not testing, it's a case of you can't test enough. I used to managed core network upgrades for another mobile operator. The vendor would test the software for 100s of hours with 10,000s of test cases, then we would test in our own labs for weeks.

    No matter how much testing was performed there was always a bug/service affecting issue found within hours in the real world. The unique combination of users, features, integrations, traffic load etc make it near impossible to find every issue before piloting.

    It was always a question of when would we find it, and how career limiting it would be

  3. Stevie Silver badge

    Bah!

    It's a good job you guys have that better and faster interwebs you were talking about last week, when you downvoted me for mentioning the small matter of provider uptime.

  4. a pressbutton

    Nigel from O2 on DownDetector

    Cracking...

    ... and his handle is N-O2

    aka Laughing Gas.

  5. Voyna i Mor Silver badge

    Re: Nigel from O2 on DownDetector - aka Laughing Gas.

    Obviously no chemists around.

    Laughing gas is N2O. NO2 is extremely toxic. It's what they add to nitric acid to make RFNA, the other half of some rocket fuels.

  6. Anonymous Coward
    Anonymous Coward

    Re: Nigel from O2 on DownDetector

    Well, it would be funny if NO2 was laughing gas. It isn't. NO2 is nitrogen dioxide an nasty brown gas that dissolves in water forming Nitric Acid, very nasty stuff. Laughing gas is nitrous oxide, N20.

  7. James Anderson

    Whats with all these Spanish owned comapnies in the UK.

    Telefonica operate a very reliable service in Spain (albeit with crap customer service).

    Sabadell in Spain manage to keep their computer systems up and their customers happy.

    AENA run some of the best Airports in the world (Badajas by Madrid is superb example of how to do an airport right) -- but the UK gets Heathrow and Gatwick.

    If you gave them Gibraltar back would they leave you alone?

  8. WolfFan Silver badge

    Wrong solution.

    Right solution? INVADE. And show the cheeky buggers who's boss!

    "Come cheer up, me lads, 'tis to GLORY we steer, to add something more to this WONDERFUL year".

  9. Voyna i Mor Silver badge

    "Whats with all these Spanish owned comapnies in the UK."

    Revenge for 1588. (Napoleon got the blame for 1805).

  10. James Anderson

    Confidentially as a member of an elete military unit cunninly disguised as Drunken Revellers and/or greey haired pensioners I can say the invasion is well under way from the Island Fortress of Ibiza and the mainland stronghold of Benidorm we have established secure beach heads on most of the Mediteranean coast (inland areas being deemed of no strategic importance).

    We are currently instructing the natives on the Joys of British culture -- Pints of Stella, lethally hot curry, shopping at Aldi, etc. -- which they have taken to quite well, although queuing and how to make a left turn at a roundabout needs a lot more work.

  11. FlamingDeath Bronze badge

    Re: "Whats with all these Spanish owned comapnies in the UK."

    Royalty

  12. Terje

    Given that quite a few networks around the world use Ericsson equipment and not every network seems to have fallen over quite this bad, I would guess it's something to do with network size / design that aggravate the issue for some, if that's the case it's probably not that easy to catch in a test environment. :(

    As I'm Swedish I'm contractually obliged to try and protect Ericsson...

  13. Ken Hagan Gold badge

    Whereas we Brits (*) are contractually obliged to slag off the home team.

    (Whingeing Poms, I believe is the preferred expression.)

  14. Anonymous Coward
    Anonymous Coward

    My work phone is on O2!!

    OMFG the world will collapse!

    do I connect it to some wifi? or do I just ignore it... well for once I have a totally legitimate reason for not giving a stuff about email! long may the outage continue!!!

  15. JoshOvki

    Re: My work phone is on O2!!

    Depends if your local has WiFi. Seems like a legitimate reason so setup camp.

  16. Anonymous Coward
    Anonymous Coward

    First Youtube was closing down, now the whole Internet has!

    I can't imagine what all those snowflake teenagers threatening suicide because they think youtube is closing down must be thinking right now . . . . . RIP the Internet.

  17. MrMerrymaker

    Re: First Youtube was closing down, now the whole Internet has!

    What a surprise, someone having a go at youth, using the alt right term Snowflake, doesn't have the sheer balls to post non-anon!

  18. Rabbit80

    It strikes me that if all the mobile operators in the country cooperated and allowed customers to roam each others networks this kind of problem - as well as the sh1te signal issues that plague the country could be pretty much non existent.

  19. Terry Barnes

    Cascade failure. What do you think the impact of suddenly having 33M devices roam onto your network might be? How much will it cost to size your network to be ready to take millions of extra users at a moment’s notice? Who would ever build our rural cells when you could just let your users roam onto the network of the mug who has?

  20. Intractable Potsherd

    Rural cells are being built?? Citation needed!

  21. Voyna i Mor Silver badge

    Rural cells are being built?? Citation needed!

    There's one on the bridge at Bradford on Avon, and there's one in Trowbridge.

    -oh you meant mobile telephone cells.

  22. Anonymous Coward
    Anonymous Coward

    tales from next month

    Will be interesting if dumb comment and downright lies drops over the period of the outage. Then we know which 32 million to blame.

    Next feeble economic figures will be blamed by the tory twats on this no doubt like when its either too sunny or too rainy.

  23. Anonymous Coward
    Anonymous Coward

    Huawei arrest

    Coincidence or not?

  24. Derezed
    WTF?

    Not the US

    What? A Chinese citizen arrested in Canada on a US warrant? I think Brexit is a more likely culprit.

  25. Joseph Haig

    This is awful

    I have to talk to people face-to-face! The horror!

  26. Wandering Reader

    Not really a crash

    It's not really a crash. With the tumblr porn filters on the way, it turns out that 4G just isn't needed any more.

  27. Jeff 11

    So their Ericsson appliances/applications have somehow failed.

    Perhaps they should have avoided this single point of failure for such a critical piece of comms infrastructure (using redundant bits of kit from the same supplier doesn't work if all instances of it are affected by the same issue).

    If other operators around the world are affected then it sounds like a botched update, or a very well coordinated cyberattack. Or perhaps they're all using some cloudy service and THAT has gone down?

  28. Anonymous Coward
    Anonymous Coward

    I just heard O2 have hired Paul Pester to sort this out

  29. Wingtech
    Facepalm

    Oh, the fallability of the BBC!

    In their report about the failure, https://www.bbc.co.uk/news/business-46464730, they quote a lady in Norwich

    "I'm disabled ... I'm in a wheelchair," she told the BBC. "So having no data but also no calls as well means I can't contact anyone if I have a fall or if I need anything."

    It makes you wonder if they used telepathy to contact her.

  30. Derezed
    WTF?

    Re: Oh, the fallability of the BBC!

    Your point appears to be that if someone comes and takes out your toilet, you'd be happy shitting out of your windows because, well, we all did in the good old days didn't we?

    They might have used this thing called "email" she used while at her home (sat at a device known as a Personal Computer or "PC"...or even a device called a "tablet" connected to a magical device called a "WiFi router").

    You can't just scoff and pretend that people don't rely on technology for certain things or suggest that because you're happy without quick access to porn that everyone else is just dandy too.

  31. MrMerrymaker

    Re: Oh, the fallability of the BBC!

    Where I work there's no signal at the best of times but data, somehow, usually anyway works. So my relatives are on whatsapp.

    You're doing the annoying old thing of assuming your situation is the only one that matters and all others are either the same or you simply don't care

  32. werdsmith Silver badge

    Re: Oh, the fallability of the BBC!

    They might have used this thing called "email" she used while at her home (sat at a device known as a Personal Computer or "PC"...or even a device called a "tablet" connected to a magical device called a "WiFi router").

    Do you not see the problem with this reply?

  33. Derezed

    Re: Private Eye always has

    Do you not see the problem with this reply?"...

    ...not unless the form factor of a laptop or desktop got miniaturised recently...

    Internet != Telephony...maybe she doesn't like WhatsApp...

  34. Doogie Howser MD
    Thumb Up

    Lyca Customers

    Time your networked moved off O2 and merged with Virgin Mobile. Then we can finally have a network called LycaVirgin.

  35. WolfFan Silver badge

    Re: Lyca Customers

    The Catholic Church will object, they own that trademark.

  36. Richard Gray 1
    Coat

    Re: Lyca Customers

    In touch for the very first time?

  37. Simon Harris Silver badge
    Pint

    Can't chat on Whatapp and Facebook?

    Try the off-line alternative, and go to the pub.

  38. Aladdin Sane Silver badge

    Re: Can't chat on Whatapp and Facebook?

    But there are people there!

  39. Captain Hogwash

    In unrelated news....

    https://www.theregister.co.uk/2018/12/06/uk_gchq_bulk_equipment_interference/

  40. Laughing Gravy

    Still out for me at 4pm, Oh well, better get some popcorn popping

  41. Rod 6

    I have no network or data in Nottingham. Emergency calls only... impressive levels of ineptitude...

  42. tfewster Silver badge
    Facepalm

    Or alternatively...O2 have maintained essential Emergency services even though their nice-to-have services have been down?

  43. Daveycoder

    Other mobile operators around the world are also affected?

    ...according to the O2 status page. Why would that be? Has everyone upgraded/changed/implemented whatever has broken the system at the same time...or is everything being routed through a single point?

    Seems a bit odd to me.

  44. tin 2

    Re: Other mobile operators around the world are also affected?

    also not heard anything about any other mobile operator in the world having similar issues. maybe searching would not reveal, but seems odd.

  45. James R Grinter

    Re: Other mobile operators around the world are also affected?

    SoftBank did. Presumably they were running one of the old software versions too.

  46. tim 13

    Re: Other mobile operators around the world are also affected?

    Would appear to mainly Japanese network Softbank are the 'others'

  47. Anonymous Coward
    Anonymous Coward

    2018/12/06 T 16:20 - "Emergency calls only" here in Cambridge

    So no nothing.

  48. Anonymous Coward
    Anonymous Coward

    Expired certificate!

    Apparently the root cause was an expired certificate....

    Press release: https://mb.cision.com/Main/15448/2694397/959040.pdf

  49. Ed 11

    This is terrible. I had to get cash out at lunch because the Korean fried chicken stall I frequent in Farringdon was unable to take cards owing to this outage. Oh the indignity of it all.

  50. Flywheel Silver badge
    WTF?

    Ericsson - they may well have found the problem

    Ericsson customers affected were using two specific software versions of the node “Serving GPRS Support Node - Mobility Management Entity” (SGSN-MME).

    “An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers,” it said.

    According to Reuters....

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018