back to article Total Inability To Support User Phones: O2 fries, burning data for 32 million Brits

Customers of O2, GiffGaff and virtual operators who use Telefonica's network in the UK have been hit by a spectacular outage across the country. Transport information services have also been affected. Ericsson, whose central user database caused O2's 2012 25-hour mega-outage, told The Reg: "We are aware of the issue and are …

  1. anthonyhegedus Silver badge

    OK so now their status reads:

    "The network issue continues to be a top priority for us and we apologise to all our customers impacted by this.

    Earlier today we suspended any planned works or updates across our systems and network. This allowed hundreds of people from both our technical teams and our third-party supplier’s teams to focus on fixing the problem, helping us to restore data services as quickly as we can which will also help reduce the high network demand we’re currently seeing for voice calls."

    Hundreds of technical people? Doesn't smack of a software problem, or at least not a simple one. It sounds like a problem that requires a lot of manpower, like manually adjusting settings at thousands of different sites.

    Today lots of things are going down for me and people I know, and we've actually had customers phone up and ask if it was due to the O2 problem!

  2. koswix

    Sounds more like PR spin to me. Can't do any planned maintenance as the network is down, so suspend those jobs and you indeed now have 'hundreds' of technical staff available to focus on the outage... Even if they're doing nothing, they're *focused* on doing nothing.

  3. The Specialist

    999

    Not sure anyone mentioned before but dialling 999 / 112 from any mob regardless of network / with or without sim card is possible. It will latch on whatever network is available.

  4. ARGO

    Re: 999

    Only part of that works in the UK. Although you can dial 999 without a SIM card, UK networks will not connect the call. This is by order of OFCOM as there were large numbers of untraceable hoax calls when it was allowed. (with a SIM card - even an expired one - it all works as standards intended)

  5. MrFacePalm

    Another result for the beancounters

    If we want to start finger pointing, first place to look is O2 probably deciding to push the envelope by continuing to use end of support SGSN software (=speculation on my part), and based on advice from an undermanned Ericsson local support team based in the UK. The second place to point the finger is to the bean counters in Stockholm who have been offshoring work to China and India for years as they plot their voyage to the bottom. (=hard fact)

  6. Laughing Gravy

    6pm and still out, having some beer with the popcorn now

  7. cantankerous swineherd Silver badge

    giff gaff a lousy advert for a network run by me. luckily gg tell me some of my colleagues are engineers and they're all over it.

    Ericsson not a good advert for Erlang it seems.

  8. Trev 2

    OMG never using O2...oh hang on!

    Chav version:

    "OMG never using O2 again...oh I just signed a 600 month contract for my fancy overpriced iPhone X so I'll never use O2 again, after the year 2068!" ;)

  9. Jeroen Braamhaar

    Related ?

    In Japan, SoftBank's mobile network also went down for 4 hours .... allegedly due to Ericcson sfotware going titsup as well ...

    (Caught this on NHK World news)

  10. Anonymous Coward
    Anonymous Coward

    Maybe O2 should be using Huawei. But hang on......

  11. tomthemac

    Ericsson blame 'expired certificate' in software version

    wow.

    expired certificate.

    yep.

    https://www.ericsson.com/en/press-releases/2018/12/update-on-software-issue-impacting-certain-customers

  12. steelpillow Silver badge

    Re: Ericsson blame 'expired certificate' in software version

    "the main issue was an expired certificate in the software versions installed"

    Can it be that carrier-grade installs don't get updated by operators as promptly as the run-of-the-mill swearware does?

    Whoever woulda thunk...

  13. Anonymous Coward
    Anonymous Coward

    Pretty embarrassing @MarkEvansO2...

    In your attempt to effectively name & shame by mentioning the third-party software company responsible for your national data outage, you spelt it 'Erricson'... whoops

  14. Anonymous Coward
    Anonymous Coward

    MME issue

    It's an issue with their MME implementation, provided by Ericsson. Source: been on calls about it all day because of the impact it's caused us.

    Happy googling, armchair service continuity managers.

  15. Anonymous Coward
    Anonymous Coward

    It's not just transport systems that it's effected. It has also hit emergency services. The trust I work for has lost connection of all the ipads that are used for patient report forms. This is extremely worrying seeing that every emergency service will be using a 4G network for the entirety of their critical communications. This outage would've put lifes at risk if ESN was live!

  16. Inventor of the Marmite Laser Silver badge

    Your ability to spell

    Seems to be affected

  17. Anonymous Coward
    Anonymous Coward

    "This outage would've put lifes at risk if ESN was live!" - Nope

    ESN is EE not O2.

  18. Ian Emery Silver badge

    Latest theory

    They were getting ready for Brexit by installed radio equipment that uses imperial wave-lengths instead of the current metric wave-lengths.

  19. Voyna i Mor Silver badge

    Re: Latest theory

    Sounds legit.

    Are we going to buy that platinum-iridium kilogramme from the French and rename it the Imperial 2.204 Pound? Then if we can just find twelve yeomen coming out of church on a Sunday we can get our standard foot back.

    Just in time for a number of large corporations to stick it right in their mouths.

  20. michael mackey

    Should have bought Huawei !

  21. Marketing Hack Silver badge
    WTF?

    I don't mean to be undiplomatic...

    But why is Britain the country that is always having these outages. You can't use your cellphone. You can't buy train tickets. Your online banking is down. Etc. Ad nauseum....

  22. DoctorMO

    Re: I don't mean to be undiplomatic...

    Is it because GCHQ has hired all the boffins away from industry? Is is because open source code that's made in the USA is first used in France and Germany and visa versa and takes years to land in blighty? Is it because IT managers in the UK are trained by Black Adder impersonators? Is it because we have abut as much respect for competent engineers as we do the guy that empties the bins? Is it perhaps because computers don't run on steam and we've been baffled by electrons ever since the end of the nineteenth century?

    All I know is, I escaped the UK and found more respect, more money and more freedom to be a good engineer in the USA. It doesn't sound like things have gotten any better since I left.

  23. Laughing Gravy

    8:30 pm and still out, on the vodka now

  24. Tim Kemp

    Using a multi-net SIM here. Feeling smug today.

    I wonder how many people finding voice calls affected are trying to use VoLTE?

  25. dank_army

    Sorry chaps I don't buy it - a software update causing global issues? Smells like a hack to me.

  26. Anonymous Coward
    Anonymous Coward

    Your confidence in O2 is entirely misplaced...

  27. MrMerrymaker

    It's back for me but not for friends

    Ah well. Hope I get a free day or something though can't complain for one day's outage when I use giffgaff for £7 every month without issue

  28. Weltschmerz

    This is cyber warfare isn’t it?

    China. Huawei. Etc.

  29. diodesign (Written by Reg staff) Silver badge

    Re: This is cyber warfare isn’t it?

    Expired software certificate, apparently...

    https://www.theregister.co.uk/2018/12/06/ericsson_o2_telefonica_uk_outage/

    C.

  30. SteveTM

    Lots of people outraged about people being outraged about people being outraged.

    Nothing new here. Apparently the last bastion of Britishness, being able to complain, is being removed in exchange for lethargic illogical tolerance.

    Imagine if the company in question has 6.5 BILLION in revenue and the supplier to that company had over 200 Billion in revenue....yet the fact that software in SGSN's worldwide was due to expire and shut everything down wasn't detected or dealt with prior to that expiry date. Stupid idea coding expiry into an SGSN anyway.

    Also, for all those complaining about people complaining, its not just people struggling to get onto Snapchat and Facebook, a lot of key services use the O2 network, because of its claimed resilience. The fact that there wasn't some degree of fault tolerance built into the O2 network is poor show considering their operating budget.

    Complaining may annoy people, but its those people complaining which have pushed providers to improve their resilience, so you should be thanking them everyday for reduced power cuts, improved mobile coverage and people being held accountable.

    The moment we go quiet and stop complaining is when we should start complaining.

  31. Anonymous Coward
    Anonymous Coward

    Well, at least the cert expiry logic worked flawlessly ....

    .... and the individual who kept the reminders 'in a spreadsheet somewhere' left O2 and/or was offloaded to an outsource supplier years ago.

  32. Anonymous Coward
    Anonymous Coward

    Highlights some pretty poor support at GiffGaff though.

    I had raised a support ticket for a completely unrelated problem with GiffGaffs "agents" the night before the system went TITSUP. I was surprised to get an update message from them during the outage and duly went to see what the resolution was. The message was telling me that they had a major problem. It looks like they decided to send the same message to every case raised regardless of whether it was related or not.

    The community support pages seem to be populated entirely by copy and paste bunnies trotting out the part line or commentards who try very hard to look like they have a clue but succeed in proving they don't.

    Sadly the only network signal I can get in the office is O2 or Vodafone and I left Vodafone some time ago due to their costs and crap customer service.

  33. jms222 Bronze badge

    giffgaff

    > community support pages seem to be populated entirely by copy and paste bunnies

    I agree entirely.

    By the way have you tried fiddling with your APN like you don't have to do on any other network ?

    Or restarting your phone like you don't have to do on any other network ?

  34. Keith Tayler

    Keith Tayler

    A Swedish company is responsible. Let us hope it makes the Swedish government think again about making a cashless economy within two years.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018