Feeds

back to article O2's titsup network struggles to find its feet

O2's mobile network is struggling to its feet after a 19-hour outage left thousands of customers unable to make or receive calls. This morning 2G should be working again but 3G remains patchy. That's according to the sluggish status page, which, between apologies, claims the 3G service is slowly being revived. The telco admits …

COMMENTS

This topic is closed for new posts.

Page:

FAIL

Doh.

"Once the network is up we'll be asking O2 what went wrong and (more importantly) what's being done to ensure it doesn't happen again."

Presumably you only have the mobes of people at O2 - presumably they're not really going to be on another network lol...

2
1
Anonymous Coward

Re: Doh.

Good point! you'd think if they had any sense they'd dish out phones to engineers on a competitors network for reasons such as this.

I'm ambivalent as to whether this is a practise that has been blocked by some fuckwit because they're using a competitor though.

0
0
Anonymous Coward

Re: Doh.

You think people are fielding calls from the Register about this, rather than working to fix it? Really?

2
1
Thumb Up

Re: Doh.

@AC...

Iron-y ... what you get if you rub yourself with iron !

3
1
Bronze badge

Re: Doh.

Fairly standard practice in telecoms to have emergency gear provided by someone else for just such a scenario. Mobile techs carry phones on other networks, telephone exchanges have lines provided that are long-lined back to a different exchange served from a different major node.

There's nothing worse than realising on a 3am callout that your first line tools and techniques aren't enough to resolve an issue and then not being able to call for help.

2
0
Facepalm

What went wrong

They spent all the money on senior manager bonuses and ignored the single points of failure in the network.

That should save O2 making something up in a reply to you.

6
0
FAIL

Apologies for the delay in resuming service...

...in the mean time, please hammer our HLR with registration attempts.

1
0
FAIL

I think this may be PR speak..

..because I'm on O2 and was fine up until recently, and now mine's just died too without explanation!

1
0
Silver badge
Meh

Re: I think this may be PR speak..

Mine was fine until this morning. I thought the SIM might be bad, reseated it, rebooted and it's fine now. Now I don't know whether I was right about the SIM or whether the network was bolloxed.

0
0

This post has been deleted by its author

Anonymous Coward

All O2...those pesky chinese.

http://www.computerweekly.com/news/2240150185/Huawei-wins-contract-for-O2-network

3
0
Anonymous Coward

Re: All O2...those pesky chinese.

Outsource followed by outage. I wonder where we have seen that before? (Natwest).

I hope they cancel the bonus payment to the fool who thought this was a good idea. Losing customers is never a good way of growing the business and increasing profits.

I'm sure Huawei will now get screwed with a nice big SLA penalty.

1
0
Silver badge
Facepalm

Re: All O2...those pesky chinese.

Post hoc ergo propter hoc.

Given the sourcing deal was signed 2 months ago, I'd be surprised if they'd even started transition (and given the majority of staff involved are going across to Huawei under TUPE it would largely make no difference to day-to-day operating). Wonder what you'd like to blame any historical outages O2 have ever had on then..

But hey, it's much easier to read "O2 problem" and google "O2 outsourcing" to see what pops up, hey? Every company in the world outsources something to some extent.

2
2
Happy

Re: All O2...those pesky chinese.

Careful - seems to be a downvote-mad pro-outsourcing and/or pro O2 contingency at work on teh threads here...

1
0
Anonymous Coward

Re: All O2...those pesky chinese.

My comments are not my own...They have been outsourced.

0
0

Throughout this issue my handset has reported full bar signal and successful connection to the o2 network from my local mast. However, inbound calls to my number were failing to unobtainable - not even to my voicemail on the o2 network - this was clearly an internal, core network issue, not simply a problem with my handset accessing the network.

1
4
Anonymous Coward

Hmmm

A core network issue you say? Good work Sherlock.

Coincidentally, that's what O2 statement said yesterday, and the BBC / El Reg reported.

4
0

Got hit by this when I tried a text this morning, did the turn it off and on again (I've worked IT for a LONG time) and problem solved.

Be interesting to see what the root cause was, but I'm guessing that with the complexity of current networks it's not going to be something simple.

0
0
Anonymous Coward

Yes

I'm guessing it was something simple.

What's anyone else guessing?

1
0
APA

Who's left?

Didn't Orange have a 3G issue earlier this year? (yep http://www.theregister.co.uk/2012/03/08/orange_down/) At least their excuse is the merging of two formerly separate networks to create Everything Everywhere.

Are outages like this expected? Do other countries have similar issues with their carriers? Not that I'm suggesting a "Broken Britain" thing (though it might be - a product of a culture that subcontracts everything to defer responsibility and not caring about quality, then wonders why things blow up later), just trying to expand the field of experience to determine whether this is "normal" and expected behaviour for the technology (or "new" technology in general)..

If T-Mobile/Orange and O2 have had their turn, who's next?

0
0
Bronze badge
Stop

Re: Who's left?

Except Orange and T-Mobile AREN'T merging their networks. They are still separate (they merely changed the setup so that users of one network could user the other and vice versa) - however any 4G network that they develop will be integrated.

0
0
Meh

Re: Who's left?

Mobile networks are surprisingly fragile. There are many single points of failure, both from the actual nodes through to cabling, failed back ups over night and a host of other things.

Given the issue seems to be a re-authentication issue it could be a few things here.

Mobiles do authenticate for two reasons, one being time, say, every 30 mins, another is when the devices actually moved around, from cell to cell, or BSC/MSC to BSC/MSC.

HLRs are often located regionally, for capaicuty and redundancy, and each is connected to an AUC, Authentication node. If these have problems, or the links between, there could be issues.

On 3G, there is the RNC, which performs similar jobs for the 3G but that the HLR/AUC does for 2G.

But none of these nodes are really redundant. A network may have, say, five HLRs, around the country, that can be load balanced between the five, but HLRs are not, usually, built in redundant pairs, which means the links are not either.

Thankfully, most Blighty and euro networks have very few issues, because the O/M procedures do work. But, given the volume of date being carried by the network, to make and place calls, CDR's etc, as well as actual usage traffic, when things do go down, it does take some hours to bring back on line.

Lets just hope that o2 and Huawei do not outsource and off-shore things like O/M etc and that this is a one off.

Time will tell.

0
0

This post has been deleted by its author

Go

Re: Who's left?

@ Piloti...

Would you be so kind as to expand the TLAs there - so I can look them up on the interwebs ?

1
1
Go

Re: Who's left?

Meh...

This seems to have it all covered:

http://en.wikipedia.org/wiki/Network_switching_subsystem

1
1

This post has been deleted by its author

This post has been deleted by its author

Re: Who's left?

Sure....

HLR: Home Location Register.

AUC: Authentication Centre

BSC: Base Station Controller

MSC : Mobile Switch Centre / Controller

RNC : Radio Network Controller.

O/M : Operations and Maintenance

CDR : Call Data Records

2
0
Bronze badge

Re: Who's left?

Generally, providers are responding to market requirements, and what the market is saying is "cheap as chips". A provider who spends a lot on resilience has higher prices and finds that they don't then have any customers.

It's not really about providers absolving responsibility or outsourcing, it's about the decisions they have to make to hit a price point that customers will pay for.

Britain has amongst the cheapest telecoms in Europe - and those low prices are driven by the market. There is a flipside to those low prices however, and that's constrained investment and engineering to meet a cost.

I buy resilient services to meet the SLAs my customers need - but the cost of that can be eye-watering. Thousands of pounds a month for resilient 100M Ethernet between two fairly close sites, for example. If you're used to paying £15 a month for consumer broadband the prices you get for proper, carrier-grade resilience look like decimal point errors. Anyway - the point of my post is this - if you want it to always work without question, you need to pay a lot of money. If you want it to be cheap, you'll get something that works most of the time.

5
1
Anonymous Coward

"a host of other things"

Was, I thought, the nicest technical term used.

0
0
Pint

Please God...

...let this come down to outsourcing...

7
2
WTF?

Going Postal

Where is Moist Von Lipwig when you need him?

Sent via the Clacks Network

5
0

Anything to do with sharing Vodafone's base stations?

and the change in internal systems that would require

0
0
Silver badge

It wouldn't surprise me

...if this weren't something to do with trying to re-prioritise their network for the Olympics. I would expect that they will try to make the London area more robust, given the expected extra traffic there over the next few months. They are the 'official' communications provider, after all.

1
0
Anonymous Coward

From an O2 service bulletin sent out....

"Ericsson have performed restarts of the LDAP processes on the CUDB in an attempt to recover service.

The Technical bridge remains open monitoring the service and assessing the impact the restart are having on customers' ability to connect.

We're also looking at other options available to us should the restarts fail to recover service."

Looks like the turn it off and back on again applies to their infrastructure as well as our phones!

3
0
Anonymous Coward

Re: From an O2 service bulletin sent out....

When was that sent? surely they havent taken 19 hours to restart their LDAP servers?

0
0
Anonymous Coward

Re: From an O2 service bulletin sent out....

That was at 5pm yesterday. I haven't seen any further detailed updates yet.

0
0
Anonymous Coward

Falback to 2G, eh?

I frequently find O2's data service falls back to sub-dialup

0
0
Joke

New O2 Tariff

I'm on the new O2 plan... Unlimited Smoke Signals 250 Pigeons a month Free messages in a bottle to other O2 users

7
0
Silver badge
Facepalm

Re: New O2 Tariff

Rats - I was just too late...

0
0
Terminator

You are all wrong.

It is Skynet.

First the banking system (RBS and Natwest), now the communication networks. Won't be long before it has us fully understood and targets all the networks.

We're doomed I tell you.

2
0
Anonymous Coward

Yes it happens elsewhere too...

Orange France fell over last Friday from about 3pm - until around 0300 on Saturday morning - so 27 milion or so subs unable to do anything - my phone kept "searching".

Apparently Ericsson and A-LU engineers were helping Orange with their inquiries and it had nothing to do with a software update about 24 hour earlier on the HLRs.

By mid evening 2 French cabinet ministers were visiting the Orange crisis control centre - which must have improved the situation no end...

Orange have offered everyone a free day of usage in September for calls, SMS and data depending on their current contract - so guess when the next big outage will happen.

It will be interesting to see what O2 do...

http://www.lefigaro.fr/flash-eco/2012/07/11/97002-20120711FILWWW00528-panne-orange-un-logiciel-en-cause.php for those who read French or can use Google Translate

Simon

1
0
FAIL

Re: Yes it happens elsewhere too...

France Telecom said Wednesday that the outage that affected its services the weekend was related to a software problem occurred on one of its equipment had suffered an update 48 hours earlier.

Heard by the Economic Affairs Committee of the National Assembly, the first CEO of French telecoms operator Stéphane Richard has elaborated on the scenario of giant failure that affected its 26 million subscribers and the operators' customers who rent its network. The incident affected a strategic asset for the operator, a set of servers whose mission is to locate and authenticate all subscribers permanently, hence the magnitude of the rare failure occurred early Friday of afternoon, the day of baccalaureate results.

Aggravating circumstance, no alert was triggered as a result of this anomaly, preventing the initiation of emergency equipment. Stéphane Richard said the expert work on the causes of the incident was not yet complete but it has already ruled out a virus attack or a traffic overload.

0
0
Alert

Is this the start of a trend?

Apolgies in advance for this rant.

I think nsld's cynical comment maybe very near the mark. If you are an affected O2 customer and were also hit by the RBS network going tits-up a couple of weeks ago you could be forgiven for worrying about other rather important tech reliant services.

O2 have been boasting that they have 200 engineers assigned to network quality. I hope that doesn't equate to 200 "engineers" doing everything needed to keep things going, from physical station maintenance to network systems admin and the rest for all those shiny 23m+ mobile devices.

The companies we rely on to deliver services essential for our day-to-day survival are in practice totally big-tech reliant these days. If the big-tech screws up, within a day we don't get our daily bread in Tesco's, Veiola don't pump our water for our cuppas, Esso don't have petrol for our school run MPV etc. etc.These companies really don't care how much we are totally in thrall to their big-tech reliant, and frequently interdependant, services, and these days we really don't have many choices apart from using the big-co's.

For most big publicly listed companies, the old style belt & braces & and a spare pair of trousers approach to vital big-tech doesn't happen anymore. Unless there has been continuing and realistic levels of investment in the infrastructure over the lifetime of the systems, these very large scale services supplier systems have become too big, complicated and fragile to NOT fail. O2 probably doesn't employ directly anymore the people who might understand the core systems issues they have experienced.

it's almost inevitable given the relentless squeeze on tech budgets these days. The IT director (in the unlikely event the business has one) has to produce cost yearly efficiency savings to get his bonus. If well managed, you can nibble off a couple of percent a year for a long time before problems become "Big", the front page headline kind of Big. Outsource it, then the outsourcer also outsources some etc. etc. Before you know it no-one has good expertise and experience of how the overall system actually functions and where the most significant practical operational risks lie.

As with the retail bank systems (and lots of others), those responsible for driving and devising these unrealistic long term operational budget strategies will usually have moved on long before the shit hits the fan, be shaking their heads sagely and counting their fully translated share option packages.

Notice that the very last budgets to get seriously cut before big problems come to light are those that service the board remuneration and to a lesser extent (dependent on shareholder identities) the shareholder divvies. If RBS / O2 style failures happen piecemeal over a few months, the pain will be soon be forgotten by the markets on which these companies rely for THEIR well being.

Once the mess has been cleaned up, it will be business as usual again -"it was just that XYZ junior techhie screwed up", or "those problems were an abberation" and " just a very unlikley set of circumstances" etc. etc.

Of course the one market segment that most fully realize their very direct fiscal dependance on big-IT systems are the market-makers and traders - they will no doubt continue to make sure THEIR systems and contingencies are really tight as the gnats proverbial and that is why they tailor their IT budgets accordingly regardless of short term conditions.

With IT security difficulties growing like topsy and yet more budget stress on the way for our big-co service suppliers, and with many supermarkets, banks, telco's etc. running on ever more complex and increasingly unpredictable tech services, those survivalist nutcases start to look less like fruitcakes and more like the main course.

I don't fancy bottled water and baked beans for the next 5 years, so any suggestions for a tasty but long shelf life dry goods diet? What's the flag for dead in the water? "M" for Mike ?

6
1
FAIL

Re: Is this the start of a trend?

I hate to say that this is not the start but more the culmination of a pattern of companies letting accountants make decisions on things they dont understand.

It generally works like this, the network architect comes up with a plan which minimises single points of failure, has built in redundancy and lots of expensive shiny, shiny added to it so its bombproof, bulletproof and tea lady proof.

This then goes to the accountants who skim over the reasons why you need this, look at the final price, reduce it by 50% and then pass it back for "remodelling" so that it fits the budget as opposed to the requirements.

Consequently all the bits that you need for a disaster are removed as the accountants dont see the relevance until the disaster happens.

Outsourcing is a classic example of this and RBs showed exactly what happens when you cut corners. Its fine whilst it works until it goes wrong and only then are the lessons previously taught taken seriously, but by then it is too late.

6
0
Bronze badge

Re: Is this the start of a trend?

"For most big publicly listed companies, the old style belt & braces & and a spare pair of trousers approach to vital big-tech doesn't happen anymore."

That's because people aren't prepared to pay the prices that such an approach would result in. Almost everyone buys on price - the cheaper the better. The choice providers face is to take out cost and remain competitive - but less resilient - or to stick with the belts and braces, lose customers and go bust.

1
0
Bronze badge

Re: Is this the start of a trend?

It's not accountants - it's the market. Customers won't pay the prices that expensive shiny shiny results in.

Margins are wafer thin and any extra expense results in higher prices or the provider making a loss.

1
0
Boffin

Re: Is this the start of a trend?

The decent engineers will spec it up a further 2x to account for the actions they know the accountants will take :-)

1
0
Meh

Re: Is this the start of a trend?

Lets not feel too sorry for NetCos and their "wafer thin" margins. They do ok.

O2, to pick a name at random do nicely with around £500 / quarter, or around £135 million / month.

That is not shabby money. Yes there is an ongoing investment in CAPEX future capacity planning etc, but lets be honest, £500m' / quarter PROFIT is not bad.

And the next time you send an sms, just think: the estimated cost of sending that sms is around 0.0001p. And they charge around 10p [on non inclusive plans] per sms.

That is because sms are carried, for most of their journey, over the C7 Signalling link. What does tis mean? Well, it is sent over the signalling that interconnects the different nodes. It is there /anyway/.

Think of is as Ceefax being delivered over the white noise......

No, I don't have /much/ sympathy for NetCo's. Some, but not a lot.

1
0
Bronze badge

Re: Is this the start of a trend?

But the cost of your text is greater than just the unit cost of the message transfer. Among the more obvious ones are the handset subsidies.

£500M profit can't be judged as good or bad unless you understand the revenue it took to achieve it. If the profit margin is 50%, hurrah, if it's 2% boo. If the margin is tiny, it's really easy for the market to turn that into a loss when the next step in the race to the bottom gets enacted - and then your loan repayments get more expensive, your share price falls and the shareholders who actually own your business start demanding even more cost cuts.

1
0

Page:

This topic is closed for new posts.