O2's titsup network struggles to find its feet O2's mobile network is struggling to its feet after a 19-hour outage left thousands of customers unable to make or receive calls. This morning 2G should be working again but 3G remains patchy. That's according to the sluggish status page, which, between apologies, claims the 3G service is slowly being revived. The telco …
Fairly standard practice in telecoms to have emergency gear provided by someone else for just such a scenario. Mobile techs carry phones on other networks, telephone exchanges have lines provided that are long-lined back to a different exchange served from a different major node.
There's nothing worse than realising on a 3am callout that your first line tools and techniques aren't enough to resolve an issue and then not being able to call for help.
This post has been deleted by its author
Outsource followed by outage. I wonder where we have seen that before? (Natwest).
I hope they cancel the bonus payment to the fool who thought this was a good idea. Losing customers is never a good way of growing the business and increasing profits.
I'm sure Huawei will now get screwed with a nice big SLA penalty.
Post hoc ergo propter hoc.
Given the sourcing deal was signed 2 months ago, I'd be surprised if they'd even started transition (and given the majority of staff involved are going across to Huawei under TUPE it would largely make no difference to day-to-day operating). Wonder what you'd like to blame any historical outages O2 have ever had on then..
But hey, it's much easier to read "O2 problem" and google "O2 outsourcing" to see what pops up, hey? Every company in the world outsources something to some extent.
Throughout this issue my handset has reported full bar signal and successful connection to the o2 network from my local mast. However, inbound calls to my number were failing to unobtainable - not even to my voicemail on the o2 network - this was clearly an internal, core network issue, not simply a problem with my handset accessing the network.
Didn't Orange have a 3G issue earlier this year? (yep http://www.theregister.co.uk/2012/03/08/orange_down/) At least their excuse is the merging of two formerly separate networks to create Everything Everywhere.
Are outages like this expected? Do other countries have similar issues with their carriers? Not that I'm suggesting a "Broken Britain" thing (though it might be - a product of a culture that subcontracts everything to defer responsibility and not caring about quality, then wonders why things blow up later), just trying to expand the field of experience to determine whether this is "normal" and expected behaviour for the technology (or "new" technology in general)..
If T-Mobile/Orange and O2 have had their turn, who's next?
Mobile networks are surprisingly fragile. There are many single points of failure, both from the actual nodes through to cabling, failed back ups over night and a host of other things.
Given the issue seems to be a re-authentication issue it could be a few things here.
Mobiles do authenticate for two reasons, one being time, say, every 30 mins, another is when the devices actually moved around, from cell to cell, or BSC/MSC to BSC/MSC.
HLRs are often located regionally, for capaicuty and redundancy, and each is connected to an AUC, Authentication node. If these have problems, or the links between, there could be issues.
On 3G, there is the RNC, which performs similar jobs for the 3G but that the HLR/AUC does for 2G.
But none of these nodes are really redundant. A network may have, say, five HLRs, around the country, that can be load balanced between the five, but HLRs are not, usually, built in redundant pairs, which means the links are not either.
Thankfully, most Blighty and euro networks have very few issues, because the O/M procedures do work. But, given the volume of date being carried by the network, to make and place calls, CDR's etc, as well as actual usage traffic, when things do go down, it does take some hours to bring back on line.
Lets just hope that o2 and Huawei do not outsource and off-shore things like O/M etc and that this is a one off.
Time will tell.
This post has been deleted by its author
This post has been deleted by its author
This post has been deleted by its author
Generally, providers are responding to market requirements, and what the market is saying is "cheap as chips". A provider who spends a lot on resilience has higher prices and finds that they don't then have any customers.
It's not really about providers absolving responsibility or outsourcing, it's about the decisions they have to make to hit a price point that customers will pay for.
Britain has amongst the cheapest telecoms in Europe - and those low prices are driven by the market. There is a flipside to those low prices however, and that's constrained investment and engineering to meet a cost.
I buy resilient services to meet the SLAs my customers need - but the cost of that can be eye-watering. Thousands of pounds a month for resilient 100M Ethernet between two fairly close sites, for example. If you're used to paying £15 a month for consumer broadband the prices you get for proper, carrier-grade resilience look like decimal point errors. Anyway - the point of my post is this - if you want it to always work without question, you need to pay a lot of money. If you want it to be cheap, you'll get something that works most of the time.
...if this weren't something to do with trying to re-prioritise their network for the Olympics. I would expect that they will try to make the London area more robust, given the expected extra traffic there over the next few months. They are the 'official' communications provider, after all.
"Ericsson have performed restarts of the LDAP processes on the CUDB in an attempt to recover service.
The Technical bridge remains open monitoring the service and assessing the impact the restart are having on customers' ability to connect.
We're also looking at other options available to us should the restarts fail to recover service."
Looks like the turn it off and back on again applies to their infrastructure as well as our phones!
Orange France fell over last Friday from about 3pm - until around 0300 on Saturday morning - so 27 milion or so subs unable to do anything - my phone kept "searching".
Apparently Ericsson and A-LU engineers were helping Orange with their inquiries and it had nothing to do with a software update about 24 hour earlier on the HLRs.
By mid evening 2 French cabinet ministers were visiting the Orange crisis control centre - which must have improved the situation no end...
Orange have offered everyone a free day of usage in September for calls, SMS and data depending on their current contract - so guess when the next big outage will happen.
It will be interesting to see what O2 do...
http://www.lefigaro.fr/flash-eco/2012/07/11/97002-20120711FILWWW00528-panne-orange-un-logiciel-en-cause.php for those who read French or can use Google Translate
Simon
France Telecom said Wednesday that the outage that affected its services the weekend was related to a software problem occurred on one of its equipment had suffered an update 48 hours earlier.
Heard by the Economic Affairs Committee of the National Assembly, the first CEO of French telecoms operator Stéphane Richard has elaborated on the scenario of giant failure that affected its 26 million subscribers and the operators' customers who rent its network. The incident affected a strategic asset for the operator, a set of servers whose mission is to locate and authenticate all subscribers permanently, hence the magnitude of the rare failure occurred early Friday of afternoon, the day of baccalaureate results.
Aggravating circumstance, no alert was triggered as a result of this anomaly, preventing the initiation of emergency equipment. Stéphane Richard said the expert work on the causes of the incident was not yet complete but it has already ruled out a virus attack or a traffic overload.
Apolgies in advance for this rant.
I think nsld's cynical comment maybe very near the mark. If you are an affected O2 customer and were also hit by the RBS network going tits-up a couple of weeks ago you could be forgiven for worrying about other rather important tech reliant services.
O2 have been boasting that they have 200 engineers assigned to network quality. I hope that doesn't equate to 200 "engineers" doing everything needed to keep things going, from physical station maintenance to network systems admin and the rest for all those shiny 23m+ mobile devices.
The companies we rely on to deliver services essential for our day-to-day survival are in practice totally big-tech reliant these days. If the big-tech screws up, within a day we don't get our daily bread in Tesco's, Veiola don't pump our water for our cuppas, Esso don't have petrol for our school run MPV etc. etc.These companies really don't care how much we are totally in thrall to their big-tech reliant, and frequently interdependant, services, and these days we really don't have many choices apart from using the big-co's.
For most big publicly listed companies, the old style belt & braces & and a spare pair of trousers approach to vital big-tech doesn't happen anymore. Unless there has been continuing and realistic levels of investment in the infrastructure over the lifetime of the systems, these very large scale services supplier systems have become too big, complicated and fragile to NOT fail. O2 probably doesn't employ directly anymore the people who might understand the core systems issues they have experienced.
it's almost inevitable given the relentless squeeze on tech budgets these days. The IT director (in the unlikely event the business has one) has to produce cost yearly efficiency savings to get his bonus. If well managed, you can nibble off a couple of percent a year for a long time before problems become "Big", the front page headline kind of Big. Outsource it, then the outsourcer also outsources some etc. etc. Before you know it no-one has good expertise and experience of how the overall system actually functions and where the most significant practical operational risks lie.
As with the retail bank systems (and lots of others), those responsible for driving and devising these unrealistic long term operational budget strategies will usually have moved on long before the shit hits the fan, be shaking their heads sagely and counting their fully translated share option packages.
Notice that the very last budgets to get seriously cut before big problems come to light are those that service the board remuneration and to a lesser extent (dependent on shareholder identities) the shareholder divvies. If RBS / O2 style failures happen piecemeal over a few months, the pain will be soon be forgotten by the markets on which these companies rely for THEIR well being.
Once the mess has been cleaned up, it will be business as usual again -"it was just that XYZ junior techhie screwed up", or "those problems were an abberation" and " just a very unlikley set of circumstances" etc. etc.
Of course the one market segment that most fully realize their very direct fiscal dependance on big-IT systems are the market-makers and traders - they will no doubt continue to make sure THEIR systems and contingencies are really tight as the gnats proverbial and that is why they tailor their IT budgets accordingly regardless of short term conditions.
With IT security difficulties growing like topsy and yet more budget stress on the way for our big-co service suppliers, and with many supermarkets, banks, telco's etc. running on ever more complex and increasingly unpredictable tech services, those survivalist nutcases start to look less like fruitcakes and more like the main course.
I don't fancy bottled water and baked beans for the next 5 years, so any suggestions for a tasty but long shelf life dry goods diet? What's the flag for dead in the water? "M" for Mike ?
I hate to say that this is not the start but more the culmination of a pattern of companies letting accountants make decisions on things they dont understand.
It generally works like this, the network architect comes up with a plan which minimises single points of failure, has built in redundancy and lots of expensive shiny, shiny added to it so its bombproof, bulletproof and tea lady proof.
This then goes to the accountants who skim over the reasons why you need this, look at the final price, reduce it by 50% and then pass it back for "remodelling" so that it fits the budget as opposed to the requirements.
Consequently all the bits that you need for a disaster are removed as the accountants dont see the relevance until the disaster happens.
Outsourcing is a classic example of this and RBs showed exactly what happens when you cut corners. Its fine whilst it works until it goes wrong and only then are the lessons previously taught taken seriously, but by then it is too late.
Lets not feel too sorry for NetCos and their "wafer thin" margins. They do ok.
O2, to pick a name at random do nicely with around £500 / quarter, or around £135 million / month.
That is not shabby money. Yes there is an ongoing investment in CAPEX future capacity planning etc, but lets be honest, £500m' / quarter PROFIT is not bad.
And the next time you send an sms, just think: the estimated cost of sending that sms is around 0.0001p. And they charge around 10p [on non inclusive plans] per sms.
That is because sms are carried, for most of their journey, over the C7 Signalling link. What does tis mean? Well, it is sent over the signalling that interconnects the different nodes. It is there /anyway/.
Think of is as Ceefax being delivered over the white noise......
No, I don't have /much/ sympathy for NetCo's. Some, but not a lot.
But the cost of your text is greater than just the unit cost of the message transfer. Among the more obvious ones are the handset subsidies.
£500M profit can't be judged as good or bad unless you understand the revenue it took to achieve it. If the profit margin is 50%, hurrah, if it's 2% boo. If the margin is tiny, it's really easy for the market to turn that into a loss when the next step in the race to the bottom gets enacted - and then your loan repayments get more expensive, your share price falls and the shareholders who actually own your business start demanding even more cost cuts.
People are referring to outsourcing but no one is specifically mentioning O2 have outsourced all there server/DB support folk to TCS (TATA), there app support folks all over the show & god only knows on the cell side so it's not pretty (BT to Erricson I recall)
While there is nothing to say if this is human versus technical failure, if human whether TCS versus Erricson versus core O2 staff (do they still exist?) or anyone else but consider what culture is created when your outsourced overlord is holding a 1 year axe over your neck?
The 1 year TUPE protection is about to end for the TCS folks, it doesn't exactly encourage "going the extra mile"
Regardless of fault I hope the folk who authored/promoted Project 1 are sweating now (especially following RBS's hiccup) about choices they sold to Telefonica with regards to mass outsourcing. Here's praying for egg on your face. Perhaps staff retention, local knowledge & experience are a good thing to promote instead of cost saving. Funny concepts eh!
Nice one O2, you've possibly reaped what you've sown, regardless it's pretty spectacular fail!
There aren't very many - lots of mergers, lots of companies who've gone to the wall. The only way those wafer thin margins can work for you is to have scale. Look at the telecoms companies around a decade ago, look at how many are still around. Sometimes the brand survives even if the company behind it has gone.
Or, the beginning of a new trend, one where the manglement ID10Ts actually have to face some real consequences for their fuck ups. And, I don't mean leaving with a Golden Parachute either.
IMHO, the consequences for those ID10Ts at RBS for their outsourcing decisions should be immediate termination, no severance pay or benefits (IOW a SKITA {Swift Kick In The Ass!!!})
If any of them squawk about it, then sue them for breach of fiduciary duty, and attach any, and all of their possessions to collect on a judgment. It is about time to start demanding that DAMAGEMENT be held accountable for its colossal fuck ups.
This bullshit has gone on long enough!
"For most big publicly listed companies, the old style belt & braces & and a spare pair of trousers approach to vital big-tech doesn't happen anymore."
That's because people aren't prepared to pay the prices that such an approach would result in. Almost everyone buys on price - the cheaper the better. The choice providers face is to take out cost and remain competitive - but less resilient - or to stick with the belts and braces, lose customers and go bust.
I disagree. Companies do spend a lot of money and effort building systems and services with "resilience" built in.
What they often don't appreciate is how much complexity this "resilience" adds, and that the complexity itself results in additional failure modes (which are harder to diagnose).
It's pretty easy to design a system which is resilient against a node powering off completely. It's much harder to design a system which is resilient against a node which starts sending out corrupt data. Google for "byzantine fault tolerance".
http://en.wikipedia.org/wiki/Byzantine_fault_tolerance#Byzantine_failures
"For example, in 2008 Amazon S3 was brought down for several hours when a single-bit hardware error propagated through the system.[2]"
With core infrastructure of large companies (RBS payment processing, 02 Telephony), it should not be possible to nuke the system, a suffcient amount of money and resources can easily render this possible.
Unfortunately the senior managers whoose bonus is based on cutting easily visible costs, and quite frankly have naff all ability at risk management, will cut these services to get their bonus, and if they are still around when the risk occurs, (a) they probably won't lose their job, and will (b) keep their bonus, despite how much money they have just lost their company(*)
This is why Barclay/Libor is a side show, it's being hanfled by the regulators, what needs serious consideration by parliment is the amount of the UK economy that is bet on technology services, that is not robust, and does not have an adequate BC/DR plan. RBS took out around a 25% of the abilitty of the UK to transact business, it is not unusual from other firms (e.g. 02) in it's approach to IT.
The modern economy needs legislation, that makes your IT and BC/DR as well audited as your accounts, shareholders and business partners should be able to read your company report, and see if they want to do business with you.
Completely agree. The key to it all is the evil tower of accountants. In virtually all companies the provision of IT services is seen as ONLY a cost centre, the imbeciles counting their magic beans try and cut funds to any department that is not a Profit Centre (except their own in most cases .. holders of the golden keys) ... This combined with management who don't really know their business creates this mess ... Whilst there is little to ague with the desire to run any department efficiently, the constant erosion of investment leads ultimately to failure...
The dimwits at the top who have got there through lies and self promotion need to wake up and see that without a good , reliable IT infrastructure their businesses will not survive ... Going for the cheapest option on IT eventually leads to reduced income/profi in the long term
It's a horrible thought and nobody's mentioned it, so I thought I'd better at least ask the question.
As I understand it, O2 have no idea what caused the outage. And I believe they are the network provider (via their "Airwave" service) for the UK Emergency Services.
Could this have been a trial run by someone who wanted to see what the effects on the Emergency Services communications would be? And do we know if they were affected at all?
Sounds a tiny bit like the plot from an Oceans X film... Take down the airwave or take the manager of that bank (Casino etc) off the air and then they can't call for help.
On the subject of call for help - what about someone who couldn't call for an ambulance or the police... Would they have a case ? or are the T&C's sufficiently wide when you take a mobile contract that paucity of coverage / service is not covered...
But then do they have a Duty of Care to their customers ?
Would it be possibly during a court case to demand the details of what did happen (same goes for RBSwest)
Whilst I can't disagree with most of the content I do have to disagree with the inferred presumption of choice. Whenever I have taken a service from a Mobile Phone company as a consumer I have never been asked if I want a best effort or guaranteed service ... therefore the point is mute.
Similarly mobile services purchased by large organisations, who spend vast amounts on building stable , redundant IT infrastructure, use the very same service ... I know which they would choose if there were different services offered.
If the TelCo's infrastructure has been built less in a less stable or redundant manner , that is purely their choice. It's a business choice, drive down costs and accept the possibility of service loss and disaffected customers , or not ...
"HLRs are not, usually, built in redundant pairs, which means the links are not either."
Back when the operational support systems in a telco were designed by engineers rather than Windows weenies from BSS or IT, HLRs often didn't need to be built in redundant pairs because they were built on fault tolerant Tandem hardware (or occasionally on VMSclusters), where the resilience is (mostly) provided by the supplied computer system. They would still need resilient connectivity, but if a telco doesn't have resilient connectivity for that kind of thing, then...
That was a decade or more ago though. I'm out of touch with what is currently fashionable (aka "best practice"?) in this particular sector of where IT meets proper engineering. Anyone got current info?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
