Three is reporting a network outage in Ireland, hours after announcing it was buying O2's Irish operation for €780m, but apparently this downtime is Virgin Media's fault. A banner across the Three.ie site blames Virgin Media for screwing up a transmission link and taking down the mobile network, which is soon to get a lot bigger …
So it's someone else's fault...
...when you've engineered a single point of failure into your systems. And then you shout about it so everyone notices?
Three has been tits up in EC2 for ages. No action yet from them to fix it.
Taking a nap having after swallowing O2?
Speak English much?
Re: Taking a nap having after swallowing O2?
@Irongut: You've got a bronze badge and you're unfamiliar with point 12 of the house rules? Tut tut.
So, if Virgin Media are providing you some backhaul fibre:
Where's your redundant cable?
Where's your redundant provider?
Where's your emergency backup?
Although a problem from Virgin Media might be the initial cause, the resulting symptoms are entirely the fault of Three.
Couldn't agree more, although it is SOP amonst mobile phone companies, pretty much the world over. The comment at the end of the article is a little off, it may be 'dodgy business' when it goes wrong, but it is how virtually everyone does it and do you want to double your backhaul costs and try and justify that to customers?
It will be interesting to see what actually went wrong, you would hope there would be some redundancy in the system where it could be done without it costing a fortune, but taking out the whole network?
Actually - It's highly likely that they did buy a resilient service. They'd be unlikely to publicly blame Virgin if it wasn't their fault - that tends to result in court cases and compensation.
It's fairly standard as well to buy your resilient service from a single provider - it's the only way to 'guarantee' that you don't pick two suppliers who both outsource the last mile to the PTT who then stick both circuits in the same cable. If you've ever tried to get two rival telcos to collaborate in delivering you a resilient service and sharing between them the cables, terminations and routes used, you'll know why. Resilience is holistic and buying from a single provider is necessary, but not sufficient, to achieving what you need.
"but it is how virtually everyone does it and do you want to double your backhaul costs and try and justify that to customers?"
A few years ago now a guy joined the company I was working for from a (now-defunct) mainland European global telco. In his debrief he told us that the unnamed company had worked out that it was cheaper to never provide resilience - despite selling it to their customers - and to pay out compensation when things did fail.
The business problem is that making a circuit resilient pretty much triples its cost. If your competitors aren't using resilience and have lower prices than you - what are you going to do? UK consumers pretty much adopt the approach of choosing the cheapest services they can find. Even after an outage like this, most consumers will stick with the £15 a month supplier and accept the odd failure, rather than shoot for the £20 a month supplier with a more resilient network. The market has spoken - and continues to speak - go cheap or go bust.
Blaming Virgin Media?
What sensible "Enterprise" size operation feeds their network without redundancy? Has someone dug a trench in Newry (Virgin Media are not in the Republic, nor are a major backhaul operator).
Once more Three demonstrate that all they are good at is selling Internet dongle contracts & PAYG and mis-labelling it broadband. It would have been better if O2 had bought Three.
An Post & Tesco use O2 infrastructure.
You'll find that Virgin Media are in the Republic, and have been for quite a while under various guises
Make sure you come to the Telefonica Fire Sale - Share Price: Slashed ; jobs :Slashed ; Networks at never to be repeated prices ; Morale : Well down.
That's Telefonica, where your Peso buys more.
You people are all so negtive at the slightest encouragement. Perhaps having recognised a weakness, no doubt put in place due to risk analysis committee, Three are buying O2 so that they can also expact their redundancy, which only just coincidently goes titsup?
Re: Negative, negative...
By the time you advertise in a paper that you are a mobile phone provider, your redundant infrastructure should already be in place. In the same way that before you do a TV ad for your website, you should have stopped running it on that old PC in your back-bedroom and moved to a proper set of servers that provide failover.
It's not a question of what's going on NOW, it's a question of why have you been selling that service to the nation without some decent backup all that time?
Hell, any secondary school you wander into will probably have a whole slew of backup / redundant servers, if not backup to another ADSL etc. But a entire nationwide mobile telecommunications provider doesn't have an independent second-line that they can switch to and so a whole part of the country can't do anything?
Like running a car without any brake pads in it - It's not a question of how easy it is to fix, it's a question of why the hell was it like that in the first place?
Re: Negative, negative...
They will feed O2 gear with their cheap Virgin Media fibre and ditch the multiple redundant O2 backhaul.
Oh and make EVERYONE in O2 Ireland redundant.
Re: Negative, negative...
"It's not a question of how easy it is to fix, it's a question of why the hell was it like that in the first place?"
It quite likely was resilient - I've had a supplier send out a tech to cease someone else's resilient circuit and accidentally take mine down - both legs - instead. Or you can find that the resilient, diverse and separate circuits you've bought have accidentally been routed by your supplier's supplier through the same card somewhere. That kind of thing is often only discovered when it fails.
But a wider point is this - resilience is expensive - making a network fully resilient makes it about three times more pricey than a non or limited resilience one - and in some cases it simply isn't possible at any price. Customers - or consumers certainly - buy the cheapest product available generally. Lots of people in the last 20 years have built resilient networks and then watched their bank balances dwindle to zero as customers vote with their wallets and buy the cheaper option. The market is driving this change, telcos are just responding to it.
Virgin Media customer support
I can just imagine the Virgin Media customer support call:
VM: "Hello Virgin Media customer support"
3: "Yes our phone network is down"
VM: "OK, can you try turning it off an on again?"
3: "It's a phone network! You can't just reset it! You need to send an engineer out!"
VM: "I can't do that until we try turning it off and on again."
3: "OK, whatever, it's turned off and on, the phone network is still down"
VM: "Can you go into your windows control panel..."
3" "This is a phone network that uses multiple broadcast repeaters, we don't run windows"
VM: "Oh I'm sorry sir, we only support windows. Goodbye"
Re: Virgin Media customer support
"Okay, we can send you an engineer... However if the fault is caused by something in your home there could be a charge of £150 or more for the engineer to put it right..."
"screwing up a transmission link and taking down the mobile network"
Maybe they just couldn't get that prism splitter inline quick enough
"but relying on someone else's infrastructure to support your customers is a dodgy business."
El Reg owns its entire network, eh? No, No you don't, you too purchase services from providers. Most would agree El Reg can be a bit dodgy, sometimes.