So it's someone else's fault...
...when you've engineered a single point of failure into your systems. And then you shout about it so everyone notices?
Three is reporting a network outage in Ireland, hours after announcing it was buying O2's Irish operation for €780m, but apparently this downtime is Virgin Media's fault. A banner across the Three.ie site blames Virgin Media for screwing up a transmission link and taking down the mobile network, which is soon to get a lot …
Couldn't agree more, although it is SOP amonst mobile phone companies, pretty much the world over. The comment at the end of the article is a little off, it may be 'dodgy business' when it goes wrong, but it is how virtually everyone does it and do you want to double your backhaul costs and try and justify that to customers?
It will be interesting to see what actually went wrong, you would hope there would be some redundancy in the system where it could be done without it costing a fortune, but taking out the whole network?
"but it is how virtually everyone does it and do you want to double your backhaul costs and try and justify that to customers?"
A few years ago now a guy joined the company I was working for from a (now-defunct) mainland European global telco. In his debrief he told us that the unnamed company had worked out that it was cheaper to never provide resilience - despite selling it to their customers - and to pay out compensation when things did fail.
The business problem is that making a circuit resilient pretty much triples its cost. If your competitors aren't using resilience and have lower prices than you - what are you going to do? UK consumers pretty much adopt the approach of choosing the cheapest services they can find. Even after an outage like this, most consumers will stick with the £15 a month supplier and accept the odd failure, rather than shoot for the £20 a month supplier with a more resilient network. The market has spoken - and continues to speak - go cheap or go bust.
Actually - It's highly likely that they did buy a resilient service. They'd be unlikely to publicly blame Virgin if it wasn't their fault - that tends to result in court cases and compensation.
It's fairly standard as well to buy your resilient service from a single provider - it's the only way to 'guarantee' that you don't pick two suppliers who both outsource the last mile to the PTT who then stick both circuits in the same cable. If you've ever tried to get two rival telcos to collaborate in delivering you a resilient service and sharing between them the cables, terminations and routes used, you'll know why. Resilience is holistic and buying from a single provider is necessary, but not sufficient, to achieving what you need.
Blaming Virgin Media?
What sensible "Enterprise" size operation feeds their network without redundancy? Has someone dug a trench in Newry (Virgin Media are not in the Republic, nor are a major backhaul operator).
Once more Three demonstrate that all they are good at is selling Internet dongle contracts & PAYG and mis-labelling it broadband. It would have been better if O2 had bought Three.
An Post & Tesco use O2 infrastructure.
By the time you advertise in a paper that you are a mobile phone provider, your redundant infrastructure should already be in place. In the same way that before you do a TV ad for your website, you should have stopped running it on that old PC in your back-bedroom and moved to a proper set of servers that provide failover.
It's not a question of what's going on NOW, it's a question of why have you been selling that service to the nation without some decent backup all that time?
Hell, any secondary school you wander into will probably have a whole slew of backup / redundant servers, if not backup to another ADSL etc. But a entire nationwide mobile telecommunications provider doesn't have an independent second-line that they can switch to and so a whole part of the country can't do anything?
Like running a car without any brake pads in it - It's not a question of how easy it is to fix, it's a question of why the hell was it like that in the first place?
"It's not a question of how easy it is to fix, it's a question of why the hell was it like that in the first place?"
It quite likely was resilient - I've had a supplier send out a tech to cease someone else's resilient circuit and accidentally take mine down - both legs - instead. Or you can find that the resilient, diverse and separate circuits you've bought have accidentally been routed by your supplier's supplier through the same card somewhere. That kind of thing is often only discovered when it fails.
But a wider point is this - resilience is expensive - making a network fully resilient makes it about three times more pricey than a non or limited resilience one - and in some cases it simply isn't possible at any price. Customers - or consumers certainly - buy the cheapest product available generally. Lots of people in the last 20 years have built resilient networks and then watched their bank balances dwindle to zero as customers vote with their wallets and buy the cheaper option. The market is driving this change, telcos are just responding to it.
I can just imagine the Virgin Media customer support call:
VM: "Hello Virgin Media customer support"
3: "Yes our phone network is down"
VM: "OK, can you try turning it off an on again?"
3: "It's a phone network! You can't just reset it! You need to send an engineer out!"
VM: "I can't do that until we try turning it off and on again."
3: "OK, whatever, it's turned off and on, the phone network is still down"
VM: "Can you go into your windows control panel..."
3" "This is a phone network that uses multiple broadcast repeaters, we don't run windows"
VM: "Oh I'm sorry sir, we only support windows. Goodbye"
>click<