Vodafone's Monday morning outage was caused by thieves who broke into the operator's Basingstoke exchange and lifted a load of switches. The break-in happened around 00.30 this morning, and the police were quickly notified. Vodafone noticed its own network collapsing and assembled its "War Room" which is supposed to deal with …
Fixed? not yet
I'm in Basingstoke. Still down
None of the users here could get a signal first thing, and we are about 1 mile from the Atantic ocean - hardly "South East England".
As for "only 100,000 people affected" - pish, tosh and other words. There were probably that many complaing on the various forums, but that is not the sum total of those affected.
Mind you, I love the comment on Twitter by someone that tried to cal someone in the Vodafone PR office, only to hear the message "Welcome to the Orange Answer phone service".
Definitely not fixed here in Guildford. I can make calls but anyone calling me is helpfully told that 'the number you have dialled is not recognised'
Data is extremely sporadic too, I had a bunch of emails appear on my Blackberry about an hour ago but the thing has been silent since.
Fixed for any given definition of not fixed
Yup - still down in Reading.
Working intermittently in Caversham (across the river) since 12 and solidly since 1.
The morning was wasted
by posting complaints to Twitter, Facebook , El Reg, the BBC and anywhere else that would take them, along with constant refreshing.
Ok so how does 1 break in knacker service delivery for such a large area, and is still in trouble 12 hours after the incident.
Maybe the customers and share holder should be asking to see vodafones BC/DR plans before handing over any more cash.
It is considerably more difficult than you think
The architecture of the "intelligent" networks and its IMS based 3GPP successors has a considerable number of elements which are very difficult to have on hot-stand by and/or in failover+load-balancing mode.
This is something inherent to the arch in question. If you take out a BSC or RNC everything slaved off that BSC is dead and you have to reprovision a significant chunk of your transmission as well as commission a whole new BSC to fix it. If you take out an MSC the story repeats just multiplied by a factor of several thousand times for all of those BSCs or RNCs that have been orphaned by the death of their master. If you take out the policy function, the diameter or some of the key bits of the EPC a big chunk of the network dies in a similar fashion.
In fact it is inherent to the idea of the intelligent network. The more intelligent the network the more vulnerable it becomes to someone taking a big axe and surgically removing key bits of the "intelligence".
One step forward two steps back?
"In fact it is inherent to the idea of the intelligent network. The more intelligent the network the more vulnerable it becomes to someone taking a big axe and surgically removing key bits of the "intelligence".
Technical issues aside, Vodaphone sell a business service, they should be planning on bits failing, or clearly stating that their service is not suitable for mission critical work.
If they want the business service profits, then they need to pay up for the backup coverage
For anyone who is still down...
For anyone who is still down, try rebooting your phone. That should get you back on the network.
I can not recieve calls here. We are just south of Reading. Any incoming call rings once then shows as a missed call on the handset. On the calling landline we tested from you get a single ring then a number unobtainable tone. Definately still down.
Swindon, this is Swindon
Finally back up in Swindon.
Swindon isn't in the South East. Your article is incorrect in terms of the areas affected, as it covered much more than just the South East, as your earlier article about this issue correctly mentioned.
Colleague in the Penines has service back but others in Reading still off air.
Yeah, took out our entire company, and everyone I know with vodafone - how exactly do they work it out at only affecting a few customers?
Service back 10 minutes ago, no calls/txt before then.
Vodafone clearly has massive single point of failures in their network, and very poor backup planning!
Single point of failure?
14:45 still unstable around Reading.
This poses serious question about disaster recovery plan at Vodaphone... If the Basingstoke datacenter was destoyed by fire it seems it would bring south (eastish) UK network down.. for how long?
I suspect other key premises are in the same situation.
Any SLA / compensation planned?
If you pay £30 a month line rental...
...that'll be 50p compensation for service lost for half a day.
Frustrating as outages are, I don't think Vodafone contractually owe retail customers more than that (and even then, it's a stretch, as they cover themselves with protective clauses in their T&Cs).
TomTom Traffic Updates
Anyone have problems in the affected areas getting Traffic Updates on their TomToms today?
These use Vodafone for the data service.
Meanwhile, on the Hampshire coast...
...no difficulties whatsoever with Vodafone. Full signal, making and receiving calls quite happily. Maybe I'm one of the 100,000 unaffected users... :)
Everything gets cheaper, the only thing we care about is how much we get nobody ever stops to question the quality, what backup is there, what resilience.
I don't imagine any other carrier is any different.
They may review their security procedures but that is it - nobody really cares and everybody will have forgotten about this when the contract comes round again, they'll care more about which network has the shiniest phone and the biggest data package / most texts.
What happens when a few affected customers finally get their service back? It jams entire capacity on the network it seems...
Corporate seriousness chart.
2. Wage level.
3. Company car.
5. Executive drinks cabinet security.
134. Infrastructure security.
135. Customer satisfaction.
As a back-up, get a POTS phone that doesn't need mains or a wireless connection. No frills. Under a tenner.
The simpler stuff is, the more reliable it is.
Never trust or rely on technology. Never trust or rely on those who provide technology or tech services. Alway back-up your data. Always have a plan B for when the tech fails (which it will).
My pigeon loft is ready and on hot standby for a situation like this.
Well... → #42
The use of POTS as a back-up is indeed a good plan, however there seems to be a move away from providing a simple piece of "wet string" in new buildings. As the IT guys seem to be responsible more and more for cable infrastructure they like Cat5, hubs and switchers. At a recent job in Turkey seven new sporting venues had been built without any from of POTS infrastructure.
Back to Vodafone, I hope they learn the lessons from this incident. They have provided me with a good service for over ten years now and, in my book, still have the best coverage.
Not sure how their network joins together and it might be a red herring, but I'm about 12 miles from Basingstoke and have enjoyed 5-bars uninterrupted service all day both home and at work. All of my colleagues are offline. The only difference is - I have a Voda SureSignal box at home & work?
Suresignal comes out of the network 'up north' somewhere. That is why I had great reception this morning at home and got my own network wakeup call at 4:28 when some of our services went down. But unfortunately once in the car and into London it was gone.
It is slowly getting back now but still very patchy despite my phone showing good reception.
The title is required, and must contain letters and/or digits.
Somebody complains about illegal interception and then suddenly their exchange gets broken into? Coincidence or somebody trying to get rid of evidence? (where's the black helicopter icon when you need it?)
what are the chances
That anonymous broke in dragging an extension cord, moved the 2nd power supplies to their UPS, and removed the equipment without powering it off.
The break in occurred when ?
I noticed a lack of communications on my handset in Oxford at around 10:20pm last night. I remember this because my Virgin broadband had died again and I was making a note of the time .
"Quite why that morning was wasted we still don't know"
Not everyone is using their mobile to play FarmVille on their mobile. Everyone in our office actively trades with people on the road who rely on their mobiles to earn their (often self employed) living. While it hasn't affected us directly, it's been bloody annoying not being able to get hold of many of our contacts.
It's guaranteed to have cost a good few people money. Sure, they're only paying Vodafone £1 per day, but it's more than a little embarrassing for Vodafone to demonstrate how bad their disaster recover is.
As somebody else has commented, you have to make cutbacks to generate £11.5 billion profits. Obviously security was one of those areas.
Still fixed, um, not quite
If you are not seeing a signal, turn your phone on then off (or enable then disable airplane mode) and you should see a signal. I have found I can place calls but not receive calls, my other colleagues in the office are experiencing the same symptoms.
Everything I have to say on this can be summed up with the below video, and one line
I hear someone stole their infrastructure - with a bulldozer, and you can't call someone for half a day? Quit your bitching.
how long ....
... before said switches turn up on Tat Bay then ? Shirley the miscreants won't be that stoopid will they ?
What I want to know is
Are infrastructure quality network switches readily tradable on the black-market and if who wants 'em?
Precisely the question I would like answered. Who would buy this pilfered kit?
Scotland or Gobi desert
I have been using Voda fone for the last two years for my phone and mobile broadband connection , I have always found their service excellent .I think a bit of commercial espionage could be behind this attack and it's not the first time this has happed.When my service went down today I rang 191 and went straight through to their bonnie call center in Scotland and as usual spoke to a polite and competent person who gave me the full details of the situation and a free £5 top-up to cover my inconvenience.When I was with T-Mobile I had to talk to some toothless old git in a tent in the middle of Gobi desert reading from a script in badly broken English.
VF still down in camberley/bracknell areas.
We have three VF handsets here. Only my tocco lite would rcv a call for a short while but no handsets would originate a call. Then an hour later ~3:30 incoming died and the balance OSD died as well. test calls at 6pm and the VF handsets are still DIW
I suspect they are still rebooting and recovering from thier outage.
My worry was in checking our PAYG balances to ensure that afterthe outage we have not lost existing credit (again!).
My last fiasco with VF was a android handset eating 1UKP/hr when switched off and battery removed. which is why I moved to gifffaff for the android handset.
Now I understand why others have more than one handset or a dual sim card handset.
Swindon - Village Near....
Still not working correctly, was getting calls flashing up for a moment then disconnecting before phone could even ring, 18:15, can make outgoing calls to non vodafone numbers, however cannot call my colleagues on Voda.
I heard on the grapevine
that they had over £500,000 worth of HSPDA kit stolen. Probably off to China/Nigeria/Brazil/Russia/[Insert Dodgy country here].
GPRS was off as well.
Our dispatch system uses GPRS to communicate bothways. It disappeared at about 7 am for about 7 hours. unfortunately the only back up is mobile phones! That's about 40 self employed people who can't earn.
Whoever the Vodafone DR person is, they must be feeling somewhat insecure at this point. and justifiably so.
however I am available should they decide to revisit the plans..
- Tricked by satire? Get all your news from Facebook? You're in luck, dummy
- Feature TV transport tech, part 1: From server to sofa at the touch of a button
- Google straps on Jetpac: An app to find hipsters, women in foreign cities
- Updated Microsoft Azure goes TITSUP (Total Inability To Support Usual Performance)
- The Return of BSOD: Does ANYONE trust Microsoft patches?