back to article Vodafone's network knackered by thieves

Millions of Vodafone customers were disconnected this morning after an overnight break-in. Calls, text and data are intermittent west of London with a fix in progress. The failures seem only to be affecting areas just west of London, in the so-called Silicon Corridor. Wales seems fine and the rest of the country is OK, but …

COMMENTS

This topic is closed for new posts.

Page:

FAIL

Where was the Data Centre?

In the middle of a field? I had the mistaken belief that they were holed up in basements about 3 floors underground.

Did someone leave a window open?

1
0
WTF?

Intermittent

Intermittent my Arse... I got a "curtsy call" at 04:15 from ADT to say the alarms PAC (GPS) signal was down, that is obviously when the scroats kicked the door in.

I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind.

2
4
Silver badge
WTF?

Erm...

"I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind."

Yeah. You relied on one single mobile communications network.

19
5
FAIL

SPoF

<quote>

I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind

</quote>

Yes, it does... why would a company so clearly reliant on mobile comms have all their staff using a single mobile network??? Surely a mixture of vodafone, o2, t-mobile, orange & 3 would be more appropriate for your company??

When you point a finger at someone, there are 3 pointing back at yourself!

3
2

No, a few other things come to mind

Two ways to look at it:

First and foremost, once upon a time Ofcom (which was called different then) and the mobile operators instituted the no-in-country roaming policy. In those days nobody looked at the mobile phone as a possible single point of contact. That time has clearly passed and having at least mvnos with multi-operator roaming is something that is long overdue. There are businesses that need that level of connectivity nowdays.

Second, you could have been prepared for it yourself (at least to some extent). For example, I do not give away my mobile (for a couple of years now) and I have a PBX which tries based on time of day, schedule information from my calendar and a few other things my fixed line(s), voip and falls back to a list of mobiles (including foreign ones) when needed. Once the list is exhausted it goes to voicemail which I get as an email as well. It takes me about 1 minute to add or remove a mobile or VOIP endpoint from that list. If you had that, you could have just told your work-force to go and buy a throw-away pay-as-you go from three, O2, T-orange or whoever else you fancy and get on with business as usual. In fact you could have told that to them on their _WORK_ phones and left them to pick up the voicemail via whatever means they find, whenever they get to it.

1
0
Pirate

First rule of geekery - ALWAYS have a backup

"I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind."

Comment from a friend of mine: "At some point last night, Voda realised they should have issued O2 or Orange mobiles to their callout engineers..."

Another nice slice of irony in our (rather major) Vodaphone-only company, though - we recently opened a new major office, where it was rapidly discovered only the Vodafone network got any decent signal in most of the building, despite perfect service levels on all networks immediately outside. Accusations of network blocking were dismissed with a 'not our problem - you can always change networks' from the brass.

Now the office has been beautifully devoid of annoying ringtones all morning, and the poor little Blackberry addicted dears are showing serious signs of withdrawal symptoms...

4
0

"communications network"

So i should not use just tcp/ip on the LAN because its a single topology,

I shouldn't use one manufacture of Hardware even though they are biggest.

Perhaps I should look at getting the coffee from three different sources in case one stops brewing

Either you work for VodaFone or a government establishment where cost is not a concern. We use VodaFone, the biggest supplier of mobile networks in the country, in order to minimize the ongoing admin and management costs of maintaining a fleet of phones.,

5
7

"Surely a mixture"

Come on join us in the real world where we actually all have four fingers and a thumb.

0
0

Engineers phones

.

.

I used to work in the telecomms business and it was normal then for engineers to have backup phones on a different network to the one they were working on. I would be surprised and slightly saddened if this had changed.

.

.

2
0
Pint

@ElbowNi

Fine, you're right, you should use just the one mobile provider. Must be a big comfort to you being so sure you're right, except you can't phone anyone to tell them.

5
0
Silver badge

@ elbowNI

Fraid I have to agree with everyone else - running something that important on one network with no apparent back up plan? Perhaps you can use today's downtime to consider how much you'd not lose by having one.

Suggestions: T and Orange allow cross network roaming on 2G; have a draw full of PAYG sims on various networks for next time and/or network coverage evaluation.

1
0
FAIL

Really? - Poor planning

I quite understand that you use one network to control the billing, the cost would be horrific if you split over multiple networks however...

How much exactly would it have cost you to call O2 and ask for a set of backup PAYG SIM cards for your staff and issue them, along with a secondary phone number list? That way when Vodamuppets bugger up their network you all switch over to the secondary network, given that the problems are rarely complete failures many SMS would get through instructing staff to switch to the backup SIMs. You can always use a web page as a secondary notification.

I quite understand your situation, we have been without a fixed line Internet connection for over two months thanks to the time wasting lazy tosspots at British Terrorcom, good job we have a Vodafone dongle and a spare router which lets us provide (albeit slower) Internet access to our network for key services such as email.

1
0
Silver badge
FAIL

Er

Several replied to you, but here would be my response: Actually, I work for schools. You know, those things with zero IT budget and what is there is spent on Visual-Basic-filled-with-clipart "curriclum-vital" software.

You're complaining about having a single point of failure when you use a single company. If you have Cisco-only gear and Cisco go out of business - you're stuffed. You can say "it won't happen" or "we need to use Cisco" until you're blue in the face. When it goes bust, nobody will care about your problems.

If an exploit hits a Cisco-only bug (say, the million-and-one IOS bugs), you're stuffed. I'm not saying you shouldn't use Cisco, but you should be prepared for the consequences of such a mono-culture. It's like saying "All Windows PC's are more susceptible to viruses". Completely correct but sometimes you "have to" use Windows. It just means: Don't be surprised when a virus hits and all your Windows stations go down (but not your Linux stations), or all your Cisco IOS gear is exploited (but not your D-Link gear or whatever), or all your engineers are out of reach because all use one mobile phone network. Complaining that MS, Cisco or Vodafone are to blame for not having their own processes to manage their mono-cultures is as silly as you relying on that mono-culture. It may be *practical*, *sensible*, cost-effective, but it's still a mono-culture.

And thus, you can't push blame to Vodafone for not doing your job. Yeah, you can complain and get service requests and claim on your business support contract (you *DO* have a written legal statement of constant availability from Vodafone, yes?), but nobody can "fix" that problem of you only using one operator because "they are the best" (this incident kinda negates that kind of thinking, I believe). Any one of your people could be using any mixture of any mobile operator without any problems at all and no difference to you except three separate bills instead of one combined. Or, in an emergency, any of them could be instructed to get / issued with a PAYG SIM on another network for the day.

My school has a dual ADSL2+ line load-balance/failover setup - we get 48MBps of Internet connectivity and can turn off any one modem without affecting the school in any way (who rely on the Internet to fill in exam papers, pay wages, contact parents, etc.). Both lines are linked to our local exchange. We *know* this and don't expect it to operate in the case of an exchange fault - it's to cover *other* eventualities, or else we'd get in a leased line from another exchange deliberately.

By a similar token, you should NOT expect your business to operate in the case of a mobile operator fault (or even the tower nearest your main site of operations having a fault), by the sound of it. Tell me, what happens if your VOIP/landline telephone system goes haywire, or are you just relying on that to "always work" too? Our actual "emergency" backup for Internet connections is a handful of 3G sticks all on different mobile operators that can run the entire school network just by us plugging them in and running a script (they are always active because we also use them for SMS reception / transmission). Latency goes up, but users continue as if nothing had happened at all. When we hit the traffic limit on one stick, we replace the SIM with some other PAYG with £1/day data. Not perfect, but it works and gets us out of trouble.

Where's *your* emergency backup, except to shout at Vodafone and say "it should work, it's been working for years!"?

And personally, we do *not* rely on one manufacturer for PC's, printers, etc. It's just too easy to have a problem with a certain support contract, manufacturer, buggy driver, etc. and get put into an impossible-to-fix-in-time situation. Even our two IT suites (13 computers each, so you can tell we're HUGE) have 13 identical computers but of two different models for that reason. There is a hot spare of each type in my office. Staff laptops are many and varied, for that reason. Printers are many and varied, for that reason (some manufacturers like to sting you on toner / drums if that printer wasn't very popular). Every server is a different manufacturer. We deliberately mix manufacturers in RAID array drives where possible. We even have two unique lines and two different types of fax modems connect to the main Linux server (which has a "hot backup" already imaged and sitting in the next room in the case of a problem) to send / receive faxes over the network.

Redundancy and avoiding monoculture isn't hard. It doesn't mean it will save your arse or cost the world. But saying "what if" to yourself about business-critical things occasionally is always a good idea. Ignore the whole thing about manufacturers and suppliers. Now just say to yourself "what if the phones don't work" and then think of several reasons why that might happen (local tower fails, landline fails, extended power failure in the local area, 9/11-like-incident swamping the phone lines so they are limited to emergency use only, etc.). You don't need to be IBM to be thinking about things like that. It was important enough for you to rant about how it's all Vodafone's fault. But not important enough for you to have ANY backup whatsoever, even in an emergency?

9
0
Silver badge
Happy

Beware the false economy then!

You pays your money and makes you're choice!

I doubt it was your choice to make, more like a dictat from on high but still the same thing applies. The management tell us to cut corners, then when it comes back to bite them in the arse, we all get to carry the can. I love working for a living!

1
0
Anonymous Coward

@Lee Dowling

Just out of interest, and somewhat off-topic, what ADSL load balancer do you use? I have been after a decent one for a while now to give the functionality you mention (a line can go down and nobody notices)

0
0
Badgers

A wild guess...

Based on the rest of his systems, I'm guessing he's using a Linux box to do the load balancing. Its a standard feature, and not at all hard to set up.

0
0
Silver badge

@Anonymous

Slackware 13.1. Has custom kernel with load-balance routing patches from http://www.ssi.bg/~ja/#routes

- Install two (or more) ADSL2+ modems on two Ethernet ports, configured to just push all config to the Ethernet port ("Bridge" mode or whatever your device wants to call it, basically DHCP on the Ethernet cable should prompt the modem to ask for an IP from your ISP and propogate THAT IP and all packets back down the Ethernet cable transparently).

- Install one network card for LAN (Gigabit with Squid proxy and 100Gb disk cache for our setup).

- Do the usual NAT / iptables magic (nothing special at all, haven't modified a single one of my usual firewall / NAT scripts)

- Put in a default route as mentioned on the page for those patches that "nexthop" to the two external Ethernet cards and make sure it's in your startup.

With the patches, you can pull a plug, and everything just continues on as normal, even when you put it back in. Without the patches, your routes disappear and you're into custom scripts to detect failed connections and rebuild routes each time - not so tricky with DHCP but still a royal pain.

And extends to any number of interfaces that present themselves as a network device - 3G stick, PPP Modems, etc. The routing just handles it all.

Only problem is remote access / hosting, which you have to take special account of. Otherwise the machine "chooses" a favoured route for each path and if your remote users don't connect to the right IP that gives that route, you get nothing. Of course, you shouldn't be hosting anything important on an ADSL connection, and I personally give each VPN user *both* IP's and tell them to try them both (it is failover, after all, so one may well be down), others have an iptables / routing exception hard-coded to always accept VPN traffic on only a single given interface.

As a side bonus, you also get a lot more torrent-peers by having two connections....

0
0
Thumb Down

SPoF

Yes it was funny!

<quote>

I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind

</quote>

But

<quote>

Yes, it does... why would a company so clearly reliant on mobile comms have all their staff using a single mobile network??? Surely a mixture of vodafone, o2, t-mobile, orange & 3 would be more appropriate for your company??

</quote>

A mixture wouldn't work! and here is why.

If a large company uses mobile Telecoms, they will stick to one provider because they need to keep costs down, having multiple contracts with different providers wouldn't work as the company would not benefit from free calls to the same network or the (friends and family) business type perks.

0
0
Headmaster

Vodafone - Third biggest

Ranking # of customers

1) "Everything to Everyone" (t-orange)

2) O2

3) Vodafone.

0
0

I'm an engineer, working in the field.

My company provides me with a Vodafone handset for general use, but also an O2 handset for backups (less features, really just for voice calls). If both those fail, then I use my own Orange handset and bill them for expenses. Failing everything else, I have memorised the number for our Fault Control, and will beg, steal or borrow a phone. That also assumes that all the signal post telephones have also failed and I cannot even communicate through a signaller (yes, i work on the railway). So, I have three methods of comms provided by my employer and at least two of my own.

Maybe you should look at your own resilience before blaming everyone else. I agree Voda should have a bit more security and redundancy in their data centres, but your company can't be much better.

0
0
Stop

Hmm

"in order to minimize the ongoing admin and management costs"...

There's your problem. Minimal cost rarely equals maximal service. If mobile comms is business critical for you, choose the option that allows you to run your business in all foreseeable situations, not the one that costs the least.

0
0
FAIL

It seems to have been edited... (the article)

BUT...

Vodafone Group plc (LSE: VOD, NASDAQ: VOD) is a global telecommunications company headquartered in London, United Kingdom.[2] It is the world's largest mobile telecommunications company measured by revenues and the world's second-largest measured by subscribers (behind China Mobile), with around 332 million proportionate subscribers as of 30 September 2010

The Reg did specifically state by profit/revenue not by UK subscriptions... Why don't you be a good little boy read the article and stay off the smack?

thanks.

0
0

Pointing finger...

I'm surprised people are criticising this guy over his SPoF remark. Is a business that procures a load of phones for their employees going to get a mixture from different service providers? Come on, they'll be looking for a bulk purchase deal, standardisation and saving hassle for their accounts department which would have to manage payments to multiple providers. Does your company have different landlines from different providers - no, I didn't think so.

Anyway, I'd be inclined to agree with the OP - you'd expect Voda's systems to have various load balancing & backup systems that should prevent a meltdown like this over such a large customer area. And where was security? Surely, a major telecoms IT centre would have 24-hour security??

0
0
FAIL

Vodafone outage - Data Roaming problems too?

We have employees working abroad who have intermittent or no Data roaming service this morning. Spoke to Vodafone and it is apparently related to a problem at one of their data centres.

0
0
Pint

Data Roaming Restored...

Seems to be fixed for our Roaming workers now.

@Offshoring: Let's hope the Germans have better security!

0
0
Happy

A title

"So i should not use just tcp/ip on the LAN because its a single topology"

TCP/IP is not a topology.

"I shouldn't use one manufacture of Hardware even though they are biggest."

That's right. Use multiple vendors. Don't use something just because it's the biggest, that's a stupid reason to do anything.

"Perhaps I should look at getting the coffee from three different sources in case one stops brewing"

Or just could stockpile coffee and change when you run out? Hardly a comparable analogy, is it?

2
0

South West

The outage is affecting most of the South West as well. None of my users can call me :)

0
0
Pint

my god

you lucky man, a beer to you to help you relax for the upcoming onslaught.

1
0
FAIL

#fail

I'm amazed that any SP's DC would be so prone to criminal attack!!

2
0
FAIL

security?

Presumably given the area affected they'd only have on the order of around a dozen of these data centres across Britain, so can they somehow not afford security staff for them? Given their global profit forecast for the year is in the order of £11.8bn, I think they can afford it!

1
0
Silver badge

Security?

You've got it slightly wrong -- one of the reasons they have such a tasty global forecast is that they've decided security is too expensive.

2
0
Anonymous Coward

You don't post £11.8bn profits

by hiring security guards

4
0

It may have been a deliberate attack, remember

We don't know the motives for the break in, yet, but Vodafone has been the named target of a number of anti-austerity groups like 'UK Uncut' and Socialist Worker. The company's alleged 6 billion pound tax evasion, inan age when ordinary folk are being expected to endure a reduction in lifestyles, is the stated reason for this. A Vodafone shop in Nottingham was occupied recently by UK Uncut, and there have been vague threats from other groups, to cause deliberate damage to Vodafone's business.

It's one thing to have a backup plan about thieves breaking in and accidentally stepping on a few cables while they made of with a server; quite another, to plan for a deliberate act of vandalism.

Like I say, it's just speculation, but I wouldn't regard a building full of servers, as being the typical sort of place to rob.

1
2
Silver badge

@ Daniel 1

It may also have been the Bavarian Illuminati, supported by the Gnomes of Zurich.

This would be a massive change in MO for UK uncut. They're about public demonstrations (hence the retail outlet occupations), not petty vandalism behind the scenes.

1
0
Troll

Tax evasion?

It's not tax evasion when you pay tax on your earnings in country, and pay the tax on earnings earned elsewhere in another place, which may happen to have a far better business climate.

Funny how no-one claimed they were dirty rotters before the election

1
0
Silver badge

Dirty Rotters?

Ask the Greeks about that. I believe Voda has quite a bad name there due to a scandal a few years ago concerning bugging of politician's phone calls...

http://en.wikipedia.org/wiki/Greek_telephone_tapping_case_2004-2005

0
0
Anonymous Coward

RE: Tax evasion?

"Funny how no-one claimed they were dirty rotters before the election"

Private Eye have been quite vocal about Vodafone's tax situation for quite some time. It's just taken the rest of the media a while to catch up.

0
0
Ed
FAIL

Had problems...

Had problems on my journey from Kensington to Victoria down to Croydon and beyond... Very intermittent phone signal and almost non-existent data signal that kept dying.

0
0

Ho Hum

"almost non-existent data signal that kept dying."

That would be my every-day experience of the data network, for me this has been mostly unusable.

0
0
FAIL

Can confirm outage affects more than just SE

People seeing problems in Cornwall/Devon also.

I agree with comments regarding security ... ^ massive fail Vodafone. Single point of failure?

Internal mail:

--------------------

Vodafone Technology Incident Management

Service Operations

.

Incident Update

Source Remedy Reference

Incident

Service Affected 3G, 2G, SMS, Voice, Vodafone One Net - Voice, Paknet, Telephony, Vodafone.co.uk, BlackBerry Connect, Email, Network Mgt, Voicemail, Fixed Link (Voice), Remote Access Service, MMS, Severity P1

Status Open Comms ID:

Start 28/02/11 00:40:00

Incident Statement Link connectivity issues are being experienced to thte Basingstoke Data Centre

Impact Multiple services are impacted. Customers may experience problems making & receiving calls, sending and receiving text messages & establishing data sessions. War room invoked, conf call ongoing to discuss action plan. Crisis Mgmt engaged.

Last Update

Next Update 28/02/11 08:00:00

[Anyone have access to this system and fancy sharing any further updates, please do?]

0
0
Coat

Some translations

"War room invoked" = Brooms and vacuum cleaner relocated to corridor.

"conf call ongoing to discuss action plan" = PR machine ramped up to "full spin". Arses being covered.

"action plan" = Headless chickens sighted in management suite. BOFH and PFY making coffee and filling in overtime claim.

"Crisis Mgmt engaged" - Press release being prepared.

It's the fluorescent yellow one with "Seecureity" on the back.

21
0

Worried

I am quite concerned that damage / loss of service at one single location can cause so many problems for a long amount of time (in communication terms anyway). Does the term System Redundancy mean anything to Vodafone?

1
0

Brave man...

... given the responses to the SPOF comment above. But I agree with you. You would have thought any Enterprise system that is business critical would have resiliency built in.

Clearly Vodafone don't think their networks are business critical. I need to re-read my T&Cs to see how long the buggers have locked me in for. :-(

2
0
Silver badge
Happy

Oh come on!

The upper management can't buy themselves £2M drums on the south coast or top-notch Ferrari's if they have to waste money on providing a decent service to the customers!

0
0
Anonymous Coward

Offshoring

@Sid_Emot: Luckily for you, roaming and non-roaming UK corporate customers will be managed from Ratingen in Germany from later this year, hopefully german sicherheits will be better - not happening quite soon enough though.

0
0
Thumb Down

Pretty far west

I have a site in Taunton and one in Redruth in Cornwall both experiencing no signal.

But only on our corporate account handsets. My personal Voda iPhone is fine.

0
0

Up and down

I'm in Bath and phone shows no signal.

Vodafone 'popped up' for a few minutes about an hour ago and changed the time on my phone (auto network update) to 12.20am!

0
0
WTF?

Customer service?

I managed to speak to one of the staff at the customer service desk around 9.00 am. He told me that it was a software fault, and that it affected a number of parts of the country, inlcuding South West, parts of the Midlands and North as well. I'm not sure if he even knew anything about the breakin.

I did suggest that perhaps they ought to have a page on their website to advise people of outages - but then I note that their web site is flaky as well this morning, so perhaps people are giving it a hammering trying to find out what the hell is going on.

1
0

Re. Software fault.

Yup, it's a software fault. And the fault is that someone has smashed up the hardware that the software needs to run on!

6
0
Silver badge
Joke

@ RichyS

Surely then it isnt really a Software fault - its user error. If there were no users to report the outage, would it really be down?

0
0

Page:

This topic is closed for new posts.

Forums