back to article Vodafone's network knackered by thieves

Millions of Vodafone customers were disconnected this morning after an overnight break-in. Calls, text and data are intermittent west of London with a fix in progress. The failures seem only to be affecting areas just west of London, in the so-called Silicon Corridor. Wales seems fine and the rest of the country is OK, but …

COMMENTS

This topic is closed for new posts.

Page:

Newsflash!

Just spoken to the people at Vodafone customer support again. They were still being told it was a software problem and were advising customers of that.

I suggested the guy read El Reg, and whilst he was reading the article, he got an email from his people advising that the problem was due to a break in.

A company telling their own people one thing, and the media something else. You really couldn't make it up....

9
0

Let Luxembourg investigate

Old bill should pass investigations onto the Luxembourg police. That's where they pay their taxes after all.

13
0
Silver badge
Big Brother

Never speak against Luxembourg

If you knew what percentage of the population were state-employed bourgeoisie looking forward to vesting pension schemes you would understand why high tax revenue is VITAL for that country.

0
0
Anonymous Coward

So no HA then?

Surely they had another DC to failover too?

I mean what if the DC had burnt to the ground - no services for weeks/months is hardly an acceptable risk for such a company is it.

1
1

No signal in Stokenchurch...

Just after I left 3 after their "No 2g fallback" fiasco... now this.... *sigh*

0
0

In devon

Tried to send some texts last night at about 1am, which failed to go anywhere, Rebooted phone this morning and full signal changed into no signal, so it looks like the fault is reaching quite far west, and i'm near to the devon/cornwall border, and there's not an awful lot west of here.

0
0

Westminster also affected

I'm in Westminster and also affected. I've got full data capabilities on my phone but no voice/text at all.

The ever so helpful callcentre explained they are doing some essential maintenance, yeah right.

PS. To the person pointing out a single point of failure...That is why some organisations pay for dual bearer services ;-) If you need that, then you pay for it

0
0
Anonymous Coward

Internal Email

Good morning to you all,

You may well be aware by now, either from your users, through myself or through the media, that there is currently a major outage on the UK Vodafone network.

Sadly this outage has been caused by the behaviour of a small group of criminals who last night attempted to break into the Vodafone data Centre based in Basingstoke. These individuals failed to breach the building but did manage to cause damage to the site infrastructure, leading to today's issues. That damage has resulted in a loss of service on 5 exchange sites, in turn causing disruption for users from Oxford down to Bristol and across as far as West London.

I would like to assure you of two facts at this time. Firstly, please be assured that there is absolutely no security risk to your business data due to the actions of the afore mentioned individuals. Secondly, the outage is a P1 concern for everyone at Vodafone, from our CEO through to myself and the customer services teams. A P1 category fix is the highest priority and we already have 7 engineers on site with all the hardware they need to execute the corrections. Everyone in Vodafone is on standby to support both you as our customer and to support the network teams in the efforts.

Some users are already restored to full service but others are experiencing anything from a full loss of service to restricted service on voice, SMS, Data sessions or voicemail. It remains our commitment to restore service to all as quickly as possible.

I am in receipt of regular updates from our support teams and I will share this with you as they come in. In the meantime if you could support me in managing your end users with updates I would be very grateful and as always, if you should have any concerns or questions, please do call me (if you can) or E-mail me and I will respond as quickly as I can.

My thanks to you for your patience and my apologies for the issues experienced.

0
0
Silver badge

Seven engineers

They're pushing the boat out for this one...

1
0
FAIL

Vodafone "hiding" internal information

Vodafone just banned me from their eForum for posting the internal information I posted above. They then had the cheek to email me saying:

"Hi,

We’ve removed five of your posts from our eForum today relating to the network outage.

The reason the posts were removed is because the information you’ve received is speculative and has not been confirmed. We are investigating this now and will provide more information as soon as it’s available.

It would be good if you could let us know who you received this message from although we understand if you choose not to.

Please do not re-post this information.

Thanks

eForum Team"

Event happened at 00:40, as confirmed in internal email - yet still no ETA, almost 12 hours later.

http://pastebin.com/GGRi9bHA

2
0
Wol
FAIL

Text and data probs in the North West

Hi,

I'm in the north west and Vodafone SMS is only working intermittently and my TomTom wouldn't get traffic data this morning.

0
0
FAIL

Something isnt right.

How can Vodaphone get away with having no DR for what is probably a cable theft at the data centre?

Why no security on the site to prevent this?

Given the amount the "saved" on tax perhaps they ought to look at the basics of disaster recovery!

0
0

northerner

It's funny how this happened a couple of weeks ago in my local village on the O2 network after thieves robbed parts from a transmiitter, yet there was no news anywhere about it. Is that because we live in an 'unimportant' village in the north of england?

1
1
Pint

tit-le

Isn't this always the case?

If it appens up norf nobody gis a shit!

0
1
Silver badge

Loose the chip

One cell site vs most of southern England.

Get some perspective and loose the chip.

You know why the north gets a hard time? Cos you lot create the expectation in the first place.

4
3
Silver badge

How many people lost service due to this outage?

Was it a few hundred, or half the country? There's a world of difference between a single cell transmitter going down and an etire data centre that appears to serve most of the (heavily populated) South of England. Hint: One is newsworthy, the other is not.

0
0

not really no ..

I think it made the news based on the number of users it affected and the fairly broad distribution of the failures.

0
0
Silver badge

Lousy customer comms

Very poor customer comms by VF.

No network status or link on their main page.

Takes about 4 links or a google to get to their network faults forum which was on posted to with this issue @ 943 am this morning.

Customer care lines engaged with no recorded message detailing the problem.

Since this issue started around 2-3a its a very very poor show. (my wife was feeding the baby and saw it go offline).

I used run some medium critical systems for a bank and I would have been shot if it had taken 7+ hours to get the first customer comms out.

Dont these guys understand basic incident management?

0
0
Anonymous Coward

Hmmm

From the memo, it sounds like the crims damage was to some external device, rather than inside the DC itself.

Given the outage, one would suspect it's the electricity supply thats been damaged in some way, although I would have expected an automated fail-over to an internal diesel generator if that was the case.

If not that, could be the microwave antenna(s) that have been taken out, which could explain the multi-site issue and the difficulty and lack of any fail-over.

0
1
Anonymous Coward

More conjecture

I think you're right. Doesn't sound like the servers in the datacentre were attacked.

On a single site basis they're likely to have dual communications, probably both fixed fibre connections AND microwave to the exchanges in question.

However, if the miscreants failed to break in, it's possible that they've damaged the aircon (high temperatures in the comms room would automatically shutdown the servers inside), they may have triggered the fire suppression system, again, this would have shutdown many of the servers.

Power supply would certainly have a backup, but damage directly to the backup kit can take down both the backup and primary, though that's less likely given it will be just as secure as the servers themselves.

Damage to the aircon systems would naturally take some time to resolve. Bringing servers back-up should be quick, but I've seen instances were certain components don't come back cleanly after the type of shutdown that comes from an instant power failure.

All conjecture.

What I still can't understand is that, if this is an MSC, why the exchanges concerned can't be supplied by an alternate MSC. More importantly, why wouldn't this switch be automatic?

0
0

MVNO too

This is also hitting their MVNO customers, our contact centres are getting hammered from pretty much all our mobile customers.

What's a little concerning is we've just had an email saying their MVNO incident mgt desk is now knackered and we've been given a mobile number as an altwrnative..

0
0

There's possibly also a separate issue...

With RIM. I have a number of directors in Switzerland at present. Those with WinMo/Nokia phones are having no issues at all. Those with Crackberries are having intermittent problems sending mail and accessing t'internet and have been since Saturday morning.

Vodafone told me on Sunday morning that this was due to outage on RIM's side and that this is worldwide.

Any one wish to confirm/deny this?

0
0
FAIL

OXFORD .. no service

although if I cover my phone with tin foil and press send just as I throw it up in the air I can get some strange looks

1
0
Flame

WTF

All the people knocking others for having a single supplier of phones get back in the real world not the world your living in.

I work for a Water utility and we have well over a thousand mobiles in use, plus laptops etc using mobile data for work allocation and emails etc.

On top of that we have hundreds of remote sites that have GSM for comunications.

These are done via Vodaphone main centre not local towers at this end

None of these is dual sim etc so doubling up means massive cost implications for remote telemetry plus all the extra sims for mobiles and laptops etc, thats not countig replacing sims.

I'm sure you would be happy to have your water bill raised to cover this cost for something that has only happened once in the 5 years we have used this system.

Come up with a lox/cost free option and fine, if not then just get back to the real world before you post such rubbish about spreading the risk.

After all do you all have Apple Macs on your desks just in case a windows virus infects everything and requires a few hours to fix.....

4
0
Anonymous Coward

Pah...

That's nothing, we use Networker at work to do our primary backups, then we go back and do it all again with NetBackup, just in case there is a problem with one or other piece of software.

0
0
Anonymous Coward

knockers

Its not a question of knocking people who have decided to use a single supplier - this started with someone who had made that choice then started complaining when that supplier let them down. As you say, using one supplier can be a perfectly rational decision arrived at by balancing convenience and cost savings vs risk of failure and costs of that failure. Having made the decision, however, one is expected to live with it (unless of course one is a politician or bank director).

1
0
Silver badge
FAIL

Low cost or free DR option?

Get some FREE PAYG SIMs from another operator (IIRC Orange and O2 do them for free). Issue these to your staff. If and when your main comms supplier fails, top these up with credit and ask your staff to swap the SIMs in their phones.

If necessary, you may have to get your provider to network-unlock the handsets. They will usually do this for a small fee, or for free depending on the phase of the moon or some such similiar thing. Even if you end up having to pay this, it is a small amount compared to the monthly contract cost.

I have a spare old handset at home with a PAYG SIM in it for exactly this purpose. I would suggest that your only difficulty is in the lack of willingness to consider your DR requirements in the first place, i.e. laziness.

0
1
Thumb Down

fail.......

And how are we suposed to do that for 7-800 remote sites on the tops of hills etc ?

Plus what about all those who forgoot where the sim is of lost it in the last 5 years when not needed......

Its easy to come up with solutions when just one office and a few people but not when spread over sveral counties involving more than a thousand phones plus several hundred laptops etc for data.

Plus PAYG sims don't do data much do they ?

0
0
Thumb Down

another fail

How does networker handle people with laptops working in the middle of no-where ?

These people connect via ethernet maybe once a quarter.....

0
0
WTF?

@ Big Ted

Since when don't PAYG sims do data ?- you can get PAYG Mbb sims from every major provider. mobilebroadbandaccess.o2.co.uk

0
0
Anonymous Coward

@Big Ted

That would have been a joke, ie: I was saying that we make two backups with different software when obviously noone does that.

As it happens, though, NetBackup does have a laptop backup option, I don't think Networker does though.

0
0
Unhappy

looks like O2 is affected too..

I am located not too far from Oxford and i have notice seriously bad sound reception on O2 since this morning. It usually works great.

As O2 and Vodafone have been sharing their network for a while it looks like this outage explains it...

0
1
Unhappy

No

Vodafone and O2 do not share a network. They have an agreement to share sites, Air conditioners and power supplies. Each network puts their own equipment into each site, independent of the others, and connect to their own internal network.

There is no overlap - you'll need to find something else to blame I'm afraid.

0
0
WTF?

Vodafone talking rubbish

Just been reading an FAQ about the outage on their forum and saw this:

Q. Have you experienced anything like this in the past?

A. Nothing of this nature.

BS - I remember on the 3rd of August a similar thing happened, a break in at one of their DC's flattened their entire Pager network for the whole day.

http://www.theregister.co.uk/2005/08/03/vodafone_pager/

0
0
Anonymous Coward

Err...

That story was from 2005, their pager network going out is very different to a full mobile comms outage, also I don't believe they even still run pager services.

They were probably not refering to the break in as the problem, rather the problems that the break in have caused.

0
0
Anonymous Coward

Not anymore

Vodafone UK sold their pager network a few years ago. Regardless, the pager network is a very different system.

Interestingly, many of the core network management folk do use the pager network simply because if the network is down, mobile phones aren't the best for informing you.

There's also a historic problem that mobile signal was far from 100% even at the best of times so it's possible the engineer lived out of range. That's less likely these days, but pagers are still used in conjunction with automatic alert systems.

0
0
Anonymous Coward

M4 corridor - Norwich?

not able to register all day - either voda are as bad at geography as they are at generally providing service

no signal or even able to register phone on network all day

0
0
Alert

Ask the Security Gaurds?

Every major corporate data centre I've walked into in recent years has been equipped with 24 hour security guards, blanket CCTV, and alarms.

There are normally several cordons of physical security... including alarmed security doors with keypads or RFID tags, and even steel cages around the most sensitive/critical infrastructure.

In short, penetrating into such a facility unnoticed, and causing serious damage, is almost impossible for a casual sneak thief to achieve.

So how on earth could someone break into a Vodafone data centre, apparently unobserved, and cause extensive damage?

The explanations being given by Vodafone at present don't sound very plausible to me.

0
0
Boffin

Blah Blah

The site has redundant capabilities with as correctly stated fibre and microwave transmitters, the issue comes from its almost impossible to expect a entire DC to fall over, if telehouse north went down the ENTIRE UK would lose internet, its not possible to predict this kind of incident, the datacentre in question is guarded by security doors and is in a secret location known by some locals but with one route in and one route out its not a manned site.

The breakin resulted in two outcomes depending on which information is believed a HLR was stolen or nothing was stolen as they failed to break in.

Depending on the methods used to get in numerous protection systems would have been activated some of these will include data destruction to protect CUSTOMERS data - as with most major companies in the event of a breakin data is logically scrambled and or physically destroyed the restoration of this data will take time, if the fire suppression system went of they will have to wait for the chemicals to dissipate (being a unmanned site they will use extreme protection systems which literally remove oxygen in a matter of seconds using very powerful gasses)

chill its not possible to predict this all the backups in the world wont help if a DC falls over unless you expect vodafone to spend some of the 12bn profit on building a redundant site for every DC they own (at a cost of at least 3bn per site!?)

get a grip, if you are failing then YOU need failover, i have a secondary mobile and redirection to a SIP number which i can access via my phone that cost me less than £10 to set up a year ago.

Just wait it will come back up.

0
0
FAIL

"a secret location known by some locals"

Is that the same secret location revealed by Googling Vodafone Basingstoke?

That leads me to a planning application on the basingstoke.gov.uk site with an address and details of the site?

Address details that you can plug into Google Streetview, to discover it is (in effect) a small shed on the corner of an industrial estate?

1
0
Oz
Megaphone

Secret location (not any more)

... and the Daily Fail's article on the break-in has a picture of the site, and more details of the equipment stolen.

0
0

Back up in basingstoke

Came back about 20 mins ago no data though just voice sms is working but is being very slow.

0
0
Thumb Down

Still flaky this evening... despite what they say!

Still getting call drop outs and instant missed calls even at home where we *normally* get great coverage... (18:51)

0
0
FAIL

I guess their update email should have been BCC'd

Rather than cc:ed to all and sundry.

I sent an email of remonstration to the Vodafone contact on the Info Email.

Somewhat (un) amusingly I got the following response.

"I'm currently out of the office with no access to emails or phone.

For any other queries while I'm out of the business, please contact the Vodafone Managed Services helpdesk on 08700 777 533. Otherwise, I will contact you upon my return."

Hey like the rest of us on Vodafone then!!!!

0
0
WTF?

Good signal

The really weird thing is that while all around me, my fellow Vodafoners have been complaining about no service, I've been getting much better signal all day. I'm on Vodafone and here in Oxford, get virtually no signal in my basement office and rarely get any 3G when outdoors in the heart of the city. Today though, I've been getting 2-3 bars in the office and a 3G signal when out in town at lunchtime. Haven't tried making any calls to be fair, but have definitely been receiving data.

0
0
Anonymous Coward

Like an onion, security is made of layers

You can have physical security with fences, gates, CCTV, alarms, Robotic guard dogs etc. but there is always the human element.

Bank robbers don't patiently queue up and politely ask the teller if they wouldn't mind filling the bag they brought with cash. They carry guns and kidnap the managers family.

On-site guards are vulnerable in exactly the same way.

While there's no information that this was the case, it could the criminals simply threatened the guard into letting them in. If a villain in a balaclava says "Open up or I'll bash you head in with my baseball bat", who here would say No"?

0
0
Thumb Down

@ AC

'@Sid_Emot: Luckily for you, roaming and non-roaming UK corporate customers will be managed from Ratingen in Germany from later this year, hopefully german sicherheits will be better - not happening quite soon enough though.'

That's news to me, perhaps I'll get my redundancy letter real soon considering corp and public sector are managed out of sunny Wiltshire.

0
0
Unhappy

I think O2 had problems yesterday as well

Yesterday (28 Feb) I kept getting silence for ages and then cut off when I rang O2 call centres on 08448090202.

About 50% of my calls were dropped during most of the afternoon.

0
0

Page:

This topic is closed for new posts.

Forums

Biting the hand that feeds IT © 1998–2018