back to article Vodafone's network knackered by thieves

Millions of Vodafone customers were disconnected this morning after an overnight break-in. Calls, text and data are intermittent west of London with a fix in progress. The failures seem only to be affecting areas just west of London, in the so-called Silicon Corridor. Wales seems fine and the rest of the country is OK, but …


This topic is closed for new posts.


  1. pompurin

    Where was the Data Centre?

    In the middle of a field? I had the mistaken belief that they were holed up in basements about 3 floors underground.

    Did someone leave a window open?

  2. Dpscott

    South West

    The outage is affecting most of the South West as well. None of my users can call me :)

  3. ElbowNi


    Intermittent my Arse... I got a "curtsy call" at 04:15 from ADT to say the alarms PAC (GPS) signal was down, that is obviously when the scroats kicked the door in.

    I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind.

  4. Sid_Emot

    Vodafone outage - Data Roaming problems too?

    We have employees working abroad who have intermittent or no Data roaming service this morning. Spoke to Vodafone and it is apparently related to a problem at one of their data centres.

    1. Anonymous Coward



      I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind


      Yes, it does... why would a company so clearly reliant on mobile comms have all their staff using a single mobile network??? Surely a mixture of vodafone, o2, t-mobile, orange & 3 would be more appropriate for your company??

      When you point a finger at someone, there are 3 pointing back at yourself!

    2. Lee Dowling Silver badge


      "I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind."

      Yeah. You relied on one single mobile communications network.

      1. ElbowNi

        "communications network"

        So i should not use just tcp/ip on the LAN because its a single topology,

        I shouldn't use one manufacture of Hardware even though they are biggest.

        Perhaps I should look at getting the coffee from three different sources in case one stops brewing

        Either you work for VodaFone or a government establishment where cost is not a concern. We use VodaFone, the biggest supplier of mobile networks in the country, in order to minimize the ongoing admin and management costs of maintaining a fleet of phones.,

      2. ElbowNi

        "Surely a mixture"

        Come on join us in the real world where we actually all have four fingers and a thumb.

        1. Anonymous Coward


          Fine, you're right, you should use just the one mobile provider. Must be a big comfort to you being so sure you're right, except you can't phone anyone to tell them.

        2. The Cube

          Really? - Poor planning

          I quite understand that you use one network to control the billing, the cost would be horrific if you split over multiple networks however...

          How much exactly would it have cost you to call O2 and ask for a set of backup PAYG SIM cards for your staff and issue them, along with a secondary phone number list? That way when Vodamuppets bugger up their network you all switch over to the secondary network, given that the problems are rarely complete failures many SMS would get through instructing staff to switch to the backup SIMs. You can always use a web page as a secondary notification.

          I quite understand your situation, we have been without a fixed line Internet connection for over two months thanks to the time wasting lazy tosspots at British Terrorcom, good job we have a Vodafone dongle and a spare router which lets us provide (albeit slower) Internet access to our network for key services such as email.

        3. Chad H.

          @ elbowNI

          Fraid I have to agree with everyone else - running something that important on one network with no apparent back up plan? Perhaps you can use today's downtime to consider how much you'd not lose by having one.

          Suggestions: T and Orange allow cross network roaming on 2G; have a draw full of PAYG sims on various networks for next time and/or network coverage evaluation.

        4. Lee Dowling Silver badge


          Several replied to you, but here would be my response: Actually, I work for schools. You know, those things with zero IT budget and what is there is spent on Visual-Basic-filled-with-clipart "curriclum-vital" software.

          You're complaining about having a single point of failure when you use a single company. If you have Cisco-only gear and Cisco go out of business - you're stuffed. You can say "it won't happen" or "we need to use Cisco" until you're blue in the face. When it goes bust, nobody will care about your problems.

          If an exploit hits a Cisco-only bug (say, the million-and-one IOS bugs), you're stuffed. I'm not saying you shouldn't use Cisco, but you should be prepared for the consequences of such a mono-culture. It's like saying "All Windows PC's are more susceptible to viruses". Completely correct but sometimes you "have to" use Windows. It just means: Don't be surprised when a virus hits and all your Windows stations go down (but not your Linux stations), or all your Cisco IOS gear is exploited (but not your D-Link gear or whatever), or all your engineers are out of reach because all use one mobile phone network. Complaining that MS, Cisco or Vodafone are to blame for not having their own processes to manage their mono-cultures is as silly as you relying on that mono-culture. It may be *practical*, *sensible*, cost-effective, but it's still a mono-culture.

          And thus, you can't push blame to Vodafone for not doing your job. Yeah, you can complain and get service requests and claim on your business support contract (you *DO* have a written legal statement of constant availability from Vodafone, yes?), but nobody can "fix" that problem of you only using one operator because "they are the best" (this incident kinda negates that kind of thinking, I believe). Any one of your people could be using any mixture of any mobile operator without any problems at all and no difference to you except three separate bills instead of one combined. Or, in an emergency, any of them could be instructed to get / issued with a PAYG SIM on another network for the day.

          My school has a dual ADSL2+ line load-balance/failover setup - we get 48MBps of Internet connectivity and can turn off any one modem without affecting the school in any way (who rely on the Internet to fill in exam papers, pay wages, contact parents, etc.). Both lines are linked to our local exchange. We *know* this and don't expect it to operate in the case of an exchange fault - it's to cover *other* eventualities, or else we'd get in a leased line from another exchange deliberately.

          By a similar token, you should NOT expect your business to operate in the case of a mobile operator fault (or even the tower nearest your main site of operations having a fault), by the sound of it. Tell me, what happens if your VOIP/landline telephone system goes haywire, or are you just relying on that to "always work" too? Our actual "emergency" backup for Internet connections is a handful of 3G sticks all on different mobile operators that can run the entire school network just by us plugging them in and running a script (they are always active because we also use them for SMS reception / transmission). Latency goes up, but users continue as if nothing had happened at all. When we hit the traffic limit on one stick, we replace the SIM with some other PAYG with £1/day data. Not perfect, but it works and gets us out of trouble.

          Where's *your* emergency backup, except to shout at Vodafone and say "it should work, it's been working for years!"?

          And personally, we do *not* rely on one manufacturer for PC's, printers, etc. It's just too easy to have a problem with a certain support contract, manufacturer, buggy driver, etc. and get put into an impossible-to-fix-in-time situation. Even our two IT suites (13 computers each, so you can tell we're HUGE) have 13 identical computers but of two different models for that reason. There is a hot spare of each type in my office. Staff laptops are many and varied, for that reason. Printers are many and varied, for that reason (some manufacturers like to sting you on toner / drums if that printer wasn't very popular). Every server is a different manufacturer. We deliberately mix manufacturers in RAID array drives where possible. We even have two unique lines and two different types of fax modems connect to the main Linux server (which has a "hot backup" already imaged and sitting in the next room in the case of a problem) to send / receive faxes over the network.

          Redundancy and avoiding monoculture isn't hard. It doesn't mean it will save your arse or cost the world. But saying "what if" to yourself about business-critical things occasionally is always a good idea. Ignore the whole thing about manufacturers and suppliers. Now just say to yourself "what if the phones don't work" and then think of several reasons why that might happen (local tower fails, landline fails, extended power failure in the local area, 9/11-like-incident swamping the phone lines so they are limited to emergency use only, etc.). You don't need to be IBM to be thinking about things like that. It was important enough for you to rant about how it's all Vodafone's fault. But not important enough for you to have ANY backup whatsoever, even in an emergency?

          1. Anonymous Coward
            Anonymous Coward

            @Lee Dowling

            Just out of interest, and somewhat off-topic, what ADSL load balancer do you use? I have been after a decent one for a while now to give the functionality you mention (a line can go down and nobody notices)

            1. Real Ale is Best

              A wild guess...

              Based on the rest of his systems, I'm guessing he's using a Linux box to do the load balancing. Its a standard feature, and not at all hard to set up.

            2. Lee Dowling Silver badge


              Slackware 13.1. Has custom kernel with load-balance routing patches from

              - Install two (or more) ADSL2+ modems on two Ethernet ports, configured to just push all config to the Ethernet port ("Bridge" mode or whatever your device wants to call it, basically DHCP on the Ethernet cable should prompt the modem to ask for an IP from your ISP and propogate THAT IP and all packets back down the Ethernet cable transparently).

              - Install one network card for LAN (Gigabit with Squid proxy and 100Gb disk cache for our setup).

              - Do the usual NAT / iptables magic (nothing special at all, haven't modified a single one of my usual firewall / NAT scripts)

              - Put in a default route as mentioned on the page for those patches that "nexthop" to the two external Ethernet cards and make sure it's in your startup.

              With the patches, you can pull a plug, and everything just continues on as normal, even when you put it back in. Without the patches, your routes disappear and you're into custom scripts to detect failed connections and rebuild routes each time - not so tricky with DHCP but still a royal pain.

              And extends to any number of interfaces that present themselves as a network device - 3G stick, PPP Modems, etc. The routing just handles it all.

              Only problem is remote access / hosting, which you have to take special account of. Otherwise the machine "chooses" a favoured route for each path and if your remote users don't connect to the right IP that gives that route, you get nothing. Of course, you shouldn't be hosting anything important on an ADSL connection, and I personally give each VPN user *both* IP's and tell them to try them both (it is failover, after all, so one may well be down), others have an iptables / routing exception hard-coded to always accept VPN traffic on only a single given interface.

              As a side bonus, you also get a lot more torrent-peers by having two connections....

        5. The Fuzzy Wotnot

          Beware the false economy then!

          You pays your money and makes you're choice!

          I doubt it was your choice to make, more like a dictat from on high but still the same thing applies. The management tell us to cut corners, then when it comes back to bite them in the arse, we all get to carry the can. I love working for a living!

        6. Anonymous Coward

          Vodafone - Third biggest

          Ranking # of customers

          1) "Everything to Everyone" (t-orange)

          2) O2

          3) Vodafone.

          1. Badbob

            I'm an engineer, working in the field.

            My company provides me with a Vodafone handset for general use, but also an O2 handset for backups (less features, really just for voice calls). If both those fail, then I use my own Orange handset and bill them for expenses. Failing everything else, I have memorised the number for our Fault Control, and will beg, steal or borrow a phone. That also assumes that all the signal post telephones have also failed and I cannot even communicate through a signaller (yes, i work on the railway). So, I have three methods of comms provided by my employer and at least two of my own.

            Maybe you should look at your own resilience before blaming everyone else. I agree Voda should have a bit more security and redundancy in their data centres, but your company can't be much better.

          2. Anonymous Coward

            It seems to have been edited... (the article)


            Vodafone Group plc (LSE: VOD, NASDAQ: VOD) is a global telecommunications company headquartered in London, United Kingdom.[2] It is the world's largest mobile telecommunications company measured by revenues and the world's second-largest measured by subscribers (behind China Mobile), with around 332 million proportionate subscribers as of 30 September 2010

            The Reg did specifically state by profit/revenue not by UK subscriptions... Why don't you be a good little boy read the article and stay off the smack?


        7. Anonymous Coward


          "in order to minimize the ongoing admin and management costs"...

          There's your problem. Minimal cost rarely equals maximal service. If mobile comms is business critical for you, choose the option that allows you to run your business in all foreseeable situations, not the one that costs the least.

      3. Select * From Handle
        Thumb Down


        Yes it was funny!


        I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind




        Yes, it does... why would a company so clearly reliant on mobile comms have all their staff using a single mobile network??? Surely a mixture of vodafone, o2, t-mobile, orange & 3 would be more appropriate for your company??


        A mixture wouldn't work! and here is why.

        If a large company uses mobile Telecoms, they will stick to one provider because they need to keep costs down, having multiple contracts with different providers wouldn't work as the company would not benefit from free calls to the same network or the (friends and family) business type perks.

      4. Bigus34

        Pointing finger...

        I'm surprised people are criticising this guy over his SPoF remark. Is a business that procures a load of phones for their employees going to get a mixture from different service providers? Come on, they'll be looking for a bulk purchase deal, standardisation and saving hassle for their accounts department which would have to manage payments to multiple providers. Does your company have different landlines from different providers - no, I didn't think so.

        Anyway, I'd be inclined to agree with the OP - you'd expect Voda's systems to have various load balancing & backup systems that should prevent a meltdown like this over such a large customer area. And where was security? Surely, a major telecoms IT centre would have 24-hour security??

    3. Anton Ivanov

      No, a few other things come to mind

      Two ways to look at it:

      First and foremost, once upon a time Ofcom (which was called different then) and the mobile operators instituted the no-in-country roaming policy. In those days nobody looked at the mobile phone as a possible single point of contact. That time has clearly passed and having at least mvnos with multi-operator roaming is something that is long overdue. There are businesses that need that level of connectivity nowdays.

      Second, you could have been prepared for it yourself (at least to some extent). For example, I do not give away my mobile (for a couple of years now) and I have a PBX which tries based on time of day, schedule information from my calendar and a few other things my fixed line(s), voip and falls back to a list of mobiles (including foreign ones) when needed. Once the list is exhausted it goes to voicemail which I get as an email as well. It takes me about 1 minute to add or remove a mobile or VOIP endpoint from that list. If you had that, you could have just told your work-force to go and buy a throw-away pay-as-you go from three, O2, T-orange or whoever else you fancy and get on with business as usual. In fact you could have told that to them on their _WORK_ phones and left them to pick up the voicemail via whatever means they find, whenever they get to it.

    4. Anonymous Coward

      my god

      you lucky man, a beer to you to help you relax for the upcoming onslaught.

    5. Jesthar

      First rule of geekery - ALWAYS have a backup

      "I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind."

      Comment from a friend of mine: "At some point last night, Voda realised they should have issued O2 or Orange mobiles to their callout engineers..."

      Another nice slice of irony in our (rather major) Vodaphone-only company, though - we recently opened a new major office, where it was rapidly discovered only the Vodafone network got any decent signal in most of the building, despite perfect service levels on all networks immediately outside. Accusations of network blocking were dismissed with a 'not our problem - you can always change networks' from the brass.

      Now the office has been beautifully devoid of annoying ringtones all morning, and the poor little Blackberry addicted dears are showing serious signs of withdrawal symptoms...

      1. Slartybardfast

        Engineers phones



        I used to work in the telecomms business and it was normal then for engineers to have backup phones on a different network to the one they were working on. I would be surprised and slightly saddened if this had changed.



    6. Anonymous Coward

      A title

      "So i should not use just tcp/ip on the LAN because its a single topology"

      TCP/IP is not a topology.

      "I shouldn't use one manufacture of Hardware even though they are biggest."

      That's right. Use multiple vendors. Don't use something just because it's the biggest, that's a stupid reason to do anything.

      "Perhaps I should look at getting the coffee from three different sources in case one stops brewing"

      Or just could stockpile coffee and change when you run out? Hardly a comparable analogy, is it?

    7. Sid_Emot

      Data Roaming Restored...

      Seems to be fixed for our Roaming workers now.

      @Offshoring: Let's hope the Germans have better security!

  5. dougporter2010


    I'm amazed that any SP's DC would be so prone to criminal attack!!

  6. Tom 260


    Presumably given the area affected they'd only have on the order of around a dozen of these data centres across Britain, so can they somehow not afford security staff for them? Given their global profit forecast for the year is in the order of £11.8bn, I think they can afford it!

  7. Ed

    Had problems...

    Had problems on my journey from Kensington to Victoria down to Croydon and beyond... Very intermittent phone signal and almost non-existent data signal that kept dying.

    1. Anonymous Coward
      Anonymous Coward

      You don't post £11.8bn profits

      by hiring security guards

    2. Elmer Phud


      You've got it slightly wrong -- one of the reasons they have such a tasty global forecast is that they've decided security is too expensive.

    3. Daniel 1

      It may have been a deliberate attack, remember

      We don't know the motives for the break in, yet, but Vodafone has been the named target of a number of anti-austerity groups like 'UK Uncut' and Socialist Worker. The company's alleged 6 billion pound tax evasion, inan age when ordinary folk are being expected to endure a reduction in lifestyles, is the stated reason for this. A Vodafone shop in Nottingham was occupied recently by UK Uncut, and there have been vague threats from other groups, to cause deliberate damage to Vodafone's business.

      It's one thing to have a backup plan about thieves breaking in and accidentally stepping on a few cables while they made of with a server; quite another, to plan for a deliberate act of vandalism.

      Like I say, it's just speculation, but I wouldn't regard a building full of servers, as being the typical sort of place to rob.

      1. Chad H.

        @ Daniel 1

        It may also have been the Bavarian Illuminati, supported by the Gnomes of Zurich.

        This would be a massive change in MO for UK uncut. They're about public demonstrations (hence the retail outlet occupations), not petty vandalism behind the scenes.

      2. David Neil

        Tax evasion?

        It's not tax evasion when you pay tax on your earnings in country, and pay the tax on earnings earned elsewhere in another place, which may happen to have a far better business climate.

        Funny how no-one claimed they were dirty rotters before the election

        1. Loyal Commenter Silver badge

          Dirty Rotters?

          Ask the Greeks about that. I believe Voda has quite a bad name there due to a scandal a few years ago concerning bugging of politician's phone calls...

        2. Anonymous Coward
          Anonymous Coward

          RE: Tax evasion?

          "Funny how no-one claimed they were dirty rotters before the election"

          Private Eye have been quite vocal about Vodafone's tax situation for quite some time. It's just taken the rest of the media a while to catch up.

    4. Miek

      Ho Hum

      "almost non-existent data signal that kept dying."

      That would be my every-day experience of the data network, for me this has been mostly unusable.

  8. some-reg-reader

    Can confirm outage affects more than just SE

    People seeing problems in Cornwall/Devon also.

    I agree with comments regarding security ... ^ massive fail Vodafone. Single point of failure?

    Internal mail:


    Vodafone Technology Incident Management

    Service Operations


    Incident Update

    Source Remedy Reference


    Service Affected 3G, 2G, SMS, Voice, Vodafone One Net - Voice, Paknet, Telephony,, BlackBerry Connect, Email, Network Mgt, Voicemail, Fixed Link (Voice), Remote Access Service, MMS, Severity P1

    Status Open Comms ID:

    Start 28/02/11 00:40:00

    Incident Statement Link connectivity issues are being experienced to thte Basingstoke Data Centre

    Impact Multiple services are impacted. Customers may experience problems making & receiving calls, sending and receiving text messages & establishing data sessions. War room invoked, conf call ongoing to discuss action plan. Crisis Mgmt engaged.

    Last Update

    Next Update 28/02/11 08:00:00

    [Anyone have access to this system and fancy sharing any further updates, please do?]

  9. Jwadmin
    Thumb Down

    Pretty far west

    I have a site in Taunton and one in Redruth in Cornwall both experiencing no signal.

    But only on our corporate account handsets. My personal Voda iPhone is fine.

  10. DRHowells


    I am quite concerned that damage / loss of service at one single location can cause so many problems for a long amount of time (in communication terms anyway). Does the term System Redundancy mean anything to Vodafone?

  11. Tony S

    Customer service?

    I managed to speak to one of the staff at the customer service desk around 9.00 am. He told me that it was a software fault, and that it affected a number of parts of the country, inlcuding South West, parts of the Midlands and North as well. I'm not sure if he even knew anything about the breakin.

    I did suggest that perhaps they ought to have a page on their website to advise people of outages - but then I note that their web site is flaky as well this morning, so perhaps people are giving it a hammering trying to find out what the hell is going on.

  12. Old Boy Si

    Up and down

    I'm in Bath and phone shows no signal.

    Vodafone 'popped up' for a few minutes about an hour ago and changed the time on my phone (auto network update) to 12.20am!

  13. Anonymous Coward
    Anonymous Coward


    @Sid_Emot: Luckily for you, roaming and non-roaming UK corporate customers will be managed from Ratingen in Germany from later this year, hopefully german sicherheits will be better - not happening quite soon enough though.

  14. Tony S


    Just spoken to the people at Vodafone customer support again. They were still being told it was a software problem and were advising customers of that.

    I suggested the guy read El Reg, and whilst he was reading the article, he got an email from his people advising that the problem was due to a break in.

    A company telling their own people one thing, and the media something else. You really couldn't make it up....

    1. RichyS

      Re. Software fault.

      Yup, it's a software fault. And the fault is that someone has smashed up the hardware that the software needs to run on!

    2. RegGuy

      Brave man...

      ... given the responses to the SPOF comment above. But I agree with you. You would have thought any Enterprise system that is business critical would have resiliency built in.

      Clearly Vodafone don't think their networks are business critical. I need to re-read my T&Cs to see how long the buggers have locked me in for. :-(

    3. Blofeld's Cat

      Some translations

      "War room invoked" = Brooms and vacuum cleaner relocated to corridor.

      "conf call ongoing to discuss action plan" = PR machine ramped up to "full spin". Arses being covered.

      "action plan" = Headless chickens sighted in management suite. BOFH and PFY making coffee and filling in overtime claim.

      "Crisis Mgmt engaged" - Press release being prepared.

      It's the fluorescent yellow one with "Seecureity" on the back.

      1. Chad H.

        @ RichyS

        Surely then it isnt really a Software fault - its user error. If there were no users to report the outage, would it really be down?


This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019