back to article Vodafone's network knackered by thieves

Millions of Vodafone customers were disconnected this morning after an overnight break-in. Calls, text and data are intermittent west of London with a fix in progress. The failures seem only to be affecting areas just west of London, in the so-called Silicon Corridor. Wales seems fine and the rest of the country is OK, but …

COMMENTS

This topic is closed for new posts.
  1. pompurin
    FAIL

    Where was the Data Centre?

    In the middle of a field? I had the mistaken belief that they were holed up in basements about 3 floors underground.

    Did someone leave a window open?

  2. Dpscott

    South West

    The outage is affecting most of the South West as well. None of my users can call me :)

    1. Anonymous Coward
      Pint

      my god

      you lucky man, a beer to you to help you relax for the upcoming onslaught.

  3. ElbowNi
    WTF?

    Intermittent

    Intermittent my Arse... I got a "curtsy call" at 04:15 from ADT to say the alarms PAC (GPS) signal was down, that is obviously when the scroats kicked the door in.

    I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind.

    1. Anonymous Coward
      FAIL

      SPoF

      <quote>

      I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind

      </quote>

      Yes, it does... why would a company so clearly reliant on mobile comms have all their staff using a single mobile network??? Surely a mixture of vodafone, o2, t-mobile, orange & 3 would be more appropriate for your company??

      When you point a finger at someone, there are 3 pointing back at yourself!

      1. ElbowNi

        "Surely a mixture"

        Come on join us in the real world where we actually all have four fingers and a thumb.

        1. Anonymous Coward
          Pint

          @ElbowNi

          Fine, you're right, you should use just the one mobile provider. Must be a big comfort to you being so sure you're right, except you can't phone anyone to tell them.

        2. Chad H.

          @ elbowNI

          Fraid I have to agree with everyone else - running something that important on one network with no apparent back up plan? Perhaps you can use today's downtime to consider how much you'd not lose by having one.

          Suggestions: T and Orange allow cross network roaming on 2G; have a draw full of PAYG sims on various networks for next time and/or network coverage evaluation.

      2. Select * From Handle
        Thumb Down

        SPoF

        Yes it was funny!

        <quote>

        I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind

        </quote>

        But

        <quote>

        Yes, it does... why would a company so clearly reliant on mobile comms have all their staff using a single mobile network??? Surely a mixture of vodafone, o2, t-mobile, orange & 3 would be more appropriate for your company??

        </quote>

        A mixture wouldn't work! and here is why.

        If a large company uses mobile Telecoms, they will stick to one provider because they need to keep costs down, having multiple contracts with different providers wouldn't work as the company would not benefit from free calls to the same network or the (friends and family) business type perks.

      3. Bigus34

        Pointing finger...

        I'm surprised people are criticising this guy over his SPoF remark. Is a business that procures a load of phones for their employees going to get a mixture from different service providers? Come on, they'll be looking for a bulk purchase deal, standardisation and saving hassle for their accounts department which would have to manage payments to multiple providers. Does your company have different landlines from different providers - no, I didn't think so.

        Anyway, I'd be inclined to agree with the OP - you'd expect Voda's systems to have various load balancing & backup systems that should prevent a meltdown like this over such a large customer area. And where was security? Surely, a major telecoms IT centre would have 24-hour security??

    2. Lee Dowling Silver badge
      WTF?

      Erm...

      "I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind."

      Yeah. You relied on one single mobile communications network.

      1. ElbowNi

        "communications network"

        So i should not use just tcp/ip on the LAN because its a single topology,

        I shouldn't use one manufacture of Hardware even though they are biggest.

        Perhaps I should look at getting the coffee from three different sources in case one stops brewing

        Either you work for VodaFone or a government establishment where cost is not a concern. We use VodaFone, the biggest supplier of mobile networks in the country, in order to minimize the ongoing admin and management costs of maintaining a fleet of phones.,

        1. The Cube
          FAIL

          Really? - Poor planning

          I quite understand that you use one network to control the billing, the cost would be horrific if you split over multiple networks however...

          How much exactly would it have cost you to call O2 and ask for a set of backup PAYG SIM cards for your staff and issue them, along with a secondary phone number list? That way when Vodamuppets bugger up their network you all switch over to the secondary network, given that the problems are rarely complete failures many SMS would get through instructing staff to switch to the backup SIMs. You can always use a web page as a secondary notification.

          I quite understand your situation, we have been without a fixed line Internet connection for over two months thanks to the time wasting lazy tosspots at British Terrorcom, good job we have a Vodafone dongle and a spare router which lets us provide (albeit slower) Internet access to our network for key services such as email.

        2. Lee Dowling Silver badge
          FAIL

          Er

          Several replied to you, but here would be my response: Actually, I work for schools. You know, those things with zero IT budget and what is there is spent on Visual-Basic-filled-with-clipart "curriclum-vital" software.

          You're complaining about having a single point of failure when you use a single company. If you have Cisco-only gear and Cisco go out of business - you're stuffed. You can say "it won't happen" or "we need to use Cisco" until you're blue in the face. When it goes bust, nobody will care about your problems.

          If an exploit hits a Cisco-only bug (say, the million-and-one IOS bugs), you're stuffed. I'm not saying you shouldn't use Cisco, but you should be prepared for the consequences of such a mono-culture. It's like saying "All Windows PC's are more susceptible to viruses". Completely correct but sometimes you "have to" use Windows. It just means: Don't be surprised when a virus hits and all your Windows stations go down (but not your Linux stations), or all your Cisco IOS gear is exploited (but not your D-Link gear or whatever), or all your engineers are out of reach because all use one mobile phone network. Complaining that MS, Cisco or Vodafone are to blame for not having their own processes to manage their mono-cultures is as silly as you relying on that mono-culture. It may be *practical*, *sensible*, cost-effective, but it's still a mono-culture.

          And thus, you can't push blame to Vodafone for not doing your job. Yeah, you can complain and get service requests and claim on your business support contract (you *DO* have a written legal statement of constant availability from Vodafone, yes?), but nobody can "fix" that problem of you only using one operator because "they are the best" (this incident kinda negates that kind of thinking, I believe). Any one of your people could be using any mixture of any mobile operator without any problems at all and no difference to you except three separate bills instead of one combined. Or, in an emergency, any of them could be instructed to get / issued with a PAYG SIM on another network for the day.

          My school has a dual ADSL2+ line load-balance/failover setup - we get 48MBps of Internet connectivity and can turn off any one modem without affecting the school in any way (who rely on the Internet to fill in exam papers, pay wages, contact parents, etc.). Both lines are linked to our local exchange. We *know* this and don't expect it to operate in the case of an exchange fault - it's to cover *other* eventualities, or else we'd get in a leased line from another exchange deliberately.

          By a similar token, you should NOT expect your business to operate in the case of a mobile operator fault (or even the tower nearest your main site of operations having a fault), by the sound of it. Tell me, what happens if your VOIP/landline telephone system goes haywire, or are you just relying on that to "always work" too? Our actual "emergency" backup for Internet connections is a handful of 3G sticks all on different mobile operators that can run the entire school network just by us plugging them in and running a script (they are always active because we also use them for SMS reception / transmission). Latency goes up, but users continue as if nothing had happened at all. When we hit the traffic limit on one stick, we replace the SIM with some other PAYG with £1/day data. Not perfect, but it works and gets us out of trouble.

          Where's *your* emergency backup, except to shout at Vodafone and say "it should work, it's been working for years!"?

          And personally, we do *not* rely on one manufacturer for PC's, printers, etc. It's just too easy to have a problem with a certain support contract, manufacturer, buggy driver, etc. and get put into an impossible-to-fix-in-time situation. Even our two IT suites (13 computers each, so you can tell we're HUGE) have 13 identical computers but of two different models for that reason. There is a hot spare of each type in my office. Staff laptops are many and varied, for that reason. Printers are many and varied, for that reason (some manufacturers like to sting you on toner / drums if that printer wasn't very popular). Every server is a different manufacturer. We deliberately mix manufacturers in RAID array drives where possible. We even have two unique lines and two different types of fax modems connect to the main Linux server (which has a "hot backup" already imaged and sitting in the next room in the case of a problem) to send / receive faxes over the network.

          Redundancy and avoiding monoculture isn't hard. It doesn't mean it will save your arse or cost the world. But saying "what if" to yourself about business-critical things occasionally is always a good idea. Ignore the whole thing about manufacturers and suppliers. Now just say to yourself "what if the phones don't work" and then think of several reasons why that might happen (local tower fails, landline fails, extended power failure in the local area, 9/11-like-incident swamping the phone lines so they are limited to emergency use only, etc.). You don't need to be IBM to be thinking about things like that. It was important enough for you to rant about how it's all Vodafone's fault. But not important enough for you to have ANY backup whatsoever, even in an emergency?

          1. Anonymous Coward
            Anonymous Coward

            @Lee Dowling

            Just out of interest, and somewhat off-topic, what ADSL load balancer do you use? I have been after a decent one for a while now to give the functionality you mention (a line can go down and nobody notices)

            1. Real Ale is Best
              Badgers

              A wild guess...

              Based on the rest of his systems, I'm guessing he's using a Linux box to do the load balancing. Its a standard feature, and not at all hard to set up.

            2. Lee Dowling Silver badge

              @Anonymous

              Slackware 13.1. Has custom kernel with load-balance routing patches from http://www.ssi.bg/~ja/#routes

              - Install two (or more) ADSL2+ modems on two Ethernet ports, configured to just push all config to the Ethernet port ("Bridge" mode or whatever your device wants to call it, basically DHCP on the Ethernet cable should prompt the modem to ask for an IP from your ISP and propogate THAT IP and all packets back down the Ethernet cable transparently).

              - Install one network card for LAN (Gigabit with Squid proxy and 100Gb disk cache for our setup).

              - Do the usual NAT / iptables magic (nothing special at all, haven't modified a single one of my usual firewall / NAT scripts)

              - Put in a default route as mentioned on the page for those patches that "nexthop" to the two external Ethernet cards and make sure it's in your startup.

              With the patches, you can pull a plug, and everything just continues on as normal, even when you put it back in. Without the patches, your routes disappear and you're into custom scripts to detect failed connections and rebuild routes each time - not so tricky with DHCP but still a royal pain.

              And extends to any number of interfaces that present themselves as a network device - 3G stick, PPP Modems, etc. The routing just handles it all.

              Only problem is remote access / hosting, which you have to take special account of. Otherwise the machine "chooses" a favoured route for each path and if your remote users don't connect to the right IP that gives that route, you get nothing. Of course, you shouldn't be hosting anything important on an ADSL connection, and I personally give each VPN user *both* IP's and tell them to try them both (it is failover, after all, so one may well be down), others have an iptables / routing exception hard-coded to always accept VPN traffic on only a single given interface.

              As a side bonus, you also get a lot more torrent-peers by having two connections....

        3. The Fuzzy Wotnot
          Happy

          Beware the false economy then!

          You pays your money and makes you're choice!

          I doubt it was your choice to make, more like a dictat from on high but still the same thing applies. The management tell us to cut corners, then when it comes back to bite them in the arse, we all get to carry the can. I love working for a living!

        4. Anonymous Coward
          Headmaster

          Vodafone - Third biggest

          Ranking # of customers

          1) "Everything to Everyone" (t-orange)

          2) O2

          3) Vodafone.

          1. Badbob

            I'm an engineer, working in the field.

            My company provides me with a Vodafone handset for general use, but also an O2 handset for backups (less features, really just for voice calls). If both those fail, then I use my own Orange handset and bill them for expenses. Failing everything else, I have memorised the number for our Fault Control, and will beg, steal or borrow a phone. That also assumes that all the signal post telephones have also failed and I cannot even communicate through a signaller (yes, i work on the railway). So, I have three methods of comms provided by my employer and at least two of my own.

            Maybe you should look at your own resilience before blaming everyone else. I agree Voda should have a bit more security and redundancy in their data centres, but your company can't be much better.

          2. Anonymous Coward
            FAIL

            It seems to have been edited... (the article)

            BUT...

            Vodafone Group plc (LSE: VOD, NASDAQ: VOD) is a global telecommunications company headquartered in London, United Kingdom.[2] It is the world's largest mobile telecommunications company measured by revenues and the world's second-largest measured by subscribers (behind China Mobile), with around 332 million proportionate subscribers as of 30 September 2010

            The Reg did specifically state by profit/revenue not by UK subscriptions... Why don't you be a good little boy read the article and stay off the smack?

            thanks.

        5. Anonymous Coward
          Stop

          Hmm

          "in order to minimize the ongoing admin and management costs"...

          There's your problem. Minimal cost rarely equals maximal service. If mobile comms is business critical for you, choose the option that allows you to run your business in all foreseeable situations, not the one that costs the least.

    3. Anton Ivanov

      No, a few other things come to mind

      Two ways to look at it:

      First and foremost, once upon a time Ofcom (which was called different then) and the mobile operators instituted the no-in-country roaming policy. In those days nobody looked at the mobile phone as a possible single point of contact. That time has clearly passed and having at least mvnos with multi-operator roaming is something that is long overdue. There are businesses that need that level of connectivity nowdays.

      Second, you could have been prepared for it yourself (at least to some extent). For example, I do not give away my mobile (for a couple of years now) and I have a PBX which tries based on time of day, schedule information from my calendar and a few other things my fixed line(s), voip and falls back to a list of mobiles (including foreign ones) when needed. Once the list is exhausted it goes to voicemail which I get as an email as well. It takes me about 1 minute to add or remove a mobile or VOIP endpoint from that list. If you had that, you could have just told your work-force to go and buy a throw-away pay-as-you go from three, O2, T-orange or whoever else you fancy and get on with business as usual. In fact you could have told that to them on their _WORK_ phones and left them to pick up the voicemail via whatever means they find, whenever they get to it.

    4. Jesthar
      Pirate

      First rule of geekery - ALWAYS have a backup

      "I now have an entire work force with no mobile communications. Does the phrase "single point of failure" spring to mind."

      Comment from a friend of mine: "At some point last night, Voda realised they should have issued O2 or Orange mobiles to their callout engineers..."

      Another nice slice of irony in our (rather major) Vodaphone-only company, though - we recently opened a new major office, where it was rapidly discovered only the Vodafone network got any decent signal in most of the building, despite perfect service levels on all networks immediately outside. Accusations of network blocking were dismissed with a 'not our problem - you can always change networks' from the brass.

      Now the office has been beautifully devoid of annoying ringtones all morning, and the poor little Blackberry addicted dears are showing serious signs of withdrawal symptoms...

      1. Slartybardfast

        Engineers phones

        .

        .

        I used to work in the telecomms business and it was normal then for engineers to have backup phones on a different network to the one they were working on. I would be surprised and slightly saddened if this had changed.

        .

        .

  4. Sid_Emot
    FAIL

    Vodafone outage - Data Roaming problems too?

    We have employees working abroad who have intermittent or no Data roaming service this morning. Spoke to Vodafone and it is apparently related to a problem at one of their data centres.

    1. Anonymous Coward
      Happy

      A title

      "So i should not use just tcp/ip on the LAN because its a single topology"

      TCP/IP is not a topology.

      "I shouldn't use one manufacture of Hardware even though they are biggest."

      That's right. Use multiple vendors. Don't use something just because it's the biggest, that's a stupid reason to do anything.

      "Perhaps I should look at getting the coffee from three different sources in case one stops brewing"

      Or just could stockpile coffee and change when you run out? Hardly a comparable analogy, is it?

    2. Sid_Emot
      Pint

      Data Roaming Restored...

      Seems to be fixed for our Roaming workers now.

      @Offshoring: Let's hope the Germans have better security!

  5. dougporter2010
    FAIL

    #fail

    I'm amazed that any SP's DC would be so prone to criminal attack!!

  6. Tom 260
    FAIL

    security?

    Presumably given the area affected they'd only have on the order of around a dozen of these data centres across Britain, so can they somehow not afford security staff for them? Given their global profit forecast for the year is in the order of £11.8bn, I think they can afford it!

    1. Anonymous Coward
      Anonymous Coward

      You don't post £11.8bn profits

      by hiring security guards

    2. Elmer Phud

      Security?

      You've got it slightly wrong -- one of the reasons they have such a tasty global forecast is that they've decided security is too expensive.

    3. Daniel 1

      It may have been a deliberate attack, remember

      We don't know the motives for the break in, yet, but Vodafone has been the named target of a number of anti-austerity groups like 'UK Uncut' and Socialist Worker. The company's alleged 6 billion pound tax evasion, inan age when ordinary folk are being expected to endure a reduction in lifestyles, is the stated reason for this. A Vodafone shop in Nottingham was occupied recently by UK Uncut, and there have been vague threats from other groups, to cause deliberate damage to Vodafone's business.

      It's one thing to have a backup plan about thieves breaking in and accidentally stepping on a few cables while they made of with a server; quite another, to plan for a deliberate act of vandalism.

      Like I say, it's just speculation, but I wouldn't regard a building full of servers, as being the typical sort of place to rob.

      1. Chad H.

        @ Daniel 1

        It may also have been the Bavarian Illuminati, supported by the Gnomes of Zurich.

        This would be a massive change in MO for UK uncut. They're about public demonstrations (hence the retail outlet occupations), not petty vandalism behind the scenes.

      2. David Neil
        Troll

        Tax evasion?

        It's not tax evasion when you pay tax on your earnings in country, and pay the tax on earnings earned elsewhere in another place, which may happen to have a far better business climate.

        Funny how no-one claimed they were dirty rotters before the election

        1. Loyal Commenter Silver badge

          Dirty Rotters?

          Ask the Greeks about that. I believe Voda has quite a bad name there due to a scandal a few years ago concerning bugging of politician's phone calls...

          http://en.wikipedia.org/wiki/Greek_telephone_tapping_case_2004-2005

        2. Anonymous Coward
          Anonymous Coward

          RE: Tax evasion?

          "Funny how no-one claimed they were dirty rotters before the election"

          Private Eye have been quite vocal about Vodafone's tax situation for quite some time. It's just taken the rest of the media a while to catch up.

  7. Ed
    FAIL

    Had problems...

    Had problems on my journey from Kensington to Victoria down to Croydon and beyond... Very intermittent phone signal and almost non-existent data signal that kept dying.

    1. Miek

      Ho Hum

      "almost non-existent data signal that kept dying."

      That would be my every-day experience of the data network, for me this has been mostly unusable.

  8. some-reg-reader
    FAIL

    Can confirm outage affects more than just SE

    People seeing problems in Cornwall/Devon also.

    I agree with comments regarding security ... ^ massive fail Vodafone. Single point of failure?

    Internal mail:

    --------------------

    Vodafone Technology Incident Management

    Service Operations

    .

    Incident Update

    Source Remedy Reference

    Incident

    Service Affected 3G, 2G, SMS, Voice, Vodafone One Net - Voice, Paknet, Telephony, Vodafone.co.uk, BlackBerry Connect, Email, Network Mgt, Voicemail, Fixed Link (Voice), Remote Access Service, MMS, Severity P1

    Status Open Comms ID:

    Start 28/02/11 00:40:00

    Incident Statement Link connectivity issues are being experienced to thte Basingstoke Data Centre

    Impact Multiple services are impacted. Customers may experience problems making & receiving calls, sending and receiving text messages & establishing data sessions. War room invoked, conf call ongoing to discuss action plan. Crisis Mgmt engaged.

    Last Update

    Next Update 28/02/11 08:00:00

    [Anyone have access to this system and fancy sharing any further updates, please do?]

    1. Blofeld's Cat
      Coat

      Some translations

      "War room invoked" = Brooms and vacuum cleaner relocated to corridor.

      "conf call ongoing to discuss action plan" = PR machine ramped up to "full spin". Arses being covered.

      "action plan" = Headless chickens sighted in management suite. BOFH and PFY making coffee and filling in overtime claim.

      "Crisis Mgmt engaged" - Press release being prepared.

      It's the fluorescent yellow one with "Seecureity" on the back.

  9. Jwadmin
    Thumb Down

    Pretty far west

    I have a site in Taunton and one in Redruth in Cornwall both experiencing no signal.

    But only on our corporate account handsets. My personal Voda iPhone is fine.

  10. DRHowells

    Worried

    I am quite concerned that damage / loss of service at one single location can cause so many problems for a long amount of time (in communication terms anyway). Does the term System Redundancy mean anything to Vodafone?

    1. RegGuy

      Brave man...

      ... given the responses to the SPOF comment above. But I agree with you. You would have thought any Enterprise system that is business critical would have resiliency built in.

      Clearly Vodafone don't think their networks are business critical. I need to re-read my T&Cs to see how long the buggers have locked me in for. :-(

    2. The Fuzzy Wotnot
      Happy

      Oh come on!

      The upper management can't buy themselves £2M drums on the south coast or top-notch Ferrari's if they have to waste money on providing a decent service to the customers!

  11. Tony S
    WTF?

    Customer service?

    I managed to speak to one of the staff at the customer service desk around 9.00 am. He told me that it was a software fault, and that it affected a number of parts of the country, inlcuding South West, parts of the Midlands and North as well. I'm not sure if he even knew anything about the breakin.

    I did suggest that perhaps they ought to have a page on their website to advise people of outages - but then I note that their web site is flaky as well this morning, so perhaps people are giving it a hammering trying to find out what the hell is going on.

    1. RichyS

      Re. Software fault.

      Yup, it's a software fault. And the fault is that someone has smashed up the hardware that the software needs to run on!

      1. Chad H.
        Joke

        @ RichyS

        Surely then it isnt really a Software fault - its user error. If there were no users to report the outage, would it really be down?

  12. Old Boy Si

    Up and down

    I'm in Bath and phone shows no signal.

    Vodafone 'popped up' for a few minutes about an hour ago and changed the time on my phone (auto network update) to 12.20am!

  13. Anonymous Coward
    Anonymous Coward

    Offshoring

    @Sid_Emot: Luckily for you, roaming and non-roaming UK corporate customers will be managed from Ratingen in Germany from later this year, hopefully german sicherheits will be better - not happening quite soon enough though.

  14. Tony S

    Newsflash!

    Just spoken to the people at Vodafone customer support again. They were still being told it was a software problem and were advising customers of that.

    I suggested the guy read El Reg, and whilst he was reading the article, he got an email from his people advising that the problem was due to a break in.

    A company telling their own people one thing, and the media something else. You really couldn't make it up....

  15. sagederby

    Let Luxembourg investigate

    Old bill should pass investigations onto the Luxembourg police. That's where they pay their taxes after all.

    1. Destroy All Monsters Silver badge
      Big Brother

      Never speak against Luxembourg

      If you knew what percentage of the population were state-employed bourgeoisie looking forward to vesting pension schemes you would understand why high tax revenue is VITAL for that country.

  16. Jean-Paul

    Westminster also affected

    I'm in Westminster and also affected. I've got full data capabilities on my phone but no voice/text at all.

    The ever so helpful callcentre explained they are doing some essential maintenance, yeah right.

    PS. To the person pointing out a single point of failure...That is why some organisations pay for dual bearer services ;-) If you need that, then you pay for it

  17. Simon Westerby 1

    No signal in Stokenchurch...

    Just after I left 3 after their "No 2g fallback" fiasco... now this.... *sigh*

  18. Anonymous Coward
    Anonymous Coward

    So no HA then?

    Surely they had another DC to failover too?

    I mean what if the DC had burnt to the ground - no services for weeks/months is hardly an acceptable risk for such a company is it.

  19. Stewart Atkins

    In devon

    Tried to send some texts last night at about 1am, which failed to go anywhere, Rebooted phone this morning and full signal changed into no signal, so it looks like the fault is reaching quite far west, and i'm near to the devon/cornwall border, and there's not an awful lot west of here.

  20. Anonymous Coward
    Anonymous Coward

    Internal Email

    Good morning to you all,

    You may well be aware by now, either from your users, through myself or through the media, that there is currently a major outage on the UK Vodafone network.

    Sadly this outage has been caused by the behaviour of a small group of criminals who last night attempted to break into the Vodafone data Centre based in Basingstoke. These individuals failed to breach the building but did manage to cause damage to the site infrastructure, leading to today's issues. That damage has resulted in a loss of service on 5 exchange sites, in turn causing disruption for users from Oxford down to Bristol and across as far as West London.

    I would like to assure you of two facts at this time. Firstly, please be assured that there is absolutely no security risk to your business data due to the actions of the afore mentioned individuals. Secondly, the outage is a P1 concern for everyone at Vodafone, from our CEO through to myself and the customer services teams. A P1 category fix is the highest priority and we already have 7 engineers on site with all the hardware they need to execute the corrections. Everyone in Vodafone is on standby to support both you as our customer and to support the network teams in the efforts.

    Some users are already restored to full service but others are experiencing anything from a full loss of service to restricted service on voice, SMS, Data sessions or voicemail. It remains our commitment to restore service to all as quickly as possible.

    I am in receipt of regular updates from our support teams and I will share this with you as they come in. In the meantime if you could support me in managing your end users with updates I would be very grateful and as always, if you should have any concerns or questions, please do call me (if you can) or E-mail me and I will respond as quickly as I can.

    My thanks to you for your patience and my apologies for the issues experienced.

    1. Dan 55 Silver badge

      Seven engineers

      They're pushing the boat out for this one...

  21. some-reg-reader
    FAIL

    Vodafone "hiding" internal information

    Vodafone just banned me from their eForum for posting the internal information I posted above. They then had the cheek to email me saying:

    "Hi,

    We’ve removed five of your posts from our eForum today relating to the network outage.

    The reason the posts were removed is because the information you’ve received is speculative and has not been confirmed. We are investigating this now and will provide more information as soon as it’s available.

    It would be good if you could let us know who you received this message from although we understand if you choose not to.

    Please do not re-post this information.

    Thanks

    eForum Team"

    Event happened at 00:40, as confirmed in internal email - yet still no ETA, almost 12 hours later.

    http://pastebin.com/GGRi9bHA

  22. Gordon 10

    Lousy customer comms

    Very poor customer comms by VF.

    No network status or link on their main page.

    Takes about 4 links or a google to get to their network faults forum which was on posted to with this issue @ 943 am this morning.

    Customer care lines engaged with no recorded message detailing the problem.

    Since this issue started around 2-3a its a very very poor show. (my wife was feeding the baby and saw it go offline).

    I used run some medium critical systems for a bank and I would have been shot if it had taken 7+ hours to get the first customer comms out.

    Dont these guys understand basic incident management?

  23. Wol
    FAIL

    Text and data probs in the North West

    Hi,

    I'm in the north west and Vodafone SMS is only working intermittently and my TomTom wouldn't get traffic data this morning.

  24. nsld
    FAIL

    Something isnt right.

    How can Vodaphone get away with having no DR for what is probably a cable theft at the data centre?

    Why no security on the site to prevent this?

    Given the amount the "saved" on tax perhaps they ought to look at the basics of disaster recovery!

  25. Arse Face

    northerner

    It's funny how this happened a couple of weeks ago in my local village on the O2 network after thieves robbed parts from a transmiitter, yet there was no news anywhere about it. Is that because we live in an 'unimportant' village in the north of england?

    1. Anonymous Coward
      Pint

      tit-le

      Isn't this always the case?

      If it appens up norf nobody gis a shit!

    2. Gordon 10

      Loose the chip

      One cell site vs most of southern England.

      Get some perspective and loose the chip.

      You know why the north gets a hard time? Cos you lot create the expectation in the first place.

    3. Miek

      not really no ..

      I think it made the news based on the number of users it affected and the fairly broad distribution of the failures.

    4. Loyal Commenter Silver badge

      How many people lost service due to this outage?

      Was it a few hundred, or half the country? There's a world of difference between a single cell transmitter going down and an etire data centre that appears to serve most of the (heavily populated) South of England. Hint: One is newsworthy, the other is not.

  26. Anonymous Coward
    Anonymous Coward

    Hmmm

    From the memo, it sounds like the crims damage was to some external device, rather than inside the DC itself.

    Given the outage, one would suspect it's the electricity supply thats been damaged in some way, although I would have expected an automated fail-over to an internal diesel generator if that was the case.

    If not that, could be the microwave antenna(s) that have been taken out, which could explain the multi-site issue and the difficulty and lack of any fail-over.

    1. Anonymous Coward
      Anonymous Coward

      More conjecture

      I think you're right. Doesn't sound like the servers in the datacentre were attacked.

      On a single site basis they're likely to have dual communications, probably both fixed fibre connections AND microwave to the exchanges in question.

      However, if the miscreants failed to break in, it's possible that they've damaged the aircon (high temperatures in the comms room would automatically shutdown the servers inside), they may have triggered the fire suppression system, again, this would have shutdown many of the servers.

      Power supply would certainly have a backup, but damage directly to the backup kit can take down both the backup and primary, though that's less likely given it will be just as secure as the servers themselves.

      Damage to the aircon systems would naturally take some time to resolve. Bringing servers back-up should be quick, but I've seen instances were certain components don't come back cleanly after the type of shutdown that comes from an instant power failure.

      All conjecture.

      What I still can't understand is that, if this is an MSC, why the exchanges concerned can't be supplied by an alternate MSC. More importantly, why wouldn't this switch be automatic?

  27. Volvic

    MVNO too

    This is also hitting their MVNO customers, our contact centres are getting hammered from pretty much all our mobile customers.

    What's a little concerning is we've just had an email saying their MVNO incident mgt desk is now knackered and we've been given a mobile number as an altwrnative..

  28. The Beer Monster

    There's possibly also a separate issue...

    With RIM. I have a number of directors in Switzerland at present. Those with WinMo/Nokia phones are having no issues at all. Those with Crackberries are having intermittent problems sending mail and accessing t'internet and have been since Saturday morning.

    Vodafone told me on Sunday morning that this was due to outage on RIM's side and that this is worldwide.

    Any one wish to confirm/deny this?

  29. Mickey Porkpies
    FAIL

    OXFORD .. no service

    although if I cover my phone with tin foil and press send just as I throw it up in the air I can get some strange looks

  30. Big_Ted
    Flame

    WTF

    All the people knocking others for having a single supplier of phones get back in the real world not the world your living in.

    I work for a Water utility and we have well over a thousand mobiles in use, plus laptops etc using mobile data for work allocation and emails etc.

    On top of that we have hundreds of remote sites that have GSM for comunications.

    These are done via Vodaphone main centre not local towers at this end

    None of these is dual sim etc so doubling up means massive cost implications for remote telemetry plus all the extra sims for mobiles and laptops etc, thats not countig replacing sims.

    I'm sure you would be happy to have your water bill raised to cover this cost for something that has only happened once in the 5 years we have used this system.

    Come up with a lox/cost free option and fine, if not then just get back to the real world before you post such rubbish about spreading the risk.

    After all do you all have Apple Macs on your desks just in case a windows virus infects everything and requires a few hours to fix.....

    1. Anonymous Coward
      Anonymous Coward

      Pah...

      That's nothing, we use Networker at work to do our primary backups, then we go back and do it all again with NetBackup, just in case there is a problem with one or other piece of software.

      1. Big_Ted
        Thumb Down

        another fail

        How does networker handle people with laptops working in the middle of no-where ?

        These people connect via ethernet maybe once a quarter.....

        1. Anonymous Coward
          Anonymous Coward

          @Big Ted

          That would have been a joke, ie: I was saying that we make two backups with different software when obviously noone does that.

          As it happens, though, NetBackup does have a laptop backup option, I don't think Networker does though.

    2. Anonymous Coward
      Anonymous Coward

      knockers

      Its not a question of knocking people who have decided to use a single supplier - this started with someone who had made that choice then started complaining when that supplier let them down. As you say, using one supplier can be a perfectly rational decision arrived at by balancing convenience and cost savings vs risk of failure and costs of that failure. Having made the decision, however, one is expected to live with it (unless of course one is a politician or bank director).

    3. Loyal Commenter Silver badge
      FAIL

      Low cost or free DR option?

      Get some FREE PAYG SIMs from another operator (IIRC Orange and O2 do them for free). Issue these to your staff. If and when your main comms supplier fails, top these up with credit and ask your staff to swap the SIMs in their phones.

      If necessary, you may have to get your provider to network-unlock the handsets. They will usually do this for a small fee, or for free depending on the phase of the moon or some such similiar thing. Even if you end up having to pay this, it is a small amount compared to the monthly contract cost.

      I have a spare old handset at home with a PAYG SIM in it for exactly this purpose. I would suggest that your only difficulty is in the lack of willingness to consider your DR requirements in the first place, i.e. laziness.

      1. Big_Ted
        Thumb Down

        fail.......

        And how are we suposed to do that for 7-800 remote sites on the tops of hills etc ?

        Plus what about all those who forgoot where the sim is of lost it in the last 5 years when not needed......

        Its easy to come up with solutions when just one office and a few people but not when spread over sveral counties involving more than a thousand phones plus several hundred laptops etc for data.

        Plus PAYG sims don't do data much do they ?

        1. Chad H.
          WTF?

          @ Big Ted

          Since when don't PAYG sims do data ?- you can get PAYG Mbb sims from every major provider. mobilebroadbandaccess.o2.co.uk

  31. Philippe
    Unhappy

    looks like O2 is affected too..

    I am located not too far from Oxford and i have notice seriously bad sound reception on O2 since this morning. It usually works great.

    As O2 and Vodafone have been sharing their network for a while it looks like this outage explains it...

    1. Chad H.
      Unhappy

      No

      Vodafone and O2 do not share a network. They have an agreement to share sites, Air conditioners and power supplies. Each network puts their own equipment into each site, independent of the others, and connect to their own internal network.

      There is no overlap - you'll need to find something else to blame I'm afraid.

  32. cranky
    WTF?

    Vodafone talking rubbish

    Just been reading an FAQ about the outage on their forum and saw this:

    Q. Have you experienced anything like this in the past?

    A. Nothing of this nature.

    BS - I remember on the 3rd of August a similar thing happened, a break in at one of their DC's flattened their entire Pager network for the whole day.

    http://www.theregister.co.uk/2005/08/03/vodafone_pager/

    1. Anonymous Coward
      Anonymous Coward

      Err...

      That story was from 2005, their pager network going out is very different to a full mobile comms outage, also I don't believe they even still run pager services.

      They were probably not refering to the break in as the problem, rather the problems that the break in have caused.

      1. Anonymous Coward
        Anonymous Coward

        Not anymore

        Vodafone UK sold their pager network a few years ago. Regardless, the pager network is a very different system.

        Interestingly, many of the core network management folk do use the pager network simply because if the network is down, mobile phones aren't the best for informing you.

        There's also a historic problem that mobile signal was far from 100% even at the best of times so it's possible the engineer lived out of range. That's less likely these days, but pagers are still used in conjunction with automatic alert systems.

  33. Anonymous Coward
    Anonymous Coward

    M4 corridor - Norwich?

    not able to register all day - either voda are as bad at geography as they are at generally providing service

    no signal or even able to register phone on network all day

  34. Anonymous Coward
    Alert

    Ask the Security Gaurds?

    Every major corporate data centre I've walked into in recent years has been equipped with 24 hour security guards, blanket CCTV, and alarms.

    There are normally several cordons of physical security... including alarmed security doors with keypads or RFID tags, and even steel cages around the most sensitive/critical infrastructure.

    In short, penetrating into such a facility unnoticed, and causing serious damage, is almost impossible for a casual sneak thief to achieve.

    So how on earth could someone break into a Vodafone data centre, apparently unobserved, and cause extensive damage?

    The explanations being given by Vodafone at present don't sound very plausible to me.

  35. Anonymous Coward
    Boffin

    Blah Blah

    The site has redundant capabilities with as correctly stated fibre and microwave transmitters, the issue comes from its almost impossible to expect a entire DC to fall over, if telehouse north went down the ENTIRE UK would lose internet, its not possible to predict this kind of incident, the datacentre in question is guarded by security doors and is in a secret location known by some locals but with one route in and one route out its not a manned site.

    The breakin resulted in two outcomes depending on which information is believed a HLR was stolen or nothing was stolen as they failed to break in.

    Depending on the methods used to get in numerous protection systems would have been activated some of these will include data destruction to protect CUSTOMERS data - as with most major companies in the event of a breakin data is logically scrambled and or physically destroyed the restoration of this data will take time, if the fire suppression system went of they will have to wait for the chemicals to dissipate (being a unmanned site they will use extreme protection systems which literally remove oxygen in a matter of seconds using very powerful gasses)

    chill its not possible to predict this all the backups in the world wont help if a DC falls over unless you expect vodafone to spend some of the 12bn profit on building a redundant site for every DC they own (at a cost of at least 3bn per site!?)

    get a grip, if you are failing then YOU need failover, i have a secondary mobile and redirection to a SIP number which i can access via my phone that cost me less than £10 to set up a year ago.

    Just wait it will come back up.

    1. Anonymous Coward
      FAIL

      "a secret location known by some locals"

      Is that the same secret location revealed by Googling Vodafone Basingstoke?

      That leads me to a planning application on the basingstoke.gov.uk site with an address and details of the site?

      Address details that you can plug into Google Streetview, to discover it is (in effect) a small shed on the corner of an industrial estate?

      1. Oz
        Megaphone

        Secret location (not any more)

        ... and the Daily Fail's article on the break-in has a picture of the site, and more details of the equipment stolen.

  36. FL1X

    Back up in basingstoke

    Came back about 20 mins ago no data though just voice sms is working but is being very slow.

    1. Anonymous Coward
      Thumb Down

      Still flaky this evening... despite what they say!

      Still getting call drop outs and instant missed calls even at home where we *normally* get great coverage... (18:51)

  37. jhth
    FAIL

    I guess their update email should have been BCC'd

    Rather than cc:ed to all and sundry.

    I sent an email of remonstration to the Vodafone contact on the Info Email.

    Somewhat (un) amusingly I got the following response.

    "I'm currently out of the office with no access to emails or phone.

    For any other queries while I'm out of the business, please contact the Vodafone Managed Services helpdesk on 08700 777 533. Otherwise, I will contact you upon my return."

    Hey like the rest of us on Vodafone then!!!!

  38. SteveK
    WTF?

    Good signal

    The really weird thing is that while all around me, my fellow Vodafoners have been complaining about no service, I've been getting much better signal all day. I'm on Vodafone and here in Oxford, get virtually no signal in my basement office and rarely get any 3G when outdoors in the heart of the city. Today though, I've been getting 2-3 bars in the office and a 3G signal when out in town at lunchtime. Haven't tried making any calls to be fair, but have definitely been receiving data.

  39. Anonymous Coward
    Anonymous Coward

    Like an onion, security is made of layers

    You can have physical security with fences, gates, CCTV, alarms, Robotic guard dogs etc. but there is always the human element.

    Bank robbers don't patiently queue up and politely ask the teller if they wouldn't mind filling the bag they brought with cash. They carry guns and kidnap the managers family.

    On-site guards are vulnerable in exactly the same way.

    While there's no information that this was the case, it could the criminals simply threatened the guard into letting them in. If a villain in a balaclava says "Open up or I'll bash you head in with my baseball bat", who here would say No"?

  40. Sean Bailey
    Thumb Down

    @ AC

    '@Sid_Emot: Luckily for you, roaming and non-roaming UK corporate customers will be managed from Ratingen in Germany from later this year, hopefully german sicherheits will be better - not happening quite soon enough though.'

    That's news to me, perhaps I'll get my redundancy letter real soon considering corp and public sector are managed out of sunny Wiltshire.

  41. devnull2
    Unhappy

    I think O2 had problems yesterday as well

    Yesterday (28 Feb) I kept getting silence for ages and then cut off when I rang O2 call centres on 08448090202.

    About 50% of my calls were dropped during most of the afternoon.

This topic is closed for new posts.

Other stories you might like