Reply to post: Re: Typical Vodafone

Manchester Arena attack: National Mutual Aid Telephony system failed

SImon Hobson Silver badge
Facepalm

Re: Typical Vodafone

Whilst they'd have trouble fighting their way out of a paper bag

That's my experience with them as well.

When I started at my last place, they had an internet connection from an outfit called YourComms, which IIRC had originally started out as Norweb Telecoms. YourComms was borged into Thus - with the loss off some knowledge as usually happens in these borgings - but still a decent outfit.

Then Thus got borged by Clueless and Witless and service went distinctly downhill. Finally C&W got borged by Vodamoan and it got worse again.

One day as we sat in our office, the internet went down - and after initial checks that it wasn't our end, I was lucky to get through quickly to their support desk and log a fault. 10 minutes later our connection came back on, and we found that a major site we managed services to was still down, and stayed down. By now their support system was in meltdown. The cause ?

Well this supposedly top tier communications company had a networking centre in London, and there had been a very simple problem - a single circuit breaker had tripped. Unfortunately, this supplied one of the power supplies for the kit, and the other one had failed. This highly professional tier one outfit had no monitoring that told them a power supply had failed, it had no monitoring that told them power had been lost to the other power supply, the diesel generator didn't fire up because there was still mains to the building - but no problem, the batteries still worked. Of course, with nothing telling them about these problems, and nothing telling them about the reducing battery voltage, they did nothing until the batteries gave out and the brown stuff properly hit the proverbial fan.

No problem, just flick the switch back on, and let the routers etc boot up again. Yup, that was what brought back our connection - but there's more. A 'kin big router didn't boot up, spares were brought in (and Cisco engineers), and guess what .... they didn't have a working backup of the config ! It took them THIRTEEN HOURS to get our other services back on - losing a complete working day for the cllients, some of whom have staff paid more per hour than I earn in a week.

To their credit they did provide a report of what had gone wrong - and promised to "update their monitoring". But really, needing such an event to find out you have suck basic monitoring in place - sadly this seems to be the level of collective professionalism at Vodamoan (hence the icon). Not having proper config backups - second vote for the icon. Apologies to those people "at the coalface" who have been good within the constraints imposed by the imbeciles further up.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019