Still TITSUP* here
Hello to you all from the balmy Jurassic Coast (Devonian section)
*Total Inability To Sustain Uninterrupted Phonecalls
Biz broadband comms provider – in name at least today – Gamma is suffering from web wobbles this morning, reportedly across the UK, as customers can't access its Horizon service. The woes for Horizon, which includes fixed and mobile telephony served up via Gamma's cloud PBX system, began just after 09:30 GMT, according to …
"P.S. This is why you ALWAYS keep one analogue line..."
Many years ago, we had a local outage of the phone system around our main office. Every field engineer within a 100 miles was ordered back to base so we could hand over our brick-like analogue mobile phones which BT would then re-route calls to (back then, only field engineers had mobiles, and not even all of them had one. Fecking expensive!!) So, it's not just digital cloudy stuff that fails and it's not new. New, is digital cloudy stuff can have a much more wide ranging impact when some digger driver up-roots a cable or a system update goes TITSUP.
Yet another case of a cloud provider overselling the product. What does Gamma's cloud run on in the backend? While you're at it, please define 'cloud', as it seems to mean something different depending who is telling it.
“It’s still down and more than 30 minutes. 8 care homes without phones. really critical resolved soon. Why can’t we divert phones as promised when entered into contract?!” link
Because when you signed the contract, you transferred your phones on a VoIP service running on the same cloud service as Gamma, with no provision for falling back to the legacy system.
My company is still down, we're based on the Horizon cloud PBX provided to us by Gamma.
We've only really heard from our internal accounts manager once we started musing about compo in emails to them due to an entire day of lost sales as most of our business is done over the phone.
The worst part of it is, the system we had in place for disaster recovery, hasn't worked, which has made this all so much worse than it should be.
It seems that my comment took much longer to get approved then from when I posted it.
We currently have gotten around the Horizon issues by activating twinning on a number of users that have mobiles and have forwarded our main hunt group to a BCP hunt group that has these mobile users as members.
The system didn't come back up at 15:00 as stated in the status updates, probably a much bigger job than Gamma/Horizon thought
The system didn't come back up at 15:00 as stated in the status updates, probably a much bigger job than Gamma/Horizon thought"
Looking at DownDetector, it appears the system (at least for the Leeds area, the only one I checked), was all back up by about 8pm, ie the graph shows reports of downtime at zero from about that time on.
Nope, former technician before me was told that it would automatically fail over and that we wouldn't be able to manually invoke a DR state.
The only silver lining in this, is that I have actually found a way to manually force calls to our main sales number to forward to a group of users that have their mobile numbers twinned with their horizon account.
However, if the horizon portal is down at the same time, I'll be up a certain creak in a leaky boat.
Critical service updates should be done in the wee smalls of sunday morning.
That way you have around 24 hours to fix things before brown smelly stuff REALLY hits rapidly rotating thingy when things don't work.
Of course PHBs don't like that as it means doubletime (and then some), but it was the normal way of doing things in telcos once upon a time - and it's cheaper than the losses if (when) things go wrong.
As I've said many many times, Cloud Tel systems are inherently flawed. When it goes down it all goes down and you can forget any portal access / call redirection failover. Its fine under certain conditions (remote workers/DDIs/cheap calls) but for mission critical its really hard to beat copper with solid failover options at many levels.
Think this has highlighted a serious design flaw in their infrastructure
Not that the failure happens, I meant we all know that IT can and will fail at the worst moments, but it comes as no surprise that service was finally restored after 5pm. Strongly suspect that the system coming back up took a lot longer than expected because everyone kept trying and trying to use the network, but come 5pm, the volume of calls drops, the number of phone restart attempts stops, and the system gets breathing room to come back up.
course I have to worry about daily stuff here, single digit (F) temps, high winds, 6-12 inches of heavy wet snow dropping tonight, etc so on and so forth. life in rural northeast maine can be....interesting.
but...as Old Dominion sings....thats a song for another time...
best of luck to everyone dealing with this, hope its fixed soon.
edit:fixed odd link to video
With the horizon platform down has no one thought about the consequence in such event if a outage beforehand ? I expect the users were attracted by the price per endpoint being attractive at the time of purchase and never thought about the loss of business in the event of an outage . Can anyone afford to lose business I expect not . Long live the onsite PABX !
Biting the hand that feeds IT © 1998–2019