Why do minions always make me feel happier?
Cisco Webex is currently suffering a seven-hour outage and counting, with biz bods alternately raging at the failure but secretly being glad they don’t have to suffer the indignities of teleconferencing. The "major outage" began at 0122 GMT, according to Cisco’s status page. At the time of writing every single service …
Gemalto SafeNet Authentication Services went down for about two hours starting at 1200 utc.
You already reported on zoho.com getting it's domain pulled. Effected all Zoho services including any ManageEngine products with Zoho callouts.
Everyone say it with me... Cloud means someone else's computer.
Inevitably, customers took to Twitter to vent their spleens. Unfortunately for Cisco, its blissfully unaware marketing teams continued to pump out soothing sales messages right in the middle of the ongoing outage…
The probably had the social media posts scheduled to post in an automated system; it would have been good if they'd turned them off though once their systems "fancied a quick break".
The incident details are here:
It appears that Cisco migrated customers to a new platform, something went horribly wrong and they addressed the issue by rolling customers back.
While they have now identified the issue with the new platform, I'm guessing there will be a publicly released breakdown of the cause, troubleshooting/customer notification process, resolution and any attempts to address these types of issue in the future.
When it works, WebEx is still the best. So light and stream-lined. You hop on a Skype or GoToMeeting conference and you really find yourself missing WebEx. Goofy pop-up windows, circus-like color schemes..blech! But WebEx has been struggling with technical issues for the past year. Not sure what's going on there.
Service is still down for us in Australia and status page confirms the down status as of writing. Complete outage for teams and associated services (WebEx boards, dx70 etc). Reverted to Google hangout to continue internal (but geographically separate) business collaboration. Some meetings cancelled.
Possibly a data breach, but with GDPR I suspect they would have announced something by now.
It appears they were migrating customers to a new video platform (Video 2.0) - not sure how far through they were, but it appears to have caused the outage. After X many hours, the remediation plan was to begin rolling customers back to Video 1.0 with the root cause of the Video 2.0 discovered after the majority of rollbacks were complete more than 24 hours after the initial issues.
In terms of engineers working on it, I suspect the majority of technical people involved in Video 2.0 would have worked on it for much more than their normal shifts -this wasn't a normal day...
Webex Teams (IM, voice, video) still down! Almost 30 hours now. And to think our company picked this over more feature rich offerings (Slack/MS Teams) because it's more "enterprise friendly". 24hr+ downtime is the most un-enterprise-friendly thing I can imagine.
It sucks for the engineers of course, pulling 24hr+ long shifts. But imagine the account execs having to explain this to furious customers, pre-sales having to defend themselves when this is brought up in 6 months time, etc.
People keep mentioning the poor engineers... Does anyone actually know the engineers? Doubt they will be pulling long shifts.. Support is global. Big professional networking company not having resilience, redundancy or contingency plans in place? Doubt it. Issue is more serious than they're letting on.
Dont get confused between webex meetings and webex teams / hybrid services. I admit I saw this post and took it to be about webex teams. Actually this thread is about webex meetings and has been somewhat hijacked :)
Webex meetings did have an issue before webex teams, I am unsure if it was related. But webex meetings has been up and remained up throughout the webex teams issue.
It does look like they are deploying from scratch and restoring a backup. All of my hybrid services config was missing from the webex admin portal today once I was able to get int after they reported "operational" status. So I had spent this morning configuring it again. One of my peers did not have this issue however and his hybrid services was working right after Cisco announced it.
Biting the hand that feeds IT © 1998–2019