Ooof! Cisco Webex has been down for 7 hours – and counting

Ah More CloudFog - Clearly its Karma for this:

https://www.theregister.co.uk/2018/09/25/cisco_hardcoded_credentials/

Yesterday Sep 24

Gemalto SafeNet Authentication Services went down for about two hours starting at 1200 utc.

You already reported on zoho.com getting it's domain pulled. Effected all Zoho services including any ManageEngine products with Zoho callouts.

Everyone say it with me... Cloud means someone else's computer.

Automated social media

Inevitably, customers took to Twitter to vent their spleens. Unfortunately for Cisco, its blissfully unaware marketing teams continued to pump out soothing sales messages right in the middle of the ongoing outage…

The probably had the social media posts scheduled to post in an automated system; it would have been good if they'd turned them off though once their systems "fancied a quick break".

Webex PMR's were only back up and working (in our tenant) as of approx 09;00BST room kit functionality is still completely borked.

They have had multiple major outages recently. I'm presuming that this is due to trying to integrate teams (formerly spark) onto their WebEx backbone.

This is great

Now we can take a leaf out of the BOFH's handbook and blame some-else for the issue

Cloud.

lol.

Coordinated attacks?

Yesterday one of Poland's largest hosting providers faced DDOS that halted their DNS services. As a result all services were down.

Coincidence?

Waiting for statements on the cause for Cisco and others.

When it works, WebEx is still the best. So light and stream-lined. You hop on a Skype or GoToMeeting conference and you really find yourself missing WebEx. Goofy pop-up windows, circus-like color schemes..blech! But WebEx has been struggling with technical issues for the past year. Not sure what's going on there.

Still down - 13 hours and counting

Service is still down for us in Australia and status page confirms the down status as of writing. Complete outage for teams and associated services (WebEx boards, dx70 etc). Reverted to Google hangout to continue internal (but geographically separate) business collaboration. Some meetings cancelled.

Still down in north europe, Can the register plese get some real feedback from cisco? my faith with webex teams is really low right now.

Still down! North Europe here.

Coming up to 24 hours outage now. In my 15 years as a network engineer I've never seen a corporate outage as long as this. I'm beginning to think that Cisco has had a data breach and has forced it down to remediate.

30 hours - still down

Webex Teams (IM, voice, video) still down! Almost 30 hours now. And to think our company picked this over more feature rich offerings (Slack/MS Teams) because it's more "enterprise friendly". 24hr+ downtime is the most un-enterprise-friendly thing I can imagine.

It sucks for the engineers of course, pulling 24hr+ long shifts. But imagine the account execs having to explain this to furious customers, pre-sales having to defend themselves when this is brought up in 6 months time, etc.

Teams is beginning to come back online now. Having some successes sending messages using teams and in other cases we're receiving immediate "failed to send". So looks like it's on the road to recovery.

Messaging is somewhat working for us again. Status update reports degraded service which is an improvement from the "outage".

Still issues

The Webex icon on my business iPhone is still reporting "Waiting...", as it has since this debacle first started. Or is that some fresh hell they've stepped into?

I don't see the point of spokespeople like WebEx's. Can't remember the last time I heard of one actually speaking something meaningful.

Less meetings, more useful work

Productivity is rising for the last 7 hours worldwide.

Looking at all of the status updates from Cisco on this, it appears they are re-deploying from scratch! Latest update is that they're about to deploy hybrid services!

Webex Meetings was still up?

For what it's worth I think WebEx Meetings was up the whole time, only Teams seemed to be impacted. I was still able to join a couple of meetings during the outage.

Dont get confused between webex meetings and webex teams / hybrid services. I admit I saw this post and took it to be about webex teams. Actually this thread is about webex meetings and has been somewhat hijacked :)

Webex meetings did have an issue before webex teams, I am unsure if it was related. But webex meetings has been up and remained up throughout the webex teams issue.

It does look like they are deploying from scratch and restoring a backup. All of my hybrid services config was missing from the webex admin portal today once I was able to get int after they reported "operational" status. So I had spent this morning configuring it again. One of my peers did not have this issue however and his hybrid services was working right after Cisco announced it.