back to article Ooof! Cisco Webex has been down for 7 hours – and counting

Cisco Webex is currently suffering a seven-hour outage and counting, with biz bods alternately raging at the failure but secretly being glad they don’t have to suffer the indignities of teleconferencing. The "major outage" began at 0122 GMT, according to Cisco’s status page. At the time of writing every single service …

  1. Anonymous Coward
    Anonymous Coward

    Despicable

    Why do minions always make me feel happier?

    1. Joe W Silver badge

      Re: Despicable

      I don't think you are the only one...

  2. Anonymous Coward
    Anonymous Coward

    Ah More CloudFog - Clearly its Karma for this:

    https://www.theregister.co.uk/2018/09/25/cisco_hardcoded_credentials/

  3. wyatt
    Devil

    Webex Teams, not the whole of Webex is having an issue. Otherwise I wouldn't be on a webex with a customer?

  4. Anonymous Coward
    Anonymous Coward

    Yesterday Sep 24

    Gemalto SafeNet Authentication Services went down for about two hours starting at 1200 utc.

    You already reported on zoho.com getting it's domain pulled. Effected all Zoho services including any ManageEngine products with Zoho callouts.

    Everyone say it with me... Cloud means someone else's computer.

    1. Anonymous Coward
      Anonymous Coward

      'Everyone say it with me... Cloud means someone else's computer'

      Yeah, but the cost-saving executive-bonus is all mine!

      Plus I'll be long gone by the time that reality hits home.

      <CEO / Senior Exec / PHB / Bungee cord manager>

  5. Korev Silver badge

    Automated social media

    Inevitably, customers took to Twitter to vent their spleens. Unfortunately for Cisco, its blissfully unaware marketing teams continued to pump out soothing sales messages right in the middle of the ongoing outage…

    The probably had the social media posts scheduled to post in an automated system; it would have been good if they'd turned them off though once their systems "fancied a quick break".

  6. Anonymous Coward
    Anonymous Coward

    Webex PMR's were only back up and working (in our tenant) as of approx 09;00BST room kit functionality is still completely borked.

    They have had multiple major outages recently. I'm presuming that this is due to trying to integrate teams (formerly spark) onto their WebEx backbone.

  7. Ochib Silver badge

    This is great

    Now we can take a leaf out of the BOFH's handbook and blame some-else for the issue

  8. Anonymous Coward
    Anonymous Coward

    Cloud.

    lol.

  9. Anonymous Coward
    Anonymous Coward

    Coordinated attacks?

    Yesterday one of Poland's largest hosting providers faced DDOS that halted their DNS services. As a result all services were down.

    Coincidence?

    Waiting for statements on the cause for Cisco and others.

    1. theblackhand Silver badge

      Re: Coordinated attacks?

      The incident details are here:

      https://status.webex.com/#/service/status

      It appears that Cisco migrated customers to a new platform, something went horribly wrong and they addressed the issue by rolling customers back.

      While they have now identified the issue with the new platform, I'm guessing there will be a publicly released breakdown of the cause, troubleshooting/customer notification process, resolution and any attempts to address these types of issue in the future.

  10. mikeymac

    When it works, WebEx is still the best. So light and stream-lined. You hop on a Skype or GoToMeeting conference and you really find yourself missing WebEx. Goofy pop-up windows, circus-like color schemes..blech! But WebEx has been struggling with technical issues for the past year. Not sure what's going on there.

    1. Anonymous Coward
      Anonymous Coward

      I'd suggest you take a look at zoom https://zoom.us

      1. This post has been deleted by its author

  11. Tonyp1700

    Still down - 13 hours and counting

    Service is still down for us in Australia and status page confirms the down status as of writing. Complete outage for teams and associated services (WebEx boards, dx70 etc). Reverted to Google hangout to continue internal (but geographically separate) business collaboration. Some meetings cancelled.

    1. nowizard

      Re: Still down - 13 hours and counting

      Same here, north europe is still down.. Any chance we could get some real feedback from cisco?

  12. nowizard

    Still down in north europe, Can the register plese get some real feedback from cisco? my faith with webex teams is really low right now.

  13. nowizard

    Still down! North Europe here.

  14. sanmigueelbeer Silver badge
    Facepalm

    I am glad that WebEx is running on redundancies and/or backups ... oh, wait.

    So does this mean that if Cisco recommends everyone to have redundancies it doesn't guarantee that Cisco has implemented redundancies?

    "Eat your own dogfood."

  15. Anonymous Coward
    Anonymous Coward

    Coming up to 24 hours outage now. In my 15 years as a network engineer I've never seen a corporate outage as long as this. I'm beginning to think that Cisco has had a data breach and has forced it down to remediate.

    1. sanmigueelbeer Silver badge

      I'm beginning to think that Cisco has had a data breach and has forced it down to remediate.

      Worst.

      We believe they've lost the database and there was no redundancy nor back-up. The entire thing was an "open beta" and someone has "monetized" it.

      1. Anonymous Coward
        Anonymous Coward

        I highly doubt that. Firstly I doubt they would put a production system on a non-HA dB. Secondly even if they did do that, then they would of had mitigation plan to at least backup the dB.

        Can you share any more info on it in case my feelings are in fact incorrect?

    2. Anonymous Coward
      Anonymous Coward

      Possibly a data breach, but with GDPR I suspect they would have announced something by now.

      It appears they were migrating customers to a new video platform (Video 2.0) - not sure how far through they were, but it appears to have caused the outage. After X many hours, the remediation plan was to begin rolling customers back to Video 1.0 with the root cause of the Video 2.0 discovered after the majority of rollbacks were complete more than 24 hours after the initial issues.

      In terms of engineers working on it, I suspect the majority of technical people involved in Video 2.0 would have worked on it for much more than their normal shifts -this wasn't a normal day...

  16. Anonymous Coward
    Anonymous Coward

    30 hours - still down

    Webex Teams (IM, voice, video) still down! Almost 30 hours now. And to think our company picked this over more feature rich offerings (Slack/MS Teams) because it's more "enterprise friendly". 24hr+ downtime is the most un-enterprise-friendly thing I can imagine.

    It sucks for the engineers of course, pulling 24hr+ long shifts. But imagine the account execs having to explain this to furious customers, pre-sales having to defend themselves when this is brought up in 6 months time, etc.

    1. sanmigueelbeer Silver badge

      Re: 30 hours - still down

      Webex Teams (IM, voice, video) still down! Almost 30 hours now.

      Don't worry. Chuck has "owned" this.

    2. Anonymous Coward
      Anonymous Coward

      Re: 30 hours - still down

      People keep mentioning the poor engineers... Does anyone actually know the engineers? Doubt they will be pulling long shifts.. Support is global. Big professional networking company not having resilience, redundancy or contingency plans in place? Doubt it. Issue is more serious than they're letting on.

  17. Tonyp1700

    Teams is beginning to come back online now. Having some successes sending messages using teams and in other cases we're receiving immediate "failed to send". So looks like it's on the road to recovery.

  18. Tonyp1700

    Messaging is somewhat working for us again. Status update reports degraded service which is an improvement from the "outage".

  19. katrinab Silver badge
    Thumb Down

    I looked at Giphy to see if there was an animated pooh that I could send them, but it seems to think I want a character from AA Milne's book.

    1. katrinab Silver badge
      Happy

      Apparently you have to search for animated turd

      https://giphy.com/gifs/funny-emoji-poop-l0MYSMV95h0NY6L1m

  20. EJ

    Still issues

    The Webex icon on my business iPhone is still reporting "Waiting...", as it has since this debacle first started. Or is that some fresh hell they've stepped into?

  21. Phil Kingston

    I don't see the point of spokespeople like WebEx's. Can't remember the last time I heard of one actually speaking something meaningful.

  22. Potemkine! Silver badge

    Less meetings, more useful work

    Productivity is rising for the last 7 hours worldwide.

  23. Tonyp1700

    Looking at all of the status updates from Cisco on this, it appears they are re-deploying from scratch! Latest update is that they're about to deploy hybrid services!

  24. Anonymous Coward
    Anonymous Coward

    Webex Meetings was still up?

    For what it's worth I think WebEx Meetings was up the whole time, only Teams seemed to be impacted. I was still able to join a couple of meetings during the outage.

  25. Tonyp1700

    Dont get confused between webex meetings and webex teams / hybrid services. I admit I saw this post and took it to be about webex teams. Actually this thread is about webex meetings and has been somewhat hijacked :)

    Webex meetings did have an issue before webex teams, I am unsure if it was related. But webex meetings has been up and remained up throughout the webex teams issue.

    It does look like they are deploying from scratch and restoring a backup. All of my hybrid services config was missing from the webex admin portal today once I was able to get int after they reported "operational" status. So I had spent this morning configuring it again. One of my peers did not have this issue however and his hybrid services was working right after Cisco announced it.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019