back to article Symantec cloud portal goes titsup after database crash

Symantec.cloud is in the middle of rebuilding its portal this lunchtime following a prolonged outage spanning more than 24 hours. The snafu stemmed from a database crash. Problems first surfaced at 1000 UTC (1100 BST) on Monday and dragged on until lunchtime on Tuesday, as detailed in a series of updates to Symantec.cloud’s …

  1. Locky Silver badge

    Message Labs

    I've had reports from a couple of suppliers who have delayed messages and they are on Symantec for their mail gateway. I suspect the impact is far reaching

  2. Anonymous Coward
    Anonymous Coward

    Far more than this.

    Since 10:30 am, multiple customers (including ourselves) lost inbound emails.

    Now the web filtering seems to have gone tits up.

  3. garethm
    Flame

    SIGH!

    We've got Symantec.cloud web filtering, and whatever they've just done has totalled our internets. Rather far from impressed!!

    1. Anonymous Coward
      Anonymous Coward

      Re: SIGH!

      As usual, with all cloud services: if continuous service is critical, then you should have two completely independent suppliers of the same service.

      In the case of spam filtering this is easy: just have two MX records pointing to two different cloud filtering suppliers.

      Of course, this is going to cost you double. Hence you have to decide whether the extra cost is worth it to you or not.

      1. garethm

        Re: SIGH!

        Completely agree. We've just switched off filtering for now, so traffic is just going direct. Web filtering isn't THAT critical for us!

        1. Anonymous Coward
          Anonymous Coward

          Re: SIGH!

          Web filtering isn't THAT critical for us!

          Don't suppose the machines there have Flash of any variety installed? :D

          1. Stevie Silver badge

            Re: SIGH!

            Yes, but fortunately the Flash functionality is cloud-hosted.

  4. upsidedowncreature

    Getting worse...

    It looks like it's not just user management now, the web filtering service appears to be knackered as well. Every page is being blocked and the Symantec message doesn't have any explanatory text.

    1. Destroy All Monsters Silver badge
      Trollface

      Re: Getting worse...

      Wake me up when they find out that it was Cryptolocker!

  5. Andy Neale
    WTF?

    Whatever they are doing to repair the portal has now broken the Web Filtering.

    Not sure if they really know what an impact it is having on customers, but not impressed from this end.

    I've had to put bypasses in to get to the Internet. I am beginning to worry that they don't actually know what they are doing.

    1. Locky Silver badge
      Coat

      Re: Whatever they are doing to repair the portal has now broken the Web Filtering.

      Beginning?

  6. Sean Kennedy

    Forgive my callousness, but if you chose Symantec for email filtering, you kind of get what you deserve.

    It's kind of like using McAfee antivirus.

    1. Nick Woodruffe

      Optional

      Are you inferring that the Messagelabs product was shite to start with or just shite after Symantec got hold if it mangled it up?

  7. retro.kid

    We're in the process of migrating to O365, so losing portal today is a total PITA.

    However not as much as in the last 20 minutes when the whole company has lost their Internet and i have users screaming at me from multiple offices.

    Symantec is making me a very unhappy IT Manager today.

    1. Vince

      I wouldn't worry, if you're moving to office 365 for some reason... this is just a 'business as usual' thing. Enjoy your future outages!

  8. Synonymous Howard

    MPKI

    Their Managed PKI Service seems to have been alright though? .. however that is the tried and tested Verisign infrastructure.

  9. gollux

    Welcome to the cloud...

    You have just hit an embedded thunderstorm. Expect to simultaneously hit updrafts, downdrafts, softball sized hail, intense rain, icing conditions and the possibility of tornado force winds.

    Hope your business survives the experience.

  10. Midnight

    What a coincidence. I wonder if this guy was posting from symantec.com...

  11. Anonymous Coward
    Anonymous Coward

    All Symantec.cloud services offline

    All Symantec.cloud services are totally borked as of 14:35 for all our customers.Web filtering and email, all blocked by a very unhelpful Symantec branded webpage.

  12. esscolab

    We were told this morning the following:-

    As a way of an update there are two issues which arose from the DC power down over the weekend. First and most pressing is the corruption of the ClientNet DB servers. Secondly Spam Manager 1 and 5 server Rack did not power back up correctly (Believed to be a power issue)

    The first issue resulted in customer portal ClientNet as well as Insight to be unavailable. The parent case for this Portal incident INC-348312 is 10364778 and currently we have 58 cases associated with this incident. We have also informed Customer Care team that inbound calls from client may spike because of the incident on the cloud side.

    As of 06:50 UTC, War Room is still ongoing. After restoring the necessary Databases and while running checks, Tier 2 engineers found errors due to inconsistencies in few Tables in Databases, which they are working on to identify the possible options to fix that. Database development team has been involved to get suggestion for the best possible options to proceed.

    Client Support enabled the Message of the Date on the Phones and updated the Splashpage to reflect the messaging that we are aware of the incident and are working with top priority to resolve the issue. Support is also focusing on inbound calls and making proactive outbound calls to specific clients (incl. those who reached by email to Support Box)

    Further updates will follow.

  13. Anonymous Coward
    Anonymous Coward

    More woes

    As a way of an update there are two issues which arose from the DC power down over the weekend. First and most pressing is the corruption of the ClientNet DB servers. Secondly Spam Manager 1 and 5 server Rack did not power back up correctly (Believed to be a power issue)

    The first issue resulted in customer portal ClientNet as well as Insight to be unavailable. The parent case for this Portal incident INC-348312 is 10364778 and currently we have 58 cases associated with this incident. We have also informed Customer Care team that inbound calls from client may spike because of the incident on the cloud side.

    As of 06:50 UTC, War Room is still ongoing. After restoring the necessary Databases and while running checks, Tier 2 engineers found errors due to inconsistencies in few Tables in Databases, which they are working on to identify the possible options to fix that. Database development team has been involved to get suggestion for the best possible options to proceed.

    Client Support enabled the Message of the Date on the Phones and updated the Splashpage to reflect the messaging that we are aware of the incident and are working with top priority to resolve the issue. Support is also focusing on inbound calls and making proactive outbound calls to specific clients (incl. those who reached by email to Support Box)

    Further updates will follow.

    1. Anonymous Coward
      Anonymous Coward

      Re: More woes

      As of 06:50 UTC, War Room is still ongoing. After restoring the necessary Databases and while running checks, Tier 2 engineers found errors due to inconsistencies in few Tables in Databases, which they are working on to identify the possible options to fix that. Database development team has been involved to get suggestion for the best possible options to proceed.

      That's priceless. The above report strongly suggests they didn't implement their backup regime very well. Super ironic, as until recently Symantec owned NetBackup - the high-end enterprise wide backup suite.

      Normally I wouldn't laugh at someone's misfortune... but Symantec are a piece of work, so laughing now. :)

      1. P. Lee Silver badge
        Facepalm

        Re: More woes

        >The above report strongly suggests they didn't implement their backup regime very well.

        Also, it points out that the service isn't a cloud.

        A cloud would have multiple replicated instances spread across various data centres, service health monitoring and dynamic DNS and IP load-balancing across known good servers.

        Well, at least they and their customers have saved money by outsourcing the function to a specialist company with in-depth expertise.

        1. Anonymous Coward
          Anonymous Coward

          Re: More woes

          Also, it points out that the service isn't a cloud.

          A cloud would have multiple replicated instances spread across various data centres, service health monitoring and dynamic DNS and IP load-balancing across known good servers.

          Only if done properly. This is Symantec we're talking about here. They prioritise hiring sales staff, and de-emphasise quality engineering.

          Some of their guys are good, but some have less than stellar attention to detail. (Case in point probably caused this outage, and/or unusable backups.)

  14. s. pam
    FAIL

    The people with the clues were probably redundant!

    Symantec gets what it has sowed -- screw over your staff, threaten them all the time, then ceremonially make them redundant.

    That means the people who had the clues are all sitting down the pub laughing their arses off that Symantec's cloud is down.

  15. Emmeran

    Wow

    Frankly I was surprised to hear that Symantec was still a company AND that intelligent people bought things from them.

    I suppose I should have known that five years of closing my eyes and wishing wouldn't make them go away but heck I even included "and please protect all IT managers from buying Symantec products" in my nightly prayers.

  16. Andrew Barr

    VERITAS

    I bet the Veritas staff are having a great day!

    I wonder what Symantec is using for it continuous backup solution?

  17. gmcfly

    I still have faith in Symantec

    1. Destroy All Monsters Silver badge

      It's like having faith in the Bible.

      1. Vic

        It's like having faith in the Bible.

        It's completely unlike having faith in the Bible.

        Unlike Symantec, the Bible occasionally brings solace...

        Vic.

    2. Anonymous Coward
      Anonymous Coward

      You are a fool

      Can't believe anything they say and get ready for more products randomly being EOL'd with little to no warning. That's their new world odour of operations. All the rats are or have left their ship!

  18. nilfs2
    Facepalm

    Whomever still use Symantec products...

    ...deserve all kinds of bad things happening to them

  19. Captain DaFt

    Oh dear, database down

    it wasn't This one by any chance?

  20. Stevie Silver badge

    Bah!

    But ... Veritas volume manager ... mirror-stripe ...

  21. ecofeco Silver badge

    How's that cloud thing workin for ya?

    You have to wonder what the lost dollar amount was for the customers. Cheaper than a decent server room and staff?

  22. JeffyPoooh Silver badge
    Pint

    Root Cause identified...

    They found some Symantec software was installed on the system.

    They're just downloading NRT now...

    1. gollux

      Re: Root Cause identified...

      Oh, you mean that awesome computer performance enhancement tool? We used it company-wide in our Windows XP days to revitalize our network and get an extra year out of the workstations so we could afford our Windows 7 roll-out. Good to know it has other world improving uses!

  23. Dieter Haussmann

    Symantec still exist?

    What about Norton?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019