back to article Sophos antivirus classifies its own update kit as malware

Sophos users woke up to mayhem on Thursday after the business-focussed antivirus firm released an update that classified itself and any other update utility as a virus. As a result enterprise PCs running the application went haywire, generating false positives reporting SSH/Updater-B malware. Sysadmins were bombarded with …

COMMENTS

This topic is closed for new posts.
  1. I'm Brian and so's my wife
    WTF?

    Is this the anti-virus equivalent of typing Google into Google?

    1. Anonymous Coward
      Anonymous Coward

      Isn't it more like an autoimmume response? Computer HIV...

  2. Anonymous Coward
    Anonymous Coward

    Easy to fix, even after quarantine...

    Run msiexec.exe /fomus {15C418EB-7675-42be-B2B3-281952DA014D} /qb in a command-line window and Sophos AutoUpdate will repair itself. Then do an update immediately afterwards.

    Simples.

    1. Phil Koenig

      Re: Easy to fix, even after quarantine...

      This sounds tempting to try, but what does it do exactly? Looks like it just repairs "required registry entries" and shortcuts?

      I was getting a "25010" with "NoUpdateInProgress" error while trying to uninstall one of 3 Sophos items in XP control-panel, apparently this has been a bugaboo for Sophos for quite a while now, judging by various online comments and kb articles.

      1. Anonymous Coward
        Anonymous Coward

        Re: Easy to fix, even after quarantine...

        It also replaces missing files (m and o refer to missing and old files respectively) if your policy was to delete quarantined files.

    2. Phil Koenig

      Re: Easy to fix, even after quarantine...

      Tried it, no joy. Same 25010 error. Still no response from Sophos 1.5 hrs after sending them diag report..

    3. Anonymous Coward
      Anonymous Coward

      @AC 09:14GMT - Re: Easy to fix, even after quarantine...

      Command line?!! But....but....users just want to point and click, they want an OS that just works without being forced to type arcane command lines.

      1. Anonymous Coward
        Anonymous Coward

        Re: @AC 09:14GMT - Easy to fix, even after quarantine...

        That puerile comment just got you a downvote... from a hard-core Linux user.

        1. Danny 14

          Re: @AC 09:14GMT - Easy to fix, even after quarantine...

          I sent a new exclusion rule to stop scanning the sophos folders and "reprotected" the clients. This was after repairing the SUM manually.

          Worked for all my 350 clients although the AV server took a kicking while they all updated. Network usage took a major spike too.

    4. Anonymous Coward
      Anonymous Coward

      Re: Easy to fix, even after quarantine...

      > Simples

      I'll assume then that windows has an "Simples" mechanism for running this command on multiple PCs without the need to do it on each individual PC?

  3. geekclick
    FAIL

    Easy to resolve...

    1. Turn-off 'on-access' scanning in all of your Anti-virus and HIPS policy.

    2. Go to the Update Managers in your Enterprise Console, right-click your Update Managers and choose 'Update now'.

    3. Wait for the update manager to finish downloading the latest updates (Download status changes to Matches)

    4. Edit all of your 'Updating' policies in Enterprise Console. Click on 'Schedule' and change the check for update time to 5 minutes.

    5. Wait 8-10 minutes.

    10. The number of false-positive Virus/Spyware detection should start falling.

    11. Enable the on-access scanner when the number of false-positive detection has fallen significantly.

    12. If there are any computers still showing the false-positive alert then they have either not received the latest update or the 'on-access' scanner was still enabled when they tried to update. The above steps can be repeated for just those computers.

    Taken from: http://www.neowin.net/news/sophos-releases-update-causes-mayhem-for-corporates

    Resolved for all our users this morning, saves having to run a command on every machine....

    1. Spoonsinger
      WTF?

      did you do for steps six through nine?

      (Don't really care though).

      1. K.o.R
        Joke

        Re: did you do for steps six through nine?

        Originally it was:

        5. Wait. Time passes.

        6. Wait. Time passes.

        7. Wait. Time passes. You are in danger of being eaten by a Grue.

        8. Wait. Time passes.

        9 Wait. Time passes.

        But then they realised they could compress this.

  4. Phil Koenig
    Thumb Down

    What a disaster

    It took me around 6 hours to get through to Sophos support on the phone.

    I didn't even want to bother after reading about everyone else's trouble reaching them, but I had at least one machine which was stuck in a loop - couldn't remove Sophos Endpoint Protection, couldn't re-install it, wasn't working properly, Windows Installer kept trying to re-install the auto-update thing every few minutes, ugh.

    I finally got someone on the line 10 minutes after the UK call center started taking calls from the pitifully overloaded Australian call-center. I've been working on this since ~18:00-19:00 PDT, it's now 02:44 9/20 and still waiting to get a response to my diagnostic report sent to them 40 minutes ago.

    Sigh.

    I was wondering what was going to happen to Sophos after they got bought by some investor group a couple years back. Perhaps this is our answer.

  5. Anonymous Coward
    Anonymous Coward

    Not just the sophos program

    Problem is if the deafult is chosen where it will delete any known (rather than suspected) virus then you will lose auto-updaters for a lot of software which means you may be at risk for security problems from that software too.

    You may also find - like we did that a lot of software stops working completely as it tries to use the updater when it first loads and bombs because the file is missing.

    How on earth this got through testing I don't know. Even the most basic of testing should've spotted this.

    1. Morphius
      Pint

      Re: Not just the sophos program

      This is what we have found... so far the list includes Commvault, Flash, Shockwave, Quickbooks, Dell Server Administrator, Java... (although most of those run, just won't update now)

      Sophos issuing instructions on their site to fix Sophos is one thing but the damage this false positive has caused by deleting the updaters is a lot worse. Thank god it happened when most of our PCs were offline so only a handful are going to need work, I pity those on US timezones where their computers were all online.

      Beer because I think some people are going to need one!

      1. Anonymous Coward
        Anonymous Coward

        Re: Not just the sophos program

        agreed. Lucky by shear chance I happen to be chaecking my work emails last night at about 2125 just as things were starting. Manage to vpn in and turn off on access scanning and we only had about 50 boxes that were effected by this. As you said pity those in the states that were in the middle of their working day! Still not quite sorted things out as some of the files on our sophos server were moved but the server does seem to be updating ok, I've sorted all the workstations

  6. Phil Koenig
    Facepalm

    It gets better..

    Now they're just hanging up on you when you call the support line after entering "4" then "2" for Enterprise Products. Did it 4-5 times in a row to me now.

    Sigh.

  7. Mayhem

    Not to mention Small Business Users are currently out in the cold

    You can easily fix as a home user, but the SUM/SEC enterprise instructions don't work with the SCC small business version. I've got two sites that have neatly dropped through the cracks at the moment - we can't fix the server, so can't update the endpoints. At least manually clearing the quarantine lists is feasible, if not a pretty option.

  8. Velv
    Joke

    Shocked and Stunned

    I'm just shocked and stunned by the fact this article was published over two hours ago and a fanbois troll fight hasn't broken out in the comments section:

    "ha ha ha, look at those muppets who've been hit by an update to software X - we're all fine over here because we use software Y which is immune to such problems"

    "ah but software Y is useless, and costs a fortune and you're just showing off how much money you've spent - , my software Z was free and is crowd sourced and open source and used by loads of people"

    repeat (until you get more down votes than up votes)

    1. Christoph
      Joke

      Re: Shocked and Stunned

      They ought to use a proper operating system, then they wouldn't be vulnerable to viruses in the first place!

      1. Ambivalous Crowboard
        Thumb Up

        Re: They ought to use a proper operating system

        +1 Internet to you, sir. Now please send me a keyboard without coffee in.

  9. Anonymous Coward
    Anonymous Coward

    What's happened to Sophos?

    We've used SAV in our business unit for many years. When the renewal came up for for the enterprise AV worldwide, I pushed hard to make the switch to Sophos from TrendMicro.

    Since then, we've also taken their SafeGuard product for full disk encryption of the laptop estate.

    I'm not sure what's happened at Sophos recently but this is the latest and most public of a long line of snafus which have caused our small team worldwide no end of problems - the most recent being an upgrade to the Enterprise Console which we were advised was just 'a few clicks' and 'no risk' which resulted in both SAV management and SafeGuard management being out of commision for several days whilst we worked through the problems.

    The analysis? You shouldn't have the SAV EM and the SafeGuard MC on the same machine. Who new? Certainly not the Platinum Patner consutants who worked with us on both implementations, nor the Sophos Technical Support guy who came along to give us a system heathcheck a couple of months previously.

    There are numerous other examples, but I can't be bothered to type them - I'm past the point where this is cathartic - and don't even get me started on the issues around encryption of solid state HDDs with SafeGuard.

    In short it appears the QA procedures are completely shot and more worryingly seem to be getting worse. Sure, shit happens and most of the major AV vendors have had similar issues in recent memory, but for our Organisation this is just another example of 'another Sophos problem' in a ridiculously short space of time.

    This last cock-up means at the decision whether to renew at the end of the license term has just made itself.

  10. Anonymous Coward
    Anonymous Coward

    Not just the US hit badly...

    ... we have 8000+ workstations and 600+ servers. We're not impressed. As stated ^^^^ how the hell did this get through QA testing?!

    Fortunately we had our quarantine configured to disable access rather than delete - we'd be thoroughly stuffed if we had. Interesting to note that Sophos' advice changed through the morning too! I'm glad to report that the latest advice - to stop the on-access scanner via the enterprise console and force an update does seem to be working

  11. Anonymous Coward
    Anonymous Coward

    Just finished sorting this out

    its only the bastards who never bother shutting down who were affected - since the fix update came out shortly after

    1. Bodestone

      Not just the overnighters

      Since the first thing to get hit was the Sophos server so it could not get the updates. It then passed on the bad one to everyone who logged on next day.

  12. Zombieman
    FAIL

    I think this might be the first time I've heard of an anti-virus/security package classifying itself as malware, though having said that no doubt it has happened before. The more typical rogue AV headline is when operating system files are involved... Bit unfortunate that it targeted the update mechanism... I have this vision of a developer standing like a scolded kid, head down, tracing a partial-circle with the toe of one foot say "uhhm... I did something silly..."

    1. Anonymous Coward
      Anonymous Coward

      CA

      CA's eTrust did the same thing a few years ago. Then of course there are those vendors who delete key system files.

      I woke up this morning to several thousand alerts. Not a happy bunny.

  13. adam payne

    We got hit with this as well. I'm certainly disappointed with Sophos over this.

  14. zaphy42
    FAIL

    Grrrr.

    Came in to 1400 errors this morning. Not impressed Sophos.

    Down to 109 to sort now. :(

  15. Anonymous Coward
    Anonymous Coward

    Message to Windows sysadmins

    Don't lose hope, lads! While you're cleaning this mess, we'll be at the pub and if it doesn't take too long, you might still be able to join us for a couple of beers.

  16. DS 1
    FAIL

    Hmmm

    Got slammed last night, went to bed at 4am.

    I was saved, not by planning or judgement, but just luck really. We were conservative in our settings and chose to use deny rather than delete. And seems the two core imlosions were along the line of delete or deny. Deny did not actually delete or move files, so when you disable on access scanning you have a path back to sanity. The delete is nuclear for some people, as the dammed ide really took out all kinds of update processes/programs/settiings.

    I was able to pull back from the brink by doing wide sweeps of knobbling on access scanning, updating, and then re-enable - starting with the central console. But not really fun, and only saved by delving to establish our own conservative setting in the on access options in our policies.

    Some people are going to need a beer, and I feel for them. Somebody at Sophos or a process at Sophos needs firing/canning, because this got past Qand A and never should have.

    How the hell does a process that eats your own product escape past QA. So far not really seeing that being answerd properly. Its all good saying sorry, but what will next week bring. A dislike to Exe files. Maybe DLL files. Or maybe NTuser.dat.

    Part of the problem with this is that in this circumstance of unknown calamity - is will the next issue we face leave us in the unforseeable worse off if we all use Deny instead of delete. My crystal ball can't say sadly.

  17. mark 63 Silver badge

    56 machines affected out of 600 - not bad!

  18. Lockwood

    Could it be...

    Lupus?

  19. cyberdemon Silver badge
    Meh

    Where's Graham Cluley's angle?

    The article doesn't seem to include a comment from Graham Cluley. I thought that was mandatory.

    1. Anonymous Coward
      Anonymous Coward

      Re: Where's Graham Cluley's angle?

      Get a Cluely (sic) is too busy eating pies !!

  20. Anonymous Coward
    Anonymous Coward

    NO_AV_Arg()

    no av at all. lightweight, and fast

  21. ElSteveo
    Unhappy

    400+ desktops and my entire serv estate was hit by this, being in a school that has kids constantly bringing infected stuff in we had set our policies to Delete.....after 12 hours of trying to sort this Ive decided to change the decision on that one!! :D

    It's more soul destroying to see the long list of other software it appears to have removed as well, as a single person department this is something I REALLY didn't need. A free mug is the least you can do Sophos! :)

    1. Anonymous Coward
      Anonymous Coward

      yep had the same on our server lots of work to fix the damage good for my overtime not good for Sophos.Informed by boss we will not be using them anymore.

  22. Ken Hagan Gold badge

    How hard can it be...

    ...to whitelist anything signed with your own private key?

  23. Euripides Pants
    Thumb Up

    Sophos has a time machine!

    How else do you explain user waking up to mayhem on Thursday and the fix being delivered late Wednesday?

    1. Euripides Pants
      Facepalm

      D'oh!

      Should have been "users"...

      1. Danny 14

        Re: D'oh!

        mainly because the original update broke the SUM too so even though they released a fix the SUM couldnt download it and sent it to clients. Also the fact clients had their autoupdates quarantined meant the fix wouldnt roll out either.

        So unless you have 24hr IT support or live in a timezone that was awake during the debacle then you would wake up to clients going mental.

  24. Anonymous Coward
    Anonymous Coward

    I've got that feeling of impending doom. I don't know how often the other half of IT has set Sophos software to push out updates. I've not seen people swearing at computers either any more than normal, so I suspect to walk into a full on shitstorm tomorrow morning.

  25. Piro Silver badge
    Thumb Up

    Dodged this

    Abandoned Sophos a while back for F-Secure.

  26. eternal cynic
    Paris Hilton

    I bet...

    ...Paris would have done it differently. She knows all about anti-virus protection...

  27. Danny 14
    Pint

    damn them all to hell

    The main pain in the arse for me wasnt the update - only the machines that were on at the time (about 350) go the "bad" update, i'd fixed the SUM by the morning so the majority of machines got the fixed update. No, the pain in the arse was the fact it quarantined a host of other apps too, java, adobe and some bespoke software we run. They will need "manual" care and attention, even using PSTOOLS remotely I will still need some hands on with them. That is not possible in some remote worker cases and they are needing scheduled RDP time.

    I have really liked sophos over the years but errors are creeping in slowing (migration to EC 5 was a nightmare for us) I think it is time to look for a different provider come renewal in January.

    Any ideas on decent enterprise AV for 1000 machines? Needs an EC style approach to managing the AV rollouts etc.

    1. Anonymous Coward
      Anonymous Coward

      Re: damn them all to hell

      There's no protection against incompetent developers because most of them are brilliant in finding ways to bring down your network. Almost each reputable AV vendor has been through this kind of event.

  28. Psymon
    Trollface

    Thank goodness for SCCM

    Thankfully, I managed to create a custom task sequence to fix all the clients.

    Using file inventory, I managed to create a collection query that listed all the machines containing the agen-xuv.ide.

    I then advertised a task sequence that ran:

    net stop savservice

    It then deleted said file (several caveats for differing install locations, x64 etc.)

    net start saveservice

    ALUpdate.exe" -ManualUpdate

    This filtered through and cleaned 6k worth of clients in about 2 hours. I'm just glad I have VPN and RDP on my massively oversized Android phone. I had 90% of the solution in place while I was still on the bus to work.

    Our poor email server is another matter - thankfully, not under my care!

  29. Mandoscottie
    Thumb Up

    Looks like Im the luckiest sysadmin (running SAV) on the planet.

    Out of 300 installs I had 1 user affected, he was outwith the perimeter so SAV endpoint on his laptop used sophos.com for the update.

    Talked him through manually updating when the fix was available and bobs your mothers brother as they say :) Ill give Sophos this one, first real issue ive had in the 6years ive managed their enterprise solution.

    no other system was affected onsite.......most odd :o)

This topic is closed for new posts.

Other stories you might like