back to article Sophos antivirus classifies its own update kit as malware

Sophos users woke up to mayhem on Thursday after the business-focussed antivirus firm released an update that classified itself and any other update utility as a virus. As a result enterprise PCs running the application went haywire, generating false positives reporting SSH/Updater-B malware. Sysadmins were bombarded with …

COMMENTS

This topic is closed for new posts.
WTF?

Is this the anti-virus equivalent of typing Google into Google?

13
0
Anonymous Coward

Isn't it more like an autoimmume response? Computer HIV...

1
0
Anonymous Coward

Easy to fix, even after quarantine...

Run msiexec.exe /fomus {15C418EB-7675-42be-B2B3-281952DA014D} /qb in a command-line window and Sophos AutoUpdate will repair itself. Then do an update immediately afterwards.

Simples.

1
0

Re: Easy to fix, even after quarantine...

This sounds tempting to try, but what does it do exactly? Looks like it just repairs "required registry entries" and shortcuts?

I was getting a "25010" with "NoUpdateInProgress" error while trying to uninstall one of 3 Sophos items in XP control-panel, apparently this has been a bugaboo for Sophos for quite a while now, judging by various online comments and kb articles.

0
0
Anonymous Coward

Re: Easy to fix, even after quarantine...

It also replaces missing files (m and o refer to missing and old files respectively) if your policy was to delete quarantined files.

0
0

Re: Easy to fix, even after quarantine...

Tried it, no joy. Same 25010 error. Still no response from Sophos 1.5 hrs after sending them diag report..

0
0
Anonymous Coward

@AC 09:14GMT - Re: Easy to fix, even after quarantine...

Command line?!! But....but....users just want to point and click, they want an OS that just works without being forced to type arcane command lines.

6
3
Anonymous Coward

Re: @AC 09:14GMT - Easy to fix, even after quarantine...

That puerile comment just got you a downvote... from a hard-core Linux user.

0
0
Silver badge

Re: @AC 09:14GMT - Easy to fix, even after quarantine...

I sent a new exclusion rule to stop scanning the sophos folders and "reprotected" the clients. This was after repairing the SUM manually.

Worked for all my 350 clients although the AV server took a kicking while they all updated. Network usage took a major spike too.

0
0
Anonymous Coward

Re: Easy to fix, even after quarantine...

> Simples

I'll assume then that windows has an "Simples" mechanism for running this command on multiple PCs without the need to do it on each individual PC?

1
0
FAIL

Easy to resolve...

1. Turn-off 'on-access' scanning in all of your Anti-virus and HIPS policy.

2. Go to the Update Managers in your Enterprise Console, right-click your Update Managers and choose 'Update now'.

3. Wait for the update manager to finish downloading the latest updates (Download status changes to Matches)

4. Edit all of your 'Updating' policies in Enterprise Console. Click on 'Schedule' and change the check for update time to 5 minutes.

5. Wait 8-10 minutes.

10. The number of false-positive Virus/Spyware detection should start falling.

11. Enable the on-access scanner when the number of false-positive detection has fallen significantly.

12. If there are any computers still showing the false-positive alert then they have either not received the latest update or the 'on-access' scanner was still enabled when they tried to update. The above steps can be repeated for just those computers.

Taken from: http://www.neowin.net/news/sophos-releases-update-causes-mayhem-for-corporates

Resolved for all our users this morning, saves having to run a command on every machine....

0
0
Bronze badge
WTF?

did you do for steps six through nine?

(Don't really care though).

0
0
Joke

Re: did you do for steps six through nine?

Originally it was:

5. Wait. Time passes.

6. Wait. Time passes.

7. Wait. Time passes. You are in danger of being eaten by a Grue.

8. Wait. Time passes.

9 Wait. Time passes.

But then they realised they could compress this.

5
0
Thumb Down

What a disaster

It took me around 6 hours to get through to Sophos support on the phone.

I didn't even want to bother after reading about everyone else's trouble reaching them, but I had at least one machine which was stuck in a loop - couldn't remove Sophos Endpoint Protection, couldn't re-install it, wasn't working properly, Windows Installer kept trying to re-install the auto-update thing every few minutes, ugh.

I finally got someone on the line 10 minutes after the UK call center started taking calls from the pitifully overloaded Australian call-center. I've been working on this since ~18:00-19:00 PDT, it's now 02:44 9/20 and still waiting to get a response to my diagnostic report sent to them 40 minutes ago.

Sigh.

I was wondering what was going to happen to Sophos after they got bought by some investor group a couple years back. Perhaps this is our answer.

0
0
Anonymous Coward

Not just the sophos program

Problem is if the deafult is chosen where it will delete any known (rather than suspected) virus then you will lose auto-updaters for a lot of software which means you may be at risk for security problems from that software too.

You may also find - like we did that a lot of software stops working completely as it tries to use the updater when it first loads and bombs because the file is missing.

How on earth this got through testing I don't know. Even the most basic of testing should've spotted this.

0
0
Pint

Re: Not just the sophos program

This is what we have found... so far the list includes Commvault, Flash, Shockwave, Quickbooks, Dell Server Administrator, Java... (although most of those run, just won't update now)

Sophos issuing instructions on their site to fix Sophos is one thing but the damage this false positive has caused by deleting the updaters is a lot worse. Thank god it happened when most of our PCs were offline so only a handful are going to need work, I pity those on US timezones where their computers were all online.

Beer because I think some people are going to need one!

0
0
Anonymous Coward

Re: Not just the sophos program

agreed. Lucky by shear chance I happen to be chaecking my work emails last night at about 2125 just as things were starting. Manage to vpn in and turn off on access scanning and we only had about 50 boxes that were effected by this. As you said pity those in the states that were in the middle of their working day! Still not quite sorted things out as some of the files on our sophos server were moved but the server does seem to be updating ok, I've sorted all the workstations

0
0
Facepalm

It gets better..

Now they're just hanging up on you when you call the support line after entering "4" then "2" for Enterprise Products. Did it 4-5 times in a row to me now.

Sigh.

0
0

Not to mention Small Business Users are currently out in the cold

You can easily fix as a home user, but the SUM/SEC enterprise instructions don't work with the SCC small business version. I've got two sites that have neatly dropped through the cracks at the moment - we can't fix the server, so can't update the endpoints. At least manually clearing the quarantine lists is feasible, if not a pretty option.

0
0
Silver badge
Joke

Shocked and Stunned

I'm just shocked and stunned by the fact this article was published over two hours ago and a fanbois troll fight hasn't broken out in the comments section:

"ha ha ha, look at those muppets who've been hit by an update to software X - we're all fine over here because we use software Y which is immune to such problems"

"ah but software Y is useless, and costs a fortune and you're just showing off how much money you've spent - , my software Z was free and is crowd sourced and open source and used by loads of people"

repeat (until you get more down votes than up votes)

3
1
Silver badge
Joke

Re: Shocked and Stunned

They ought to use a proper operating system, then they wouldn't be vulnerable to viruses in the first place!

4
1
Thumb Up

Re: They ought to use a proper operating system

+1 Internet to you, sir. Now please send me a keyboard without coffee in.

2
0
Anonymous Coward

What's happened to Sophos?

We've used SAV in our business unit for many years. When the renewal came up for for the enterprise AV worldwide, I pushed hard to make the switch to Sophos from TrendMicro.

Since then, we've also taken their SafeGuard product for full disk encryption of the laptop estate.

I'm not sure what's happened at Sophos recently but this is the latest and most public of a long line of snafus which have caused our small team worldwide no end of problems - the most recent being an upgrade to the Enterprise Console which we were advised was just 'a few clicks' and 'no risk' which resulted in both SAV management and SafeGuard management being out of commision for several days whilst we worked through the problems.

The analysis? You shouldn't have the SAV EM and the SafeGuard MC on the same machine. Who new? Certainly not the Platinum Patner consutants who worked with us on both implementations, nor the Sophos Technical Support guy who came along to give us a system heathcheck a couple of months previously.

There are numerous other examples, but I can't be bothered to type them - I'm past the point where this is cathartic - and don't even get me started on the issues around encryption of solid state HDDs with SafeGuard.

In short it appears the QA procedures are completely shot and more worryingly seem to be getting worse. Sure, shit happens and most of the major AV vendors have had similar issues in recent memory, but for our Organisation this is just another example of 'another Sophos problem' in a ridiculously short space of time.

This last cock-up means at the decision whether to renew at the end of the license term has just made itself.

1
0
Anonymous Coward

Not just the US hit badly...

... we have 8000+ workstations and 600+ servers. We're not impressed. As stated ^^^^ how the hell did this get through QA testing?!

Fortunately we had our quarantine configured to disable access rather than delete - we'd be thoroughly stuffed if we had. Interesting to note that Sophos' advice changed through the morning too! I'm glad to report that the latest advice - to stop the on-access scanner via the enterprise console and force an update does seem to be working

2
0
Anonymous Coward

Just finished sorting this out

its only the bastards who never bother shutting down who were affected - since the fix update came out shortly after

1
0

Not just the overnighters

Since the first thing to get hit was the Sophos server so it could not get the updates. It then passed on the bad one to everyone who logged on next day.

0
0
FAIL

I think this might be the first time I've heard of an anti-virus/security package classifying itself as malware, though having said that no doubt it has happened before. The more typical rogue AV headline is when operating system files are involved... Bit unfortunate that it targeted the update mechanism... I have this vision of a developer standing like a scolded kid, head down, tracing a partial-circle with the toe of one foot say "uhhm... I did something silly..."

0
0
Anonymous Coward

CA

CA's eTrust did the same thing a few years ago. Then of course there are those vendors who delete key system files.

I woke up this morning to several thousand alerts. Not a happy bunny.

0
0

We got hit with this as well. I'm certainly disappointed with Sophos over this.

0
0
FAIL

Grrrr.

Came in to 1400 errors this morning. Not impressed Sophos.

Down to 109 to sort now. :(

0
0
Anonymous Coward

Message to Windows sysadmins

Don't lose hope, lads! While you're cleaning this mess, we'll be at the pub and if it doesn't take too long, you might still be able to join us for a couple of beers.

0
0
FAIL

Hmmm

Got slammed last night, went to bed at 4am.

I was saved, not by planning or judgement, but just luck really. We were conservative in our settings and chose to use deny rather than delete. And seems the two core imlosions were along the line of delete or deny. Deny did not actually delete or move files, so when you disable on access scanning you have a path back to sanity. The delete is nuclear for some people, as the dammed ide really took out all kinds of update processes/programs/settiings.

I was able to pull back from the brink by doing wide sweeps of knobbling on access scanning, updating, and then re-enable - starting with the central console. But not really fun, and only saved by delving to establish our own conservative setting in the on access options in our policies.

Some people are going to need a beer, and I feel for them. Somebody at Sophos or a process at Sophos needs firing/canning, because this got past Qand A and never should have.

How the hell does a process that eats your own product escape past QA. So far not really seeing that being answerd properly. Its all good saying sorry, but what will next week bring. A dislike to Exe files. Maybe DLL files. Or maybe NTuser.dat.

Part of the problem with this is that in this circumstance of unknown calamity - is will the next issue we face leave us in the unforseeable worse off if we all use Deny instead of delete. My crystal ball can't say sadly.

2
0

56 machines affected out of 600 - not bad!

0
0

Could it be...

Lupus?

0
0
Meh

Where's Graham Cluley's angle?

The article doesn't seem to include a comment from Graham Cluley. I thought that was mandatory.

2
0
Anonymous Coward

Re: Where's Graham Cluley's angle?

Get a Cluely (sic) is too busy eating pies !!

1
0
Anonymous Coward

NO_AV_Arg()

no av at all. lightweight, and fast

0
0
Unhappy

400+ desktops and my entire serv estate was hit by this, being in a school that has kids constantly bringing infected stuff in we had set our policies to Delete.....after 12 hours of trying to sort this Ive decided to change the decision on that one!! :D

It's more soul destroying to see the long list of other software it appears to have removed as well, as a single person department this is something I REALLY didn't need. A free mug is the least you can do Sophos! :)

0
0
Anonymous Coward

yep had the same on our server lots of work to fix the damage good for my overtime not good for Sophos.Informed by boss we will not be using them anymore.

0
0
Gold badge

How hard can it be...

...to whitelist anything signed with your own private key?

1
0
Thumb Up

Sophos has a time machine!

How else do you explain user waking up to mayhem on Thursday and the fix being delivered late Wednesday?

0
0
Facepalm

D'oh!

Should have been "users"...

0
0
Silver badge

Re: D'oh!

mainly because the original update broke the SUM too so even though they released a fix the SUM couldnt download it and sent it to clients. Also the fact clients had their autoupdates quarantined meant the fix wouldnt roll out either.

So unless you have 24hr IT support or live in a timezone that was awake during the debacle then you would wake up to clients going mental.

0
0
Anonymous Coward

I've got that feeling of impending doom. I don't know how often the other half of IT has set Sophos software to push out updates. I've not seen people swearing at computers either any more than normal, so I suspect to walk into a full on shitstorm tomorrow morning.

0
0
Silver badge
Thumb Up

Dodged this

Abandoned Sophos a while back for F-Secure.

0
0
Paris Hilton

I bet...

...Paris would have done it differently. She knows all about anti-virus protection...

0
0
Silver badge
Pint

damn them all to hell

The main pain in the arse for me wasnt the update - only the machines that were on at the time (about 350) go the "bad" update, i'd fixed the SUM by the morning so the majority of machines got the fixed update. No, the pain in the arse was the fact it quarantined a host of other apps too, java, adobe and some bespoke software we run. They will need "manual" care and attention, even using PSTOOLS remotely I will still need some hands on with them. That is not possible in some remote worker cases and they are needing scheduled RDP time.

I have really liked sophos over the years but errors are creeping in slowing (migration to EC 5 was a nightmare for us) I think it is time to look for a different provider come renewal in January.

Any ideas on decent enterprise AV for 1000 machines? Needs an EC style approach to managing the AV rollouts etc.

0
1
Anonymous Coward

Re: damn them all to hell

There's no protection against incompetent developers because most of them are brilliant in finding ways to bring down your network. Almost each reputable AV vendor has been through this kind of event.

0
0
Trollface

Thank goodness for SCCM

Thankfully, I managed to create a custom task sequence to fix all the clients.

Using file inventory, I managed to create a collection query that listed all the machines containing the agen-xuv.ide.

I then advertised a task sequence that ran:

net stop savservice

It then deleted said file (several caveats for differing install locations, x64 etc.)

net start saveservice

ALUpdate.exe" -ManualUpdate

This filtered through and cleaned 6k worth of clients in about 2 hours. I'm just glad I have VPN and RDP on my massively oversized Android phone. I had 90% of the solution in place while I was still on the bus to work.

Our poor email server is another matter - thankfully, not under my care!

0
0
Thumb Up

Looks like Im the luckiest sysadmin (running SAV) on the planet.

Out of 300 installs I had 1 user affected, he was outwith the perimeter so SAV endpoint on his laptop used sophos.com for the update.

Talked him through manually updating when the fix was available and bobs your mothers brother as they say :) Ill give Sophos this one, first real issue ive had in the 6years ive managed their enterprise solution.

no other system was affected onsite.......most odd :o)

0
0
This topic is closed for new posts.

Forums