How many times does this have to happen
And for those who can't reboot to apply the new file?
Enterprise customers of a widely used McAfee anti-virus product were in a world of hurt on Wednesday after an update caused large swaths of their machines to become completely inoperable. The problem started around 2 pm GMT when McAfee pushed out DAT 5958 to users of VirusScan Enterprise. The virus definition falsely identifies …
We've been bitten by this. The immediate response of our IT people was to tell everyone to start yanking network cables - fair enough, as it looked like a day-0 worm spreading like wildfire across all our sites.
Ironically, it's only people not at their desks or bloody-minded enough to ignore IT that have survived, because their machines were still on the network for the virus definition rollback. There are hundreds of PCs that are going to require a bit of TLC to fix because they don't boot far enough to be fixed remotely.
Me? I'm a smug Mac/Linux admin.
yep. the day was going well until about 14.25. then it all went Pete Tong. been a rather interesting
last few hours at work. we took multiple steps to stop windows systems from getting the DAT file without just pulling the internet plug. sort of worked...we estimate just 400 machines need sorting out - better than the c. 8000 it could have been.
False positives forced me to abandon McAfee for Avira years ago. Once identified, the module would go into quarantine with no way to use it except to turn McAfee off completely. McAfee had no mechanism for me to report a false positive, instead telling me to boot a repair disk and scan the system again, fruitlessly. Avira lets you ignore a false positive and continue to use the module, and allows you to submit the module for analysis which, once found to be false, is fixed in a day or two. What a difference!
I managed to get mine out of the reboot cycle and back up and working by disabling all McAfee services via Safe Mode and registry editing (Network Polices prevents the Service Manager from doing it).
Some other guys in the office reported svchost.exe was deleted by it (ouch) and were less lucky.
Why won't our sys admins get avast :(.
You have a trade off between the potential of the AV updates to cause problems and the potential of not releasing the updates to allow a new virus to spread through the network.
To test every DAT file quickly enough you pretty much have to have someone dedicated to doing that on a daily basis. It has to be tested on every variation of machine you have, every OS, every OS level, every critical app. We quarantine engine and product updates, but not DAT files, we simply don't have the resources to test them and get them out quickly enough to avoid the potential risks of un-patched machines.
We could of course use the "previous" branch in ePO to update, then we'd have time to delete the DAT's from current if problems are reported. But again the problem then is that if a new virus gets into the network and we don't have the latest DAT's it can cause far worse problems.
On the plus side, at least now I have more fuel to use in my recommendation that it's time to ditch McAfee.
Doug fully agree with you !
My wife's machine has died tonight of the same ailment. Might be able to get it back but really not hopeful !
What you really have is POS security/antivirus running on POS o/s. True recipe for disaster.
The lesson to be learnt ? Don't use m$ shitware in the first place for mission critical services. That way you don't have to rely on retards like mc-crappy to fuck things up even further for you !
(1) what is a "safe" waiting time?
(2) if you delay reports of disasters, isn't the overall population in the same spot?
(To wit: now EPO users less likely to be hit, others more likely; so everybody else installs EPO with same values; so population ends up as initially, just slightly longer infective for viruses due to delayed definition install).
I pity the poor IT dept that has to use that load of rubbish. It's bad enough at home having that ransomware on your machine, with pop-ups appearing all the time saying "pay up or your computer gets it!". Isn't there a more grown-up anti-virus that enterprise users can take advantage of?
MCAFEE basically sent a virus out to their entire customer base! MORONS! If i had the decision power behind our software selection for antivirus, i would DUMP THIS PROGRAM!
Where is their CHANGE MANAGEMENT process? Where is there IMPLEMENTATION REVIEW process?
I would not be surprised if MCAFEE loses a crap load of customers over this. Their stock is already down .20 cents today. not enough if you ask me. but this is my opinion.
Old dinosaur companies that are too slow to respond to years of failure by McAfee are now being removed from the breeding pool. Why do people keep buying this crap? Its not even like its bought and paid for, you have to ante up every year.
Ever since AVG 8 turned my computer into a POS I've been surfing naked. I've never had a virus scanner find an actual virus since the days of the STONED virus that spread on floppy in the early 90's.
I know I shouldn't feed the troll, but here goes anyway...
I ain't McAfee's biggest fan - truth be told I ain't a fan of them at all - so I'm not trying to defend them nor any of their competitors, but if you're not using any AV software then how do you know you've not been infected?
I think user education is more important than any software solution, and I do agree with your recommendation of using a firewaa to minimise exposure, but I'd not rely on the firewall and a Flash-free browser alone to ensure I was virus free.
Not that I really care as I don't use Windows on my own PCs anyway, but I do have to use this abomination of an OS in the workplace.
Best alternative to McAfee?
How about an Operating System where little things like privilege separation and non-executable files are baked in, rather than crude hacks bolted on from the outside.
And a culture where Source Code is passed around, shared and re-used; as opposed to treated as though it were allergic to daylight, with the consequence that everybody is forced to rewrite common functions from scratch, occasionally missing an awkward edge case.
I ceased to use McAffee in 1994, when it successfully destroyed NATAS. .. only to curl up and die because of an "unknown" virus. That "unknown" was DIR II.
I reverted to MS Antivirus back then (remember CPAV? MS bought them!), and later to Norton. I'm currently using avast!, though I had a brush with ZoneAlarm/Checkpoint... until they also brought upon me a bad false positive. Whoops!
At first we thought a virus had hit our Domain controller and pused out to all the boxes. So everyone assumed the best way to avoid it was to update Mcfee..FAIL
I feel sorry for the IT bods, they will be having to manually fix a couple of hundred network PC's over the next couple of days ! :S
I fail to see how they could defend any legal action.
It would appear even the most basic testing should have picked up it canning a windows system file.
Go on someone please take them to court for your costs caused by this update. That way they might actually do their job properly.
Personally I stopped using their software quite a few years ago (having been a fan for quite a few before) as I started having problems with it.
I've used AVG ever since, never had any issues with their software or any infections.
Yay.. what fun. So I've stopped the reboots... and now somehow sound does not work and various programs just gave up. It was crazy to come into the office today and have everyone gone.. guess they just gave up and said hey nice weather... wish I could have done that.. but being the drudgen that I am.. I cannot.
"Bricking" reduces the utility of a computing device to that of a brick. It happens to game consoles and shitty phones that are so locked down that software bugs can render them unusable. But how the hell do you brick an average computer? Okay, maybe this means that you can't boot your primary OS. Does it not still boot from other partitions or devices?
Call me pedantic, but I don't think a device is a brick if you can have it mostly recovered, by yourself, by the end of the day.
tbh that infrastructure security head is the IT equivalent of a shrieker. I'm in a multinational bank with thousands of PCs affected. Boot in Safe Mode with Networking - downgrade using the 5957 superdat (use the /F switch to force the downgrade). If the PC is off the network do it from CD or any removable media but my experience was that 90% of pcs were still on the network. If the svchost.exe file was quarantined copy it back in to %systemroot% from CD also or from C:\Quarantine - it's the 14k file. Reboot and voila.
Of course you want to stop the 5958 update being deployed too but that's a no-brainer.
Well done NAI. You're making a habit of this - http://www.theregister.co.uk/2009/07/03/mcafee_false_positive_glitch/
From the McAfee Blog:
"The faulty update was quickly removed from all McAfee download servers, preventing any further impact on customers. We are not aware of significant impact on consumers. We believe that this incident has impacted less than one half of one percent of our consumer base and enterprise accounts globally."
And from the McAfee Newsroom:
"In most developed countries, critical infrastructure is connected to the Internet and can lack proper security functions, leaving these installations vulnerable to attacks. Without the appropriate protection combined with the current lack of preparedness, an attack on these infrastructures would be detrimental and will cause more destruction than any previous cyber attacks."
Well there it is, by their own words no less.
It is rumoured that they managed to pretty much take down Intel, and the NYT describes their stuff as 'beserk' :
Is this a good time time for resignations? We have a "recognized authority on cybersecurity" who's company just launched an attack on the availability of every single properly patched copy of the worlds most popular OS that their AV product was installed on.
Ironically enough, If you did have a machine running McAfee that was compromised by a 'retrovirus' you would have been spared this debacle.
As most electronic devices have a flashable bios of some sort it's likely that most devices, e.g. PSP, etc, could have the chip removed, reflashed good and replaced. Or just replaced.
Hence also not a brick. Depends on the lengths to which you wanna go.
It is a brick until it is not a brick.
I've bricked a system before, not a happy experience. These are not bricked, a quick BIOS change and a Knoppix CD gets you out of most continuous reboot sequences. And allows you to mangle McAfee so it won't start... And allows you to replace svchost.exe... Or whatever else file McAfee decides to eat for lunch that day.
Besides, I thought everyone had shut off that "Reboot on serious error" cruft that Windows XP ships with after the first bad XP patch got pushed out.
Our EPO server only gets updates once per day, so we were still distributing 5957, other sites were not so protected. Some admins have the EPO server getting updates every hour and distributing these to the machines every hour, useless.
With EPO if you need a new dat you manually pull it down and push a new managed update job to all your machines to get the new dat for a critical 0 day.
I agree McAfee needs to do some better testing and checksum the OS files that are NEVER rogue.
They also need an option to quarantine/copy a file, so later IT can look at the file attributes to determine who and what downloaded it. Auto deleting viruses is bad. The user downloading the virus needs to be dealt with.
Now I would like to see one of those comparisons on the TCO between Windows and Macs or Windows and Linux machines. :-)
The Average Joe
Well I have not had a virus in the last 15 years, until now. Now I have had the pleasure of dealing with the Mcafee virus. It affects not just the Enterprise packages, it also affects the Home packages as well. Run a tight ship and you get the Mcafee virus. It looked like I had a virus, so guess what I ran a scan. Bang lost all networking and was not able to run a restore. Now have the pleasure of running an install of the whole bloody operating system. Fortunately it was my netbook with no original data on it.
How in heavens name did they manage to let that one through testing. It looks like any PC using Windows XP SP3 is affected and the signs of a problem show pretty quickly.
I suspect that the numbers affected are bigger than Mcafee say, and the people most likely to be affected are those who keep their software up to date.
"The faulty update was quickly removed from all McAfee download servers" say Muckufee
Errrm you might want to re-evaluate "quickly"...
I am on the platinum support contact list... first I got was an email SEVEN HOURS after this all kicked off...
Oh and this morning the latest status update email says:
"Our initial investigation indicates that the error can result in moderate to significant performance issues on systems running Windows XP Service Pack 3."
Moderate eh guys? Errrrm yeah.
Ever since I was forced to use McAfee ePO 4.x I insisted that we always run -1 on the DAT. This is the third time that this approach has saved the companies bacon and do I get any thanks? Noooooooo, they still wax lyrical that we ‘aren’t up to date, the network is at risk.’
Seriously though – I’m glad this has happened (My apologies to all the BOFHs out there cleaning this mess up) hopefully this will convince the company to ditch this POS when the contract is up. And I can go back to real work instead of babysitting 1 piece of software.
McAfee, Thou art weighed in the balances, and art found wanting.
Er, it removes svchost.exe! That's a lot of vital windows services that stop working (and reboot your machine in some cases, i.e. RPC). My internet stopped working after the forced reboot so I was lucky I had a second PC to try and work out what was going on. I applied the mcafee hotfix suggested which worked for a while but then it quarantined svchost again! I tried to apply the new definition file in safe mode as suggested and it said "Error: No qualifying McAfee products found"! I now use AVG...
It's a relatively easy fix. I did the 20 affected machines on our floor in under an hour, on my own, with just a pen drive. OK, it caused all manner of panic when it hit, and I had everyone pull their cables (which stopped the not yet updated from getting the dodgy file) until I had gathered enough info from the support forums to produce a response. Don't let's go over the top here.
In a statement, McAfee said the false positive "can result in moderate to significant performance issues"
Not being able to boot is classed as a moderate to significant performance issue? I call it a complete lack of ANY performance issue due to the fact that if you can't bot your pc ou can't DO anything!
It only seems to affect machines running the 8.7i engine and XP SP3.
Fortunately we had not rolled out 8.7i to everyone yet, so only a small subset of our machines were affected.
I found the easiest solution was to copy the dat files from a 8.5i machine with the 5957 dats and also take a copy of svchost.exe.
Boot machine normally, when you get the DCOM is shutting down crap, open a command prompt and abort the shutdown, open viruscan console, turn off Access protection then go and shut down the Mcafee services.
Copy the dats into C:\Program Files\Common Files\Mcafee\Engine.
Copy svchost.exe to c:\Windows\System32
Reboot, job done :)
It came with McAfee and one of the first things I did was rip it off with malice aforethought.
The first thing that came up during the uninstall was a message saying: "You've got a year's subscription here, why would you want to do this?" to which I mentally answered "Because you're shit".
Turns out that should have been: "Because you're shit and you know you are".
Can we have a steaming turd icon please?
I love how people arepaying McAfee to basically DOS their machines.
This sort of impact is a virus writers wet dream and companies have handed over cash for the fun of having their networks taken down.
Is there any way to reclaim lost earnings / costs of cleaning & restoring from McAfee over this? If not, I would say that there is no real benefit to having their service and you may as well use a freebie package. Even if McAfee only b0rk you once every two years, its still about as often as AVG will let a virus through. The virus risk isnt treated or transferred by using McAfee so why bother?
Now we just need to stop every home PC coming with mountains of McAfee / Symantec rubish which is almost impossible to remove....
How about this plan:
When the tool is installed, the antivirus will securely checksum and store fingerprints of existing executable files, with date.
When a system file is updated, it is checksummed again.
A virus signature is given a "birth" date, before which the virus is presumed not to exist.
If a virus is "detected" in a file whose contents have not changed since before the virus birth date, then it isn't a virus.
Oh yeah - and they do this test BEFORE they publish the virus signatures to the world.
Is it possible that somebody in the company was motivated - even paid - to make them look really, really bad?
This problem is really sorting the Sys Admin men from the boys.
OK.. the error has quite severe symptoms, but the fixes available are all quite simple... even though in some cases you may need to touch every infected machine in the process.
Though obviously - as already mentioned - anyone running business and enterprise networks should really be staggering untested updates at a bare minimum.
Mines the one without the P45 in the pocket.
Recently Mcafee on my PC stopped working for no reason. I got mesasges that my antivirus was not current and when I tried to get into the security center it would just hang. I called mcafee technical support and all they did was persistently push this professional service that could remove this so called virus (that only attacks Mcafee). I'm almost positive they pushed this bug to my machine in an update (although this would be hard to prove) and now they want $89.95 to fix it. I agree with another user that when you call tech support they are sales driven instead of customer support driven. Someone should investigate this new professional service they are offering, since they can easily drum up their own business using innocent customers like me. Think about it - download a problem in an update, and then charge customers to fix it. Easy money - don't think it doesn't happen
The real irony here is McAfee is rubbish at new detections - every time I upload a suspicious file that McAfee says is not a virus to virustotal.com McAfee is consistently not detecting anything. They have a potentially great solution called Artemis that does a DNS lookup of a hash of a file that, if it resolves, means it's possibly malware, the idea being the second their labs have a hash the whole planet can be aware, however it's crap and I have submitted files to their labs, had them positively identified, an extra.dat emailed to me, deployed it with GPO, and Artemis *still* says the file is clean.
When I've logged support calls to pull them up about it I've been fobbed off with pathetic techno babble excuses; when pressed they have actually used the excuse that the system was undergoing 'quality control testing'. Well they do fail consistently I'll give them that.
The double irony is Artemis was a Greek goddess that fought in the Trojan Wars. And lost. Fail.
To quote Vader, You have failed me for the last time McAfee. When I get back to work I'm looking at F-Secure or Sophos.
Biting the hand that feeds IT © 1998–2019