July last year
I spent most of my evenings fixing this:
Many enterprises, including police departments and hospitals in the US, were hit by a false positive from McAfee on Wednesday that labelled a core Windows file as potentially malign. A detection update from McAfee (DAT 5958) falsely labelled the svchost.exe as the Wecorl-A virus, sending a core Windows system file into …
I spent most of my evenings fixing this:
Yep, were lucky in my shop. We only lost one part of the org and half dozen IT guys had to spend the night manually patching and fixing about 150 desktops ready for the morning.
What the hell happened in McAfee QA for crying out loud?!
And this from a company that was the first and for a long time the best.
You are the weakest link. goodbye.
I've thought that for a long time now!
I don't understand what your inane comment has to do with this story whatsoever. This is about a bad mcafee update and has nothing to do with windows aside from the fact that it was windows suers who were impacted? Grabs the clue-bat.
If anyone is still looking for fix information you can also check out my blog at http://cosine-security.blogspot.com which has information on manually fixing the problem.
but it has Windows at it core.
This problem is starting to become very serious now. How many lives have been put at risk here because of hospital IT infrastructure going down.
Corporation's and Government really need to start thinking seriously about moving away from the Windows platform.
Companies being unable to trade for a day or two is one thing, but threats to emergency services and even national security are quite another.
which is what you two are implying, the crooks would soon be working full-time on exploiting it just as they do now with Windows. Bearing in mind that these guys have billions of dollars to pour into circumventing OS security, I doubt it would be long before Linux would be just as aswarm with malware as Windows now is. Bear in mind that most malware gets installed by convincing an ignorant user to do something to install it; e.g. scanning for viruses, looking at a jpeg of some girl's tits, whatever. Right now, these problems don't exist on Linux because the vast majority of it's users are too tech-aware to be phished. But if Jane and Johnny Sixpack start using Linux, how long will it be before some malware site convinces them to sudo some rogue software into their system?
The issue is not Windows or Linux or MacOS, it's the popularity of the platform that determines malware-purveyors' targets.
This had a huge impact to enterprise and carrier class business. Since it was workstations the impact wasn't as visible to the 'outside' world, but i'd have to think from just what I saw that the actual cost in lost work and recovery efforts will be in the tens of millions.
A core file.... identified as a virus. A core file... that's part of every windows installation.
The only conceivable way that something like this does not get found immediately in even the most cursory testing is that.... They don't even bother to test at all!
Truly, an occasion for the WTF? icon.
"Getting advice directly from McAfee is a far better option."
Doing the same thing repeatedly and expecting a different outcome is a definition of "insanity". Why bother asking the company that fsked you up for advice on how to unfsk you?
The best option ("far better" even than your suggestion) is to abandon the offender entirely in favor of better options, of which there are many. I can't believe that McAfee and Symantec are still in business. The number of security issues those companies have caused by screwing up their programming are on par with Microsoft, itself. There's a lesson, here, for those willing to observe.
The blacklist simply doesn't work anymore... time to move on. The trouble is even though more secure options exist, AV is so ingrained that now some places are legally required to have it. Even if it makes them less secure.
Don't get me wrong this is an epic fail by McAfee, but some admins make life more difficult for themselves by not using DAT-1 this is not allowing for failure by the third party and admins should remember this type of event does seem to occur all too often.
If using McAfee, then EPO is essential and in the 4.0 documentation it alludes to 'you can schedule a DAT Repository Pull once a week' (p.136) and it states you can make use of the Evaluation branch for testing of DAT (p. 144).
Its obviously a fine balance of being protected but giving yourself a 24-48 hour window to stop the dodgy DAT from being downloaded seems like a good idea if you don't want to visit 40,000+ machines or potentially affect your career prospects. When you get the email saying theres a major new threat and they have released a DAT which protects against it, force an update out asap then. By being at least 1 DAT behind I can rely on all those other admins testing it for me.
All AV suffer from this kind of event. Hopefully in the near future improved whitelisting technology will reduce the risk of these situations.
This has caused massive grief at home (where I was stupid enough to go for mc-crappy on a cost constraint basis), and I have a doner machine to get svchost from. mc-crappy help was useless in resolving the problem they caused. a$$hole$ ! What about your single machine home user ? No internet without svchost so no way to fix. Brilliant. Thanks to el Reg and all the info posted I was able to fix the problem. Only possible as my main system is not windo$e.
What ever happened to testing ????? mc-crappy clearly don't do it !
Also you get an insight into how lazy and sloppy the mc-crappy code is. The machine that got screwed was told to update immediately prior to a scan, as was another machine in the house. One machine had been online already that day, and had already downloaded the problem update. IT DID NOT DOWNLOAD THE LATEST (FIXED) UPDATE WHEN INSTRUCTED ! Consequently it got turned into a useless brick. The other machine did update to the fixed version and was OK. This is inexcusable.
The consequence of svchost.exe missing was mc-crappy died completely. The affected system was screwed and completely insecure ! That wonderful m$ architecture shows it's true colours yet again. USE THIS SHIT AT YOUR PERIL (I don't by choice). Why the fuck critical system files like this are not truly READ-ONLY (as in like a ROM) is beyond me.
I need a pint to wash out the bad taste this has left behind.
Flame aimed at mc-crappy.
I got the luck of getting dragged into helping someone from another 40K end user company (name withheld) when they got zapped. The symptoms popped up rather rapidly. I had time to get Task Manager open after boot, and McAfee console to begin a scan and the 60 second countdown to shutdown popup appeared. The only way McAfee could not have seen this is if they just rolled this update right out the door without even loading it onto a test machine in their QA lab. By the time I got the console open it had already identified svchosts.
IMHO we might as well assume the cockroach principle here: if you see one, how many more are there? In other words, how many other updates have been just going out the door? And that is the end users' and admins' responsibility to test before deployment? There is an implied trust that has been broken here. This is gross negligence.
Finally, what kind of a piss poor response is it they shovel out on their customers who did get affected, all downplaying it so that they can save face in the industry? Only a small number of machines? WTF.
Taking 10% of the machines offline for the better part of 24 hours at a government agency is no minor impact, particularly when the impact removes the capability of having an automated fix because the systems can no longer phone home. The 60 second reboot timer was no minor annoyance either. Pretty much the whole IT staff in my building for 2000 people was working on nothing but cleaning up this colossal mess for two days. We were still getting alerts on day 2 even though the network admins pulled the update as soon as it was identified.
DAT-1 is no option when the security people have a policy in place that requires automatic full scans to protect the environment. The Internet is simply too toxic these days to go with less than full MOP gear. Which makes it critical that outfits like McAfee get their shit right.
received "in error" contained the following within 10 minutes of the bad DAT:
- Over 100 phonecalls in the support queues - lines are now full in all regions and customers are getting engaged tone
- Support tickets open in Europe from 7 customers with either BSOD or DCOM and then a shutdown since 5958 DAT
- 8000 machines at HBOS, more at Evolution Group Services and Cadburys
130 out of 160 machines on my network corrupted...
Have told our account exec we want several years free licenses to even start thinking about keeping the product, otherwise we're switching - probably to Symantec. I can't find any major chaos caused by them in the press archives, whereas some other vendors (McAfee included) are quite adept at getting headlines from cockups.