Security researchers are close to formulating plans to overhaul anti-virus testing amid growing concerns that current tests can be misleading. Anti-virus packages are traditionally tested for their effectiveness in detecting a sample of malware packages known to be in circulation. Products that fail to detect a sample get a …
Wow, looks like we've gone full circle.
I remember my anti-virus software on my Win 3.1/DOS box detecting viruses with both signatures and heuristics. It was fairly decent at it too. Of course back then anti virus was actually a technical challenge what with viruses deploying stealth techniques like polymorphism, IAT hooking etc rather than an exercise in generating hashes of files.
Good, but not good enough
The proposal is a step in the right direction - but it is not good enough. If the tests are conducted as the proposal suggests, they will essentially test obsolete (months-old) heuristic scanning capability. This isn't good enough, although it's better than the current situation.
The proper way to do it is to install an up-to-date version of the AV software and attack it with *live* malware. Don't just let it scan a "dead" collection of samples. Start executing the malware - and then see if *any* part of the package manages to prevent (completely!) the infection of the test machine. It doesn't matter whether this is the scanner, the heuristic analyzer, the behavior blocker, or anything else that stops the malware - it's sufficient if the malware is stopped.
There's the problem with the antivirus industry
If you ask me - this is what is wrong with the antivirus / endpoint security industry today. Too many people patting themselves on the back for fighting malware, and not attention paid to real-world effectiveness. This article just sent me off on a rant - http://bit9.com/blog/home/tabid/15398/bid/2456/Antivirus-Protecting-Against-Yesterday-s-Malware.aspx
Up-to-date heuristics/out-of-date signatures
Is it not possible to mate, for the purposes of testing, an old or even blank/minimal signature file with the latest heuristic engine if you want to test the capacity of the heuristics to detect threats?
It sounds like a great idea to me.
They want to test the ability to protect against previously unknown threats.
The best way to obtain test against unknown threats would bet to travel one week into the future and obtain the latest real world nasties.
However until they get their time travel machine working, they decided to do the next best thing.
Today's threats vs AV software that has been frozen in time for a week.
As far as it being unfair because the AV software doesn't have the latest updates, I wish I lived in a world where AV software became dramatically more effective on a week to week basis. :)
Blank signature file
Anonymous Coward: Yes, theoretically, it is possible. In practice, however, it is not. First of all, practically no AV vendor will supply you with a "blank signature file". We (F-PROT) used to do it only for our macro malware signatures and nowadays even we don't do it any more. Furthermore, the term "signature" is misleading. Contrary to popular belief, it's not a collection of scan strings for known malware. Nowadays it is a complex database containing whole programs for detecting malware. Often even the scanning engine of the AV product is updated by this database. So, if you use an old database, you're running the risk of using an old (even buggy) AV engine.
- Mounties always get their man: Heartbleed 'hacker', 19, CUFFED
- Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
- Samsung Galaxy S5 fingerprint scanner hacked in just 4 DAYS
- Feast your PUNY eyes on highest resolution phone display EVER
- AMD demos 'Berlin' Opteron, world's first heterogeneous system architecture server chip