Can you trust anti-virus rankings?

It seems no-one can agree on a universal set of tests for anti-virus software, with Eugene Kaspersky the latest to weigh-in on the topic, criticising the well-known Virus Bulletin 100.

While major AV vendors promise solid protection for your PC, you might think twice if you read any of the major security bulletins, which regularly pan the big brands.

Kaspersky is one of several big anti-virus brands to fall foul of the VB100 tests, reportedly failing to pass a recent test of security software on Windows Server 2008, along with F-Secure and Computer Associates.

And if that doesn't make admins leap for the "uninstall" option, perhaps this one will. Security outfit Secunia tested 12 major Internet security suites against 300 exploits, and came up with the stunning conclusion that "major security vendors do not focus on vulnerabilities".

Unsurprisingly, anti-virus vendors are irritated at statements like this, including Kaspersky CEO Eugene Kaspersky.

"I don't want to say it's rubbish," Kaspersky told PC Authority. "But the security experts don’t pay attention to these tests. It doesn’t reflect the real level of protection."

"The products which have a very poor level of protection, they have the certificate, while products which have a very high level of protection, they don’t have the certificate."

It seems the industry still can't agree on the best way to rank AV vendors.

Kaspersky criticised anti-virus tests that focus on static exploit scanning alone. "To have true test you have to put malicious file on the Web, then click it, or have an infected exploit which sends exploit to machine. These tests are really expensive and take a lot of time and hardware."

The Secunia tests in particular seem to have generated disagreement over testing methodologies. Secunia's recent drubbing of most major brands brought the issue of PoC (Proof of Concept) testing to the forefront - a method that involves code that triggers a vulnerability, but doesn't actually carry a virus "payload". Real exploits were also part of the test.

At Kaspersky, bloggers have pointed out that they don't focus on detecting PoCs, calling it a "dead end", and saying their antivirus database focuses on "real threats and exploits."

Other have questioned the relevance of major anti-virus rankings altogether, with some criticizing signature-based detection as the problem (something anti-virus vendors are supplementing with behaviour-based detection).

Kaspersky says behaviour based blocking, that stops applications once suspicious behaviour is detected, would be a good feature in Windows 7.

"Application control is one of the most promising technologies which can bring the net a layer of security. Just to run apps with different rights. I call it application harassment."

"The problem is that in the industry there’s no other complete tests," says Kaspersky. Many sites including PC Authority refer to VB100 results in their reviews, though we also refer to behaviour based blocking where relevant.

Until anti-virus vendors figure out a way of replace signature-based scanning entirely, the best thing you can do, in addition to having full anti-virus protection, is to patch your PC.

Read our review of Kaspersky Anti-Virus 2009.