Security companies forge an anti-malware standards organization

Security computer companies recently met in Spain to officially create the Anti-Malware Testing Standards Organization (AMTSO), with the objective of formulating new guidelines for the proper review of anti-malware software.

Originally conceived during the first International CARO workshop in Iceland last year, the organization's first meeting in Spain featured more than 40 security gurus familiar with the basic pitfalls the security industry faces today.

"Users often regard tests of anti-malware products as important, but most tests have a different winner, and most of the time the testing methodologies are not described," said Righard Zwienenberg, chief research officer of member company Norman Data Defense Systems. "If they are, rather often they are outdated, not compensating the new technology created by anti-malware vendors to detect new threats. And then these tests do not include the new technologies within the methodology, then the test result is flawed and not representing results in the real world."

AMTSO has three pro tem committees made up of industry security professionals from participating companies.

Senior research engineer at the Kaspersky Lab, Roel Schouwenberg, publishes an unofficial quarterly blog giving both security experts and regular computer users a glimpse into how experts evaluate the viability of security software.

"Reviews published by computer security/virus specialist magazines can provide you with information which may be useful in determining a product's strengths and weaknesses because they have a distinct advantage," reads a white paper published by AMTSO this morning by Sarah Gordon and Dr. Richard Ford, research analysts with member company Command Software Systems. "These reviews tend to be well done and informative, focusing on the ability of products to meet published criteria."

But there are still several basic problems that need to be addressed, the team pointed out. For example, most tests available today have a very limited scope: Real world scenarios produce results for a wide variety of situations, as opposed to a single test environment. Most reviews they find only cover one software performance area, rather than the whole gamut.

The AMTSO hopes to eliminate the complexity of reviews commonly published in computer magazines. The average reader, it believes, is unable to understand the phraseology, possibly leading to a misunderstanding of what anti-malware categories actually do, and how they're distinguished from one another.

As malicious users continue to create more complex software to damage PCs, reviewers are left with no guidelines and very little realistic ways to test how software programs work.

The lack of guidelines has brought Symantec, Trend Micro, Panda Security and other companies together to develop new methods of properly evaluating software. Since the complexity of evaluating anti-malware technology is so high, AMTSO will call upon companies, researchers and vendors to contribute methods and ideas for different ideas.

The AMTSO plans to create a bulletin board for users to discuss topics related to product reviews, and to help create an outline on how users can properly review security products. It has yet to publicly reveal its milestone dates for these efforts.

AMTSO members will gather together on April 30 in The Netherlands, before heading to the second International CARO event.