Malware testing gains some structure

An international group formed in May to lay some ground rules for anti-malware testing has delivered a pair of documents setting forth basic principles.

The Anti-Malware Testing Standards Organization (AMTSO), which includes representatives from most of the major vendors on its membership roster, has published two documents setting forth fundamental principles (PDF available here) and best practices for dynamic testing (PDF available here).

There's a little echo of Asimov in the list of nine fundamental principles, which begin with the dictate that "testing must not endanger the public." (Public to AMTSO: Hey, thanks!) Malware undergoing testing must not be accidentally released into the wild, and new malware can't be created simply for testing purposes.

Most of the other principles involve balance, fairness. and clarity of communications. Testing must be unbiased, reasonably open, and transparent. Product effectiveness and performance must be measured in a balanced way, and testers must take "reasonable care" to check that test samples or cases have been accurately classified as malicious, not malicious, or invalid. The testing process has to mesh with the purpose of the testing. Results should be valid and conclusions should be based on those valid results. And everyone involved -- vendors, testers, and publishers -- has to be actively available for testing-related correspondence.

Meanwhile, most testers agree that dynamic testing -- the kind involving live threats, such as a piece of malware that tries to install something on the test machine -- is a much bigger pain than testing static functions such as on-demand scanning. The best-practice guidelines for the goldfish-juggling process that is dynamic testing include advice on reproducibility, product and sample selection, checking for false positives, the factors to balance when building a testing environment, and the use of networks and virtual machines.

There are also more specific notes about the hands-on portion of the testing process, covering logging and auditing the malware's behavior, possible measures of success, how to handle user-interaction moments, and the benefits of testing one piece of malware at a time versus testing in multiples.

Comments from AMTSO's 40-odd members responding to the publications of the guidelines were, not surprisingly, positive about the effort to bring some standards to the wild (and occasionally wildly inconsistent) world of anti-malware product testing. Karel Obluk, CTO of AVC Technologies, spoke for many when he noted, "Consumers of anti-malware solutions can only benefit from being able to recognize instantly whether or not a test has been conducted using universally accepted standards and make their purchasing decisions accordingly."