Product testing plays an important role in the cyber security industry, allowing buyers to make informed decisions and helping make our products better. This is increasingly important as attacks continue to escalate in both quantity and sophistication. The availability of real-world, standards-based tests that are verifiable, statistically valid, and objective will result in better outcomes for our customers and the larger cyber security community.
Surprisingly, the cyber security industry has never had a set of standards for product testing. While many tests may be done in an honorable and ethical way, the lack of a consistent standard has allowed some questionable testing practices to emerge, harming the cyber security community by providing inaccurate information, and eroding confidence in security product testing in general. In response, earlier this year the Anti-Malware Testing Standards Organization (AMTSO) – an organization comprised of both security vendors and testing organizations - ratified the first ever standard for testing cyber security products.
AMTSO is a non-profit organization dedicated to improving the conditions related to the development, use, testing and rating of anti-malware products and solutions. These standards ensure that important information about the test and the participants are disclosed. This includes critical pieces of information such as which vendors (if any) had a chance to configure their product, which vendors (if any) had the opportunity to dispute their results, and which vendors had feedback or commentary on the methodology of the test. While adherence to the standard does not guarantee the quality of a test, it ensures a degree of transparency, allowing buyers to draw their own conclusions.
The emergence and ratification of the AMTSO standard is an important milestone in our industry. Now both vendors and testing organizations have the opportunity to embrace this standard. We support this standard because we believe that the cyber security industry will advance if all security products have their capabilities tested by organizations that conduct open, transparent and ethical tests. We believe that our customers and industry are best-served by tests that apply standards (such as AMTSO) and are built upon the following principles:
- Openness: Full testing results should be free for all members of the security community and the general public.
- Transparency: Testing methodologies should be transparent, with conclusions verifiable and drawn from statistically significant sample sizes based on the results of the test.
- Integrity: Vendors should not introduce features for the sole purpose of scoring better on the test and testers should not favor products based on their degree of participation or patronage.
- Collaboration: Testers and vendors should communicate before, during and after a test. This communication can facilitate product access for the testers, provide the vendor with the opportunity to actively participate in the test (or decline participation), and allow the vendor to respond to or dispute results.
- Legitimacy: Testers should adhere to accepted industry standards as they emerge.
We applaud the testing organizations SE Labs, AV-Comparatives, and MRG Effitas for their early adoption of the AMTSO standard. We are also encouraged by the work of MITRE (another not-for-profit organization) who are conducting high quality security product testing in a collaborative and transparent way.
We encourage organizations to insist on standards-based tests that comply with transparent and ethical guidelines, allowing buyers to make objective decisions when selecting security products. To learn more about AMTSO standards-based security product testing, visit the AMTSO Standards Homepage.