FireEye Stories

FireEye’s Support of Standards-Based Third Party Testing

Every single day FireEye protects thousands of organizations from an ever-increasing cyber security threat landscape. We take our customers’ trust seriously and are completely focused on helping them effectively navigate uncertainty and minimize the consequences of cyber attacks.

To make informed choices customers must have objective, transparent, and accurate information on cyber security product testing. Standards have driven the growth of IT since the early days of computing, but surprisingly, there has been a lack of standards-based cyber security testing. Moving forward, FireEye is fully embracing standards-based testing, and we’re announcing our support for the Anti-Malware Testing Standards Organization (AMTSO).

AMTSO is a non-profit organization dedicated to improving the conditions related to the development, use, testing, and rating of anti-malware products and solutions. These standards ensure that important information about the test and participants is disclosed, and the tests are consistent, transparent and unbiased. The organization lists 9 fundamental principles of testing:

  1. Testing must not endanger the public.
  2. Testing must be unbiased.
  3. Testing should be reasonably open and transparent.
  4. The effectiveness and performance of anti-malware products must be measured in a balanced way.
  5. Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.
  6. Testing methodology must be consistent with the testing purpose.
  7. The conclusions of a test must be based on the test results.
  8. Test results should be statistically valid.
  9. Vendors, testers and publishers must have an active contact point for testing related correspondence.

While adherence to standards does not guarantee the quality of a test, it ensures a degree of openness and transparency, allowing buyers access to the most transparent, ethical and fair assessment of cyber security products, and setting ground rules for vendors, including:

  • The right of a vendor to receive advance notice on a testing.
  • The right of equal treatment of vendors during testing.
  • The right of feedback on the design and implementation of testing.
  • The expectation of fair play from vendors.

AMTSO has garnered significant support, and now lists over 50 (security vendors and testers) members. We applaud the early adoption of the standard by testing vendors such as AV-Comparatives, MRG-Effitas, and SE-Labs.  

We are also working closely with The MITRE Corporation, a not-for-profit organization "dedicated to solving problems for a safer world". Shortly before RSA Conference 2018, MITRE announced that they will offer evaluations for security product vendors.

FireEye is fully committed to standards-based tests that allow customers to make informed decisions about cyber security products and ultimately create a safer future for all of us. We encourage the cyber security community to learn more about standards-based testing and the AMTSO at: AMTSO Standards.