Vulnerability Test Sets
CVE testing in RFC 9411 falls under Security Effectiveness Evaluation, defined in Appendix A. Rather than being a standalone test, the document is primarily focused on performance benchmarking, but it is RECOMMENDED to validate the security features configuration of the DUT/SUT by evaluating security effectiveness as a prerequisite for the performance benchmarking tests. In other words, a device must first be shown to detect known threats before its throughput and latency numbers mean anything.
Selection Criteria
Minimum Set Size
The vulnerability set should reflect a minimum of 500 CVEs. This floor ensures statistical breadth — no cherry-picking a handful of easily-detected exploits.
Date Range (Age)
CVEs should come from no older than 10 calendar years to the current year. This keeps the test set anchored to the threat landscape that real enterprise networks actually face, rather than ancient vulnerabilities no longer in the wild.
Target Software (Application Relevance)
CVEs should be selected with a focus on in-use software commonly found in business applications. This directs testers away from obscure or niche platform vulnerabilities and toward the software stack most representative of enterprise environments — web servers, databases, productivity applications, and similar.
Severity Threshold (CVSS Score)
CVEs should be selected with a CVSS Severity of High (7–10). Using the Common Vulnerability Scoring System as a filter ensures the set reflects genuinely dangerous exploits, not informational or low-impact findings.
Test Sets
The test sets currently are made up of three components:
-
The full test set
-
The public test set
-
The private test set
The full test set is split into a public set and a private set. These, together, constitute a subset of the total number of samples in the full set. This is done to prevent DUT vendors from "gaming" the test. The full set will be released, as will the public set. The private set will be known only to the test tool vendors and the accredited labs. For example, a full set could contain 1,000 samples, a public set 700 samples, and the private set only 150 samples. The full set will be a known quantity, as will be the public set. But no one will know which of the remaining 300 samples will be used in the private set. The theory is that if a DUT vendor tries to game the test, the detection rates could differ significantly between the public and private sets. If this were to happen, the NetSecOPEN Certification Body could be notified, which could lead to a conversation with the DUT vendor.
Version 1 of the test set was released January 15, 2023.
Version 1.1 of the test set was released February 2, 2024. The only difference between version 1 and version 2 is the removal of CVE-2016-3368 as the sample was found to be encrypted.