Updated: Jun 7, 2019
A couple of years ago while I was employed as a programs manager at a U.S. network security testing lab many vendors were encouraging us to get into the performance testing business. I had always been resistant to taking that step. The reason was that I felt it would be almost impossible to stand up a credible offering that was both fair and an accurate reflection of the real-world. There were plenty of ways I could hurt myself, I figured. Why add more to the list?
Well the pressure didn’t let up. The vendors were looking for performance testing that was open, transparent, fair and actually reflected reality. They wanted a standardized approach that wouldn’t change based on the testers whim.
I started thinking about this problem and talked to a number of security product vendors and enterprises. Everyone, and I do mean everyone, said that the current state of performance testing was subjective and for the most part meaningless. Results were presented without explanation on how or why the set of results was important nor was there any guidance on how to interpret results. Consumers of these test results were left wondering why it wasn’t possible to get test results that would support an apples to apples comparison. Or worse yet, just blindly accepted the results and made purchasing decisions based on dubious data.
After discussing this problems with various vendors and enterprises, it was determined there was a real need for an alternate to the existing performance testing offerings. After getting commitments that resources would be made available to support the creation of a transparent and open set of performance testing standards this project was launched.
The ultimate goal will be to produce an Open Performance Testing Standard that would be available to use by all of the stakeholders. Stakeholders are – testing, enterprises, network product vendors, and tool vendors.
NetSecOPEN in conjunction with product vendors, tool vendors and enterprises will be working together to develop an Open Performance Testing Standard.
The initial phase of this effort is focusing on Next-Generation Firewalls (NGFWs). Subsequent phases will see the expansion into other technologies.
More details on this effort will be discussed in upcoming blogs.
If you have any questions please feel free to e-mail me at firstname.lastname@example.org.
About the Author:
Brian Monkman, A NetSecOPEN projects/programs manager has been working in the network security industry for over 20 years. Prior to NetSecOPEN he worked at ICSA Labs for 17 years. His responsibilities included managing multiple technology specific consortia and programs. ICSA Labs, an independent 3rd party network security testing lab has been in existence for over 25 years. Monkman has also worked as a customer service manager, product manager, sales engineer amongst other roles at various networking companies.