Why Methodology Matters: Guidelines for Evaluating a Real-World Security Test

In the last year, we have seen security vendors and well-known testing labs go toe-to-toe in headlines, blogs and social media over the methodology used to produce final test results. Claims of flawed methodologies, out-of-date software, improperly configured devices, or solutions unable to access cloud intelligence have been made. This creates a climate of contention where little can be agreed upon, and even less progress is made toward a consensus on improving the reliability of results.

Similar to other vendors, Websense is often approached to participate in independent security bake-offs. Recently, we had to decline one such invitation from a well-known testing lab because we felt the methodology used was flawed and would produce misleading results. Like many in this industry, we know that real-world security isn’t something that can be accurately tested in a static Lab. Enterprise buyers want reliable performance tests to help guide their own purchasing decisions. The gap between agenda-based security tests and what’s needed for effective results led us to explore the current challenges and best practices for a successful testing methodology.

Lack of Methodology Agreement and Shifting Threat Landscape Impair Security Effectiveness Tests

When there is a lack of an agreed-upon testing methodology to validate the performance of security devices, testing security solutions can be a challenge. Even if the industry could agree upon testing protocols, the rapidly shifting threat landscape shortens the shelf-life of a test almost immediately. Accurate assessments and comparisons of solutions can only be done simultaneously and “in the moment” of the current threat landscape.

Web threats, for example, could be hosted on a site that was “clean” yesterday, infected two minutes from now and then “clean” again an hour later.

In addition to “live” threats, you also need “recent” threats to test against – malware, exploits and redirects that have just emerged. In today’s security landscape, where a quick automated code change is all that is necessary to avoid signature-based detection efforts, each sample must be verified malicious at the time of testing with each unit under test. 

Bottom line, creating an effective testing methodology is difficult and when the process is flawed, it creates more confusion and controversy than answers for those in charge of making security product purchasing decisions.

Testing Methodology Should Involve Cooperation Between Vendors, Testing Labs and Enterprises

In some ways, the past year's intense scrutiny on testing methodology is good for the industry, opening the door toward more effective and clear communication on what needs to happen to create a true, real-world security effectiveness test. Rather than getting caught in the negative feedback loop, there is a real opportunity to build the framework for future testing that inspires confidence, rather than confusion, in security buyers.

Let’s explore some of the necessary requirements needed for any credible future testing protocols:

·         Test methodology and benchmarking the performance of network and content security elements must be properly defined for optimal results. We need to precisely describe what we will be measuring and how we will measure it.

·         A well-executed test should reflect what is important in selecting modern threat protection by testing a representative diversity of relevant attack types that enterprises face today.

·         Testing labs should combine their own expertise with vendors’ knowledge of current real-world applications. Security vendors are usually the first to encounter evolving or new threats; either by stopping them, or by missing them and then reviewing why the threat bypassed existing security measures.

·         Any testing should accurately measure the real-world performance characteristics of a typical enterprise network in both small and large scenarios.

o   To do so, live and active, not archived threats, must be included in the set and performance must be monitored and assessed simultaneously. The reason a security solution has taken action or identified a threat must also be assessed to avoid rewarding solutions with inaccurate detections and penalizing those with real-time classification capabilities.          

Yes, It IS Possible

Ultimately, we believe an accurate, industry-standardized security effectiveness test can be developed. However, it will take open communication and sharing by security vendors, independent testing organizations and enterprises seeking to protect themselves from today’s advanced attacks.

Testing labs and vendors should share information and processes to produce optimal and accurate testing results. In doing so, all parties need to check egos and marketing pabulum at the door. Our collective focus needs to be on the enterprise as the beneficiary of more accurate testing results.

By working together to remove confusion and discord from the marketplace, we believe we can create better information security for us all.

photo credit: https://www.flickr.com/photos/111692634@N04/