Stop Pen Testing Like It’s Y2K – How to Implement Continuous Penetration Testing

It’s 2:00 AM on a Friday and your phone begins to ring. On the other end is the desperate voice of your IT Security manager trying to communicate that your company has just been hacked and that it looks like customer data may have been stolen. “How can this have happened?” you exclaim. “Didn’t we just complete our annual penetration test a couple of months ago?”

This very real scenario highlights one of the significant challenges surrounding annual or semi-annual penetration tests. As an industry, are we doing enough to protect ourselves from the ever-increasing risk of cyberattacks?

Hackers don’t schedule annual invasions of your infrastructure, so why are most companies stuck with the yearly penetration testing model? This article explores the benefits of the continuous penetration testing methodology and how organizations can implement it cost-effectively.

Annual Penetration Testing Is Not Effective

Did you know that in Y2K, there were 1,438 publicly disclosed information security flaws (CVEs)? Compare that figure to the whopping 21,085 vulnerabilities discovered in 2023, and it becomes apparent that the pace of security is increasing at a gallop. While no organization will be open to all 21 thousand vulnerabilities, all it takes is one unpatched weakness to give an attacker the necessary advantage. The problem with annual or semi-annual penetration testing is that they can be rendered obsolete within a few weeks to a month as thousands of new vulnerabilities are discovered.

Skilled threat actors, intent on compromising the defenses of your IT infrastructure, maintain lists of technology they’ve learned you employ through their reconnaissance of your digital footprint. When a new vulnerability is publicly released, they often gain a tactical advantage by exploiting the window of opportunity between disclosure and patching.

 

Automated Vulnerability Scanning & IDS Is Not Enough

Automated vulnerability scanning and intrusion detection systems (IDSs) are valuable tools in the arsenal of an organization’s active defense, but as stand-alone tools, they’re not enough to prevent compromise. This is primarily because both utilize signatures, like digital fingerprints, to detect potential vulnerabilities and unwanted activity. In many cases, especially with newly disclosed vulnerabilities, there’s not enough information on how the vulnerability is being exploited in the wild to identify it while it’s happening.

Take, for example, the recent theft of Personal Health Information (PHI) of close to 4.5 million customers. From a postmortem examination of the incident, it is obvious that the organization had substantial detection and response mechanisms in place, but even that proved to be inadequate in preventing the breach. Although unconfirmed, it is likely that the compromise was the result of an unpatched vulnerability.

According to Andrew Costis, chapter lead of the adversary research team at AttackIQ, “…Organizations can leverage the common tactics, techniques, and procedures (TTPs) used by threat actors, testing them against their current security measures to identify gaps or potential blind spots. Simulating these attacks through continuous testing will help promote a more proactive and efficient response.”

This is why skilled penetration testers, sometimes called Red Team Operators, are a vital part of an organization’s cyber defenses. Unfortunately, employing a dedicated staff of full-time Red Team Operators can be cost-prohibitive. This is why many companies choose to outsource their Red Team operations to third-party organizations in a Continuous Testing as a Service (CTaaS) model.

What Is Continuous Penetration Testing?

Unlike traditional single-point-in-time annual or semi-annual penetration testing, a continuous penetration test is an ongoing adversarial attack simulation that closely emulates real-world threat actors’ Tactics, Techniques, and Procedures (TTPs). It begins with a Baseline Assessment designed to identify unpatched vulnerabilities and provide a roadmap to remediation. This stage usually lasts 1-2 weeks, depending on the complexity of the organization’s attack surface.

The next phase is the Threat Modeling phase, where a list of all software in use is collected (called a Software Bill of Materials or SBOM), along with the exact version numbers. Attack trees are constructed that describe how each software component if found to be vulnerable, may impact the confidentiality, integrity, or availability of the organization’s IT assets. The threat models and attack trees are updated to reflect the changes whenever software components are modified or updated.

The third phase plans and executes a series of Directed Attacks that simulate adversarial behavior from various standpoints, such as Initial Access Attempts, Assumed Compromise, and Post Exploitation activities. This is the heart of continuous penetration testing, and a large part of it involves continuously researching modern TTPs, and designing attack patterns to determine how an organization would detect and respond to various threats. An excellent resource organizations can use to understand current trends in attack patterns is the MITRE ATT&CK framework. Organizations can use this resource to design simulated attacks that more accurately reflect real-world threats than traditional (read spray and pray) penetration testing.

This phase also includes monitoring security advisories and cross-referencing them with the Threat Models and SBOM in real time to expose organizational vulnerabilities before a malicious actor exploits them.

 

Continuous Penetration Testing Is More Cost-Effective

The primary reason most companies are not yet performing continuous testing is because they believe it is cost-prohibitive. However, when compared to the cost of annual or semi-annual penetration tests, the year-over-year costs of continuous pen testing are often significantly less expensive when outsourced. This is due to several factors:

1. Continuous testing actively monitors the changes in an organization’s IT infrastructure and focuses on testing the deltas instead of taking a broad-spectrum (read shotgun) approach.
2. There is no recurring ramp-up time. In a typical annual penetration test, anywhere from 1-3 days of consultant ramp-up time is embedded in each engagement. This allows the consultant time to set up testing environments and understand the organization’s critical infrastructure. This ramp-up time typically costs between $2,000 – $6,000 of non-testing time each year. With the continuous testing model, there is only one ramp-up event, and in many cases, this fee is waived.
3. In an average annual penetration test, 1-3 days of reporting time is embedded into each engagement. This cost accounts for an additional $2,000 – $6,000 of non-testing time each year. With continuous penetration testing, the bulk of the report is generated once, and after that, the ongoing updates take minutes, not days.

With traditional, annual penetration testing, one-third to one-half of the engagement is spent on costly non-testing activities. After the first year, the ROI of continuous security testing becomes readily apparent, translating into significant savings. More importantly, given the rampant rise in publicly disclosed information security vulnerabilities, continuous testing is helping organizations close the risk window between annual or semi-annual penetration tests.

Summary

Given the prevalence of publicly disclosed information security vulnerabilities, averaging over 2,000 every month, and highly skilled threat actors are constantly attacking, companies can no longer afford to consider penetration testing an annual event. Without continuous testing and ongoing updates of the penetration test report, an annual penetration test can be rendered null and void just days, hours, or weeks after it is finalized.

Moreover, continuous penetration testing should be informed through a deep knowledge of the techniques, tactics, and procedures (TTPs) used by today’s threat actors.

With the right approach, continuous penetration testing can offer better insights, greater security, and more value while maintaining higher ROI and cost-effectiveness than its Y2K ancestor. Annual penetration testing worked well decades ago when only a little over a thousand new vulnerabilities were disclosed yearly. Today, continuous testing emerges as the leading methodology to secure organizations from the genuine threat of compromise.

If you want to learn more about L&C’s “Continuous Testing | Immediate Insight” offering, contact us today and schedule a free continuous penetration testing consultation.