This blog post is meant to provide details on patch management including the importance of a documented patch management process, how to implement the process successfully, and some common issues and roadblocks to avoid when doing so.
What is a Patch & Why is Patch Management so Important?
A patch is a piece of code that is implemented to correct a coding flaw (system vulnerability). Patching is a very important part of an organization’s cyber hygiene to ensure its systems do not fall victim to those looking to exploit known vulnerabilities.
Given the very public breaches in the past related to exploited vulnerabilities (e.g. Equifax, Yahoo, Facebook, Marriott, etc) and the recent unprecedented Solarwinds 2020 supply chain attack, it is very important for organizations to reduce the risk of a breach. (Read more about what the Solarwinds attack taught us here.) One of the important ways is to review (or create) their patch management processes, to make sure that it is being followed, and that gaps are identified and filled.
What is the Difference Between a Patch & a Hotfix?
A hotfix is typically considered a patch that would need to be addressed quickly to remediate a typically critical software flaw immediately vs waiting to install the fix on the next patching schedule. On the other hand, a patch is a less urgent fix that would follow the normal patching cycle.
What is a Patch Management Process?
When patches to vulnerabilities need to be implemented, it is very important that a consistent and repeatable process is followed. This will ensure all patches are reviewed, tested, and validated prior to implementation. Developing a patch management policy should be the first step in this process. A patch management policy outlines the process an organization is to take to update code on a consistent and reliable basis to ensure systems are not negatively affected by the change.
How Do You Implement a Patch Management Process?
Implementing a Successful Patch Management Process
To reiterate a flexible and responsive security patch management process is critical to maintaining proper cyber hygiene.
There are many different methodologies and guidance to help with building a quality patch management process. The key takeaway is to make sure you implement a process that aligns with your organization’s people, processes, and resources.
The process implemented must be repeatable and there must be buy-in throughout the entire organization, from the administrators installing the patches all the way up to the executives and board of directors. If there is no buy-in, it does not matter how great your process is because the chances that the process is being followed are very low.
If you do not have a process in place or are taking this time to review and update, the SANS Institute InfoSec Reading Room has provided a good methodology on how to implement a patch management process. At a very high level the methodology is:
- Baseline and Harden
- Develop a Test Environment
- Develop a Backout Plan
- Patch Evaluation and Collection
- Configuration Management
- Patch Rollout
- Maintenance Phase – Procedures and Policies
Another guidance provided by IRS.gov is very similar and focuses on:
- Assess
- Identify
- Evaluate and Plan
- Deploy
- Maintain
Patch Management Procedures & Best Practices
Quality Inventory Management Process
If you do not know what you have in your environment, then how are you supposed to protect against threats?
Most organizations maintain a very basic asset inventory which usually includes Operating Systems, IP Addresses, location, and sometimes the owner. While this is helpful and way better than not having an inventory at all, it does not include enough details to properly understand the organization’s threat vectors.
A best practice is to expand the inventory list to include at least the OS version, installed applications and their versions, the function of the asset (i.e. Domain controller, web server, database), interfaces/services/protocols, and a criticality score. Feel free to include additional details that are helpful to your organization. If it can help in the discovery, assessment, testing, and patching of an asset, then add it in.
Reduce Threat Vectors
This is much easier said than done. As organizations grow, so do the number of assets and applications they must manage. Some organizations can have distinct operating systems and applications and others just a handful. Both scenarios can lead to a breach or increased risk if the assets and apps are not patched and maintained. However, the organization with thousands has a lot more work to do and they have far more threat vectors to manage and monitor.
How Often Should Patch Management Be Performed?
Best practice is to periodically review the operating systems and applications and work to reduce the number as much as possible. While Bob from accounting may really like using MS Works, it may not be in the best interest of the organization to continue to allow him to use it. Even if there is one legacy system or application in your environment that is unknown, out of date, or no longer supported, it can expose protected data and lead to an increased risk to the environment.
Discovery
You can’t secure what you don’t know about. The best way to know if a vulnerability exists is to employ discovery and scanning capabilities. A proper discovery service uses a combination of active and passive scanning features and the ability to identify physical, virtual, and on and off-premise systems.
In addition to scanning the network, you also should define a reliable system for identifying new vulnerabilities. Waiting for the operating system or application to release a patch is not sufficient. In some cases, a new vulnerability may not have a patch available and the organization must mitigate or remediate the risk by other means.
Best practice is signing up for notifications for all the operating systems and applications that you have in your environment that provides this feature. It is also best practice to subscribe to security bulletins or vulnerability reporting services that can help you identify emerging threats and new vulnerabilities. Review these notifications and compare them against your inventory to ensure you are up-to-date.
Patching is Not Just for the Operating System
It is surprising to see how many organizations focus their patching solely on operating systems. They either ignore or prioritize application patches much lower than the actual risk warrants because operating system patching is “easy” and everything else is perceived as hard. As was the issue at Equifax during their 2017 breach, an application vulnerability was identified but it took months and a breach for the organization to patch it.
Therefore, don’t just focus all efforts on the operating system. Make sure to also take into consideration installed applications, services, libraries, and devices. With the growth of easy-to-install apps, IoT, and connected devices, everything matters. Review the threats and vulnerabilities to determine the risk and prioritize. Don’t just lower the priority or not patch something because is hard or complicated.
Metrics & Verification
How are you doing with your patching? Without measuring your performance, it is hard to determine if you are meeting your target and organizational goals. Metrics can help to validate that your patch process is effective and provide valuable information that can demonstrate the security posture to the business in a meaningful way. Metrics like percentage of systems up-to-date, percentage of failed patches, number of hours/days to patch, etc. are all valuable to the organizations.
Along with the metrics, an additional item that often gets missed is to verify the patch actually remediated the vulnerability. Many organizations just assume that once a patch has been installed, they are good to go, without going back to verify. Ideally, this is done in testing before the patch is installed. If not, rescanning the environment or testing a few machines after the patch was installed is necessary to determine if the vulnerability has been remediated.
Common Issues & Roadblocks with Patch Management
Without getting into too much detail, below are some high-level issues and roadblocks organizations may run into when trying to implement and follow a patch management process.
- Not reviewing exceptions: Exceptions are generally written when something can’t be patched. Many times these exceptions are never reviewed after they have been written. Remember to go back and reassess all exceptions periodically to make sure they are still necessary and not introducing additional risk that was not originally defined.
- Being forced to support aging software: This is touched upon above. Sometimes you must support old software. If that is the case, work on trying to limit access, network segregation, and adding additional monitoring to reduce the risk.
- Not viewing risk as a whole: Most risks are written for each threat or vulnerability, but no one took a look at all the risks as a whole. A low risk somewhere else in the organization may increase the risk for a given threat or vulnerability. Also, if you have hundreds of low risks, maybe the overall risk is now medium, or even high.
- Not enforcing a patch management policy: If you have a policy but don’t follow it or don’t enforce it, then why have it? Be sure to hold teams accountable for not meeting the targets and goals.
What is the Difference Between Patch Management & Vulnerability Management?
In short, vulnerability management consists of processes and tools implemented to identify and remediate vulnerabilities. Vulnerabilities identified from these processes and tools may require patches that would trigger and initiate an organization’s patch management process.
What are Patch Management Tools?
Patch management tools can simplify and automate an organization’s patch management process across many different applications and operating systems. Techradar provides details on the top-rated third-party tools of 2021. You can learn more about some of these tools here:
Summary
In summary, implementing a patch management policy and patch procedures are critical to ensure that vulnerabilities are being rectified in a timely manner to ensure the risk of a breach is greatly reduced.
Linford & Company provides multiple services like SOC 1, SOC 2, FedRAMP, and HIPAA audits that are designed to assess an organization’s security management and regulatory compliance effectiveness. Contact us if you would like to discuss our services further.
This article was originally published on 8/7/2019 and was updated on 3/3/2021.
Linford & Co., LLP, founded in 2008, is comprised of professional and certified auditors with specialized expertise in SOC 1, SOC 2, HIPAA, HITRUST, FedRAMP and royalty/licensing audits. Our auditors hold CPA, CISA, CISSP, GSEC licenses and certifications. Learn more about our company and our leadership team.