An End-to-End Patch Management Process
Patch management in an operational technology (OT)/industrial control system (ICS) setting is full of challenges. From proprietary hardware and software to a lack of staff, inadequate or non-existent testing equipment, and regulatory reporting and system maintenance, many organizations struggle to determine what is in scope. This results in unmanaged patches.
To combat the challenges with patch management, Verve Industrial Protection developed a multi-staged approach to streamline the patch management process. Combining our world class software with cost effective, scalable services, this blog outlines the various stages of a patching process and highlights how we help.
What is OT/ICS Patch Management?
Software patching is often thought of as a basic cyber security process. On the surface, it appears to be a straightforward practice: simply apply updates to your OT systems.
The software updates are provided by the vendors that are intended to close any security or functional holes in your systems. This is so basic on paper that it is often overlooked or neglected by many security teams and system operators.
Patch management is defined as a comprehensive cycle of ensuring baseline data, identifying available patches and known vulnerabilities, reviewing patches for applicability and OEM-vendor approval, designing deployment or mitigation strategies, executing patch deployment and confirmation, and finally re-establishing baselines.
But as it turns out, patching is not so straightforward after all. In fact, it is likely the single most time-consuming task that the North American power industry faces in adhering to regulatory expectations.
This is due to a combination of factors, most notably:
- Lack of automatic inventory/monitoring of end systems
- Difficulty in monitoring patch releases for all systems/applications
- Time and expertise to review, approve, or mitigate patches in a workflow
- Testing and individually assigning patches to groups of endpoints
- Time to deploy on each device and confirm update working as appropriate
- Time to document changes and update baselines
Because of these patch management challenges, we created a six-step, end-to-end patching process. Using a combination of our Verve Security Center (VSC) software and our Verve Engineering Services (both off site and on premise), we significantly reduce the time and complexity and improve the quality and compliance-readiness by integrating each of the critical steps in a single-flow process.
6 Steps to Effective OT/ICS Patch Management
Step 1: Establish Baseline OT Asset Inventory
The first problem many organizations face is gathering a comprehensive asset inventory to understand what assets they have plugged in, where they are located and what software is deployed. Some organizations have managed to compile a reasonable list of assets, either manually or through extension of existing corporate tools or agent based technologies.
However, almost all industrial operator networks struggle to connect on a regular basis (let alone automatically) to the non-windows machines. In a typical operational network these proprietary systems constitute up to 75% of all assets.
Verve Security Center provides visibility into all 100% of connected assets. Through a combination of our best-in-class tools for agent-based systems and our proprietary Agentless Device Interface (ADI), our clients inventory and monitor all assets in the OT/ICS network.
Even more importantly, they monitor all endpoints in even the most complex or challenging environments (like across low baud serial connections). Verve does so in a very low-cost solution without the need for taps or other hardware collectors by using a 100% software-based approach.
Step 2: Gather Software Patch and Vulnerability Information
The second challenge is the ability to monitor what patches are available and required. The core components of Windows, Linux, Unix, Office and other products like Adobe are straightforward (either from Microsoft or the OEM vendor-approved MS patches). Third party apps, however, usually require manual review of the vendor’s website to look for new updates.
Operators need to research patches to determine what, if any, security components are addressed. The sheer volume of these apps makes the task exponentially difficult. In fact, one of our clients in the power industry is currently monitoring just under 300 third-party apps that fall into this category at just one facility.
Verve Engineering Services leverage the scale across clients to provide a much lower cost solution than any individual company can provide on their own. The client simply provides a list of third party app, which we monitor for appropriate vendor communications. When a patch or update is published, we integrate that information into the Verve Security Center patch solution to close the loop and remediate the vulnerability.
But patch availability is only half of the equation. Effective patch management requires robust vulnerability assessment capabilities. Traditional IT tools with scan-based approaches are not effective and/or safe for OT/ICS systems due to the sensitive nature of the devices and their firmware.
Therefore, a specific OT/ICS vulnerability assessment is required to use the data available from the robust software and asset inventory described above. Verve Security Center takes this agent/agentless data collection and defines vulnerabilities across endpoints without the need to scan deploy expensive taps and hardware. Read this for a comprehensive review of vulnerability management.
Step 3: Identify Vulnerability Relevancy and Filter to Assign to Endpoints
One of the most challenging elements of patching is using the asset inventory to determine which assets should apply which updates – or filtering in other words.
Many companies gather lists of potential patches available for software, but linking it back to assets to ensure whether that particular patch is relevant becomes a logistical headache and labor burden.
Verve Security Center allows users to automatically filter on the specific assets that are in scope for a particular patch. VSC sorts by any number of characteristics on the end device from type of OS to NERC CIP criticality ranking to any other specific characteristic of the target system.
This filtering process significantly speeds the analysis of what patch is required and on which systems.
Step 4: Review, Approve, and Mitigate Patch Management
Many patch management processes end there and leave the approval and action to another set of tools or processes. Verve Industrial brings the approvals and actions into the same toolset. The VSC patching module allows users to build baselines for patches that are approved and unapproved.
Furthermore, these baselines reflect those approved by a specific vendor. Once this baseline of approved and unapproved patches exist, the Verve dashboards are filtered to report on only those patches that have been approved, saving the complexity of trying to manually remember which patches should and should not be deployed.
Users create as many baselines as they like to group patches in any way they’d like.
Step 5: Test and Deploy Vulnerability Patches
Testing software patches in cyber security is often a luxury that clients do not have time to conduct. Verve uses several techniques to ensure the patch provided is the one that is approved and delivered from the vendor.
VSC allows users to take the next step in the patch management process to programmatically deploying patches across OEM Windows/Unix/Linux devices right from the console. Importantly, the interface allows you to schedule deployment on one or two assets initially to test that the update is working appropriately on less critical devices.
It rolls back updates that are not working appropriately, and additional roll-outs can be scheduled at any sequence. [Note: Automatic deployment of patches is recommended only under controlled circumstances and is only an option on agent base devices.]
Additional controls such as rebooting (or not) the end device, displaying a message or retrying in case of failure are also configured in the console and are sent to the end device.
For those devices that cannot have a patch delivered, we offer professional, experienced staff who come on-site on a regular basis to deploy those patches to the agentless assets. Our engineers are exposed to all patches on all manner of equipment and as such have significant experience in the testing and deployment of patches.
Moreover, they are well-versed in operational knowledge, so their respect for and understanding of control equipment is unparalleled. Many customers manage the administrative review and approval of patches then leave it to our engineers to support and manage the deployment of the approved packages, allowing company staff to focus on their operational tasks instead of repetitive compliance tasks.
Step 6: Profile and Document Systems Pre- and Post- Patching
One of the more tedious regulatory and managerial tasks related to patch management is the requirement to baseline systems before and after the application of a patch. Any changes to that baseline need to be captured and entered into corporate change management workflows in order to secure the new configuration and maintain compliance.
The baseline configuration before and after is automatic with Verve Security Center. The agent-based systems automatically flag any changes to target systems. Even more powerful is the fact that the Agentless Device Interface that inventories the 75% of agentless devices gathers and confirms the patch updates have been deployed on networking, relays, PLCs, and other embedded devices. This patching tool allows customers to run a baseline after an update and confirm the latest version is installed.
Finally, Verve's services team assists in the collection of baseline changes and submits them to regulatory workflows and cyber security reporting tools within your industrial organization.
Simplify OT/ICS Patch Management
While OT patch management seems to be a straightforward function on the surface, it is actually quite difficult and time-consuming. Without automatic collection and monitoring tools, the time and effort burden can be significant.
Manual tasks are much more prone to error and increase time and effort to rework, potentially introducing cyber security risks to your OT systems and regulatory standing.
Fortunately, Verve Industrial’s end-to-end patch management solution with its combination of on- or off-site services, powered by our innovative technology, greatly increases the accuracy of patching efforts while simultaneously reducing the time and effort to complete these tasks drastically.
The patching solution is flexible and scalable, as any or all of the products and services outlined in this document are scalable to fit any client situation.
We welcome the opportunity to provide an initial diagnostic on the time and effort required of your current work process. We'll implement some or all of the controls suggested here and measure the time and accuracy of the program once the services have been tested. The resulting gain in time and accuracy will meet or exceed any corporate expectations for return on investment.
Please contact Verve Industrial to discuss your ICS patch management challenges or grab our guide to OT Systems Management: