This is the 4th in our series on Vulnerability Management (VM) in OT. If you have been following along we have so far discussed:
- An overview of what a VM program looks like (from planning to protection)
- The value of a real-time, automated OT inventory in designing and supporting an OT VM program
- The relative merits of Scan vs Agent based tools in cataloging and reporting on asset specific risks.
Today it is time to talk about remediation of risks or, as our introductory image showed, is called the ‘treat’ stage. This is where the task of vulnerability management actually requires action. However we know that patching in OT is not as straightforward as simply deploying the patch, rebooting and on we go. Specifically OT is usually challenged when a new patch or update comes out on a multitude of fronts. (To recap our specific discussion on how the recent Remote Desktop update went for 2 very different customers read our blog here)
Specifically OT is challenged by:
- Different classes/importance of assets means different patching techniques
- When assessing how many patches are required in an OT environment you usually see a very large number but not all assets are created equal. Buried in those numbers are the more or most critical assets as well as other contributing factors to what order or sequence you likely need to proceed. Knowing your assets by location, OS type, criticality to operations, its configuration (or lack thereof) of security hardening parameters, is the system redundant? Can we apply compensating controls? Does it have a recent restore image? All of these are key components in triaging the order and urgency of patching. This is why we push our clients so hard on establishing a robust, real time inventory.
- Huge amount of assets possibly (probably) in scope
- Lets be honest. The number of IP enabled assets on the plant floor or along the pipe/transmission lines vastly outnumber the number of staff tasked with supporting them. In addition, the range of asset OS types (vintage?) and varying software and OEM warranty requirements means large scale automation of patching is very uncommon. Solutions that allow for global visibility but last mile OT oversight allow for scaling scarce resources to support multiple sites.
- Different actions are sometimes required other than or instead of applying the patch
- In reality there are many reasons why a patch cant be deployed immediately or completely throughout an OT environment. Whether it is a permanent deferral due to a lack of vendor support or a temporary one while you roll out your test plans other controls (like disabling individual ports/services/accounts, etc) provide a necessary and valuable compensating control.
It is clear – an OT patching program is not easy, straightforward nor does it look like anything you typically see in the IT world where latest patches and update cycles can be automated and enforced on what is typically a pretty standard set of assets. So what do we recommend for standing up and executing an OT based VM remediation (patching) program? It is pretty straightforward:
An agent based, automated asset inventory that allows you to append tribal knowledge, third party security information (like backup or whitelisting status) and profiles the asset beyond just missing patches (like system hardening parameters) will provide the single biggest insight and value to triaging a host of patches across a patchwork of systems. It is the single fastest and most efficient way to cut through what is otherwise a seemingly insurmountable raw list of assets in need of a patch. Look, for example, at the screen shot below. A VM dashboard that showed over 35,000 total patches (low through to critical) has been filtered down to just show critical assets (operations has deemed these to be critical to safe operations) with a critical risk that also have failed their recent backup and don’t have whitelisting in lockdown mode. You can see we eliminate 34,934 risks and focus on our highest priority.
Central Oversight and Support combined with last mile OT oversight – otherwise known as ‘Think Global, Act Local’. A central visibility into multiple sites allows for a small, central team to research, test and help to automate the deployment of patches. Our central reporting can filter views and therefore target assets by region, criticality, owner, site, OS type, etc. Secondly, our agent based approach allows for the deployment and installation of the patches as well. But even more importantly, that central patch deployment can be tuned to only load the files, not install, to run at certain times of day, to throttle bandwidth to that asset or that segment, the patch cycle can disable auto reboot of the command can come in the form of an ‘offer’. This means that a local OT rep has to log into the specific console and accept the command to install the patch. If he/she does not do so then the patch will not be installed. This is central visibility and specialty coupled with boots on the ground OT oversight. The best possible example of an IT/OT convergence in action.
Finally what about compensating controls? An agent based approach can enumerate any number of end point security settings like disabling the guest account from initiating remote access protocols or listing then disabling known bad ports/services if they are not needed. In short an agent can tune just about any parameter on the endpoint you might need to tune. Thereby providing the ability to filter your at risk assets, target specific compensating controls to just those assets and then automate the execution of applying those compensating controls.
In essence, an agent based technology, coupled with a central reporting capability allows for the most effective yet OT safe approach to the remediation portion of a VM program. If you would like to hear more please reach out and let me know where you are on your cybersecurity program initiatives. I look forward to hearing from you!