When composing the top nine takeaways from S4x20 cyber conference, I neglected to discuss the first industrial cyber security (ICS) Pwn2Own competition that took place earlier this year, and for good reason: I’m on the fence regarding the value of the vulnerability identification competition.
I respect the work and risk that go into the competition, vulnerability management, research, and awareness to help asset owners understand risks inherited from owning or deploying cyber security solutions. But this is different for Operational Technology (OT) or critical environments.
The challenge for OT asset owners is addressing should I patch, should I rip it out, and should I panic when a new vulnerability appears?
Let's imagine the life of Bob and Jane, your local site operators who maintain OT systems. They have a lot of tasks to complete with limited resources and time, but also pre-dictated schedules and pre-approved work orders. In either case, the activities are limited to specific systems, or even maintenance windows, leaving little time to understand the vulnerabilities published daily.
In a perfect world, devices would not have flaws, but if they did, we could patch and fix as we please. Unfortunately, this is not the case. Should you have other compensating controls such as correctly architected secure networks, firewalls/access control lists, end-point protection, VPNs, and backups – why should having a vulnerability with a CVE score of 10.0 on a legacy device really make a difference?
Vulnerabilities and cyber risks should be managed by determining the applicability of a vulnerability and its potential threat to an operational environment.
The context of a vulnerability is determined through several factors:
- Deployment idiosyncrasies and other implementation considerations (e.g., MD5 is insecure, but in a correct HMAC MD5 implementation, it is secure)
- Tribal knowledge of a particular asset and its configuration (e.g., Blue Keep is present, but it can’t be executed because RDP is disabled)
- The deployment, the system's impact, and surrounding layers of protection
While it is a great practice to be aware of vulnerabilities, most asset owners with some cyber security awareness already know that many systems have undocumented features that are easily misused or that their devices can easily “roll over and die” with the slightest deviation.
This also extends to our fictitious Bob and Jane – they already have a huge list vulnerabilities to manage, but perhaps they did their job and have a number of protections around critical assets such as HMIs that enable process visibility. The last thing they want is their CISO panicking when the residual risk factor is minimal.
On the other hand, the researcher in me applauds some of the findings, especially on the DNP3 gateway work. This cascades across many devices, and hopefully this protocol is only used under protected conditions. I suspect that it is not, and this has value from my perspective.
The real question for the operator is, "What do I do with this information?" There is no question that having additional information of vulnerabilities is better than not having it. By the same token, as an operator, I am likely already dealing with hundreds, if not thousands, of unpatched vulnerabilities. In fact, one can almost assume that all of the embedded devices in the OT environment contain some unknown vulnerability.
If a cyber attack reaches critical OT assets through a network, it is game over. If an attacker gains physical access, or an insider threat intends to be devious, it is game over. A wise man once said to me, "Never come with a problem without a (feasible) solution," and we need to consider that on behalf of asset owners with realistic and effective OT technology solutions.
Solutions for vulnerability management:
- Begin by understanding the greatest risks to your environment; not just at a device-level, but how that device impacts the operations of the process or plant, as well as its role relative to other devices
- Understand the "context" of that vulnerability on that asset; does it require certain configuration settings to be enabled, etc.
- Maintain a 360-degree view of the potential compensating controls you can implement, short of ripping out that old device. What are the surrounding controls you have in place? If you can't readily patch the asset, what can you do through configuration, whitelisting, firewalls, etc. to protect it?
To conclude this brief opinion post, I enjoyed watching the competition, and research has value. We want to know what vulnerabilities exist. But research for its own sake without understanding practical solutions that can be deployed in the field won't move the industry forward. We need to combine the "geeky" part that I love with the practical actions I love even more.