More organizations are adopting cyber security programs and requirements for OT environments. This is a positive development to those who have advocated for years for more robust cyber security programs and the development of a security culture in OT (operational technology).
Many of these organizations drive new cyber security initiatives as direct orders from the board or executive level down through the CISO and corporate IT teams. In turn, this creates an IT perspective bias, where IT teams with IT tools and IT backgrounds are expected to help drive OT security programs. This can be very problematic if the IT teams are not fully aware of their IT bias when it comes to rolling out OT programs.
The three common bias conditions that regularly create a disconnect between IT and OT are segmented into three groups: philosophy, project and technology biases. The philosophy biases include perspectives of the operational technology function as a whole. Project biases discuss field constraints, both physical and mental, that impede progress. Technology biases focus on selecting the right tools for OT-defined projects with the backing of OT support, budget and perspective.
IT vs. OT: Philosophy Bias
Operational Technology is not a recognized category in analyst research
IT organizations regularly turn to traditional IT research sources for guidance and insight. Analyst research firms spend considerable time, resources and effort in the research and analysis of a large range of IT tools. However, these analyses are conducted by IT people, for IT people and to be deployed within IT environments. Selecting technology and mapping it to components for an OT environment becomes skewed and misguided with differences between IT and OT programs. Or to state this a different way, IT tools don't always fit in an OT environment. At least not in the way most IT tools are intended to be used.
OT network systems are not homogeneous
Many IT teams build towards outsourced skill sets and centralized tools to manage a fleet of homogeneous, nearly identical systems. It is how you manage hundreds and thousands of assets from a single tool set or small, centralized or offshore team. In OT, there are dozens of IT-looking assets, but they are made up of different configurations, software, modifications and special needs. What this often means is that a tool chosen for a specific vintage or profile of OS can overlook and therefore possibly not apply to all manner of assets in the OT realm. And it is obvious that any tool selection made that only applies to a subset of assets will fail to provide adequate coverage. For example, SCCM as a corporate standard does not assist with the 1,000+ Linux or Unix operational assets often found in operational environments.
OT security should focus on basics before sophistication
How many times have you heard of a board receiving a report dictating the operational side of the business has no perimeter monitoring or a SIEM or SOC type of oversight? This is, of course, a critical component of a robust security program, but the challenge is when alerting or monitoring takes place it is an after the fact alarm. It is not the basic building blocks of security (like patching, backups, system hardening, least privilege,) that for years has been overlooked or neglected in so many OT environments. If you want to make a step change in OT security, start with the basics.
IT vs. OT: Project Bias
Operational Technology is tied to immovable objects
We know OT has legacy, outdated, no longer supported OS types and equipment, and simply upgrading to Windows 10 is not an option. These legacy systems often run proprietary software and/or communications protocols which are key to the safe operation of the facility. If the vendor doesn’t have an upgrade or the facility does not have budget or downtime to upgrade the software, test it, document it, and return the facility to regular operations, then the option to upgrade the asset is not realistic. In some cases, the assets in question manage entire portions of an operation. The time and money required to upgrade a DCS or SCADA system is not trivial and takes production offline for long periods of time. When planning for an OT upgrade or asking for a system upgrade, understand that it is not as simple or isolated as simply upgrading a single OS . There is much more at stake.
Operational technology systems require OT services and support
This is true on more than one front. OT teams need to be comfortable with anyone in their environment touching or modifying their assets. This is especially true of when corporate IT pressures the facility to patch software or try a new machine or technology. There is a prevalent and very real difference in perspective between IT and OT, and trust must be built between the two. Building trust takes time but is key to success in deploying and maintaining security tools in OT.
The second OT support mechanism and buy-in must come from the OEM vendors. More often than not, OEM vendors push back on OT teams who want to move forward with security solutions because of the support level they offer critical systems. The relationship between OT and OEM vendors can be full dependence (i.e., the plant completely relies on the vendor to support and maintain their operations, deferring to any OEM objections around security modifications) or the OEM vendor pushes back contractually if the OT team attempts to use tools the OEM vendor themselves have not tested or endorsed. In either case, the need to loop in OEM vendors and understand their role in plant operations is another key hurdle to overcome in OT that IT does not have experience with.
IT budget should be separate from OT budget
Many CISOs or IT executives push back on security proposals for OT because they don’t realize the significant number of assets in an OT environment. There can be tens of thousands of assets for larger facilities or global companies. In many cases, the number of OT assets exceeds IT assets. When an OT project asks for large budgets to secure the plant, they get denied or are asked to reduce the project. The deliverables are asked to be rolled out in stages and already over-tasked operational personal are tasked with deployment and maintenance to reduce cost. This results in a project never fully deployed or maintained. While many OT environments are months or even years behind in basic security hygiene, the initial cost to deploy technology and secure the assets is a significant upfront cost.
IT vs. OT: Technology Bias
IT management solutions assume relatively robust endpoints
The reality is quite different in IT and OT. Most scan-based IT tools are invasive and have a track record of knocking more fragile, proprietary OT systems offline. Using scan-based technology requires scaling down the scan, allowing additional time to include OT staff overseeing the system, and only scanning on offline systems or during outage. By the time you allow for all those conditions to align, you are getting very minimal security coverage from scan-based security tools. To succeed, you need proven safe, OT tested profiling and data collection tools to maximize asset coverage and automate asset insight.
IT best practices break OT systems
A typical system hardening practice is to have end points run a logon banner when a system starts up. The theory is that a user will see the logon banner and be reminded they are working on a corporate owned or critical system. The problem in OT is that OT is 100% uptime, meaning those assets are usually configured to auto-reboot and auto-login for redundant safety and monitoring systems. The introduction of logon banners breaks the auto-logon process for these critical OT systems. Requirements like these are why most OT environments only apply 40 to 50 of the CSC20 top 100 security controls. Many do not apply or actually interfere with operations.
Service Level Agreements (SLAs) in OT exceed those in IT
In IT environments, most users expect the internet and mail or file servers to be ready and able when they connect. If not, they usually go about their day while IT sorts out the issue and restores connectivity. These outages or scheduled maintenance windows take up three to four hours from start to finish, during which the end user does not have access to the system or service. In OT, if a switch or communication point is rebooted or misconfigured, it immediately interrupts safe operations.
For most industries, this results in either a loss of product specifications, and quality. For others, it is a safety issue (no visibility to safety, pressure, flow, temperature, speed, etc.) and instant product degradation or even shutdown. This is a significant loss to production, impacting revenue. This issue is often exacerbated in industries where production is more complex than turning a conveyor belt on or off. For example, coal-fired generation units take 25 to 30 hours to reach full capacity after a shutdown; Refining and petrochemicals take hours and days to return to proper product specifications.
I recall one particular OT security presentation to an operating company. The company recently experienced a significant cyber incident in the corporate network, causing a lot of damage. During my presentation, the IT team peppered me with dozens of ways the security controls could be circumvented. No surprise there, security is not bullet proof. They said they had it covered, which I later learned meant they turned off internet access for all operational facilities.
I expressed concern that these plant managers were likely using sneaker net and USB drives to bring data, updates, and files in and out of the facility. They didn’t believe their plants would do this, as there was a policy denying the use of USBs. Sure enough, when we went to the facility later that day, the plant manager’s desk was littered with USB drives. I asked him why he was ignoring the corporate policy about USBs. He smiled and said, “How much trouble do you think I would be in if the plant stopped producing? I am pretty sure I will get a pass on USB use.”
If you work for an operating company, protecting production is your first priority. Do you need to help OT make it more secure? Absolutely. Can you force security for the sake of security onto operational requirements? Probably not. Instead, educate yourself on the differences between IT and OT requirements, challenge the IT bias, be patient, be creative and know that security will improve. Set expectations that security will not improve quickly or without challenges.