Threat Research Blog

Understanding the Importance of Industrial Control System Security, Interview with Dan Scali, Manager Industrial Control Systems

Mandiant recently held the webinar From the Front Lines: The Industrial Evolution - Transforming Security for Critical Infrastructure, which focused on threats to Industrial Control System (ICS) and the challenges organizations are facing to secure these gaps and vulnerabilities. I caught-up with one of the presenters, Dan Scali, Manager, Industrial Control Systems, to further discuss ICS Security and what can be done to mitigate the risk to these systems.

[JB] What is ICS Security?

[DS] Industrial Control Systems (ICS) allow operators to monitor and control industrial processes, including those in the oil and gas, nuclear, power transmission and distribution, manufacturing, chemical, and other industries. You can find ICS in lots of different places - from running the power grid, to regulating energy use in a building or managing the process of brewing beer.

Some security researchers and reporters use the term "SCADA" to describe these systems, but that's not quite technically accurate. We prefer the catch-all term ICS to encompass a diverse set of technologies including Supervisory Control and Data Acquisition (SCADA), Distributed Control Systems (DCS), Programmable Logic Controllers (PLCs), Remote Terminal Units (RTUs), and any other type of technology specific to running an industrial operation.

When I talk about ICS security, I'm talking about keeping these specific systems secure, which in contrast to other information security disciplines, is less about securing data and more about keeping things up and running and about ensuring that the picture displayed on the control room screen matches what's actually happening on the plant floor.

[JB] What's your philosophy on ICS Security?

[DS] I believe that a successful ICS security strategy combines prevention and response. This contrasts with most of the work being done today in ICS security which is very focused on prevention and compliance. Using history as our guide: If PCI-compliant retailers can be breached and millions of credit card records stolen, what makes us think that NERC CIP-compliant or ISA99-conformant asset owners will avoid the same fate?

I also feel that network security monitoring (NSM) is particularly well-suited to ICS due to its passive nature. If it's too hard or too costly to make changes to control systems, you ought to at least keep a very close eye on them. You can't hope to understand if your ICS is compromised without appropriate visibility and instrumentation to look for indications of compromise.

[JB] What are some of the limitations of NSM? What about attacks that require physical access or exploit insecure ICS design? For example: USB-delivered malware or uploading malicious ladder logic to a PLC.

[DS] Common challenges in NSM are encrypted traffic, extreme traffic volume (and cost of storage), and privacy concerns. All of these are actually easier to overcome in ICS than in IT. ICS networks are rarely encrypted, ICS networks have lower bandwidth and generate less traffic, and communications are largely machine-to-machine.

Of course, a limitation of network security monitoring in ICS is that it only considers the data that can be collected. If an attacker were to communicate or plug-in directly to an ICS device and update its logic or configuration, it would be hard to detect on the network unless the device later attempted communicate with a command and control channel. In the enterprise space, we use host-based data to complete the picture. We can get some but not all host-based data for ICS. Software agents are valuable but are particularly challenging in ICS- they often require approval or testing from the ICS vendor and don't often support uncommon platforms like Real-time Operating Systems. Additionally, not all ICS devices generate useful logs. However, this is changing as some newer devices support syslog. Windows-based systems generate Windows Event Logs that can be a very useful source of data - errors, logins, CPU consumption, etc.

Despite these challenges, we find that if you put in the time to analyze what data is available in the environment, you can get a pretty good sense of the security posture of your ICS.

[JB] Is NSM possible in small organizations?

[DS] This strategy works for both small and large organizations! It may be easier in a big organization where some of the capabilities or infrastructure (for example, a Security Operations Center) already exist. In smaller organizations, IT and Operational Technology (OT) teams necessarily have to wear many hats. Congratulations, you're now responsible for ICS security!

[JB] What's the difference between IT and OT? Are there security issues that are specific to ICS other than high resistance to change?

[DS] It's best to think of ICS technology as being slower to evolve because of business constraints in the industrial space - for example, razor-thin margins and lack of internet-level scale - drive certain mindsets and behaviors: system lifecycles measured in decades instead of years, lower staffing levels, and heavy dependence on vendors or system integrators. So while we all understand that security is important, the business constraints and the technology ecosystem tend to keep us in a vicious cycle of poor security decisions.

There certainly is a cultural element as well. The high availability requirements of ICS are often cited as a reason for why they can't be updated - "If it's working, don't touch it!" However, these same systems often lack a backup or failover. If Facebook went down for a few hours the world might revolt - yet Facebook's engineers push new code twice a day. In truly mission-critical ICS systems, forward-thinking engineers need to advocate for a DevOps-like culture that allows the system to be quickly changed but also stay up-and-running.

It's also a common misconception that ICS attacks must exploit ICS-specific vulnerabilities - rather, they are just ICS-targeted. For example, the Stuxnet worm took advantage of both Windows zero days and a few software vulnerabilities in the Siemens control systems software that could be easily mapped to the vulnerability categories in the OWASP Top 10 or CWE/SANS Top 25 Most Dangerous Software Errors.

[JB] While technologies are often the focus of ICS vulnerabilities, cultural differences related to how OT and IT teams have evolved are often the greatest challenge to resolving them. In your experience, what strategies and tactics have worked for you to get IT and OT on the same page for cyber security?

[DS] We do find that often there is lingering tension between IT Security and OT over an ill-fated Windows patch, a security policy that didn't make sense for the OT environment, or just general distrust. It's important to remember, as you correctly point out, that successful ICS security programs are about people. In many cases, there's really not a secret sauce: take your OT counterparts out to lunch or coffee, visit them face-to-face, ask questions and try to understand their concerns. Ask them what aspects of IT Security are working well and what aspects aren't.

One strategy that we find particularly effective is the "working group" concept. Establish a team of interested individuals from across the stakeholder groups and establish regular meetings to develop the ICS security strategy, mission, and goals together. The group will become an important source of data, a sounding board, and a de-facto stamp of approval for new security policies and initiatives. To limit complexity, focus on finding common ground and establishing high-level wording that can be applied to both IT and ICS.

Also consider bringing in non-technical team members. Organizations that identify an executive-level security leader who is responsible for security across both IT and OT will move faster than organizations that rely on grass-roots efforts. One caveat is that the leader must be respected by both the IT and OT teams. Often the Legal or Risk Management departments are very interested in this topic and can be powerful allies and champions of the ICS security effort.

[JB] Are there any standards for ICS security?

[DS] There are numerous ICS security standards, including many that are industry-specific. The most notable and broadly applicable standards are IEC 62443 and NIST SP 800-82. However, we do find that the existing standards focus primarily on prevention rather than detection and response.

[JB] What are your thoughts on IDS/IPS for industrial control systems?

[DS] Signature-based detection is one aspect of an effective network security monitoring strategy. In our experience, these are potentially valuable resources that often go untapped. Very few asset owners have IDS/IPS deployed and configured appropriately at the boundary between the Enterprise IT and ICS networks.

While we encourage deployment of these types of security technologies in ICS, it is important to also recognize some the limitation of IDS/IPS. First, they only address known vulnerabilities discovered by researchers or used in past attacks. In the recent ICS-related BlackEnergy malware campaign, known vulnerabilities were used. However, new vulnerabilities can be used in the future. The other limitation is coverage - there are a limited number of signatures and they may not be representative of the equipment that you have in your environment. However, if you see a hit on your IDS for any potential ICS-related exploit it's probably worth investigating the source of that attack.

[JB] There are a number of government organizations (ISACs, ICS-CERT) that publish or share intelligence about ICS threats. What's the current state of information sharing in ICS?

[DS] There is a lot of good work being done here but also lots of room for improvement. The good news is that ICS-CERT and other organizations are starting to release indicators of compromise to asset owners. The bad news is that many asset owners lack the capability to consume and apply that intelligence to their ICS environments. We feel that better information sharing would lead to better ICS security and we're supportive of organizations like ICS-CERT and the ISACs that advance this mission.

Also driving our philosophy behind network security monitoring for ICS is the hope that private organizations can start finding evil on their ICS networks and sharing the indicators of compromise or other lessons learned with their peers instantly. One challenge inherent in the government-led programs we've seen across the world is that they have lots of restrictions on who can see the information and the intelligence can't always be communicated in a timely fashion

Take ICS-CERT's latest release on an ICS malware campaign that has been associated with the "BlackEnergy" malware campaign. While the information provided was valuable, it took over three years from the point where exploitation was first observed to broadly notify the industry. This isn't a criticism of ICS-CERT's capability. It's anecdotal evidence about the agility of advanced attackers vs. the speed at which we, as an industry, are currently able to react to ICS compromise.

[JB] Mandiant has documented attacker tactics such as stealing credentials and escalating privileges. What are some considerations for defending ICS against these common patterns of attack?

[DS] Authentication in ICS is a foundational control that should be designed with attacker tactics in mind. Primarily, we want to make ICS access difficult for an attacker who has compromised a user's Enterprise IT credentials. To this end, ICS credentials should be managed separately from Enterprise IT credentials - for example, in an ICS-specific Active Directory Domain. Where possible, require two-factor authentication for remote access to the ICS. Similarly, separate the two-factor authentication token from the token used for enterprise VPN or other systems.

Things can get tricky within the ICS environment itself. Some ICS software or devices may not have an authentication capability or may have hardcoded default credentials. Short of replacing these devices, the most effective strategy is reducing their exposure and monitoring them closely for unusual events. You should first ensure that they are not placed at the perimeter of the ICS network. Next, implement a network security monitoring strategy that generates alerts for logins, changes, or other behavior that you'd like to monitor.

To learn more about how your organization can improve the security of their ICS and SCADA systems based on a complete evaluation of their security program click here.