The Journey Ahead is the Challenge in ICS

On the occasion of this, our 200th episode of the Recorded Future podcast, we welcome back our very first guest, Robert M. Lee, CEO of industrial control systems security company Dragos.

They recently published their 2020 ICS security year in review report, and Rob joins us to share some of the insights he and his team have gained over the past year, as well as the long term security trends they’re tracking.

This podcast was produced in partnership with the CyberWire.

For those of you who’d prefer to read, here’s the transcript:

This is Recorded Future, inside threat intelligence for cybersecurity.

Dave Bittner:

Hello everyone, and welcome to episode 200 of the Recorded Future podcast. I'm Dave Bittner from the CyberWire.

On the occasion of this, our 200th episode of the Recorded Future podcast, we welcome back our very first guest, Robert M. Lee, CEO of industrial control systems security company Dragos. They recently published their 2020 ICS security year in review report, and Rob joins us to share some of the insights he and his team have gained over the past year, as well as the long-term security trends they’re tracking. Stay with us.

Robert M. Lee:

I started my career in the US Air Force side of the house, but I spent most of my time over at the National Security Agency. While there, I ended up building out the US government's mission, looking at various state actors breaking into industrial sites around the world. Enjoyed my time there, spent most of my time on the defense, a little bit of time on the offense.

And then when I left, or as I was leaving I jumped out to SANS because I was pretty passionate about growing the workforce, and thinking about how people need to be a major portion of that. So, I built the ICS515 class at SANS, which is the monitoring and incident response class for industrial control, and I also built the Forensics 578, which is a cyber threat intelligence class there.

So, a lot of fun, but basically what I saw was, when people were trying to approach OT security, a lot of it was thinly veiled, copy and pasted, IT security controls into the plant, and we have different threads, different missions, different focuses, different risks. It's just can't be copy and paste stuff.

And, when I ended up leading up the investigation into the 2015 Ukraine attack, which was the first ever cyber attack to take down electric power anywhere, and the output was people were talking about security controls from IT and patching, and stuff that had nothing to do with the attack. And long story short, at a very selfish desire from myself and my son to have lights and water when he grows up, I said, "We need to make a company." Let's make a company focused on OT security and show what right looks like. Let's take an approach to try to build out the community while we do it. But we'll be a technology company that does visibility and monitoring and industrial networks, but let's keep it informed by having an intel team and a services team that does things like incident response and the hunting, and so forth.

So, that was kind of how we all brought it together. We started in 2016, and we're a little over 250 or 60 folks now, and keep growing, and having fun.

Dave Bittner:

When you started Dragos, I mean, was anyone else really paying attention to the space and in the way that you felt it required?

Robert M. Lee:

No. I think there's a lot of good people trying to tackle the problem in their way, but I fundamentally disagreed with the strategy that some were trying to take.

So, there was an emerging ICS security market, probably in 2013. 2012, 2013 timeframe, we started seeing some players enter the space, and a lot of their focus was just, help me get an inventory, or help me know that the configuration management of what's in my industrial environment.

And, those are very useful products, but I disagreed with the approach of, "Let's just, when we have that, we'll just use the machine learning to highlight anomalies, and if there are anomalies, then you guys can figure it out."

And, that was never going to get you in to understanding threats and responding to them, or so forth. And so, while our company also does visibility and help people understand what they have in their environment, it's really the starting ground to then also be able to search for threat behaviors, and how to respond to it and so forth.

Or said differently, I didn't think any of the companies were... That's not true. I think I thought some of them were malicious. I didn't think most of them were malicious, most of them were doing it with well intention, but I firmly believe that none of them had had the experience of going the journey, and I wanted to make sure there was an option in the community for folks that thought like us, which was, "Yeah, we got some challenges today, but the journey ahead of us is the real challenge," especially when we start talking ICS-specific focus threads.

Dave Bittner:

Well, and for folks who aren't in that world, I mean, how do you describe those specific ICS-focused threats? What are some of the things that, in the ICS world, are different from those who are doing cybersecurity in a corporate environment, for example?

Robert M. Lee:

Yeah, and when we look at OT or ICS, it is very common for an IT security professional, very well-intentioned, to come in and go, "Oh, that's just a Windows system that's running that human machine interface. Let's patch it, let's put anti-virus on it, let's put strong authentication on it, whatever, I'm good."

And the reality is... The way that I like to think about it is, enterprise security often, and this is a broad generalization, but often, is a combination of things like data security and systems security. I want to protect the system. I want to make sure the adversary can't elevate privileges on it, get access to it, compromise it, et cetera. I'm going to patch it to make sure that doesn't happen. I'm going to use encryption on my data. I'm going to have strong passwords, et cetera, et cetera, et cetera, system and data security.

In industrial security, it's systems of systems security. It's not that I can compromise one system and do anything, it's how can I interact with the human machine interface to open up a valve, and modify the control.

And, it's also a physics challenge. What is the physics? What can be done in these environments? How is electric power generated? How is it done? What keeps the rail system from having an unsafe event?

And so, very broadly stated, there's a lot of native functionality in those environments, and it's not that it's insecure, it's just native functionality of exactly what the operators need to do to interact with physics. And, that native functionality provides a larger ability for the adversary to do maliciousness than the necessity for a malware, or vulnerabilities, or exploits.

I'm not saying there isn't ICS malware, there are. But the overall reality is, is the threats are different, the mission is different, the risks are different, the systems are broadly different. Not everything has an end point and so forth, and so if the mission threats, risks, et cetera, are all different then I don't know that the security strategy can be the same.

Dave Bittner:

What about the culture differences there as well, right?

Robert M. Lee:

Big time, and so those operators that are keeping the plant running and going and so forth, they've got commitments for real time operations.

Oftentimes, they've got commitments about availability, and resilience, and safety. They've got legal requirements around safety and environmental protections.

And so, the idea that you're going to come in and potentially disrupt that, or slow that down, or et cetera, when they're already working well over 12-hour days, and similar, to patch a vulnerability when they don't even understand the risk, and very often, there's not risk on most of the vulnerabilities, like there is in IT, it causes a lot of culture clash.

I mean, there's been more power systems brought down by IT than China, Russia, and Iran combined. So, it's just, how do we go in empathetically and understand their mission? How do we jointly agree on what the risks are, and then how do we put in solutions to address those risks while we provide some other value as well?

A good example is, even for the customers that we have that understand that threats are real and they care about it, they understand that these are low-frequency events. In enterprise security, you have high-frequency, low-impact events. Tons of phishing emails, tons of scams, et cetera. But, not very often is one of those going to kill people at your site or do something majorly impactful.

But, in ICS we have low-frequency, high-impact events. Now, we're not going to compromised all that often, I hope. We're not going to see targeted adversaries as often, though it does happen more than people realize. But when it happens, it's literally life on the line, core intellectual property for the entire existence of that company, et cetera, stuff that we're dealing with.

So, how do you provide value in the meantime as well? So, again even though we like to focus a lot on detection response because we care about it, we provide a lot of focus on, "Hey, here's the visibility you need to even understand your systems, and keep them more well-maintained, and operations value, and similar."

And so, the way you approach the problem, and the mission, and the culture is definitely just different.

Dave Bittner:

Is there a Rosetta Stone? Is there a common element that you can use as a point of departure to bring the IT and OT folks together?

Robert M. Lee:

It is the mission. When we come in and we talk about any discussion, it's cool. What are you in the mission here for? What is the mission?

And when you realize the mission isn't the email server, but it's that the power is getting produced, or that the rail lines online, or the pharmaceutical vaccines are going out, et cetera, you can align on, okay, then what's critical to the mission? Let's look at that. What is the actual requirement? Because too often people look at requirements in the framework of NIST's cybersecurity framework, or NERC CIP, or something else, and it's like, "Look, hold on. No, no, no, no, no. Those are abstractions, and we're talking about security requirements, backup."

Security isn't the mission, security is there to compliment the mission. What is the mission? What are we in business for?" Now, let's look at the ways that we can better enable that. Some of it's going to be security, some of it's not, and you kind of come together and do that.

Dave Bittner:

You and your team at Dragos recently released your most recent Annual Industrial Control System Cyber Security Review, this is for 2020. First of all, what prompts the creation of the report every year?

Robert M. Lee:

Yeah, when I was starting out in Industrial Control System Security, there was not really any actual dataset to get started with. It was a lot of anecdotal insights and, "Oh, I heard one time at this plant, this guy had this," or, "I heard this one time this gal had this."

And, it led to people being able to say almost anything they wanted to, and the hype was palpable on, "Oh, there's all these threats." And, there's not like 500,000 cyber attacks, calm down. Or, it led to the opposite of, "Ah, there's no such thing as ICS-specific threats."

And, there was these extremes formed and anecdotal experience formed, and if you really just wanted to get data-driven answers, it was very, very difficult to do, and if they existed, it cost a lot of money, and there were whole things that people would try to sell you, but then weren't open to any sort of public analysis as well.

And, I was inspired, mid-career, by the M-Trends reports from Mandiant, especially the Verizon DBIR report. I thought these were just good reports of sharing knowledge out to the community. And I said, "Look, we've got an opportunity. We are the ones that get called in."

We're not just a technology provider. We get called into all these service engagements, and answer response cases and stuff. I think we're probably the team most people turn to, and we're the only team that's tracking the ICS-specific threats. Okay, let's share those insights.

And so, about four years ago, we created them and it was extremely well received, far more than we could have hoped for. And what I saw, even in a year's timeframe, is the security professionals at the power, manufacturing, water companies, et cetera, were able to take our reports as a public document, go to their leadership and say, "Look, forget the assessment for a second. Here are the insights that they have across the community. Here's the no kidding facts, the data. This is what's happening."

And we're very proud of this, but there are industrial security programs that now exist. And, there's people that have been welcomed in the community because hiring got open for it, and training and workforce development happened, and et cetera, that can be mapped directly to the Year In Review reports, and people share those insights with us all the time of, "Hey, we only even are doing this because of the 2017 report or '16 report," or whatever else.

And so, it's just so much fun to be able to put together a data-driven report, make it out and available to the community. And, some people will find value in different sections of it, some might find the whole thing, but either way, it's a no kidding, here's the data, here's what's actually happening. It's not a point of view. It's not a here's the marketing. It's not a here, go buy our product. It's, here's the top recommendations on what we see and the data to back it up.

Dave Bittner:

Well, let's go through the 2020 report together. What are some of the things that stand out to you?

Robert M. Lee:

Well, I think the first highlight that we put on the website piece of it, so there's a whole report, but we kind of have like this interactive website this year. The first one that probably isn't a shocker to security professionals, but is incredibly useful outside of that circle, is that it's extremely limited or no visibility at all in OT environments.

And, when you look at things like, even on the Hills of solar winds, there is an expectation and understanding that's incorrect amongst many of our infrastructure owners, Congress, anybody, that we're doing a ton of work in industrial security. And, they think that because there's been a lot of projects over the years, there's been a lot of regulation compliance stuff, but actually what's taken place is a lot of preventative work.

And so the security strategies that most people adopt, especially for ICS is prevention-based, segment it from the internet, firewalls, patching, antivirus, et cetera, even if it's the wrong controls, it doesn't matter.

It's extraordinarily prevention-based, and when you get into having to understand, is that prevention still at the same state that it is? Are the firewall rules fine, or, hey, is there anything happening in here? Are there new assets in the environment? Is there remote connections that I didn't know about? Is the controllers online? All that kind of stuff is visibility and detection and they don't have it.

And so, what we found in 2019 was that 81% of the people that were coming to us had limited or no visibility at all. In 2020, we found 90%. And so, it looks like, "Oh, the community's kind of slid backwards 9%."

I think it's a bit of a bias on kind of, the folks that are coming to us. Where the people that were coming to us in 2018 and 2019 were probably a little bit more early adopters in the community. ICS security is pretty much a broad discussion now, and people all over the community are coming to us to engage us. And again, from a security professional standpoint go, "Yeah, of course, if they're going to drag us, like they probably don't have visibility into the networks yet, duh."

But, to actually go talk to your executives and talk to others and go, "hey, like 90% of the people that engage with them at all, don't have this. We're probably in that same boat," is a real eye-opener for a lot of people outside of the security community to realize that we're not in a place yet to fully know what's happening in those environments, or to be able to detect and respond to anything. And, that a prevention-only strategy is probably not going to work.

Dave Bittner:

Help me understand. When we say we don't have visibility, what are we talking about here?

Robert M. Lee:

Nothing. So, no logging, no asset inventory, no monitoring, no traffic analysis, nothing.

Dave Bittner:

All right, but when I imagine an ICS environment I'm thinking of the person in the master control room at the power plant, or something like that, someone standing in front of a wall full of racks that have lots of blinking lights, buttons, switches, and dials, right?

So, I guess what I'm looking for is you to tell me how far off am I on that? And, do those buttons, blinking lights, switches, and dials, aren't they providing feedback on what's going on all over the plant?

Robert M. Lee:

Ah, good. No, that's a good separation though.

So that's, that's one type. That's probably a representation of those panel boards would be like, and like electric power pipelines and water. There's also a lot of industrial sites that would be more distribute control systems, which are more localized and similar, but either way, you're right, and that's a good clarification that we're talking about cybersecurity.

From a plant operations perspective your operators do have a lot of visibility into the process, if you will. And, they have a lot of feedback points, data historians, and SCADA alarms, and similar to say, Hey, the valve is open. The valves closed, et cetera.

There's a lot of visibility into what is the plant doing, and how is it operating? But, when you get into how it's all connected and the network, and you start talking about the security aspects of it, they don't have anything below that.

And so, it's not uncommon for... Well, first of all, it's not uncommon for that environment to change and they have no idea, or some vendor comes in and connects with a wireless access point and now they're directly connected to the internet. Or, they had a firewall, but it had an any-any rule in it, and communications have been coming in and out. Or, an OEM, an Original Equipment Manufacturer is coming in outside their contract hours and making control logic changes to the system when they're not supposed to. Or, the software does wrong, the HMI reads wrong, and you have a voltage shift across a major portion of the infrastructure. You have no idea what actually happened because you don't actually know what commands were sent across the network.

So, they might have visibility. They usually have visibility in their operations. The network, and host, and everything else visibility that would help for cybersecurity is completely absent.

Dave Bittner:

I see. Okay. No, I'm glad I asked. That's an interesting distinction there. What are some of the other things that stood out then? What are some of the things you want people to take away from the report?

Robert M. Lee:

Yeah. I think another one that I thought was interesting is, 100% of our answer in response cases involved shared credentials for lateral movement. And, generally a big problem was lack of separation between IT and OT user management.

So, it goes into the land of IT security here, but it's an important IT security component. I'm not saying no IT security is useful in ICS.

What we see is, domain controllers and active directory as an example. Where an active directory or domain controller is being shared between the enterprise and the plant, and an adversary compromises the IT, or the enterprise domain controller, and just rides that sucker down in the plant. There's a lot of ransomware cases that have taken down plants this year as well, based off of that.

And, in 2020, a hundred percent of our Incident Response Cases also happened where the adversary access the ICS from the internet, where either they accessed it directly from the internet, or once in the ICS, they were getting back out through the internet

And, the reason that's interesting is, a lot of enterprise security teams think, well, you have to come through the enterprise to get to the ICS and vice versa. So, we just monitor the enterprise, and naturally we're going to see stuff that comes in and out of the ICS.

That's not actually true. There's a lot of other connections and integrations and similar that go out to vendors and access points and remote access software, and so forth. So, there's a lot of stuff happening down in there.

I usually like to joke around that it's kind of like Schrodinger's ICS. There's a lot of stuff happening in there, we just don't look in the box and see if the cat's alive. And, our data will support, from the Incident Response Cases, that lateral movement's happening a lot, shared credential's happening a lot, and internet access directly to the ICS is happening a lot, and by a lot I mean 100% of the time.

Dave Bittner:

Looking at the big picture, I mean, that you've been doing this report for multiple years now. What are some of the long-term trends that you're tracking?

Robert M. Lee:

Yeah. Long-term trends probably relate to the recommendations. Number one, consistently is around network visibility. That sounds biased because obviously we're in that business, but everything starts there.

The ability to tune the preventions that you have, the ability to get to detection, the ability to get the response, et cetera. All of that depends on actually knowing what the hell is in your network and how it's connected. So yeah, number one is to increase OT network visibility.

Number two recommendation is to identify and prioritize crown jewels, so what's actually important to you. When we think about an oil company, as an example, they might have a wellhead that generates enough oil to generate $1000 a day off of that. It's probably not valuable enough to do all the security efforts on, but maybe they've got a refinery that's generating a million dollars an hour in value.

There's probably a lot of value in protecting that. And so, don't try to boil the ocean, figure out what the top 25, 30% of your assets that matter. I don't mean network assets now, I mean like, plants and sites, because each one of them are going to have their own networks and similar. It's not like there's an enterprise network. There's an industrial network, and for each one of these locations.

And so, I would identify and prioritize those sites, and then in those sites what matters. A safety system matters more than just a normal engineering system.

Number three would be to boost Incident Response capabilities. We still find, and this is of the people who are already calling us in for proactive Incident Response work, and so these are already mature clients to start with that 42% of the IR service engagements discover that the organizations did not have suitable Incident Response plans. And, 75% of them had difficulty with declaring a cyber incident.

And so the, the IR plans that they have are usually an enterprise IR plan, and there's like a paragraph or something about ICS, or like call the plant manager or something else. It's woefully under prepared, and we usually recommend to boost that capability.

We Usually start with something like a tabletop exercise with your operations and IT staff together and go through a scenario, and how would it unfold, and who actually is on point?

Number four would be validated the network segmentation. A lot of companies have depended on the strategy of network segmentation. That is not a good strategy anymore, but if you are still depending on that, especially in the short-term, go validate it.

88% of our engagements found improper network segmentation. So, it wasn't effective for people, and what I mean by that too is, all these plans used to think if we just stay off the internet, we'll be okay.

The reality is, the digital transformation is taking place at a fast scale, which largely just means hyper-connectivity. These plants are getting connected up more and more to cloud, and infrastructure, and OEMs, and integrators than ever before. And, a segmented off strategy is failing quickly, and so it's not that it was a bad idea before, but you must get a better strategy going forward, but at least for now, validate the one you have.

And then lastly, you have the separation of IT and OT credential management. I should not have, if it's reasonable to maintain, a domain controller in the enterprise that is also the domain controller for my critical plant safety systems or operation systems.

So, those are the five big trends we see year to year and we've pulled out again this year.

Dave Bittner:

Our thanks to Robert M. Lee from Dragos for joining us. You can find the 2020 ICS year in review on the Dragos website. Don't forget to sign up for the Recorded Future Cyber Daily email, where every day you'll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.

We hope you've enjoyed the show and that you'll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast production team includes Coordinating Producer Caitlin Mattingly. The show is produced by the CyberWire, with Executive Editor Peter Kilpe, and I'm Dave Bittner.