ICS Security Concerns Explained

This is Recorded Future, inside threat intelligence for cybersecurity.

Dave Bittner:

Hello, everyone. Thanks for joining us for episode 57 of the Recorded Future podcast. I’m Dave Bittner from the CyberWire.

Our guest today is Joe Weiss. He’s the managing partner of Applied Control Solutions, a firm that provides consulting services to optimize and secure industrial control systems. He’s been in the industry for over 40 years and has earned a reputation as an outspoken and sometimes contrarian advocate for improved ICS security. He’s been a featured speaker at dozens of conferences, has written countless books and articles, and has testified before Congress multiple times.

Our conversation centers on what he sees as critical shortcomings in the current approach to securing critical infrastructure, including the electrical grid, manufacturing, railways, and water supplies. Are IT and OT professionals simply talking past each other, or is there more to it than that? Joe Weiss has strong opinions on that and many other topics, opinions formed from a long, fruitful career fighting to keep those systems safe. Stay with us.

Joe Weiss:

I’m an instrumentation control system engineer, which I’ve been doing my whole career. I was at the Electric Power Research Institute for about 15 years. The first five years, I ran the nuclear plant instrumentation diagnostics programs. Then, the next five years, I ran the fossil plant instrumentation controls programs. I was the institute’s lead on advanced sensors controls and communications.

Then, because I was getting all kinds of calls from colleagues, that resulted in starting the Y2K embedded systems program. When that was done, that was really why we started the control systems cyber program. We were under the impression that because there was such phenomenal information sharing with Y2K, that there would be the same level of information sharing on cyber.

I can only speak for myself. I had no idea that it was a problem. It’s just, we had such great information sharing, we didn’t want it to go to waste. But the thing that really got me and sent me along the path that I’m in is two things. One was, every time I would go to an industry cybersecurity meeting or conference, it was really IT focused. Every time I would bring up specific control system issues, I would be told that it was the first time anybody had ever brought it up.

The other thing was, because I have had and currently have contacts all over the world, primarily in the control systems space, I was getting people providing me, basically, case histories. They’ve had these problems — “Has anybody else seen the same thing?” That was the genesis of where my database of actual control systems cyber incidents come from, which by the way, is now well over a thousand.

Dave Bittner:

So you’ve really experienced, in real time, over the course of your career, this transition of having these systems … for lack of a better term, hosed up to the internet?

Joe Weiss:

Yeah. In fact, you could take it a step further. To watch them simply get connected and go from being, if you will, dumb electromechanical devices to smart communication devices.

Dave Bittner:

Can you speak to this tension that’s ongoing? I’m not saying it’s an unhealthy tension, but a tension that exists between the IT folks and the OT folks.

Joe Weiss:

Well, and I’ll add another couple of tensions, if you will.

Dave Bittner:

Okay.

Joe Weiss:

One is between the safety and security people. There’s another tension between the forensics and the ICS world. What you’ve got is a community, if you will — the ICS community. And I hate to use the word “ICS” because this is also building controls, and defense, and everything else. The operators and maintainers of physical infrastructures and essentially, the people that provide networks. That’s really different.

Dave Bittner:

How so?

Joe Weiss:

In the sense that our systems, whether it be in industrial, commercial, you name it — they were designed years ago to run … You could call them a network, but they were basically point to point. They would work whether the operator could ever see anything or not. The whole point about Windows and the interconnectivity really wasn’t an issue with the internet, it was an issue with the microprocessor. Because internet or no internet, if all you had was dumb zeros and ones, nobody would care. What made everything change was the microprocessor that actually gave you information.

That’s what everybody wanted. So what happened was, there was a decision made early on that, “Gee, let’s go to a standard interface.” Because it wasn’t just the plant floor — I hate to use that term — that was interested in this. But it was others, too. That’s where the Windows HMI came in. It’s just, we got used to, and now need, a view of the process. We’d sure like to know what train is on what track. The train will continue to run whether you know it or not, but it may not be safe if you don’t. If you’re the dispatcher and you don’t know the train is on the wrong track…

Dave Bittner:

But help me understand, Joe. There’s a big difference … To continue along the lines of what you’re describing here, to use that metaphor. There’s a big difference between being able to view what train is running on which track and being able to switch the tracks to put a train on a different track.

Joe Weiss:

Correct.

Dave Bittner:

So, can you suss that out for us? When it comes to ICS, that difference between being able to view things and being able to control things.

Joe Weiss:

Well, there’s another big issue too. That is, if you will, timing. We’re looking at things … When you start looking at sensors and controllers, these are things that operate in real time, as much as in microseconds to milliseconds. The operator is totally out of the loop. The operator doesn’t get involved until you’re talking about things that can happen, really, in minutes.

Our concern is in two layers. One is what happens instantaneously. Because if you think about an automated … Whether it’s a rail system, or a power plant, or whatever, you’re monitoring, say, the position of a train, or the temperature of a process. You’re monitoring that in real time.

The control systems are smart enough that they’re making decisions, basically, in milliseconds, or even faster. That if they see, for example, the temperature is too high or too low, they’re going to automatically change the process to bring that temperature back to where it belongs.

Dave Bittner:

They’re not necessarily going to wait for human input to make that decision.

Joe Weiss:

No, they don’t.

Dave Bittner:

It’s not practical.

Joe Weiss:

They’re not necessarily … It’s not safe.

Dave Bittner:

I see.

Joe Weiss:

It’s also not reliable. The electric grid operates, again, in the microsecond to millisecond range. You’re not waiting at all for an operator to make a safety decision — that’s done automatically. It’s done, like I say, in microseconds to milliseconds.

Dave Bittner:

So in what ways do the adversaries take advantage of that?

Joe Weiss:

Well, if they can, for example … I think you’ve heard me talk about or seen this whole issue about level zero and level one in the Purdue reference model. That’s the sensors, actuators, and drives. Let me, again, give you kind of a funny analogy.

If you’re a doctor and you can’t trust your temperature or blood pressure readings, how do you make a diagnosis? Yet, when it comes to security, there are unbelievably rare exceptions. Everybody is working on the diagnosis end and simply assuming that the temperature and blood pressure readings must be okay. What’s happening — here’s the big issue. I’ll explain the cultural reason behind it. None of our vendors, nobody to this day, makes secure, authenticated process sensors, actuators, or drives. The reason for that is, cybersecurity is essentially being led by IT, even in the ICS world, regardless of what people say.

For IT, security is the network. Sensors, actuators, and drives are considered to be engineering systems. To this day, there are no requirements at all for what constitutes a secure sensor, actuator, or drive. The issue is, if you go in — and you can, this is absolutely possible to do — if you manipulate the sensor — so the sensor tells you you’re at 80 degrees when you’re really at 100 degrees. The controller is going to change, in real time, the movement of valves and the movement of dampers, or whatever — all of this mechanical equipment. Or turn the motor speed up or turn the motor speed down.

But it’s going to do all of these real physical things to be able to bring the process back to where you want it to be. This is all happening instantaneously. The operator is made aware of this, but by the time he’s made aware of it, it’s many, many, many seconds, to minutes. So if you compromise — tell a motor at 80 degrees when you should be 100 degrees — you’re going to have the system try to bring it up. It could cause catastrophic failures. This is what we’re worried about. This isn’t somebody sitting at a console fiddling with things. These are things that are happening before the operator can do anything.

Dave Bittner:

What are the odds of these sorts of things happening? Are we seeing these sorts of catastrophic failures in the real world? Are people successful?

Joe Weiss:

Oh, there’s been many. Oh, God, there’s been many. But the point being is this — remember I told you that there’s a tension between forensics and ICS?

Dave Bittner:

Right.

Joe Weiss:

The ICS world has a lot of very, very good forensics. We can tell you pressure level, flow, temperature, motor speed — you name it. What we cannot tell you is, in a sense, who has changed any of those and whether they were being done maliciously or unintentionally.

So what’s happening is, we don’t have very good cyber forensics for ICS. What’s more, when something big happens, the only difference was motivation. Did whoever do it, do it because it was, “Oops, it was unintentional,” or did they do it because they were upset or had an ulterior motive, which then makes it malicious? There’s no forensics to tell you the difference.

Dave Bittner:

But in terms of the people who are running these systems day to day, in your opinion, what is the appropriate place for threat intelligence?

Joe Weiss:

There’s a big need … By the way, let me rephrase another thing — its definition. Threat intelligence. Just start with the word “threat.” In your parlance, or in certain … In the “threat intelligence” world, it’s all about malicious or malevolent actors.

In my world, a threat is something that could upset the system. In other words, turn the lights on, have a train crash, have a pipe break. A threat could be a totally non-malicious event.

Dave Bittner:

A janitor tipping over a bucket of water on a console is as much of a threat as a malicious actor.

Joe Weiss:

Absolutely. If it shorts out — the console — and you no longer have the ability to type anything in. But meanwhile, if that’s what you’re dependent on, then that’s a heck of a threat, isn’t it? You know, a cup of coffee.

Here’s the thing. If you want to cause specific damage and you do not want to be found, with no attribution being a big deal, then yeah, you probably need to be a nation state or have nation-state funding and capability behind you. But if you don’t care about attribution, and you don’t care about what you break just as long as you break something, you don’t need to be a nation state.

Dave Bittner:

But in terms of the folks who are, again, running these operations day to day. The ability for them to receive a stream of threat intelligence …

Joe Weiss:

What would they do with it?

Dave Bittner:

Well, that’s what I’m asking you. There must be information that … If you know someone is targeting you, you can respond to that information, right?

Joe Weiss:

No. I’m using “no” right away for a reason. If I’m operating a power grid, my concern is operating that power grid in a safe, economic, and reliable manner.

Dave Bittner:

Sure.

Joe Weiss:

If somebody tells me I’m being targeted, I’m going to go tell the security people to go do something about it. I’m not going to turn the lights off because somebody tells me that I’m being targeted. That isn’t going to happen — or it shouldn’t.

Here’s the worst part. In a sense, it did happen with WannaCry because you had a number of facilities shut down in fear of what WannaCry was going to do. They didn’t shut down because the systems were being hacked or compromised, they shut down because they were scared that they could have been. This is part of where the lack of forensics comes in. Because everything — and I mean this for a fact, all of the network monitoring companies, including for ICS — every one of them starts with the ethernet packet, which means they do not have a view of the actual process — they make an assumption that they do, but they don’t.

This gets back to … If I compromise the sensor before it goes — the serial to ethernet converter, the gateway — the network monitoring will never find it. I use the word “never” because it’s going to take that packet and assume it’s gold. All of the network monitoring is going to be about making sure that that packet stays pristine. But it has no way of knowing — zero — whether that actual underlying data that converted into this ethernet packet is actually good or not. None of them have that capability.

Dave Bittner:

Help me understand here, because I think … To play devil’s advocate, I could say to you, “Well, Joe, the lights are on. The trains are running. The plants are making power. The world keeps on spinning.” So is this —

Joe Weiss:

Sure. Yes, until it doesn’t.

Dave Bittner:

Well, and that’s where I’m going with this. Is this just, people whistling past the graveyard hoping that catastrophe won’t happen?

Joe Weiss:

Let me put it this way. There’s already been a thousand deaths to date from control systems cyber incidents. So when you go whistling by the graveyard, you could actually wave. I’m sorry for saying it that way, but yeah. We’ve had more than a thousand deaths to date.

You’ve had all of these discussions about a cyber Pearl Harbor. Will we have a cyber Pearl Harbor? Here’s the point — will we know if it’s cyber or not? Would the government be willing to acknowledge if it was cyber or not? That’s a totally separate question, and the answer there is, probably not. You can’t hide when a pipe breaks, or a train crashes, or the lights go out. But you may not know that it was cyber that caused it. That’s the issue.

Dave Bittner:

When you talk to the folks who are on the shop floor, who are responsible for these devices …

Joe Weiss:

Yes.

Dave Bittner:

What sort of things do you hear from them?

Joe Weiss:

They think — and here’s the real issue — they keep being told about cybersecurity, but their function in life is not to prevent cyber events. Their function in life is to make electricity, to make and deliver electricity, or to have natural gas or gasoline flow through a pipeline, or to make plastics, or to do something. That’s their job. Their job has zero to do with security.

So the only reason security becomes an issue to them is if somebody can tell them and show them where security has actually impacted their job. There is this complete, total lack of connection, if you will, between vulnerability and impact. You can tell me, until the cows come home, that there are all of these cyber vulnerabilities. Until you can point out and say how this affects the safety of my plant or the ability to produce widgets as designed, I don’t care what you say.

Dave Bittner:

How do we bridge that gap? How do we ease this tension in —

Joe Weiss:

This is what I’ve been … First of all, I’ve been reasonably unsuccessful, in a sense, for 18 years. We’ve gotten to the point … And again, people have to understand that because you’re sitting in a security world, you think everything is good, in the sense that people are taking this seriously.

But the people that are taking this seriously are the security people. The people not yet taking this seriously are the people who actually operate and are worried about safety and reliability of these systems.

Dave Bittner:

For the organization in ICS who is trying to figure out what part threat intelligence should play in their defensive posture, what’s your advice there? How should they approach it?

Joe Weiss:

My feeling is, again, you’ve got to rethink what the words “threat intelligence” mean. Threat intelligence is all about cyber intelligence, cyber threat intelligence. To anybody operating a system, that’s just not that important to them. It’s really important to the IT security organization. If you extend the term on what the words “threat intelligence” mean to incorporate threats to the system, regardless of where from, then that starts becoming a very big deal.

When I was working on the predictive maintenance of equipment 20 years ago, that was of critical interest to the operators. When you were talking about something like a nuclear plant, or for example, a ship going out to sea for nine months, you want to know that your critical equipment is going to be able to operate for that nine months on a ship’s voyage, or the 12, to 24, to 36-month fuel cycle of a nuclear plant.

That’s really, really important. The intelligence that’s telling you the state of your equipment is absolutely critical to operations and maintenance. Somebody telling them that Iran, or North Korea, or China, or ISIS is looking to hack them, is of marginal value to anybody who is directly operating a facility.

Dave Bittner:

What is your message, then, to the folks on the cyber side? Is everyone just talking past each other?

Joe Weiss:

Yes. When you kept talking about whistling by the graveyard, that’s where we are. People keep talking about vulnerabilities, but they’re not tying it back to, “What does this really mean?” Unless you’re looking at and understand what all of this means to the actual equipment, you’re not making a lot of progress. You’re sure not going to have the operations people all of the sudden want to sit up and take notice.

When I started all of this back in 2000, it took until maybe around 2014 or 2015 to get people to care about monitoring control system networks. They’re now doing it. There are a whole bunch of companies who are doing a terrific job of monitoring control system networks. The security world is not the right one, period, to be addressing sensors, actuators, and drives. They need to be part of the ride, but they cannot lead it.

There are an awful lot of very, very, very smart people that are involved in this overall area, whether it’s on the security area or the operations area.

Dave Bittner:

Our thanks to Joe Weiss for joining us.

If you enjoyed this podcast, we hope you’ll take the time to rate it and leave a review on iTunes. It really does help people find the show.

Don’t forget to sign up for the Recorded Future Cyber Daily email, where every day you’ll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.

We hope you’ve enjoyed the show and that you’ll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast team includes Coordinating Producer Amanda McKeon, Executive Producer Greg Barrette. The show is produced by Pratt Street Media, with Editor John Petrik, Executive Producer Peter Kilpe, and I’m Dave Bittner.

Thanks for listening.