Isolating Control Systems

This episode has been automatically transcribed by AI, please excuse any typos or grammatical errors. 

Steve King: [00:13] 

Good day everyone. This is Steve King, the managing director at CyberTheory. And today’s podcast episode is featuring Joe Weiss, who’s the managing partner at Applied Control Solutions, a frequent keynote speaker in operational technology and ICS and SCADA and a widely known industry expert on all kinds of control systems and electronic security of control systems. Joe has been at this a long time, more than 40 years of experience in industrial instrumentation controls and automation, and part of that was 14 years at EPRI in San Mateo, where he led a variety of programs including cybersecurity for digital control systems programs. He’s also served as taskforce lead for review of information security impacts on IEEE standards. He’s also a director on ISA Standards and Practices Board, has provided oral and written testimony to three house subcommittees, one Senate committee and a formal statement for the record to another house committee. He’s also an invited speaker at many industry and vendor user groups, security conferences, he’s chaired numerous panel sessions on control system security. And he’s frequently quoted throughout the industry. If you want the ultimate knowledge base about almost any question, relative to industrial controls, you go to Joe, and he’s published over 80 papers on instrumentation controls, diagnostics, COVID-19, etc. He’s a registered professional engineer in the state of California, certified information security manager, and he’s certified in risk and information systems control. So, welcome, Joe. I’m delighted that you were able to join us today.

Joe Weiss: [02:20]

Well, thank you for the invite. I look forward to these kinds of things, because it’s important to get the message out to the mainstream as much as we can possibly get.

Steve King: [02:34]

That’s absolutely right. I agree with you 110%, especially these days, where we’re seeing real increase in the number of reconnaissance operations around IT, and particularly in manufacturing and energy. I’m going to ask you about that in a minute. But some of the major challenges in securing industrial environments have been initial design and ongoing maintenance. From my point of view, the initial design challenges assume that networks were safe due to physical separation from the enterprise, no connectivity to the outside world, and the assumption that attackers lacked the specific knowledge to carry out a security attack, but we’re all living in a different world. Now, what specifically do you recommend we do to harden our OT networks today?

Joe Weiss: [03:28]

Okay! Let me just – before I answer that – give one other point. And that is essentially what is a control system? And a control system consists of instrumentation controls, field devices, sensors, actuators, drives, things like that. The engineering devices, those are generally not IP type devices. And then you have the OT networks. The OT networks are almost always Ethernet or IP. And then there is a connection, generally through a DMZ to the IT networks, which are also IP. So, almost all of the discussions going on are solely on the OT networks and the actual devices where cybersecurity is not addressed in any of these discussions. So now going to specifically what you ask, and ISA, the International Society of Automation, through the ISA/IEC 62443 standards, have guidance on segmenting networks. We have what are called zones and conduits. So, what is important is not only to “firewall” – often I use the term firewall in quotes because it doesn’t have to be just a conventional firewall. But to isolate not only the control system networks from the IT networks, there is also a need to isolate the OT networks from each other depending on safety significance. So one of the reasons you have what are called data diodes, is because you don’t want to have communications going from “a more secure or higher level security zone.” It’s okay to go from there to the lower level one, but you don’t want a lower level zone going to a higher level. So part of where I’m coming from is segmentation has to be on two fronts. One, you have to segment or isolate the OT networks from the IT networks. But you also have to make sure that even within the OT networks, they’re adequately isolated from each other based on a sensitivity or priority.

Steve King: [06:31]

Those are all great points. If we look beyond the endpoints, the communication infrastructure and shared computer resources are then built to comply with modern standards, as you point out. In fact, their communication methods and protocols, and particularly, Modbus, is like a late 70s, let’s see, that’s 50 years old – a 50-year-old communication protocol for acting as a hub for all these devices on most industrial control systems.

Joe Weiss: [07:07]

Yes, and again, if you’ll bear with me, I’ll add a little bit to this. First of all, Modbus comes in two flavors, you have Modbus Serial, which is the 1970s-80s protocol that has no security. And then you have Modbus IP, which has – I want to be careful – some security in it. When you get down to the lowest level, often, here, we are in 2022, Modbus Serial is still used fairly extensively. But on top of that, discriminating, to start with, you’ve got your engineering devices. And their protocols are things like Modbus Serial, but also things like HART, HART protocol is Highway Addressable Remote Transducer in abbreviation. Anyway, the point being HART comes in a couple of flavors to generally have wired HART and you have WirelessHART. Wired HART is built on Frequency-Shift Keying protocols, which date back to the 1970s and they sit on 1200 baud modems. That is the most popular sensor actuator, in other words, field device protocol in the world. And then you also have “the Siemens part” which is Profibus, which is, in a sense, it’s not the 1200 baud modem, but it’s also protocols that don’t have much – I’m giving credit where credit may not be due – security in those protocols.

Steve King: [09:03]

Yeah, and didn’t Alan Bradley make a version of that too?

Joe Weiss: [09:07]

Yeah, I mean, all the vendors have their offshoots, which has also led to a thing called OPC. Where, because we have the Tower of Babel, you have an overall control system in a power plant or refinery or water system is a mix of different vendors, you’re going to have Emerson, might have Honeywell, you may have Siemens, you may have ABB, just to name a few, Schneider. And so each of them has their own proprietary protocols. And so there needs to be an overall translator which has turned out to be OPC, Microsoft OPC, which is what allows all of these different vendors with their unique protocols to be able to communicate with these windows-based operating systems and either SCADA systems or distributed control systems. And so, the bottom line is, there is a mix of protocols there. In our world, there isn’t just one. And invariably, it’s like in the electric industry, it’s DMP, the mp3. Yeah, it’s been around since the 80s or so, all of these started out without any cybersecurity, there is work going on within all of them, you could look at, for building controls, you have Pac-Man, all of these started off strictly as communication protocols without the concept of cybersecurity. And people are now trying to backfit cyber on to these. And there have been more than a few cases where the security upgrades have caused some real impacts.

Steve King: [11:18]

Absolutely. And, we just noted, we do a quarterly report on what we find in the wild here, and we’ve seen a 2200% increase in recon, specifically around Modbus in OT, energy and manufacturing just in the last nine months. So somebody is interested in what those configurations look like. And it seems very fragile to me that we should place all of our eggs in that basket. So what do we do? Do we rip and replace? Or how do you de-risk that kind of an environment?

Joe Weiss: [12:02]

Well, I’m going to give you a different answer. Part of the reason we have such a risk, and, by the way, we will never be able to be secure until this particular problem is overcome. And it’s not technical, it’s culture. And that is, what has happened without being flippant, the tail is wagging the dog. The organization responsible for the security of all of these systems, the OT or “control systems” is the CISO. You know, the chief information security officer and his or her minions. Well, the CISO is not responsible, nor in most cases have any knowledge of what the actual devices and systems are that they’re trying to secure. But the point is, the people who do have that knowledge are not under the purview of the CISO. They’re under the purview of vice president, engineering, vice president, manufacturing, vice president, power delivery, in other words, the vice president of the actual operational organization, and they’re not part of this cybersecurity policymaking or anything else organization.

Steve King: [13:34]

Do you think that – you’re suggesting that’s at the core of the problem here?

Joe Weiss: [13:39]

That is the core of the problem. Everything else emanates from there. When you don’t understand how that programmable logic controller works, or how, for example, a boiler is working, there’s no way in the world you can adequately secure it, period. Now, the reason I’m going further is … look at what was done with Stuxnet. And with Triton, being the Russians wanting to … Okay, in those cases, the only way for that to work is that physical security, IT security and control system knowledge people had to work together seamlessly. That doesn’t happen with the defenders. So I’m going to be talking about in Minneapolis, this is his whole point about the square peg in the round hole. Networking, whether it’s IT or OT, is the square peg, engineering is the round hole. And until we start having either a square peg in a square hole or a round peg in a round hole, it’s not possible.

Steve King: [15:11]

That’s stunning to me. Because, when I was running IT, I also had responsibility for OT. And it quickly became clear to me that nobody was going to shut down the plant, because I requested it anytime in my lifetime. So, I raised the issue to our joint boss, who, in this particular case, was the CEO. But, ultimately, he came down on the side of the plant manager and I got that. That was like my first experience with, “I see. So we’re not going to shut the business down to accommodate me patching network server. I guess I won’t ask him again.” But that was like 120 years ago. I find it hard to believe we still have exactly the same problem.

Joe Weiss: [16:00]

It’s worse, it’s 100 times worse. Because now, who does the CISO and the board listen to? The OT network people. Not OT, OT network people. They’re not listening to the vice president of engineering. And here’s the other point I wanted to get across to. You gave a very great answer. But in a funny sense, it was halfway there. And that was, you weren’t going to intentionally got that plant down, patch a server. However, we’ve had umpteen cases where the plant was shut down, because the patch that went onto that server was incompatible with the control system. And, by the way, this is also why ISA has a separate – we’re not sure if it’s going to be a standard or recommended practice or whatever. But it’s called patch management for control systems. Because what people do is they follow IT patch management guidelines. That doesn’t work.

Steve King: [17:15]

There’s nothing about the IT protocol for networking or even otherwise, as much of anything to do with IT.

Joe Weiss: [17:25]

Here, part of the problem is two-fold. One is, IT basically says as soon as they’re notified, you have to expeditiously patch, which means shut a system down, which is exactly what you ran into. But here’s the other point. In IT, when Microsoft puts out a patch, it’s a generic Microsoft patch. But in the control system world, Honeywell or Siemens or Schneider, they use Windows, they don’t use plain vanilla Windows, they’ve modified Windows a little bit for their applications. So when you pull a plain vanilla Microsoft patch off the website and try and stick it on one of these systems, it’s going to fall down.

Steve King: [18:20]

Are you suggesting is a strategy then a way to shut the network down so we can apply?

Joe Weiss: [18:28]

Absolutely. And that’s happened often. This is part of the reason ISA 62443-2-3, that is patch management for control systems, was started. Our problem is people use the same terminology. And it means totally different things. You think patch management is a general term. At the top level, it is, but not at the implementation. I’ll give you another one, because I got to do a presentation right after we get off, and I was dealing with definitions. And there are two definitions that to me are important. But one is people talk about anomaly detection kind of standard in OT network security. Anomaly detection there is looking at anomalies within the network. So you’re looking at are they the right bits and bytes? Are they coming from the right places? What’s the header correct? All the rest of that stuff. For the engineer, anomaly detection is, is vibration increasing? Is temperature going up? What’s going on with my voltages and currents? Until I got into security, my whole world was anomaly detection. But it was anomaly detection of physical properties. So you have both of these organizations mouthing the same words meaning something completely and totally different. So you’re saying this all boils down to the semantic problem? Oh, absolutely. I mean, and here’s the other thing, too. This whole term of OT – OT was a term coined by Gartner because they didn’t know what a control system was. So they came up with OT and said, “If you’re not IT, your OT,” so people basically say, “All of this control system stuff is OT.” That’s not right. OT is not pumps, valves, heat exchangers, relays, transformers. It’s not the instrument engineer, the relay engineer, the manufacturing engineer, the safety engineer, OT are the networks. And we keep getting “wrapped around the axle.” Because people throw the term OT as if that’s everything. It’s not. And many of the engineers to this day, if you use the term OT by them, they would look at you and wonder, “What are you talking about?” Now, again, unfortunately, that term has spread throughout the security world. It is not spread throughout the engineering world, or the maintenance engineering, maintenance world. So when you talk about that, that goes back to culture, the plant manager is not OT, the plant manager may be operations. But he doesn’t know what OT means.

Steve King: [22:11]

Yeah, I get that. We work with lots of different kinds of companies. And every factory automation company that I’ve worked with, they all lack the ability, to your comment about networks, they all lack the ability to identify threats at Level Zero. And yet, they all boast that they can capture telemetry and analyze it. Data Level 2, for example, but the electronic electromagnetic signals that come out of a sensor actuator PLC, they fly below Level 2. It’s down in the concrete. So I don’t understand why our industrial automation companies don’t partner up with – I noticed a small handful, maybe five firms that can now detect that telemetry at Level Zero and supplement their network monitoring, their classic IT network monitoring partners, and appropriately.

Joe Weiss: [23:17] 

That’s the greatest question in the world to me. But these OT monitoring companies view the engineering type things as competition. They try to tell them it doesn’t make any sense because one is complementary to the other. It’s not one in lieu of the other. But they view this. And it goes back to networking. Our computer science people, Level 2 or 3, certainly Level 3 on up, are network people. Level 01 are engineers. Now, that is what’s broken, this is what I keep speaking about. And it’s not getting better, it’s getting much worse. And honestly CISA, DOE etc. are self-propagating that problem by all the guidance they put out. When you look at that, I don’t know if I’ll put it out because it’s – I won’t say it’s controversial – but people don’t like it. But CISA has never put out and I use the word never and that includes when it was DHS before it became CISA, never put out any guidance on a Level Zero device ever. Every single thing they have put out, including all of their vulnerability disclosures, are all on Level 2 or up.

Steve King: [23:35]

They’re all ex-IT guys, right?

Joe Weiss: [25:10]

Yes. That is exactly the discussions I’m having now. Because when I’m giving a presentation, October 26, as it’s going to be my first in-person presentation since the pandemic started. So it’ll be the first time in two and a half years I’ve been face-to-face. And one of the people that’s going to be there is one of the deputy directors from CISA. And I’ve been having discussions back and forth. And one of the things that’s going on, is they’re saying, “Talk to our SMEs, subject matter experts. The subject matter experts for CISA are not instrumentation and controls, cybersecurity SMEs, they are OT network SMEs. And so what keeps being provided to see some management is it’s a network problem. This is what I’m going to be talking about. And honestly, it’s dangerous. Because you try to apply these network recommendations or network guidance to Level 01 devices. They have damaged or shut down or in some cases actually killed the devices. They’ve had to be replaced.

Steve King: [27:01]

I get that. I understand it. I don’t know what we’re going to do about it, though. Do you think, in the context of old dogs, new tricks, do you think education is a solution?

Joe Weiss: [27:11]

I’m trying, as we speak, to get to the head of both the new cyber organization and CISA. Let me throw one other thing out, because I know we’re getting close on time.

Steve King: [27:30]

Yeah, I just say that we’re about done here. So, final comment.

Joe Weiss: [27:36]

The problem we have, and this is, like I said, made its way all the way up to the top is our field devices, which is where you have no cybersecurity, no authentication, no real cyber logging, no cyber forensics. I’m talking the newest stuff out there, not just the old 4-20 analog. What’s happening is the offensive people know this. Not just our offensive people. But Russia, Iran, China, I assume North Korea can’t speak. But I know the first three. They know this. The defenders, for whatever reason, refuse to acknowledge this even exists, refuse, and all the guidance coming out of Washington is only IP network related. Without any, it would be even semi-passable if that guidance would be benign to these devices. Unfortunately, it’s harmful to these devices. This is the message that I’m going to be giving October 26. And I have to give presentations at a couple of universities, actually defense universities, on exactly this. And this is what I need to get out. Is IT cybersecurity or OT cybersecurity, policies, procedures, etc. can actually be harmful than the devices?

Steve King: [29:42]

That’s the right message. And I wish you all the best and we need to solve this problem. And based on that data I gave you earlier, a 2200% increase in recon, we probably need to solve it sooner than later too, I don’t know if that’s going to happen by chatting with folks at the federal agencies, but perhaps education is the key here. And we’re certainly going to include that in our CyberEd programming. Because, this is a real problem, for sure. Anyway, I know we’re out of time, Joe, and I really appreciate you taking the time today. I think this was very illuminating. After your pitch at the end of October, let’s get together for Phase 2 on this because there’s a bunch of things we didn’t talk about I’d like to that are appropriate, and kind of see where we are then.

Joe Weiss: [30:09]

See, that’d be great. Thank you for reaching out and having me on.

Steve King: [30:43]

All right, Joe Weiss, Applied Control Systems expert. Again, thank you and thank our listeners for spending 40 minutes of their time with us, as well. I hope this was useful to everybody. Until next time, I’m Steve King, signing out.