Protecting Industrial Security

Steve King: [00:13]

Good day, everyone. This is Steve King, the managing director at CyberTheory. Today’s guest will be Mark Cristiano, the commercial director for Global Business Services for Rockwell Automation. Mark spent the last 30 years in IT and enterprise and manufacturing systems leadership, about 19 of which were spent with Rockwell Automation. Rockwell is a global leader in industrial automation and employs about 25,000 people with customers in about 100 countries worldwide. Currently, Mark leads the global commercial team that focuses on industrial IoT and cybersecurity programs and managed services development for Rockwell’s connected services growth initiative, with an emphasis on facilitating IT and OT convergence. Mark is able to leverage that 30 years of IT and OT experience to help bring companies into the 21st century and into the fourth industrial revolution as we as we know it today. So, welcome to the show, Mark. I’m glad you could join us.

Mark Cristiano: [01:29]

Thank you, Steve. Great to be here.

Steve King: [01:32]

Let’s jump in my first question. Let’s help me and our audience understand what all this means. So, take us on a journey through Rockwell’s move into cybersecurity: when it started, how and why it’s become the fastest business unit in terms of growth at Rockwell.

Mark Cristiano: [01:56]

It’s hard to believe, Steve, but Rockwell’s business that I lead, the network and cybersecurity services business, is now a decade old. It’s been quite a journey. The way we went to market initially was to solve a specific issue with our customers. I used to say to customers, “The good news is every device you buy today has an Ethernet port on it. So you get to plug it in and it works. It’s time to value is realized, it’s efficient.” All of my OT customers and engineers love that plug and play features of Ethernet-enabled shop floor devices. And then I turn around and say, “The bad news is every device you buy has an Ethernet port on it.” And the reason being is because the networks and OT grew up out of necessity rather than proactive planning like an IT network. And these engineers would just plug device after device into the network until they hit what I call the tipping point, they plugged that one last device and maybe an IP camera usually, and the entire network would go down. And on the OT side, availability is essential. So that was a market pain point that we address going early on in the go-to-market strategy for the business. We go in and help customers with network assessments and designs and then implement traffic segmentation following the CPwE reference guide. And that paid off well, we helped our customers with improving the bandwidth and the stability of their networks and we continue to deliver that today. We also still deliver our virtualized compute platform, the industrial data center. And we talked about cybersecurity for the first five or six years of the business and people listened. But we didn’t see a lot of expenditure. And then in 2017, NotPetya and WannaCry hit, and that got the attention of my largest customers. So, a couple of them got severely impacted. And we mobilize to help them recover after those severe attacks. And it was in 2017, from a business perspective, that we completely pivoted to focus solely on cybersecurity. So, all of the investments that we’ve made in our portfolio expansion and the acquisitions that we’ve made have all been focused on expanding our cybersecurity services portfolio from a countermeasure perspective, services perspective, and from a managed services perspective, as well. So, we continue to build out the cybersecurity portfolio. We recently stood up at an OT-specific secure operations center and our Avnet facility in Israel. And that’s where we’re headed. We’re going to continue to develop and expand countermeasure deployment services for our customers. And more importantly, using those solutions as data sources to feed into our OT secure operations center. We are going to be the premier MSSP from an OT perspective for all of our customers globally that are in 100 different countries. So that’s how we grew up, where we’re at today, and where we’re going.

Steve King: [05:04]

That’s great. Thank you. You mentioned NotPetya attack and ever since the war in Ukraine began, I’ve had to scratch my head over Putin’s strategy and from my point of view, all you had to do is continue to release these viruses and watch them take on minds of their own and see what happens. It’s hard to believe that was five years ago. Given not only the state of vulnerability in the IoT sectors, which, in my mind, the Chinese and the Russians have exploited to demonstrate their cybersecurity superiority. And I’m referring to the oil and gas pipeline attack here. That was a year ago, doesn’t seem quite that long ago. And JBS Foods and so forth. We’re also seeing these attacks. And it seems like the larger customers are now understanding the urgency but most of these industrials are years behind. Why do you think that is? And why do you think it’s hard for people to listen to this when it’s happening right in front of their eyes?

Mark Cristiano: [06:25]

You’re right, Steve. We are in many of the large global industrials and food and beverage and CPG life sciences as well, maybe some heavy industries. And the big ones, their awareness has been raised and they are actively pursuing and deploying comprehensive cybersecurity programs, deploying these countermeasures that I alluded to, and then leveraging third parties for managed services for OT-specific cybersecurity monitoring. But you do point out something that we’re seeing in the industry. It’s the smaller guys because they’re overwhelmed, they now are aware that they have to do something, they don’t know where to start. That’s the most common question that I get from the smaller industrials which is, “Mark, we know we need to do something, but we have no idea where to start.” So, it’s an educational exercise. The other dynamic that we’re seeing is the elevation of the role of the CISO in organizations, which is a good thing. However, the majority of the CISOs that I talked to have grown up in the IT space. And in other media events that I’ve participated in, I’ve always discussed the fact that the uniqueness, the dynamic nature of the OT environment is something that’s foreign to CISOs. So, there’s another educational aspect that we’re going through with our community of CISOs. And I know you guys are doing that as well. So, just to educate them on how different that environment is and the way it needs to be approached, from deploying cybersecurity countermeasures to help protect that OT environment. So, that’s a dynamic that we at Rockwell, we’ve got CISO boards that we talked to, our CISO Nicole is always talking to our customer CISOs to educate them on what we’re doing because we’re a manufacturing company as well. And she’s leading the effort to deploy cybersecurity programs, all of our plants. So that CISO dynamic is getting better, but there’s still work to be done. The other thing, Steve, is that deploying cybersecurity slows things down. And it’s for a reason. Passwords need to be entered, credentials need to be checked. And when you look at the philosophy of the shop floor, it’s all about availability uptime. Those two dynamics cause friction a little bit in terms of having to bring a plant down to microsegment or anything that contributes to the loss of production is something that’s scrutinized. So I think that that dynamic in the OT environment is something that my customers are struggling with. And we’re very sensitive to that. So think about maybe staging things ahead of time and then deploying rapidly. So we take that into consideration in terms of that loss of production that we always try to minimize that. If we go back to the larger companies, one of the struggles that we see with them, although they are embracing it, is because they’re in every continent and in many different countries, and they’ve got plants in all these different countries. There’s this standardization, the desire that they have, but they don’t necessarily have the ability to standardize and deploy in a standardized manner at the scale and the speed that they want to, globally. And that’s something that we recognize and that’s one of our value props and differentiators is, I always say to customers, “We can probably run faster than you can.” And that’s all about the mobile scale that we have. And more importantly, that standardization that contributes the day to support and lower TCO. If you’ve got that same solution on every single plan, then that support becomes a lot easier and the total cost of ownership gets driven down.

Steve King: [10:17]

So your message to me then is that we’re still dealing with the reluctance or inability, or however you want to put it, to bring down a system long enough to upgrade it or apply patches on the OT side.

Mark Cristiano: [10:36]

It’s a dynamic we have to deal with, for sure. I think the good news is they understand that they are going to have to take some pain, and they’re going to have to experience some loss of production in order to do this because think about the alternative. You do nothing, you drive without insurance, for example. And I can guarantee you that that small loss of production is going to pale in comparison to how much money you’re going to have to spend if you get hit with ransomware.

Steve King: [10:58]

Yeah, sure. It was about 114 years ago, I had the pleasure of running both process automation and IoT, and operational technology for Memorex. And we had the largest cassette capacity manufacturing facility in the world. And there was no question; that conversation ever even happened right beyond the first time about whether we’re going to bring that plant down, or whether we weren’t going to bring that plant down. And we never did. So, I’m surprised that we haven’t figured out a way to replicate that function or module in some modular fashion, move portions of that to offset facilities. But you’re right, that’s still the primary consideration.

Mark Cristiano: [11:51]

And I think OT providers like ourselves, who understand that criticality and availability are approaching deployments differently, like maybe stage and cutover, those types of strategies to try to minimize that. I grew up in a shop floor as well. I was in IT for 15 years at two different manufacturing companies. So, I have similar experiences to you. But I think when OT customers go to IT providers, they don’t have that sense of urgency or that knowledge of the criticality of uptime. And that’s a differentiator that sometimes customers have had bad experiences with IT providers who just don’t understand that.

Steve King: [12:30]

So, touching a little bit more on those last two points. We’ve got new government mandates around cybersecurity. Can you explain what those are, and if that’s going to have an impact on industrial clients?

Mark Cristiano: [12:48]

Yeah, absolutely. So post Colonial Pipeline, JBS Foods, Oldsmar water, the government has stepped up in overseeing CISO, and DHS are implementing new reporting standards around incidences, specifically around critical infrastructure, also minimum recommended security practices that critical infrastructure is going to have to implement, that’s getting the attention of our customers within critical infrastructure and they’re asking for help there. They need to mobilize, they need to get ready. And that starts with assessing against some of the standards that the government’s putting out. And that’s something that we’re leaning into heavily to try to create and quantify that risk profile for our critical infrastructure customers. So, we’re helping them with that. And then once you’ve identified the risk profile, we can help with the remediation of the risks that we identified, but this was coming fast. And it’s going to continue to increase, not decrease by any stretch of the imagination. So I imagine there’ll be more. Another one is, if you want to do business with DOD, there’s a standard called CMMC. And those are specific cybersecurity levels that if you don’t adhere to that standard, you cannot do business with the DOD. So these are three examples of what’s coming down. There are other sub industries in critical infrastructure that have their own boutique standards that are coming down as well. And again, I see those not decreasing, but continually increasing. And it goes back to the confusion that I talked about with not even critical infrastructure customer but these customers that are under these mandates to adhere to these standards, they need help. They don’t know how to dig out from a hole that they’re in today. And that’s something that we’re leaning into to help them. The other thing is all industrials are seeing the writing on the wall around regulations. Not just critical infrastructure, but what’s been around forever like the food industry. That’s been around forever. It’s not necessarily cybersecurity, but we’re going to be seeing electric and oil and gas and whatnot, other industries that are going to be under government mandates to both deploy countermeasures as well as to enhance and decrease the time of incident reporting. So everyone’s aware of it. And I think that there’s an opportunity to assist all of our customer base to try to get their arms around on the frontend about how to deal with these directives.

Steve King: [15:26]

Yeah, sure. And let’s talk a little bit more about that point about clients driving around with no insurance and that you had mentioned as well here, and you also indicated that zero trust showed up after the Colonial and JBS and Oldsmar attacks, though it’s been around for 12-13 years now, as somebody’s idealized standard, Forrester and Palo Alto Networks’ idealized standard for strategy and architecture for building an excessive trust network. It’s interesting to me then that wherever the teeth are coming from, however, the shop floor guys react to that, there’s an inevitability of that government contracting and what you need to do in order to stay on that list or be on that list or continue to pursue your business. Is there a generalized acceptance of regulatory pressure? And what we’re going to have to do here?

Mark Cristiano: [16:39]

I think in the past, it was the old, “if I get hit, not when,” so there are little acts, they could roll the dice, cybersecurity was a lower priority and other things took precedence and they were able to get by, but we see this every day now. Industrials are the highest hit sector from a cybersecurity perspective. So clients are attacked constantly and relentlessly. And I think now they’re all waking up that if you don’t have that insurance of putting those protections in place, you’re going to get hit. And the price to pay is millions, if not hundreds of millions of dollars. And I loop back to that, I’m going to lose a half, our production run about $750 million, that Merck had hit right back in 17 or 18. That is just colossal. And so I think we’re seeing a general acceptance. And we’re seeing a mindset of “when, not if,” anymore, so everyone’s waking up. And that’s frankly why our business is the fastest growing business in Rockwell. It’s because of the general acceptance and awareness in the market of industrials that they have to do something. That’s one of the things. I talked about the confusion of customers, and my advice is always do something, don’t try to put the perfect plan together. And because you’ll never get started, just get out there and start to do something. I think we’ll talk a little bit about what some of the suggestions are later. But we’re definitely seeing a general acceptance. And those costs that I referred to, downtime is the most obvious. But safety: people can get killed when this stuff happens. This is that serious. And then there’s litigation. You could get sued, you can lose your IP. There’s so many different facets of costs that are at risk here by doing nothing. I think that the market’s waking up and we’re seeing a significant uptake in taking this seriously and starting to put programs together aimed at combating this.

Steve King: [18:57]

So I assume that Department of Defense is going to be overseeing this. But I also noticed that we just fired up another agency within the State Department, which is going to have a certain degree of oversight into these problems and the regulatory impact. So the question is, what are the right protections? And I’m sure each client has some custom needs, but what are the minimums in terms of industrial protections that organizations need to have in place today?

Mark Cristiano: [19:31]

Yep. So, at the foundation, we advocate the NIST cybersecurity framework that advocates identify, protect, detect, respond and recover. That’s every single conversation I have with customers to educate them on the phases of cybersecurity protection. I’m old school, Steve, and I always advocate to my customers that they start with that network infrastructure, segmentation. And I often get pushback that that’s not cybersecurity. And I say, “Sure it is. When you segment your network, the traffic only goes where it needs to go. And it allows you, should you get hit, to isolate quickly.” And then furthermore, the most obvious function of segmentation is to implement an industrial DMZ to separate that business network from the OT network and protect each other, should there be a breach. I think the majority of these ransomware is coming in on the IT network through email and then permeate down into OT, because there’s no DMZ to protect those two zones. So, I start there. But I always advocate to the customers, that’s basic blocking and tackling. And then, once that’s done and your infrastructure is robust, that segmented asset identification is absolutely essential, you can’t protect what you can’t see. And when you look at the industrial install base, that is a daunting task. They’ve got 20 control systems, they’ve got different disparate assets from different OEMs and vendors. And it’s important to get your arms around what you have installed there to identify what needs to be protected and then it becomes a prioritization exercise. There’s critical processes that probably should take priority when you start to deploy those types of countermeasures, like threat detection, for example, whitelisting, endpoint protection, those are what I continue to refer to as deploying those active countermeasures, USB kiosks, things like that. But given the large disparate number of assets that are out there, it’s important that we work with our customers to prioritize which assets to protect and when. Some might not even need, they’re running this process that doesn’t add a lot of value or whatever it happens to be. But those, especially in life sciences, there are some processes that if they go down, because of the global supply chain, people can get sick, or worse. So I think that prioritization is important if we continue to refer back to those disparate assets. Patching is a huge challenge for my customers. And it’s due to all the things I just talked about in terms of the age of the assets, the operating systems that are running on them, things like that, and that’s something that we certainly can help. And then if we look at that continuum, you have to assume, even though you’re going to be taking countermeasures, that you will get hit someday. So, incident response planning is important. And that’s something that we just went to market with Dragos. So we’re now offering insert response retainers and proactive cyber hygiene services with them, best-in-class industrial knowledge from both of us to bring that value, that protection to our customer base. And then recovery services, backups, blocking and tackling backups, restoration and planning. Anything that you can do, should you get hit, to minimize downtime is important. And then, there are other steps that can be taken, vulnerability assessments, for example. Just trying to get a handle around what your risk profile looks like, so that you can then formulate a comprehensive, probably multi-year plan to go after those risks. Security mode access with the pandemic overnight became absolutely required. And the complexity associated with security, remote access deployment and configuration and maintenance is something that shouldn’t be underestimated because you got to think about in terms of personas, you’ve got an employee, a knowledge worker who has one set of characteristics. You’ve got shot for people who are running production, you’ve got third-party OEMs that need to get in and see their equipment, but in a secure manner. So, where we used to see secure remote access was one size fits all, we, over the past several years, have identified that need to be granular in terms of personas and access, credentials, logging, auditing, things like that, endpoint management I alluded to, as well. And then at the lower level, level two and below, Rockwell has put a focus on what we call SIP security, where the author of that’s a generally adopted protocol that’s used not at that control layer. So, it’s not just that high, level three at level two and up to three and a half. It’s down at the control layer. There are specific countermeasures that can be taken as well. And I alluded earlier to that CPwE or the converged plant-wide Ethernet design guide that we co-authored with Cisco that is basically anything that you can think about from a network infrastructure perspective or a cybersecurity protection perspective. There’s a white paper on our section in that document that addresses best practices from an OT perspective. It’s about 800 pages, so I always tell people to download it, but definitely don’t print it. But it’s the Bible of OT infrastructure and cybersecurity.

Steve King: [25:07]

So that’s interesting to me because I’ve always thought that one of the challenges here is that while it’s all good and they say, “We retract telemetry at level two and above, because we’re partners with X, Y and Z.” And they do that for IT, the electromagnetic pulses that happen at the concrete level are not being tracked, and that seems to me where the attack vectors are going to occur. So I was always curious about whether you follow the zero trust approach or not. At some point, you’ve got to monitor what’s going on and that whole 24/7 threat detection step needs to be in there. And you have to be able to capture those attack signals. And if you don’t, then all the rest of it is boilerplate happy language, because you’re not going to get those attack vectors that are happening at the physical level.

Mark Cristiano: [26:17]

That’s interesting you say that, and I’m interested if you’ve seen what I’ve seen lately, but I’m starting to see a solution set that’s aimed – anomaly detection – that’s aimed specifically at level two and below, where they interrogate that electrical signal between devices. There’s a couple of different commercial players that we’ve been talking to along those lines, I’m not sure if you guys have seen that same thing. But it seems to be gaining momentum in the market and awareness of the needs to protect those lower layers.

Steve King: [26:46]

It needs to gain attention in the market. So happy to have a conversation about that at some other point here. But what do you tell clients who have got all this legacy equipment? We haven’t changed the thing in 50 years or something, this stuff isn’t easily patched. Much of it wasn’t even designed to be patched. Older controllers, for example, what do you tell clients about that?

Mark Cristiano: [27:14]

Yeah, you’re right. There’s 20-30-year old controllers still out there. And I think I always talk to people, the fact that when they were deployed, they were black boxes. There was no data that needed to go in and out of them. They were designed to be just closed systems. And now, with the need for digital transformation to even get data on those legacy systems, they’ve been opened up. So there is a level of vulnerability there. I guess it’s a challenging question, I can’t tell a customer to upgrade their thousands of control assets in one swoop. So, one of the solutions is to upgrade some newer equipment that has security built into it. It’s not realistic. So, there are a couple other things that we do talk to customers about, specifically as it pertains to compute, that from our perspective, that’s probably the easiest one to mitigate, getting rid of XP, and some of the different things and our solution is the industrial data center, which is a virtualized solution. I strongly advocate to my customers to migrate all of their physical systems and into a virtualized environment, whether it’s ours or someone else’s, that’s fine. But what that does, it allows a one-stop shop for either the customer or someone like ourselves, Rockwell, from a data services perspective, to be able to patch consistently on a quarterly or monthly date, whatever it happens to be, but that facilitates that patching process that has to occur. But it’s a complex challenge. As I talk about the old control systems and I talked about the old workstations that are out there as well, there is another emerging type of countermeasure that it doesn’t eliminate the need to patch but it buys you time. And it’s this notion of these whitelisting applications that are out there these days. They reside in memory, you can only run a set list of applications on them. And if something else tries to run it, it blocks it. So if you can conceptualize that, that does ease the burden or the timing associated with patching. I’ll stop short of saying it doesn’t eliminate it completely. But it is a kind of a buffer that we’re seeing as that market starts to emerge.

Steve King: [29:38]

Yeah. And I assume you offer that as a managed service as well?

Mark Cristiano: [29:43]

Yeah. So everything I talked about is always like the industrial data center. We offer that stack and the deployment services, but then we’ll remotely monitor that and administer and patch it 24/7 365 for customers, should they choose to take advantage of those services. That’s for our IDCs but if we put network in, we’ll do that same thing for firewalls and switches. And as I talked about our enhanced security operation center is the same mindset and paradigm.

Steve King: [30:14]

Yeah. Going back to the Biden administration for a moment that has demonstrated beyond a shadow of a doubt that they have no problem spending money, there must be lots of grants available to companies that could take advantage of that, out of the US infrastructure bill from last year that has a lot of funding in it. Talk to me about critical infrastructure and what could be supported through that kind of funding.

Mark Cristiano: [30:48]

So, we talked a lot about it today. But that’s an important set of articles that we need to shore up as quickly as possible to protect human life. That’s how serious this is. As we work with Colonial Pipeline, the production of critical goods and services. And I always allude to life sciences, the criticality of that supply chain, our opinion is that organizations need now, probably it’s even a little late, they got to get in front of this to start to prepare plans for grant submissions because the money is out there, it’s available. But it’s the steps that you need to take to be able to show the US government what you need and why you need it. That’s the challenge. And that’s something that I’ve alluded to it, we’re leaning into that for our customers, we just redid our entire website to provide free quick assessments to try to get some level of knowledge or visibility into how bad it may be within your organization. And we’re continually working with you guys as well to add to those proactive self-service type analysis that they can do. It’s not going to do everything for everyone. But it’s great to give them a headstart. And then we can come in and help them with those types of risk and vulnerability assessments, starting to audit against the standards that we alluded to, to help them for what’s going to be required to get those grants and help them with that submission process. And then zero trust, we’ve alluded to it already. That’s gaining quite a bit of publicity and momentum, I think we have a job to do in the industry, Steve, to educate people on what zero trust is and what it isn’t. I feel like some of my customers think it’s a single technology. And it’s not, it’s an approach. And by the way, it doesn’t deviate that far from what we’ve done historically, it’s just more overt. So, we, at Rockwell, subscribed to zero trust in our own facilities. As we continue to build out that portfolio I talk about, we’re all about reducing that attack surface and the prioritize protect surface. And then taking a look at network segmentation and access policies, and it is doubling down on that and taking a look at the current policies, and maybe even making them more robust. So, we’re all in on that. But it’s a very solid framework and the US government is all onboard as well. I think we have a duty to educate the market on what it is and what it isn’t, though.

Steve King: [33:35]

Yeah, I agree 100%, and as you’re well aware of our activity in that regard, I had a thought that occurred to me, as you were talking here that what we should do is grab your most influential CI customer, bring them to the US infrastructure folks, with an idea that we’re going to build a small black box, and we’re going to stick on all of that critical infrastructure, and we’re going to monitor and track anomalistic behavior at level zero telemetry and we can even then refer to that as our zero trust monitoring component and see if they’d like to fund that. And that becomes the industry standard. And if it has RA on it, all the better.

Mark Cristiano: [34:29]

I like that idea, Steve.

Steve King: [34:33]

Great, you work on that in the next half hour. It was great having you here today. I’m conscious of the time. I know we’ve got started a little bit late. I appreciate you taking the time out. I know your busy schedule. And so, thank you, and I hope that our audience enjoyed this as well. I think we, through discussion of these issues, have brought a lot of this to the surface. It’s not an area that people, generally speaking, spent a lot of time thinking about. But the next time they’re standing in gas line in North Carolina on a Tuesday afternoon, they may want to give that a little more thought.

Mark Cristiano: [35:16]

Absolutely. Thank you for the time as well. I enjoyed spending time with you as always, Steve. Thanks a lot.

Steve King: [35:21]

Until next time, I’m your host, Steve King, signing off. Thank you once again, to our audience for spending a little bit of time in the wacky world of cybersecurity. Take care.