Exploited: The Cyber Truth, a podcast by RunSafe Security. 

[Paul] Welcome back everybody to Exploited: The Cyber Truth. I am Paul Ducklin, and today, I am joined by Joe Saunders, CEO and Founder of RunSafe Security. Welcome back, Joe. 

[Paul] Hello, Joe.

[Joe] Hey, Paul. Great to be here. 

[Paul] Well, we’ve got a big, deep, wide there be dragons topic today, Joe. Can we fix OT security? We’ve kind of been flung into what I believe is known as a Betteridge’s law of headlines, haven’t we?

[Paul] Which is, if you’ve got a headline that ends in a question mark, the answer must be no because otherwise, the headline writer would have jolly well told you what the answer was. So it’s not all bad, is it Joe? It’s really more a question of, well, obviously we can fix it if we really want to, but what is it going to take to get us to want to do it, and how are we going to get there? So maybe I can kick off by just asking you to give, if you like, a brief definition of what we mean with this overarching term OT. We used to talk about ICS, industrial control systems, and SCADA, which is a bit of a mouthful, Supervisory Control and Data Acquisition.

[Paul] And then about fifteen to twenty years ago, the term OT took over. And it’s kind of everything that doesn’t fit under the normal IT umbrella, isn’t it? 

[Joe] It is. I mean, we think about operational technology differently today than we do information technology. And, of course, information technology is all those things that we have come accustomed to process data or to support our productivity applications.

[Joe] Think about your CRM. You think about your word processing. You think about other applications like that. That falls in the realm of information technology where we’re analyzing a bunch of data. We’re storing data.

[Joe] We’re building product. We’re doing things like that. Operational technology, though, is a little bit different. It’s not meant for employees. It’s meant for industrial processes.

[Joe] So operational technology is the software, the hardware, all that stuff that comes together to help manage and control industrial equipment and processes. And, yes, that includes SCADA. And, yes, that includes ICS. But what we have found over time is there’s sort of layers of operational technology that do different functions. And so OT has become a great umbrella word.

[Joe] I like to think of it simply as the technology that makes infrastructure operate. 

[Paul] So in IT, we might be concerned with payroll and customer information and how many hits we had on the website. With OT, we’re more concerned with what’s the wind speed? What are the weather conditions in the shipping channel? Do we need to open the sluice gates?

[Paul] Although things like payroll are obviously super important, there’s potentially a lot more at stake when you’re dealing with, should we turn this power on or off? Should we let the water flow or block it? There are some much bigger questions that sometimes OT devices have to answer, and they’re not always up to it, are they? 

[Joe] Well, they’re not always up to it if they’re not operating correctly. You know, there’s a lot of things that can go wrong.

[Joe] In general, OT devices, OT systems are designed to do pretty specific events, manage processes that are quite discreet. And so, you know, on one hand, they can be simplified. You might have a low power, low processor device doing a very, very important function on time, every time, all day long. And so OT plays a vital role. And yet to your point, ultimately, if something goes wrong, there can be a grave consequence or or a very severe consequence if one of these devices in infrastructure doesn’t operate.

[Paul] And very many of them were designed and built and are still operating perfectly fine from an era when paying $5 a year to connect a device with a SIM card to the Internet so they could be accessed remotely just wasn’t a possibility. And they were built, as I understand it, in many cases with the assumption that their cybersecurity would be determined by their physical security. So they might be locked away in a big Victorian era stone pump house with a secure lock. And if you wanted to go and fiddle with the thing, you’d have to break into the pump room first. And if you could already get into the pump room, then the reliability of the OT device might be the last thing you needed to worry about.

[Paul] But suddenly, OT and IT kind of merged together, didn’t they? 

[Joe] Yeah. I mean, if we knew all this stuff would be connected to the Internet from day one, we may have designed things differently. I liken it to the automotive industry, which is one of my favorite things to say is if you’re designing a brand new car from the ground up with no restrictions, you might build the protocols and the networking, the connectivity on that brand new car in a very different way. And the same is true if you realize the potential for connecting some of these OT systems or networks to management layers outside of the infrastructure itself.

[Joe] That comes with great benefit. Of course, it also comes with exposure to the Internet. So I was thinking of an example. Campbell Soup connected their manufacturing plants to the Internet. I think it was around 2018, 2019 when, you know, they really had to think about what are we gonna do.

[Joe] There’s this great opportunity for modernization of systems, of management systems, but we have all these devices that have been around ten, fifteen, twenty years. And lo and behold, they’re now exposed to the Internet. So being exposed to the Internet is one aspect of it, but there’s also another trend in the industry I like to point out. Before, it was real time operating systems, which are proprietary in nature, operating very specific functions on various discrete tasks on limited power, limited processing requirements on a device. And that model has changed.

[Joe] More and more organizations are also adopting open source. They’re adopting embedded Linux operating systems, and they’re bringing in more and more open source to these devices. So it’s the Internet and it’s the open source technology that are changing the attack vectors ultimately. 

[Paul] Now, Joe, the NSA, the National Security Agency of the United States of America, has published a brand new guide hoping to get everybody involved in OT to raise the bar a little bit. Now that has a very fancy name.

[Paul] The title of that paper is Operational Technology Assurance Partnership colon smart controller security within national security systems. And it’s quite a complicated read if you’re not an insider, but the story that it tells is quite compelling, isn’t it? So in that document, what leapt out at you the most of the things that the NSA is warning about and that they say, actually, we can fix this? It’s not as hard as you think. 

[Joe] Well, I think the NSA did a great service.

[Joe] They analyzed eight different vendors really in an anonymous way, not to expose the vendors themselves, but to show across these common vendors that are often selling their technology into critical infrastructure. So they analyzed eight vendors, and they identified the threat categories. And they enumerated, based on the vulnerability analysis that they did, the most common types of vulnerabilities they found in these devices. And lo and behold, when they list out the seven most common vulnerabilities found across these eight devices, there are some common suspects that we see in all types of software, anything from cross site scripting and SQL injection. But no surprise here as you’ve listened to this podcast in the past, memory based vulnerabilities rise to the top.

[Joe] The top two common vulnerabilities in this analysis are buffer overflow and memory corruption. And, certainly, those are addressable. And, certainly, CISA and the NSA and a whole slew of actors in the community are trying to address the memory safety vulnerabilities in critical infrastructure. 

[Paul] I think 21 out of 50 bugs that they categorized for the purpose of the report were buffer overflow or memory related bugs, which is a very high percentage really. If you just fix those two things, 40% of the vulnerabilities would go away.

[Paul] But even if you do that, there are plenty of others, as you said, to choose from. And that makes me think of what we discussed in the last podcast, which is that whole idea of Secure by Design. So you’re not saying, oh, we’ve got a buffer overflow vulnerability, let’s go in and patch it. You’re saying, let’s take a step back and try and build something where those vulnerabilities just won’t get coded in in the future in the first place, and we will emerge into a happier place. 

[Joe] Exactly right.

[Joe] If I may make a joke. 

[Paul] Of course. 

[Joe] That 42%, you only need one other vulnerability type. And in the parliamentary system, you have complete control of, in this case, the network and the OT network. So, you know, you’ve got a majority with just one other category.

[Joe] But we do see 40 to 70% is actually quite common. So I think that lends credibility to the NSA report. But to your point also, it does mean that things like Secure by Design, Secure by Default, Secure by Demand are important steps to help address these things. And the first key thing, though, is to understand what are the threats we’re looking at, and, of course, memory jumps to the top. 

[Paul] And when you think about patching, which we’ve become used to whether we use Linux or Mac OS or Windows, clearly, that’s a fantastic way of dealing with a clear and present danger in a product we’re using today.

[Paul] But when it comes to things like mobile phones, we’ve kind of got used to the idea that, well, if there’s a bug, there’ll be an automatic update. It might take thirty seconds and then as I think you equipped last week, and then you can get back to playing your game. Now with a lot of OT devices, they may be connected to the Internet and therefore exposed to danger, but that doesn’t necessarily mean that they’re able to be patched over the air. So it’s not quite as simple in the OT arena, is it? You have to get in a little boat and sail across the channel, and go into the weather station and go inside in your Southwestern and do the update by hand.

[Paul] And if you have to do that 500 times around the coastline of The United States, that is a very different job from saying, hey. The update’s ready, folks. Click to accept. 

[Joe] Right. And I think that window to patching is pretty significant.

[Joe] And that’s far different than, say, a web based application where you can push out an update and render it the very next time someone clicks on that link to a website. It’s, on average, a hundred and eighty days or more for patches to get applied, and that’s if they do get applied. Some of them sit out there and are not updated for years. And so what that means is if there are vulnerabilities exposed, there’s a greater window of opportunity for bad actors to do something nefarious or not something malicious. 

[Paul] Yes. There’s a great, if slightly chilling, read in that NSA document. There’s a sort of a section that if I were to paraphrase very loosely is, what could possibly go wrong question mark, I counted them, there were 12. I can’t read out them all, there’s too many. But it starts with, this could cause trouble for national security, e.g. terrorism or disruption, injury or death of employees, injury or death in the community, ruined equipment, release of toxic material, environmental catastrophe, violation of regulations, product contamination, and I haven’t even finished the 12. 

[Paul] There are some very different things at stake here. Clearly, we have to do something, but the sort of whack a mole approach that still holds in some parts of cybersecurity. Oh, I found a flaw, I’ll fix it. Oh, I found a new bad website, I’ll just add it to a block list.

[Paul] It’s twelve hours late, but maybe we’ll be okay. That’s not really going to work here, is it? We have to have a slightly different mindset in how we actually approach the problem. 

[Joe] Yeah. I think it needs to be systematic.

[Joe] I think it needs to be taking into account a risk management framework, help you prioritize, you know, what you do and when you issue the updates and all of those things. If you think about the likelihood of an attack multiplied by the consequence of an attack, you then get the severity of a potential attack as a result of that. Consequence times likelihood is a good way to do that. And I actually think that’s an interesting chart. If you put on one axis consequence and the other axis likelihood, you can break that into a simple two by two and focus in on those areas where there’s high likelihood of attack or potential reachability of an exploit along with consequence.

[Joe] And, unfortunately, in OT, a lot of these things have high consequence. But that’s why we also have great procedures. We have regulations, testing procedures to help minimize any potential downside effects. It’s interesting that the NSA lists all these grave or severe consequences because I think there’s a lot of people in the industry will say, we don’t really have a cyber issue in OT systems because systems aren’t reachable. But the ones that are reachable that do have a high consequence, they’re such a grave problem that could result as the NSA enumerating.

[Paul] We started out with devices that we thought we could rely upon remaining physically secure, so they wouldn’t be connected. And even though it might not have been necessary to connect them for them to carry on doing their job, the convenience and the apparent benefits of doing so were just so huge, and the cost savings so large. And the NSA makes it obvious by saying, OT systems are more vulnerable to attack primarily for two reasons. One, lack of security by default. In other words, that we thought they’d be locked away.

[Paul] And two, we have increasingly integrated these systems into an IT infrastructure, which has its own vulnerabilities, which is not a great situation to be in. Is it? 

[Joe] If you think about the Purdue model, you think about the different levels of devices inside OT networks, you think about the level zero, those would be actuators, the sensors, even the robots on the assembly line. And then you go up a level to the controllers. And then you have kind of the management layer, if you will, the HMI devices, the human machine interface, and the SCADA systems that are collecting data across those controllers and what have you.

[Joe] And so you can see that there are layers. And if you can gain access into those layers, then you can move around and administer your exploit. To your point then, ultimately, those levels keep going. We’ve got workstations. We’ve got data storage.

[Joe] We have IO servers. Maybe we have firewalls that are trying to protect those devices, but we can get around those. And, ultimately, we are connected to web servers and workstations inside the enterprise. As the IT and OT is converged, then you can see there’s different levels still. But if you can break through, if you can get to that HMI device, you can get to the controller.

[Joe] And that’s really the risk is there are ways to break in and get to these devices. 

[Paul] So, Joe, RunSafe Security focuses on eliminating vulnerabilities literally and figuratively at source rather than doing the whack a mole security theater stuff. In other words, let’s try and rebuild things so they come out not just with vulnerabilities patched, but with vulnerabilities not baked in in the first place. Can you walk us through what that means and how that works with a real world example that’s accessible to technical and nontechnical listeners alike? 

[Joe] Absolutely.

[Joe] So, you know, in some ways, you can say it’s patchless remediation in that if you affect things at build times such that a vulnerability cannot be exploited out in the field, whether that vulnerability is known or unknown, if you can eliminate entire classes of vulnerabilities, then your technology should endure much longer, and you don’t have to do as many updates throughout the year. So what do I mean by all of this? What I mean is if you insert security into the code that you deploy and then ship it out into the field such that you don’t have to do patches for sixty, seventy, eighty percent of the vulnerabilities that are out there because you’re already preventing the exploitation, then the infrastructure benefits, the product manufacturers who ship that technology benefit because they all reduce their operational cost in general. And so how do you do that? And it gets to the inherent weaknesses in software.

[Joe] And we alluded to the memory based vulnerabilities in these OT systems earlier based on the NSA report. And what we found is if you can address those at the root instead of after the fact through a patch, not only are you affected, you can avoid that whack a mole strategy. It’s not efficient to identify vulnerability, develop a patch, test it, and then push it out and hope that everybody applies the updates in a timely fashion. That is a losing model over time. You can never keep up.

[Joe] The bad actor will always be one step ahead. So the alternative is to insert memory based protections and even relocating where functions go into memory for the technically astute folks out there every time the software loads on a device and an exploit would fail before it even attempts to succeed. 

[Paul] So Joe, maybe another way of me asking that. I didn’t tell you I was going to do this before the podcast, so I hope I’m not putting you too much on the spot. Imagine that instead of being CEO of RunSafe Security tomorrow, you’ve woke up and found that you were Chief Information Security Officer of a CISO of an OT company, somebody who sold this kind of device.

[Paul] Where would you start? What would be the first two or three things that you’d want to do? 

[Joe] And I may even try to answer this, from two different angles. 

[Paul] Great. 

[Joe] For the product manufacturer first, since that was your question, what I would do first, I would identify all the components that goes into the product that I have.

[Joe] And what I mean by that is I would generate a Software Bill of Materials for every single device that I’m shipping out into the field to my customers. And then with that Software Bill of Materials, I would associate them all the vulnerabilities associated with that. And then the third thing I would do is I would rank order and categorize the best way I can reduce my attack surface. What is that trade off? And if it’s looking for ways to, you know, eliminate entire classes of vulnerabilities, great.

[Joe] If it’s looking for adopting secure open source components, great. If it’s looking for, you know, shoring up devices so that there’s secure boot in place so that I know what I ship is actually what gets loaded on the device out in the field, then great. But I do think you need to start with all the software components, then a full picture of all the vulnerabilities, and then a prioritization of how to tackle the problem from there. But what I was also gonna do is answer it if I’m a CISO at a data center or some kind of information security person or cybersecurity person for infrastructure as well. And I would do almost the same approach, but I’d be looking at my supply chain.

[Joe] I’d be looking at my suppliers. I’d wanna know from the cooling system. I’d wanna know what vulnerabilities are in all those systems. I would wanna look at the power system and know what are all the vulnerabilities in the power system. I would wanna look at the connectivity in the security physical security side, where can people break in, and look at where the vulnerabilities are.

[Joe] And I wanna take all my vendors across all my OT systems and rank order them based on their Software Bill of Materials, the vulnerabilities, and the things that I can attack. So for me, it’s identify the components, identify their vulnerabilities, and then rank order through some form of prioritization where to begin with my security journey. 

[Paul] So while a lot of people are inclined to say things like SBOMs, Software Bill of Materials, are kind of like security theater. Oh, it’s just a list. Are you telling me you really know what each and every one is? Why bother? 

[Paul] The alternative way of looking at that is if the vendor is not going to tell you what’s in it, it kind of implies that they’re not really sure themselves. And that matters an awful lot, doesn’t it? Because if there is a vulnerability, if you can’t cross reference that against a list of those ingredients, even if you don’t quite know what they’re all for, you’re never going to be able to chase down where the problem is and what you need to do to fix it. You’re just going to be on a wing and a prayer, aren’t you?

[Joe] Yeah. I think the issue is that the software supply chain for these systems is rather complex. There are simple firmware products that have a thousand components in it. And so it’s hard to manage and know what every single component does. And what if I told you that 700 or 800 of those components on that simple software package out of the thousand, if 70 or 80% come from the open source community, wouldn’t you wanna know who those authors are?

[Joe] What are the vulnerabilities with all those components? And open source software, I must say, I’m a big fan of open source software. 

[Paul] Me too. 

[Joe] You know, you get the community effect of identifying problems, fixing bugs, fixing vulnerabilities, and the like. But you can see that the software supply chain gets complex, and that’s for a simple system.

[Joe] What if you’re building an aircraft? What if you have hundreds of vendors inside your data center or inside your industrial automation facility? Then all of a sudden you take that thousand or 5,000 components and multiply it by a hundred or 200, And all of a sudden you’re thinking about, wow, I’m looking at a hundred thousand software components, and I need to know what the vulnerabilities are. You need tools that can interoperate across the supply chain itself so things can be automated and so that you can score things and help you pick out where to apply to your brains and where you wanna fix things. 

[Paul] And as you mentioned, even for apparently very modest OT devices, even just something as simple as, hey, here’s a device that you can put in the basement of a building that has a sump pump that will try and detect when it starts flooding and pump out the water.

[Paul] That could have, as you say, hundreds or thousands of open source components. There could be one team that provided, say, the TCP stack. There’s one that provides the code that deals with wireless connectivity. There’s one that formats strings, thinking of Log4j, so that you can actually present log messages. There might be a separate module that deals with, oh, I want to get the timestamps right so I don’t mess up with Daylight Saving.

[Paul] The number of components can actually be very, very much bigger even than some experienced programmers think. So that’s the Secure by Demand side of Secure by Design, isn’t it? Where you become an informed consumer who knows the right questions to ask and knows how to make sense of the answers and to push back when you don’t get the answers you need. 

[Joe] It is. And it also makes the case for automation for developing more mature software development life cycles, investing in CICD, DevOps, build tools, and deployment tools so that these processes are automated.

[Joe] And, of course, you wanna spot check everything. You don’t wanna just rely on automation because there, over time, it is an amount of bias in scanning tools and tools that are looking for vulnerabilities. You do need to take a defense in-depth approach. With all the components, with all the potential for vulnerabilities along the way, you can see you need a fully mature software development process and a fully mature software supply chain discipline.

[Paul] And that supply chain discipline really does work at both ends of the chain, doesn’t it? Both in the people who are putting stuff into the stream so they can flow down and those who are experiencing things coming downstream.

[Paul] The catchment error is absolutely enormous, particularly with this rich and broad open source supply chain. It is possible to master it. You just have to want to do so, and you have to reasonably expect your suppliers to want to do so as well. Right? 

[Joe] Yeah.

[Joe] I think if you manage infrastructure, it’s very reasonable to ask your suppliers for a Software Bill of Materials. You just wanna be curious what’s in there and what are the known vulnerabilities in there. Because you can scan all those devices, the scanning tools may not know all the components that are on that software and they may miss vulnerabilities. So I do think it’s an ecosystem. I think you need to manage your suppliers.

[Joe] You need to ask your suppliers for information and work with them. As you say, if you are the operator of the energy grid and you have suppliers that ship to that, well, those suppliers have their own suppliers. They have third party software developers. They have open source software. And so at each stage of the value creation, there is new software components being added, new testing procedures being in place, and more and more assumptions about what’s been done upstream.

[Joe] And so, we need to think about it as a complete process, not just through the steps. And if you are the end customer deploying the technology, you don’t wanna just analyze what’s before you. You wanna get information upstream so that you have a more complete picture. 

[Paul] So, Joe, let me, if I may, finish up with a bit of future gazing. Do you think that we will improve OT security steadily, collectively, and reliably in a way that heads off any major incident?

[Paul] Or do you think, sadly, perhaps there are some people who are going, you know what? I’m just gonna wait and see. Do you think it’s going to take something bad like a massive outage across half of the energy grid in The United States to focus our minds enough to want to do something about it? Clearly, the NSA would not have published the document they just did if they didn’t think, a, it was important, and b, we could actually do something about it proactively. 

[Joe] Yeah.

[Joe] I think over time, we will make steady progress towards more and more secure OT systems and OT networks 100%. And I do think that as the ability to do more and more builds of software through automation, through automated testing, and through processes like that are more and more efficient as we rely on more large language models. Believe it or not, as automation continues in the software development process, we will also get more and more secure. The key is we can’t solve everything manually. The key is automation.

[Joe] Now the other key though, of course, is who bears the risk? Does the end customer bear the risk? Does the nation state who has national security implications bear the risk? Does the product manufacturer bear the risk? And if you look at it, they all bear some risk, and something has to help catalyze everyone to move.

[Joe] And so I think all players need to ask each other what they are doing. The end operators, the asset owners should ask their suppliers to boost security so the operators can reduce their risk. And I do think that the product manufacturers need to really embrace automated tools so they can be more efficient, but then kinda weed out some of these vulnerabilities that are further upstream to them from the open source community, from their third party developers. And then I think at the country level, I do think regulation is needed. I do think that things like the Cyber Resilience Act and others are helpful, and that helps move people along.

[Joe] If there is a risk of having to pay a fine that is disproportionate to the cost of a good, then that will get people’s attention, and people will start to comply. And I’m not saying everything should be a stick. There should be some carrots out there as well, but I do think regulation plays a role. So I think the asset owners have something to lose, and that is disruption to their service. So they should be asking questions, and I think product manufacturers should be doing stuff, and I think the regulators should be doing stuff as well.

[Joe] And we’ll get there. Ultimately, automation and good processes will raise all boats, as they say. 

[Paul] Joe, I think as a way to finish, that’s fantastic because we’ve answered the question, can we fix OT security? Yes. Clearly, we can.

[Paul] There are ways to do it. Are we going to? That’s very likely if we take your advice and to borrow the words from Secure by Design if we all, producers and consumers, if you like, sign the pledge. So thank you so much for your time and your passion once again, Joe. And thanks to everybody who tuned in and listened.

[Paul] That is a wrap for this episode of Exploited the Cyber Truth. If you found this podcast insightful, please be sure to subscribe and share it with everyone else in your team. Remember everybody, stay ahead of the threat. See you next time.