Exploited: The Cyber Truth, a podcast by RunSafe Security. 

[Paul] Welcome back, everybody, to Exploited: The Cyber Truth. I am Paul Ducklin. I’m joined by Joe Saunders, Founder and CEO of RunSafe Security. Welcome back, Joe.

[Joe] Hey, Paul. It’s great to be here. 

[Paul] A slightly more focused than usual topic in this episode, Joe, but a very, very important one. Embedded risk, what CISOs can’t ignore. Let’s start by discussing embedded risk and embedded devices. How does the threat differ from what you might expect in your usual IT network where you’ve got laptops, desktops, printers?

[Paul] How does that compare to the factory floor? 

[Joe] Well, I could even see flipping this title of this episode on its head and say, hidden risk, what you have to discover. 

[Paul] Yes. Well said. 

[Joe] The issue, of course, is we have all this automation, all this industrial automation, and there are complex systems ensuring that the factories are operating in the efficient way.

[Joe] And on all those hardware components that make the factory floor operate are hardware devices controlled by software, and that software is embedded on those hardware components. We used to say cyber physical systems. We used to say embedded systems. In certain situations, we say operational technology. But the big difference is these are items that are for making things operate.

[Joe] They’re not for productivity tools for employees. The big difference then is the consequence if something goes wrong, if a cyberattack occurs. Just as we are concerned about safety in automotive, safety in aviation, we want blades and machinery to also operate without consequence and ensure the safety of the employees in the plant or ensure that the productivity of the plant remains and things operate on time and on schedule. So there’s a tremendous economic cost, and there’s a tremendous safety concern if things go wrong in these systems. So the embedded software on these cyber physical systems have a much greater consequence if things go bad than if, for example, your word processor isn’t available for the next five, ten minutes.

[Joe] Maybe a deadline’s on the line, but life’s not on the line. 

[Paul] Exactly. 

[Joe] I think the consequence is the big difference. The methods in which you analyze the security is a different story also, but we can get into that more detail. 

[Paul] So given that it’s hard enough for people to secure things like Windows and Mac OS and Linux with all the incessant updates and software changes and whatnot.

[Paul] If that’s hard where we expect the device to be on the Internet and have a fast connection where updates do come out regularly, I imagine that for a device that you bought maybe twenty, twenty five years ago that controls a lathe or a welding machine or a tidal floodgate or something like that. It’s a very different exercise to update that even if it can do over the air updates because the consequences of a failed update are much worse than, oh, no. You’re gonna have to go and swap your computer. 

[Joe] Yeah. No doubt.

[Joe] If you think about this manufacturing process, for example, I’m really excited to talk about industrial automation and manufacturing for some reason today. If you think about that end to end process, you have these different components. And as you say, even within a component, there could be embedded software that operates one piece of it. Maybe it controls the arm. If that breaks down, you have to go fix it.

[Joe] And how do you get updates to those components? You certainly need to schedule them. You certainly need to find ways to get software applied on these industrial equipment, and that’s not always easy. Certainly, if you can minimize the updates and ensure security and ensure safety, then that’s the best case scenario because these are not general purpose operating systems on which these applications are running. Generally, they are focused real time operating systems that allow you to build, you know, these embedded software systems and devices that do a very specific task.

[Joe] But to your point, despite that focus and that reliability, still things go wrong. There needs to be anticipated processes for that, and some of those are to schedule updates. And I think a big plan for that is not to schedule too many updates. Right? You have to find that happy medium.

[Joe] If we’re security conscious professionals who are also safety conscious and were operationally efficient, we wanna come up with ways that minimize downtime, whether it’s from cyberattack or from patching and fixing things that could take, you know, a manufacturing process offline. And so the whole idea is to minimize offline time, maximize security, maximize safety, and optimize that whole equation.

[Paul] Joe, maybe for our listeners, we can take a little subroutine call, if you like, into real time operating systems and real time computing. I think the average listener probably thinks, well, everything happens in real time these days. When you have a meeting, we don’t record it and then send it later like we used to with email.

[Paul] But in industrial automation, real time means something very, very strict and specific, doesn’t it? If you’re playing a video game and it gets a little bit laggy, you get annoyed and you probably go and do all kinds of tuning on your computer to try and minimize the lag. But in an industrial control device, that may be a hard legalistic requirement. You must respond to this incident within one millisecond and cut off the power, and nothing longer is acceptable no matter what else is happening on the computer at the time. And that is a very different sort of programming problem to solve than, hey, can I get my frames per second up in my video game, isn’t it?

[Joe] It is very different. And I think the key discriminator is looking at the purpose of the operating system. In general, the operating systems we all know and love that we have on our laptops, we have on our servers, the ones on which applications are typically deployed for enterprise IT systems. Those are generalizable operating systems. So things like Linux, things like Microsoft Windows, They do many things, and they have to support many functions of all different types.

[Joe] And the problem is in the operating system, you might have competing resources that affect the timing. You can’t necessarily guarantee that a function will fire at a certain point because there’s so many competing things. So the notion of a real time operating system that can help ensure that the operating system delivers what it’s supposed to deliver at the when it’s expected and there’s not those lags is generally the concept of a real time operating system. It’s not trying to be a generalized operating system. It’s trying to do very specific functions to ensure that timing is maintained in these systems.

[Joe] There is a focused set of things that a real time operating system does to ensure performance and reliability, and it’s much more scaled back than what you would see in a Linux operating system or a Windows system so that you have these performance guarantees and in a lot of cases, the security guarantees. 

[Paul] And particularly in the case of older devices, where these days we’re thinking, oh, will 16 gigabytes of RAM be enough for my next laptop? Should I pay the extra and have the 32 or the 64 gig? Should I have a 10 core processor or a 16 core processor? Some of the older industrial control devices, not just for price reasons, but for ratifying their correct performance, might have megabytes of memory, maybe kilobytes of memory.

[Paul] So any changes to the software that introduce new features could have alarming effects if they’re not very well controlled and well looked after. 

[Joe] Yeah. 100%. I mean, we wouldn’t wanna process a lot of analytical, you know, overhead on a constrained device that is really meant to open and close a valve. So if you’re doing massive amounts of AI processing, that would be a big mistake.

[Joe] The idea is, especially these legacy systems, you have constrained hardware that are doing very specific tasks. You’re not trying to be a general computer resource. With that, you do find limited processing power, limited capacity to compute. And with that, you still need to maintain performance. When these systems were first built, and you mentioned age a couple times here, fifteen, twenty, twenty five years ago, they weren’t necessarily considered exposed to the Internet or connected devices, and there weren’t as many concerns about security access to these devices to administer an attack.

[Joe] That has changed over the last twenty years. These networks are now connected. They might be connected to the IT networks. They might be connected to the Internet. They might be connected to a broader set of operational technology, and there’s an entry point into the facility.

[Joe] And so, attackers have found ways to get access to these devices, which were historically not defended because there wasn’t that much risk of a negative consequence. That risk has changed, and the consequences are high. And so with that, creates the need for figuring out how to protect those legacy devices themselves. 

[Paul] Yes. We spoke about this in last week’s podcast, didn’t we?

[Paul] The idea that in the early days of adding computerized digital control to things like pumps, the designers of the equipment that replaced the pump room, the pump room is locked up. If someone wants to get into the device and start hacking it, they have to get into the pump room first. And if they’re already in the pump room, then the game is so far over that the last thing we’re going to worry about is the little computer that’s turning it on or off. Yet suddenly the idea that, hey, what if the devices could call home? The OT network just became part of your IT network.

[Paul] But even worse, it meant that not only did the person not have to break into the pump room, they didn’t have to know where the pump room was, They could be quite literally on the other side of the world. Pick any country you like and yet, they could reach into many cities in many countries all at the same time. Sounds like a little bit of a recipe for disaster, doesn’t it? 

[Joe] It does. And, I  keep thinking of an analogy, that I’d love to to share, which is imagine if you’re a CISO of a castle.

[Paul] Now that sounds like a cool job. 

[Joe] Think about all the defense in-depth that you have in a castle. You have high towers you can look out on the horizon, a moat to ensure that no one gets close, and then you have a big door that’s impenetrable. And you have high walls that people can’t climb and 

[Paul] Don’t forget the portcullis, Joe. 

[Joe] Exactly.

[Joe] Spikes. Boiling oil. Yeah. But then once you get inside, there’s still further defense mechanisms. You know, there’s little slots that you can shoot out of, but you can’t shoot into.

[Joe] You know, you go up the staircase, and the staircases are different heights and different widths, and they even turn a different way so that the attacker going up has to swing left handed where the defender from the top is swinging right handed and mostly, you know, that’s dominant. And yet, what if you connect that to the Internet? 

[Paul] Yeah. 

[Joe] And what if what if you allow outside parties to give you stuff that goes inside that castle? And that’s really the question.

[Joe] So a CISO has a complex supply chain. They are not only connecting their systems, but they’re also receiving technology from outside parties. And, of course, most of those parties are well intentioned, intending to do good, but suddenly your castle’s defense is not just the castle. It’s all the stuff that comes into the castle, and you have to think about protecting those supplies that you get from the outside world and thinking about what could go wrong if those are compromised. 

[Joe] In the cyber means, that’s what we have. We have the adversary inside your castle maybe navigating where you can’t see them. And so connectivity is a big issue. Supplies coming in is a big issue. You know, the supply chain, and the CISO has to broaden their thinking as a result of all that. 

[Paul] A great example perhaps, which will resonate with home users, with office users, and certainly with people who run warehouses and factories, is how the things that they choose for their own security can turn against them.

[Paul] You know, cheap and cheerful webcams, it’s a camera with WiFi embedded in it. You don’t need a cable. Oh, it’s brilliant. You just stick it on the wall. And then while you’re at home, you can see if anyone’s broken into your factory.

[Paul] And there are countless examples of bugs in those devices that mean that the crooks can actually use your cameras, so that they can see when the security guard’s having a nap and they know exactly the right time to break in. So the thing that you put there so that you could keep an eye on your factory to make sure it was safe, actually turns out to be a poison chalice and it’s the thing that compromises your security as much as anything. You’ve actually made a bad thing even worse. 

[Joe] Yeah. You think about a facility. There’s cameras. There’s sensors. There’s different items that are reporting on the perimeter and what’s outside the building and what’s inside the building, and there’s temperature controls. And there’s all these items where we do have to monitor the operations to ensure the environment is exactly what it should be so that everything can operate efficiently and on time and in the way that everyone expects. At home, we have routers.

[Joe] We have cameras. We have doorbells that expose who’s in front of our house, and so it’s very similar. So if you think of all the different entry points to your house, we have all these different devices now that we don’t even think about that make our lives more convenient, more efficient, and that we begin to depend on. That efficiency comes, in some cases, at a cost of increased risk. And in the industrial automation arena, there are countless embedded systems like this that help make the processes work efficiently.

[Joe] Sometimes there’s trade off and you have to look at the risks. So that’s why I think CISOs and these facilities, they’re really enabling business to operate efficiently and looking for ways to prevent negative consequences. 

[Paul] So, Joe, if you are a CISO who’s stuck with legacy devices that either you don’t want to have to upgrade because it would be enormously expensive, or perhaps that you can’t easily upgrade because there isn’t a new device on the market yet that meets the regulatory requirements for, say, human safety. So you can’t just go, well, let’s rip it out and put in a different operating system. If you can’t do that, what can you do to back fit security into devices like that without causing yourself a regulatory pain and without needing to throw out the baby with the bathwater, basically?

[Joe] Yeah. It’s something RunSafe thinks about all the time. And the idea is that you can retrofit some of these devices even when they are constrained by power or constrained by power or processing capacity and what have you. And so the idea then is to add in security that doesn’t require new software agents and changes to the systems because those things might be brittle and could break, and they’re operating on outdated versions of software. So there are things like binary randomization that can be applied, which sounds like a scary term, but you can apply those to retrofit security and prevent exploitation of those devices even when a patch is not available.

[Paul] It sounds like a casino game. 

[Joe] It is. 

[Paul] Play binary randomization, but it’s a bit more structured than that, isn’t it? When you rebuild the program, you have exactly the same code, but you cast it into a form in which exploits or attacks that may have worked against the previous version are almost certain A) not to work against the new version, and, B) to be detectable if somebody tries them. 

[Joe] Exactly right. And the whole idea is these are the kinds of attacks that occur at run time when that device is operating in the infrastructure. And what you really wanna do is prevent the ability at that point of the attacker to deliver their payload and exploit that device. If you can relocate where the vulnerabilities are in a way that doesn’t affect how the system processes, then you have a chance of defending because the attacker can’t find the vulnerability any longer. And that becomes a very easy way to defend existing devices, including legacy devices. What I would say for CISOs then is to think about it holistically and really analyze which ones have the potential for likelihood of attack and which ones do you not have other options for.

[Paul] Yes. This idea of prioritizing is always important because you can’t do everything at once. Provided that you don’t use it as an excuse to keep the things you don’t want to do down at the bottom of the list so you never get to them. Maybe that is a nice opportunity for us to segue to the idea of bills of materials Joe. We’ve talked about these in several of the previous podcasts.

[Paul] SBOMs, not bomb with a b, won’t explode. Software Bill of Materials. Very few software binaries stand alone, do they? And if you don’t know what goes into them, then how can you predict where the greatest risk is going to come from? What do SBOMs do for us? And how can a CISO who hasn’t crossed that bridge yet, either preparing them for their products or asking for them from their vendors, how do they get started? 

[Joe] If you look at a device and say, oh, is this device high risk or low risk? Does it have a high consequence or low consequence? At the device level, you might have one view of that. However, knowing what’s inside the device, knowing the full enumeration of all the software components to your point earlier where you said, you know, maybe something is always a low priority, Having more data underneath the at the device level and say, what are the thousand components of software that comprise this one device, and how risky are some of those components?

[Joe] That’s what an SBOM can illuminate for you, all the individual components that are in it. And using that, you might be able to reshuffle the deck, if you will, in terms of your prioritization and say, there’s good reason this one is more risky than I originally thought when I looked at it, just a high level visibility. And so I think it brings a lot of data to you about recalculating your risk assessment and your prioritization of all the devices out there because you see all the individual components and you can see how risky some of those are. And your original notion of whether something’s exploitable or not might change very much if you look at the data underneath the device. And that should be easily consumable for CISOs.

[Joe] Vendors should be able to generate an SBOM with lots of ease, and there’s easy ways to calculate even a two by two matrix. You say, how risky is it and what’s the consequence of it? You can recalculate that two by two matrix easily and see what has high consequence, what has high exploitability, and reshuffle the deck, so to speak, so you can focus on a new set of ones that are really truly putting your infrastructure at risk. 

[Paul] I guess the average person might think, well, I’ve got a device. Let’s say it’s a dash cam for your car.

[Paul] What bill of materials do I need? I’ve got the bill of materials. It’s here on the receipt. It is a car dash cam and it was made by Videos R Us. Isn’t that the bill of materials?

[Paul] And as you explained last week, even a tiny device like that, that simply streams some video, digitizes it, records it, and then overwrites it after thirty minutes, could have hundreds, maybe even thousands of different software bits in it. No one person sat down and coded that whole thing and then burned it into the firmware and shipped it, did they? There’ll be components coming in from absolutely everywhere. And even a tiny device can have a very big footprint when it comes to its supply chain. 

[Joe] I think analyzing this data in a world of automation can be very straightforward where CISOs see the end result of analyzing those individual components.

[Joe] It doesn’t mean if it’s a greater complex problem to analyze. There’s easy ways to automate and analyze these different tools to look at their underlying risk of all those thousands of components. But imagine one variable in that device, one component in that device could be written by a group in China or Russia. And so imagine if you didn’t know that and you thought that device was trustworthy or  good. That’s just one example of a data point country of origin that is being used.

[Joe] The reason for that is because there’s underlying risk to it. And, you know, maybe not the same risk that you thought originally. You know, I think my device comes from the US, but some of the software doesn’t. And so I I think these are the kinds of things that can be illustrated, and a CISO might be able to reshuffle the deck when they think about what are the components I may wanna replace, what are the components I wanna bolster with more security, what are the some of the components that are less risky because they’re not reachable and not exploitable. 

[Paul] So if you are a CISO and you haven’t confronted the whole bill of materials yet, don’t be afraid of doing so because what you don’t know really can hurt you. 

[Paul] If you’re afraid of the fact, oh, I’ll get so much information, I won’t know what to do with it. That’s a very bad reason for not collecting it, isn’t it? And as you say, the collection can be automated and the initial analysis can be largely automated, leaving humans, including those CISOs, with a smaller amount of information that’s very, very informative for them to analyze and manage their risk. 

[Joe] Absolutely. And, you know, if you think of it as pragmatic analytics and automated processes ingesting SBOMs so that you can reshuffle the deck and analyze your risk in a different way, then let’s call that pragalytics, pragmatic analytics. You might find some things interesting.

[Joe] And so I think looking at your supplier’s technology and reassessing risk in a different way means that you can learn some new things and create better security posture overall, and you can prioritize when you wanna do that. So I think it’s an easy analysis that can be done, and the way to do that is to ask your suppliers for that Software Bill of Materials, and use that to create that analysis I was talking about, and I think it can be easily done. 

[Paul] There’s kind of an element of what I like to call self serving altruism in there, isn’t there? You’re doing it because it will benefit you. But if you pass some of the things that you find on to the community, say to the people who are working on the software upstream who didn’t realize that they were getting tainted components from somewhere else, then not only do you help yourself, you actually help the community at large.

[Paul] You help your fellow citizens. And you help them protect themselves from all those people online, whether they’re cyber criminals or state sponsored actors who actively and continually wish us harm. 

[Joe] It also simply puts visibility, as you say, on the overall problem. And I suspect that just asking helps to keep the suppliers aware that their customers care. And so I think demanding an SBOM might be an aggressive term.

[Joe] We’ve talked about secure by demand in the past. 

[Paul] Yes. 

[Joe] If you do it in the context of, hey. I’m trying to elevate my security posture. Please send me an SBOM so I can understand the underlying risk of the devices.

[Joe] Let’s see what we find, and I’m gonna look for ways to prioritize my risk reduction. Then everyone’s on the same page, and I think those suppliers will support. And I also think those suppliers will pause and think and say, okay. The CISO is looking at this in a unique way. I’m gonna improve my game too, and I’m gonna look at it in a unique way.

[Joe] What that does is spawn collaboration, and that collaboration will help boost resilience and security overall. 

[Paul] Joe, given that I think the jargon term is convergence, it’s been something that we’ve talked about for 10 years at least, maybe even 15 years. But that seems to be accelerating by which I mean that it’s increasingly unlikely that you would have an industrial plant that is not online, that is not uploading stuff. It’s very, very difficult these days to buy a CPU or a system on chip device that you would put into a modern embedded device that does not include WiFi and Bluetooth, you choose either or both. So how does CISOs confront that given that being disconnected is becoming harder and harder and harder?

[Paul] So how do you adopt the future without giving away some of the safety that we had in the past? 

[Joe] You mentioned convergence, and I think convergence leads to modernization, which is what you’re describing there. More and more capability is put onto those chips, greater connectivity. Why? For the benefit of more efficient and more effective manufacturing processes in general and integrating those systems to automate more and more.

[Joe] The flip side of that is if you’re simply trying to maximize your profit margin and drive down the cost to zero, you know, or whatever the target is, then you might miss a variable, and that variable could be security, and that security could have a consequence. The drive towards modernization can be good. Modern society is operating on modern technology. Right? But with that said, we have to be mindful of the security aspect.

[Joe] So what can you do? You can make sure you understand all those components that are on those chips and on that hardware and how the software interacts with them. You can get a Software Bill of Materials. You can understand what’s being used, what’s not being used, and what things are available. And so then you can start to have a more complete discussion around the security steps that you need to take to ensure that you’re not further exposing the business to greater likelihood of attack for a negative consequence.

[Joe] And just because the latest new system on chip at the same price as the old one has all these new features doesn’t mean that you have to bake that into your software. It should be an informed decision which features you want to use. In the same way that your car can probably do a 120 miles an hour, but that doesn’t mean that that should be your target every time you get on the freeway. Life doesn’t have to be like that. 

[Joe] 100%.

[Joe] Yeah. And so I I think looking at everything holistically and understanding the core objective and then prioritizing what you need to do, get more data about what’s involved in a simple product so that you can boost your security and boost your productivity and efficiency at the same time. As you say, convergence has been a hot topic for a while. I’d like to just add to it that convergence leads to modernization. And with modernization, we need to think about security enabling that modernization so that systems still run on time and we’re not created in greater risk to the enterprise overall.

[Paul] Joe, one final question, if I may. If you could solve just one cybersecurity problem for industrial CISOs and you could do it magically overnight, which one would you choose? Or is that choice a bad thing to do in the first place and actually always need to consider everything? 

[Joe] Well, you always need to consider everything, but I think it’s a valid question. If I had to choose one thing, I would wanna understand in all these embedded systems what is my exposure to memory based vulnerabilities across the board, and I would wanna know which of my devices can be compromised at run time in a way that could result in a very negative consequence.

[Joe] With that, I think analyzing your suppliers, analyzing what technology is used, analyzing what software is on those components. So leveraging a Software Bill of Materials and then figuring out a way to solve the memory based vulnerabilities that lead to runtime attacks on these devices is a critical thing all CISOs should do. If I could wave a magic wand, I would say eliminate all memory based vulnerabilities across your supply chain so that you’re no longer exposed in a major way to runtime attacks. And so that’s my approach. And the way to do that is to analyze the Software Bill Materials and to find ways to protect all those devices that have high exposure and high consequence in a way that doesn’t cost you a lot of money. 

[Paul] So some, you may have to replace. Some, you may be unable to replace, so you’ve already discussed how you can deal with those. And others, you might be able to manage the risk in other ways. The bottom line is that saying that has become a bit of a cliche, but it’s a cliche or a truism because it’s jolly well true. If you can’t measure it, you can’t manage it.

[Joe] And if you can’t see it, how do you stop it? 

[Paul] Yes. Exactly. It’s not just that the devices are embedded. The risks are embedded with them, and your job is to make that exposed.

[Paul] Joe, thank you so much for your wonderful insights and passion as always. Thanks to everybody who tuned in and listened. That is a wrap for this episode of Exploited: The Cyber Truth. If you found this podcast insightful, please be sure to subscribe and share it with everyone else in your team. 

[Paul] Don’t forget, stay ahead of the threat. See you next time.