Exploited: The Cyber Truth,  a podcast by RunSafe Security. 

[Paul] (00:04)

Welcome back, everybody, to Exploited: The Cyber Truth. I am Paul Ducklin, joined as usual by Joe Saunders, CEO and Founder of RunSafe Security. Hello, Joe.

[Joe] (00:20)

Greetings, Paul. Great to be here.

[Paul] (00:21)

You’re on the road again, aren’t you? Just in case the drinks get delivered in the background and people wonder what the background noise is. You’ve taken time out of your traveling schedule to be with us, so thanks for that. We have a very special guest today, and that is Ralph Langner. He is Founder and CEO of Langner Inc. Hello, Ralph.

[Ralph] (00:45)

Hi Paul, thanks for having me.

[Paul] (00:47)

A pleasure. Now, Ralph, many of our listeners will recognize your name and associate you very specifically with the Stuxnet virus, and I’m sure we will have at least a bit to say about that. However, our topic is much more general than that.

Our title is “Smarter Vulnerability Management in OT Systems.” I’m sure we will talk about all sorts of vulnerability management because it’s rare that you have an OT system that is not interconnected to some IT system at some point these days. However, before we start, I’d like you and Joe to give our listeners an insight into what drew you into operational technology industrial control systems rather than a more traditional sort of IT career.

[Ralph] (01:40)

Well, for me, that was a very long journey because it started very early in the 80s when I got fascinated with cyber-physical systems, and you may not believe i,t but at the time I was studying psychology.

[Paul] (01:54)

That probably came in handy later on, right?

[Ralph] (01:57)

Yeah. And so I was focusing on things like psychophysiology, where the mind and the body interconnect. And a couple of years later, I found myself working in software, and it certainly happened that this field also attracted my attention, where information meets physics. So I focused on developing software products for connecting the first PCs at the time with factory automation equipment.

And then a couple of years later, the whole industry shifted from point-to-point serial connections to Ethernet and to networks as we know them today. And since I had a pretty good idea of how those protocols operated, etc., it was totally clear for me that the whole industry was moving into one gigantic cyber risk scenario. So from that day, I shifted my attention to cybersecurity and especially in the OT space.

[Paul] (02:56)

Joe, your turn.

[Joe] (02:58)

In my case, I followed Ralph and joined the party much later, I would say around 2015, 2016. And for me, it was the intersection, maybe not of psychology, but of economics and technical issues, that when you combine those, it becomes a national security issue. Given the rise in geopolitical tensions and the targeting of energy or data centers or other systems like that, the consequences are quite high and the problems are a bit challenging because it’s very hard to update systems. 

For me, it became sort of an economic principle of how do you solve a large swath of the vulnerabilities in an area that’s constrained both technologically and operationally. For me, it’s a very interesting angle, but the national security implications are quite high at the same time.

[Paul] (03:49)

And we go back to, well, around 2010 and talk about matters of national security, Ralph, that does bring us to the infamous Stuxnet virus. For me, one of the most fascinating things about that malware when it was decompiled is that there was still a part of it that was essentially impossible to understand, namely, what was it supposed to target? It’s going to mess with some motor control systems, but which ones, where, and why?

[Ralph] (04:23)

For me, it was very fascinating. The fact that we did not know the target and we didn’t even know if the malware had been executed already or not. And that certainly got me alerted because I was under the impression, well, we might have a chance to prevent disaster from happening. The vendor in question at the time, Siemens, in their public statements, they said as much as well. We might never know what the target is. So it pretty much got me up on my heels because, understanding the sophistication of the malware, I saw that disaster might strike, and it might well have struck in the United States or in Germany. And this is something that I thought would be worthwhile preventing.

[Paul] (05:13)

So, in an OT-type attack, if you can’t be certain what the intended target is, do you have to start with the assumption that it could be anybody? And if so, how do you prioritize which potential attacks you should take on first?

[Ralph] (05:32)

Get back to the first part of your question. Let’s just say during your evening walk, you come across a rocket launcher. That’s scary. But you will figure out rather quickly that rocket launcher, whoever put it there, whoever is intending to use it, he’s not going to use it against the bakery next door or the pizza parlor, right? So the size and nature of the weapon, especially if it’s custom built, tells you something about the intended target. 

When you see the sophistication that you can see in Stuxnet, when you see what is possible with cyber-physical attacks, you will certainly get the impression,  this is going to be the next area of warfare. We’re going to end up in cyber war. That is certainly what I had expected. And then fast forward, it actually never happened. And as a striking example, if you look at the ongoing war in Ukraine, cyber doesn’t play a role. And honestly, I have to say, I’m very surprised about that. What we see instead is old-fashioned kinetic war with the addition of drones.

[Joe] (06:40)

I need to jump in there. I mean, there are active cyber attacks that have happened in Ukraine. And in fact, the cyber attacks precede many of the kinetic attacks. And if you think about acting groups like KAMACITE and Electrum, who are active in Ukraine and are associated with Russia, there are significant cyber attacks happening in the Ukraine. I look at Taiwan and I see that there’s 2.4 million cyber attacks a day. And a lot of them are originating from China itself. 

Part of the activity that happens in Ukraine that I think is instructive for Taiwan is that folks like KAMACITE will enter by way of IT systems and provide access for acting groups like Electrum that are targeting OT systems. I have seen plenty of data and I’ve seen heat maps following the cyber attacks that precede kinetic attacks. And it’s all in the western part of Ukraine where the kinetic attacks occur.

[Paul] (07:38)

Joe, would you also say that there’s a significant risk, even with vulnerabilities or exploits that are not specifically used for what you might traditionally call an attack, where you’re aiming to break in and force somebody else’s industrial control systems to do something unintended? Because there’s an awful lot of data available that can be milked out of the average industrial control system if you think of something like a power grid or a sewerage control system or traffic control systems across a large city. And all of that matters too, doesn’t it?

[Joe] (08:18)

I think it does matter. We’ve seen movement in that area in the past couple of years in the United States. And part of it is knowing and testing. If you have a water facility, what is their ability to respond? So you can check and test and see what the responses are and learn a lot about a system in order to think about if you’re an attacker, think about how might you approach other water centers across the country.

[Paul] (08:44)

Managing all of this is a little bit different from a traditional IT system and very different from a traditional web app where you can just change the code and the next person who arrives at the website gets the new version. So what are the biggest challenges and considerations, Joe and Ralph, that you find in what we might refer to traditionally as vulnerability management in OT or industrial control systems?

[Joe] (09:10)

Well, from my perspective, we’ve got the product manufacturers who produce these technologies. And of course, then you have the operators of infrastructure who are deploying those assets inside their infrastructure. The operators need to protect the network itself, but there is separation of cost and consequence. If the operator bears the risk, ultimately, but the product manufacturer produces that technology, there needs to be a lot of coordination between the organizations. And so I think the difficulty in getting updates out into the system, I think the capitalized expense, this is where my economics interest comes in. 

There’s purchases of equipment that’s expected to be capitalized over 10, 20, 30 years. And these gadgets, these devices, these systems are intended to survive or to operate for a long time. Their age in part creates part of the vulnerability. And as we’ve seen, memory-based vulnerabilities on these embedded systems, on these cyber-physical systems. The vulnerabilities have been around since the 80s. Now there are human elements that help, and there’s technical approaches that help, but I think the distributed nature, the separation from product manufacturer to operator, and then the complexity in getting updates out to these systems all contribute.

[Ralph] (10:29)

So let me give you my perspective on this. First of all, I think it’s important to sort out the landscape and you think about OT vulnerabilities because there are so many different aspects of it. Most people, when they hear the term OT vulnerability management, for them, it’s just about knocking down CVEs. And if you try to do that, you face a unique challenge because there are hundreds of thousands of vulnerabilities, known vulnerabilities, for which you have a CVE.

So you would need an army of engineers to actually fix even half of these. But the other thing, since we have been talking about cyber war, et cetera, what most people don’t know. When you consider a competent attacker, they would not exploit those CVEs. So just to give you an example, for pretty much every single PLC model with an embedded web server, you will have CVEs because it appears impossible for mere humans to design a web server that you put on a PC that doesn’t allow for cross-site scripting and all that nonsense. 

But the competent attacker would never try to take advantage of these vulnerabilities because the competent attacker knows if they want to succeed in OT, they just need to exploit features. Those are not considered bugs. They are not considered vulnerabilities. They are just legitimate product features. And the backstory is that those OT systems, those controllers, etc., they’re insecure by design and they’re going to stay that way for quite a while. So this is what we have to deal with. 

Depending on the context that you discuss this in, like when you think about predicate infrastructure and national security, et cetera, the CVEs don’t play that much of a role. It’s just more the question, what would a competent attacker do? And this is something that you can analyze and protect pretty decently against it.

And then there is even a third group of vulnerabilities in this context that you also have to consider, which is mere stupidity when it comes to configuration. And since we have been talking about municipal water facilities and they are hacked, what the problem is, somebody put the controller or HMI on the public internet without even thinking about a decent password. Those three topics are very, very different. Unfortunately, as I see it, the majority of organizations in this space, meaning the asset owners, focus too much on shooting down the CVEs, which is not the most rewarding job.

[Joe] (13:01)

Just add that the quantity of vulnerabilities is one thing. I think the severity of vulnerabilities is another thing, and you can start to prioritize what does matter. I think the traditional thinking of patching every single one is in fact a losing proposition in that there needs to be a more asymmetric shift in technology that doesn’t target vulnerabilities per se, but looks at classes of vulnerabilities and eliminates the exploitation of those. 

Then specifically, I’m thinking about memory-based vulnerabilities that a lot of these acting groups are using. And so if you apply asymmetric technology to prevent exploitation for a majority of them, then your problem in your funnel of known vulnerabilities goes way down, and your ability to adapt goes way up. And certainly, there are ways to prevent exploitation of potential zero days, without even knowing what the attack vector is. If you apply that stuff in the build side of the equation, then you have a much deeper set of resilience so you can focus on the other areas of defense and depth.

[Ralph] (14:12)

Let me add another facet to this context, and that would be just look at the present threat landscape. It is very clear that the majority of actual cyber attacks in the OT space for the last couple of years were in one specific region, and that is simply ransomware. Luckily, we didn’t see any real cyber-physical attacks, but we were seeing thousands of ransomware attacks. So even though I don’t think that these ransomware operators are really targeting specific industries such as manufacturing or creative infrastructure. I think those are opportunistic attacks, but it’s very clear how this whole thing plays out. It always is associated with a Windows box, with a vulnerable Windows box. And this is where, in OT, you find a target-rich environment.

[Joe] (15:04)

Considering what could go wrong. And you look at China’s pre-positioning and critical infrastructure, and you look at things like Volt Typhoon and Salt Typhoon and the like, it’s well beyond ransomware that the problem occurs. I think what most people in the industry are concerned about is also the loss of operational capability, whether it’s taking out data centers or others, and the vulnerabilities in those systems by virtue of the HMI and then connecting to the thermal controllers and the cooling systems and things like that. 

The problem is complex. I see a great opportunity to improve the security posture by shoring up the vulnerabilities related to memory-based attacks. And just as Ralph talked about, well, what could go wrong? There’s a lot that can go wrong with all these memory-based vulnerabilities across critical infrastructure. When we have already seen attacks in Ukraine, attacks in Taiwan, and critical infrastructure, and then certainly pre-positioning by China in US critical infrastructure. I think the stakes are high and growing, and there’s certainly risk that needs to be mitigated.

[Ralph] (16:11)

Let’s just focus our attention on the more interesting cyber-physical attacks. Let’s focus on taking down a data center or multiple data centers, for that matter. You would not attempt to do this by trying to attack all those NVIDIA machines directly. You would go through the building automation part, which is considerably easier because usually those systems are not well protected. And unfortunately, the cybersecurity guys that are in charge of the security for that data center, they never even considered that all that network shit that is basically driving that whole building automation and the lack of security that you usually find there. But this could be fixed fairly simply.

[Paul] (16:56)

Now Ralph, earlier you mentioned, almost in passing, about OT systems that, in most cases, they are effectively insecure by design. So what do we do, both for OT systems, for IT systems, and their nexus, the network connections that bind them together? What do we do to change the world so that secure by design becomes something that we can take for granted rather than something that organizations seem to avoid either because they’re afraid they’ll never get there or they think it will simply cost too much.

[Ralph] (17:36)

That is a wish that OT security experts had for decades. Well, why don’t we just try to push the automation vendors to actually build security in, but so far, it never happened. And should it ever happen, you and I will not live to see it. That’s for a very simple reason. You have already addressed a long lifetime of these systems. If you just think about the installed base. 

So presently, a rough estimate would be globally, we are talking about 300 million industrial control systems. And pretty much all of them are insecure by design. So if you just imagine how long it would take to even replace a 10th or maybe 20 % of that installed base, it’s going to take decades. What I could envision is that through some breakthrough technology shift, we would see more secure products. Like when you think about the revival of U.S. manufacturing then that could probably involve new designs, new architectures that would, over time, actually address and fix that problem. However, I’m actually not that much concerned about those insecure by design controllers, RTUs, actuators, sensors, you name it, because, well, we have certainly learned by now how to arrive at secure network design.

[Paul] (18:59)

And yet we have ransomware attacks all the time, well-documented, that affect people across the entire industrial base.

[Ralph] (19:08)

Don’t get carried away because so far the ransomware text that we have seen didn’t involve the industrial control systems.

[Paul] (19:17)

What I’m saying is that they did involve the network, so the idea that we have solved the network security problem seems to be, how can I put it, wishful thinking.

[Ralph] (19:25)

Hear me out. So certainly we haven’t solved network security. We will never be able to solve network security, but we are making incremental improvements.

[Paul] (19:35)

Why can’t we make those incremental improvements in OT systems as well, without having to go after each vulnerability one by one?

[Ralph] (19:42)

Yes, certainly we are making progress in that area. And one strong push would come if you start implementing the basics. What everybody should try to accomplish rather quickly, if they haven’t done so already, is to separate the enterprise network or the IT side from those process networks with a DMZ. 

For reasons beyond my comprehension, that hasn’t happened so far. And we have seen cases where the asset owner was under the impression that they have a DMZ when in fact they didn’t. Happened in the Triconics attack, where the safety controller was compromised. The asset owner thought it separated properly when a risk assessment had shown later on after the fact, no, it’s not a DMZ. So that would be a good starting point, to just segregate those two worlds from each other.

[Paul] (20:35)

Joe, where do you issue relating to liability and lifetime come in? And I’m thinking specifically of discussions that we’ve had in the past about things like the CRA, the Cyber Resilience Act in the European Union, which as I understand it is trying to be a stick to press manufacturers to be liable for poor decisions that they make for insecurity by design and also requiring them to commit to the lifetime over which they will actually support their products, whether they’re IT or OT or IoT or whatever they might be. Do you think that that’s absolutely needed if we want to get anywhere?

[Joe] (21:20)

Well, I think it can be helpful. Do I think it’s absolutely needed? No. But I do think that the stick does help raise part of the awareness. And I also think that it does start to change the thinking in terms of economic implications. And today, I think, yes, maybe reputation could be a problem if you are Schneider Electric and your gadgets ultimately were targeted inside an operator. Well, that comes down pretty hard on Schneider Electric, even if it was Saudi Aramco that was attacked for example, there is reputational risk, but I do think that a stick that tries to impose liability and also requires other aspects does start to elevate people’s awareness about what kind of responsibilities they can have. 

Secure by design, building security into your products is a good idea and also good practice. And it may in fact lead to improvements in code quality and byproducts like that by simply looking at what your development processes are and what your methods are for building and compiling and testing and automating those processes to ensure that you’re producing technology that is ultimately resilient.

[Paul] (22:30)

This certainly sounds as though it will lead to a world in which we will not have the kind of response that Ralph talked about earlier in the Stuxnet incident, where it sounds like they just threw up their hands and said, well, how will we ever know? We may never find out. That seems to be a very self-serving and defeatist attitude. And if manufacturers and vendors did not think that way, do agree that the world would be a better place from a cybersecurity perspective?

[Joe] (22:58)

Yeah, I think if people improve their software development processes, then we’ll have higher-quality code and more resilient code in the first place. And it does take folks like Ralph who thought about what could be the target and what could be going on here, despite the manufacturer in that case, not focusing on it. It has a major implication for how we approach cybersecurity going forward. So I’m grateful for Ralph for what he has done and what he is doing. 

And I agree that there are human elements. There’s also technical approaches to create asymmetric shifts. And I often think about things in both a silicon-based approach and then a carbon-based approach. I think we need to look for a technology solution that does create an asymmetric shift in cyber defense. But I also concede on the carbon-based side that we do need a human-based approach to ensure that you are managing the operations of your security programs effectively as well.

[Ralph] (23:59)

Let me add another dimension here, which is money. And in my experience, that’s the most important dimension, because if you look back at what the automation vendors did, let me put it in simple terms. We know how to build secure by design products. That’s not a mystery. The real problem is once you do that, your secure by design controller, all of a sudden it is let’s just say, 50% more costly. It’s more expensive than the regular one that you’ve been using now for 20 years. That other model of which you have in store, let’s just say 5,000 pieces. That is the real issue. 

We have seen that play out many, many times after Stuxnet. For example, we used to consult automation vendors on how to design secure controllers. And then we saw that those projects had been stopped because the vendor realized, oh, we will never be able to sell this thing because it’s going to cost, let’s just say 50% more.

[Joe] (25:01)

We always promote from our side that for a 5% increase in cost, you can solve 80% of the problem. So I think that 5% is a good number in some situations. It would be calculated that way if you look at it on an individual product basis. What I’m talking about in terms of updating software development processes, these manufacturers are quite large. And so if you spread that redesign process over multiple teams, it’s far less than the 50% that you would suggest or see in an individual case.

[Ralph] (25:32)

Okay, so let me make this clear. I’m talking about end-user cost. And first of all, what you have to factor in there, that new product will certainly have a higher price ticket on it. But then also if you think about the cost that comes with the additional effort it takes to train your existing workforce to actually use those secure by design products properly, that’s another cost factor.

The way that I pitched it back in the day, well, you know, think of it as a business opportunity because all those insecure controllers will be replaced and that’s a gigantic business opportunity. But according to their calculations, it didn’t pan out so far.

[Paul] (26:10)

Ralph, I’m conscious of time, so to conclude, can I put a question and just give you and Joe a chance to give a pointed, suggestive answer to our listeners? If you could recommend just one change that manufacturers or vendors could make in the next year, what would it be?

[Ralph] (26:30)

Just be open about your security issues, I would say. And we have seen a lot of progress in that area. So what I would like to see is that every automation vendor makes their vulnerabilities and their other problems publicly available. Downloadable via a REST API port, something like that, because you need automation to actually process it. That would be a huge improvement.

[Paul] (26:57)

Joe?

[Joe] (26:58)

Solving memory-based vulnerabilities at a class level instead of individual CVE and vulnerability level would result in an asymmetric shift in cyber defense and is easy to implement. So I would solve the memory-based vulnerabilities as an easy first step.

[Paul] (27:13)

Excellent. I’ll summarize that into three words: onwards and upwards. Ralph and Joe, thank you so much for your insights and in particular for discussion of the history, the current, and the future of OT security. I’m sure our listeners have enjoyed it greatly. 

So thanks to everybody who tuned in and listened. That is a wrap for this episode of Exploited: The Cyber Truth. If you enjoy this podcast, please don’t forget to subscribe so you know when each new episode drops. Please like and share us on social media as well, and don’t forget to tell your whole team about us so that they, too, can listen to Joe and Ralph’s opinions and expertise. Once again, thanks to everybody who tuned in and listened, and remember, stay ahead of the threat. See you next time.