Exploited: The Cyber Truth,  a podcast by RunSafe Security. 

Paul Ducklin (00:06)

Welcome back, everybody, to Exploited: The Cyber Truth. I am Paul Ducklin and I’m joined for this episode by Doug Britton, who is Chief Strategy Officer at RunSafe Security. Hello, Doug.

Doug Britton (00:20)

How are you? Good to see you, Paul.

Paul Ducklin (00:22)

And our special guest for this episode, Andrew Ginter, who is VP of Industrial Security at Waterfall Security Solutions. Hello, Andrew.

Andrew Ginter (00:33)

Hello, nice to meet you.

Paul Ducklin (00:35)

Andrew, let’s get straight underway. Our title for this episode is “When Cyber Attacks Become Physical: The Truth About Operational Technology Security Risk.” So we’re very much focusing on the OT side, not the traditional IT side. So if you don’t mind, I would like to start by asking you to tell us a bit about your background and perhaps most particularly, how you came to end up in OT security rather than the more traditional field of IT security.

Andrew Ginter (01:08)

It’s a bit of a long story, but I’ll make it short.

Paul Ducklin (01:10)

Haha!

Long stories are great. Cybersecurity was never simple, so it can’t be told in a second.

Andrew Ginter (01:17)

You can see from my gray hair, I’m on the end of a long career. I’ve been doing stuff for forty years. As a young man, I bounced around; the grass was always greener. A lot of young folk do this. I settled in at Hewlett Packard designing industrial control systems. My code is still, believe it or not, thirty years later, automating the world’s largest liquid petrochemical pipeline. Did control systems for a number of years.

And then moved to Agilent Technologies, where I was asked to lead the development of the world’s first IT/OT middleware that connected OT systems to SAP. This was the thing to do in the mid-1990s, connecting a lot of IT and OT networks, which contributed to the cybersecurity problems.

Paul Ducklin (02:05)

It’s very nice of you to say that.

Andrew Ginter (02:08)

I got religion. I wound up the Chief Technology Officer at Industrial Defender, developing the world’s first industrial SIEM security information and event management system. Today I work with Waterfall. I work with some of the world’s most secure industrial operations. I learn from those operations. And again, at the end of my career, I’ve taken to writing books. This is my most recent engineering grade, OT security, books about what I’ve learned from the world’s most secure industrial operations.

Paul Ducklin (02:38)

Andrew, it seems that these days surprisingly many companies who decide they want to, if you like, make the OT side, the SCADA side, the industrial control side of their business more secure just sort of lift up their received wisdom from IT and try and splat it down onto the OT side of things. And that doesn’t really work very well, does it? Because there’s a bit of an impedance mismatch there.

Andrew Ginter (03:08)

It doesn’t work for a number of reasons. The OT space is vast. The intrinsic difference between most OT processes and IT processes is consequence. If a subway switching system is misoperated and high-speed trains full of people collide at rush hour, hundreds of people die, mass casualty event. What’s the worst that happens if you break into that same subway’s IT network and steal all of the PII, personally identifiable information about all of their passengers?

Paul Ducklin (03:41)

And I guess for very good reasons, there’s a long history of regulatory controls in many parts of the OT world that have focused on safety, human safety, machine safety, if you like, perhaps at the expense of security. So it’s important that the system, when presented, for example, with correct data, will perform correctly within some hard real-time period.

But the system has not necessarily been tested for how it might respond to data that was deliberately misconstructed in order to cause harm. Would you agree with that?

Andrew Ginter (04:22)

Absolutely. You hit the nail on the head. It’s a truism in cybersecurity that a reliable system is one that always does what it’s supposed to do. A secure system is one that never does anything else. And proving a negative is very difficult. A lot of the differences people point to between IT and OT systems, it’s difficult to patch OT, it’s difficult to anti-virus, it’s difficult this, it’s difficult that. A lot of it has to do with safety, has to do with engineering change control. It’s a whole process to control the risk of making a really bad mistake. This engineering change control process flies in the face of standard IT practice, not all of which, but many of which, amount to constant aggressive change. The latest security updates, the moment they’re available, the latest antivirus signatures, the moment they’re available.

Paul Ducklin (05:01)

Yes.

Indeed.

I’m trying not to smile and desperately trying not to say the word Crowdstrike. They pushed out a data change that caused a buffer overflow, that caused a crash early on in Windows boot and brought the system down. A failure like that, for say a valve that needs to shut in an emergency. It’s not just that you can’t afford 90 minutes while the thing gets patched and rebooted. What I believe you refer to as cyber-informed engineering, in OT that’s a very, very different discipline to mostly getting software right in the IT world.

Andrew Ginter (05:55)

I’ve talked about consequences. One of the things that is poorly understood is that these changes, engineering change control, tends to make OT systems harder to secure. They can’t be patched as aggressively, and thereby makes OT security weaker than the average of IT security. Worst-case OT consequences tend to be truly unacceptable, whereas worst-case business consequences tend to be acceptable. We can buy insurance for that. 

The obvious advice that we get from IT people is why can’t you use zero trust everywhere? Why can’t you patch everything? Why can’t you encrypt everything? Well, even if I had a magic wand and I patched everything and I encrypted everything, I would now have an OT network that is as strong as my IT network, because that’s what I’ve done on IT. I need an OT network that is materially stronger than IT. How do I get that?

This is where CIE comes in. CIE is the Idaho National Laboratory initiative funded by the US Department of Energy, credit where credit’s due. Cyber-informed engineering. It is a new way of looking at the OT security problem. It’s a lot of the same puzzle pieces put together in a different way.

Paul Ducklin (07:13)

So that’s not just about engineering for cybersecurity. That is engineering things like information flows, safety, and security altogether, so that you’re not trying to make your OT network like your Zoom network, where you’d actually probably more concerned with keeping people who shouldn’t be in the meeting out than actually making sure that meetings we’ll always go ahead no matter what.

Andrew Ginter (07:43)

Vaguely, I mean, the way I describe CIE is officially, formally, CIE is a body of knowledge that spans elements of safety engineering, protection engineering, automation engineering, network engineering, and most of the six pillars of the NIST Cybersecurity Framework. It’s a big body of knowledge. And it’s a body of knowledge focused on the resilience of industrial processes. 

Unofficially, informally, cyber-informed engineering positions OT security as a coin with two sides. One side of the coin is cybersecurity. Teach engineering teams about cyber threat, teach them about cyber attacks, teach them about the limitations of cyber tools that we’re using in IT and OT so that they can correctly evaluate residual risk. The other side of the coin is engineering. Cyber-informed engineering is not one or the other. When you spend a coin, which side of the coin do you spend? The engineering side or the cybersecurity side? Well, what a silly question. When you spend a coin, you spend the whole coin. You’re not choosing one side or the other. We need it all. The essence of cyber-informed engineering is engineering-grade mitigations as well as IT-grade mitigations to address threats to safe, reliable, and efficient physical operations. I’ve invented the phrase network engineering.

It’s a collection of techniques that deterministically control the movement of attack information at consequence boundaries.

Paul Ducklin (09:19)

So when you say network engineering, you’re talking about managing the flow of information inside an OT network, inside an IT network, and very importantly, between them, so that correct data emerges and correct instructions go in the other direction. Is that roughly what you mean?

Andrew Ginter (09:40)

Fundamentally, let’s step back and understand. There is actually a second fundamental difference between most IT networks and most OT networks. And that is in IT networks, information is the asset. Information is what we need to protect: the confidentiality, integrity, availability of the information. In OT networks, physical operations is the asset, and information is the threat. 

Step back for a second. 50 years ago, we’re talking 1973, the American military awarded a research contract to a bunch of university researchers trying to figure out how to protect classified information in the era of mainframes. They put a theory together that is still applicable. A lot of us still learn it in school. When we go to school and learn cybersecurity, Belle Lapadula put a theory together of how to prevent theft or leakage of really important information. What most people forget is that two years later, 75, Biba came out with another theory, used all the same ideas, used all the same terminology, but inverted the theory and said, if the goal is to prevent sabotage rather than to prevent espionage, theft of information, what we need to do is control the movement of attack information, not protect the information. Information is not the asset, information is the threat.

The only way a control system can change from a normal state to a compromised state is if attack information enters the system somehow. Controlling the movement of attack information is a fifty-year-old theory. And so this is the theory behind network engineering.

Paul Ducklin (11:20)

So is that sort of the idea of saying, whilst it might not be great if an attacker could learn exactly what temperatures and pressures were inside XYZ boiler at any moment of time, it’s very much worse if they can actually make you see that information incorrectly and change it without you realizing it. In other words, it’s a modification and the manipulation of the system that you should be worrying about, whereas perhaps in IT systems, we’re sort of more worried about the corporate database getting stolen.

Andrew Ginter (11:54)

Generally, yes. Most OT systems don’t have that much valuable information. There’s exceptions. Aircraft manufacturing, even automobile manufacturing. A lot of manufacturing has robotic programs that are trade secrets. You want to protect them. But what you absolutely need to do first, you know, aircraft manufacturing, prevent the bad guys from getting in and introducing subtle defects into the products so that aircraft fall out of the sky ten years later.

Not leaking the design of the aircraft is good too, but the first priority is no aircraft fall out of the sky. Thank you. So again, preventing attack information entering the system. It’s about sabotage, not espionage.

Paul Ducklin (12:36)

So Andrew, at this point, do you want to say something about a couple of buzz phrases that are understandably big and important at the moment? Namely, secure by design, and I guess the market’s flip side of that, secure by demand, where the market says we will spend our money with companies that actually take this whole thing more seriously. How do those two faces of the die fit into modern OT security?

Andrew Ginter (13:02)

There’s also skewer by default, which is when product comes out of the box, all of the security features are turned on. First thing is, I really don’t like the terminology. I mean, these are good approaches, but I really don’t like what they’re called. Who invented these terms? Marketing. Why did they invent these terms? Because back in the day, certain large companies who shall remain nameless had a reputational problem as being insecure.

Paul Ducklin (13:09)

Yes.

Andrew Ginter (13:28)

So they invented these secure by design methodologies and said, what are we going to call this? Let’s call it secure by design, because that means if we use it, then we must be secure. And we are using it, therefore we must be secure. Security is not a yes or no thing. It’s a spectrum. The question, how secure are we, has an answer. The question, are we secure has no answer. The question, how secure should we be, is even more important. But to say we’re doing secure by design, therefore we’re secure, is complete nonsense.

Paul Ducklin (13:59)

But would you agree that as a starting phrase, it can be quite handy, particularly when you look at things like the IoT, the home market of OT, if you like, that products are rushed to market where they haven’t even made any effort to do any kind of security by design, it’s been completely ignored. At least you’ve chosen a trajectory that might get somewhere useful. Would you not agree with that?

Andrew Ginter (14:24)

Methodology is good. It’s not secure by design. It’s security by design. If you do it, you have some security.

Paul Ducklin (14:33)

Okay, I see what you mean. You’re avoiding making it sound like an absolute; you’re just making it sound more descriptive, I get you.

Andrew Ginter (14:41)

That’s right. Too many people coming into the space, you know, they see, I’ve got secure communications, therefore I’m secure. No, you have encrypted communications.

Paul Ducklin (14:51)

Yes, I guess it doesn’t help that secure by design trips off the tongue more easily than maybe the marketing people sort of want out there.

Andrew Ginter (14:59)

Once we leave that behind, these are all very useful design patterns, methodologies. They are part of the CIE umbrella over on the cybersecurity side. By all means, do all of the secure by stuff to the degree that it can be done in the systems that you’re looking at. And the systems deeper into the stack, closer to the physical process, especially the safety-critical systems. You have to be careful with that. I mean, can you encrypt communications for a safety-instrumented system?

Well, I suggest that’s probably a bad idea unless you’ve managed to partition the system into multiple CPUs, which I think modern systems do. But back in the day, if you had a CPU that was doing your safety, the code that CPU was running was as small as you could make it. If you throw an encryption library in there, you just tripled the amount of code in your safety instrument system.

Paul Ducklin (15:52)

If you’re lucky. It’s probably worse than that.

Andrew Ginter (15:55)

Precisely. And every new line of code is an opportunity to make a mistake. And you cannot make mistakes in safety systems.

Paul Ducklin (16:03)

And what about the corresponding problem that, particularly in industrial control systems, you can’t just easily rip and replace like you can have a three-year laptop exchange program in your IT department? So how do you bring security by design into a world where you might be stuck with some legacy devices that you don’t want to change because they do actually work as originally designed?

Andrew Ginter (16:31)

The short answer is use engineering tools. Use network engineering. Use deterministic protection as well as IT-style probabilistic protection. Right. If we’re gonna make OT security stronger than IT, this is one of the things we have to do. We have to use tools that are available on the engineering side that don’t exist on the IT side. That’s how we make OT stronger than IT. Deterministic protection is one of the powerful tools we have to make OT systems dramatically more secure than IT. 

And legacy equipment is one of the differences between IT and OT that tend to make OT weaker. It’s why we need CIE so very urgently. Another pet peeve is people talk about brownfield, brownfield. I’m sorry, everything is brownfield. Let’s say we have a brand new solar installation, brand new equipment, brand new computers, brand new everything.

Before that system goes live, the system’s integrator assembles all of the computerized hardware in their lab, puts it all together, programs it all, configures it all, tests it all, documents it all, and carries out a factory acceptance test. And then they package it up, they drive it to the site, they connect it all out, and you do a site acceptance test. After the site acceptance test, if it’s passed, if everything works, you go live. That’s a six-month process.

Six months before you go live, you have started locking down the factory system. Because if you don’t, you risk making mistakes. And you don’t want a mistake to show up in the site acceptance test. Engineering change control kicks in six months before you deploy.

Paul Ducklin (18:17)

This is very different from the wilder world of web apps, isn’t it? Where hey, you can update tomorrow and if you don’t feel like waiting till tomorrow, you can update before the next guy actually sends an HTTP request. You can try out ten different approaches for the next ten different users. You absolutely cannot have that approach in an OT network where you want particular real-time performance and behavioral promises to be met.

Andrew Ginter (18:46)

Precisely, so everything OT is legacy. A better way to say it is there’s no such thing as greenfield. It’s all brownfield. Even if I had a magic wand, even if I could in the space of one second upgrade the entire thing to the very latest patches and software and hardware and security capabilities, it would all be as secure as my IT network. I need it to be stronger. How do I make it stronger? CIE is charting the way.

Paul Ducklin (19:16)

Doug, do you want to say something at this point about some of the things that RunSafe has been doing in order to bring about what you might call an element of self protection to software that allows it to behave in a sufficiently similar way to the way it did before, but it still meets its deterministic needs, without just stuffing it full of more security agents, more antivirus, more EDR, more behavior blocking, more filtering, this, that and the other as a way of bringing about security even when you don’t get a choice to change the hardware.

Doug Britton (19:52)

Yeah. We’ve observed a very similar economic problem. You don’t get to just right click, replace everything. So now what? The underlying economic proposition we focused on was using some modern software methods. Can we teach a piece of software how to protect itself against attempts to interdict and get, as you described, Andrew, unintended behavior while maintaining its intended behavior?

Paul Ducklin (20:00)

Yeah.

Doug Britton (20:21)

What we’ve been able to do is identify ways of keeping software in its state diagram, particularly this is essential in safety-critical systems. Keep software in its state diagram, but add a separate state that if there’s an attempted interdiction into something that would otherwise go off state, it now has a place on a state diagram. You jump into deterministic and predictable behavior without changing the underlying performance.

Paul Ducklin (20:48)

So don’t bet it will be for things like buffer overflows, for attempts to change control flow illegally, stuff like that.

Doug Britton (20:56)

Precisely. Right now, attempts to interdict software or get it to go into unintended places, those unfortunately go off a state diagram because there is no state that describes opening a valve under this voltage condition. It’s not envisioned by the designer. But what you’re able to do is when someone attempts these invasive software mechanisms that use memory in unintended fashions, that is defined state, and the system has a response that prevents the intended interdiction and it does it in a way, again, that doesn’t impact performance, can be done to existing software without requiring rewriting things.

Paul Ducklin (21:33)

And without injecting new libraries at runtime and wrapping it in this and that and the other, that no matter how hard you try is going to change its performance and its memory footprint and all of that.

Doug Britton (21:43)

Well, exactly. And that’s particularly important when you, again, going back to the Brownfield observation, you don’t get to add more memory, you don’t get to add more bandwidth, and you don’t get to add new crypto modules. And so that’s inline protections that allow that deterministic behavior of safety-critical systems. There’s been a lot of economic benefits.

Paul Ducklin (22:03)

And I guess that also means that you don’t have to rely on the fact that we can’t think of any other way to deal with this newly reported vulnerability other than to patch and therefore rush out and change everything where change is not necessarily a desirable thing in an OT network.

Doug Britton (22:22)

Precisely, particularly in the face of the onslaught of AI-discovered vulnerabilities, the update cycle for OT systems isn’t going to go up by 5%. It’s going to go up by multiple X, 10X, 100X. Teams are already at their breaking points with respect to the volume they’re able to update. And so now one postulate is you’re going to have to reboot OT networks 10 or 100 times as frequently.

But that’s not workable either. So methods that can be added in that make it so that you don’t have to have 10X or 100X reboot frequency become quite important in those conversations.

Paul Ducklin (23:04)

Gentlemen, I’m conscious of time, so I’d like to finish up by asking either or both of you a fairly specific question. If you’re what we might call a critical infrastructure leader, if you’re a CXO in that space, and perhaps you hadn’t got far along the road of improving both safety and security in your OT networks, what would be the one thing you could start with that would get you on the best path as quickly as possible?

Andrew Ginter (23:36)

Learn about CIE, start using engineering-grade mitigations as well as IT-grade. Do what Doug has said, do what I’ve said to make your OT network stronger than IT. That’s the goal. In particular, I would point out that network engineering is the most universally applicable CIE technique. Unidirectional gateways, which is what Waterfall produces. I work for Waterfall, which is the most widely used kind of network engineering. This is why I study this stuff.

Paul Ducklin (24:05)

Is that what some people call a data diode?

Andrew Ginter (24:08)

Yes, that is what some people call a data diode. It’s a data diode with software that makes copies of Pi servers, copies of Oracle servers out to IT networks to simplify integration. But to Doug’s point, AI is coming. AI is finding the vulnerabilities faster. Zero days is developing exploits, chaining together multiple low-severity zero days into high-severity exploits and automating, outright automating the entire attack process. AI is gonna change the game. It’s not a game, it’s gonna change the nature of the threat in the next 24 months. 

We urgently need to make our OT systems stronger. CIE, network engineering, unidirectional gateways, lots of other OT-specific technology is urgently needed. That’s one answer. A very short second answer is the first step that pretty much everyone has to take is go and take your passwordless HMIs off of the internet, people, do it on an emergency basis.

Paul Ducklin (25:10)

There are lots of exciting things you can do, but if you start with the basics, you’ll be making a better start. Doug, do you have anything that you’d like to add?

Doug Britton (25:19)

Yeah, I think we need to figure out how to get out of the sense of it’s okay to just keep track of patches. Patches are structurally reactive. If you’re judging your success on patch management time, an Army Special Forces captain I worked for years ago said if we adopt a defensive posture, the only logical conclusions are surrender or defeat. And so patch management discipline is a defensive process.

We’ve said we’re gonna keep getting penetrated and we’re gonna just try and deal with it. The mindset needs to be more aggressive than that. And cyber defense can be aggressive. And what you want it to be is how am I getting rid of entire classes of risk? How am I making it so that classes of attack become irrelevant? Focus on methods that make entire classes of attacker activity irrelevant.

Paul Ducklin (26:11)

And sometimes that might be as simple as disconnecting a device that doesn’t need to be connected in the first place. Or it might be reworking your or recompiling your code so that it behaves, as you say, in a safer state machine without needing to re-engineer it completely. Get the basics right and don’t try and fix everything all at once. I guess with what’s happening in the AI world. Best start today, don’t leave it till tomorrow.

Gentlemen, thanks so much for your passion and your insight and your huge experience in this field. I wish we didn’t have to stop. I wish we could just keep on talking about where cybersecurity and cyber integration in OT is going. But that is a wrap for this episode of Exploited: The Cyber Truth. If you enjoy this podcast, please like and share us on social media and leave a nice comment if you listen to us on a podcast feed.

Thanks to everybody who tuned in and listened. And remember, stay ahead of the threat. See you next time.