Exploited: The Cyber Truth,  a podcast by RunSafe Security. 

Paul Ducklin (00:03)

Welcome back, everybody, to Exploited: The Cyber Truth. I am Paul Ducklin. For this episode, I am joined by Shane Fry, CTO at RunSafe Security. Hello, Shane. 

Shane Fry (00:20)

Hello. Thanks for having me.

Paul Ducklin (00:23)

And our special guest today is Andrew McPhee. Andrew, you are the solutions manager for industrial security at a company that needs little introduction, I guess, namely Cisco. Welcome.

Andrew McPhee (00:36)

Thank you. Happy to be here.

Paul Ducklin (00:37)

Andrew, our title for this week is not only a broad but also a deep and very important topic. “Seeing the Invisible: The Reality of OT Security.” Why don’t we kick off, Andrew, by telling us how you got into cybersecurity in general, and perhaps more interestingly and importantly, how you got to focus on industrial control systems and OT.

Andrew McPhee (01:05)

I actually got into OT before I got into security. I was a mechatronic engineer, that was my specialty. Coming out of university, I took a trip over to the US. I am Irish and there was an intern position at a company called Cisco. I was like, sure, let’s let’s work for Cisco for a summer while I explore the US a little bit. So I’m on an eleven-year internship, as I call it, as I’ve been here ever since. I started off as a software engineer.

Paul Ducklin (01:31)

Hardly started.

Andrew McPhee (01:34)

A software engineer within our IoT business unit. I used to work in our innovation labs, where we’d show you what you could do in this IoT space when it was new and exciting. We spun up an automotive business unit within Cisco where we focused on vehicle networking. So when I was ready to move into big Cisco from all of the mini startups within it, I moved to the security business group. And then, full circle moment, our OT business group that focuses on this as their day job. And they needed a full-time OT specialist to work on security.

Paul Ducklin (02:07)

Now it sounds as though you’re in the position that very many people who started out in OT probably are, in that when they started, cybersecurity wasn’t either their first, second, or even third interest, and wasn’t as important as it is today. Would you agree with that?

Andrew McPhee (02:28)

For sure, yeah. I didn’t know what a TCP or UDP packet was when leaving university. We understood that we needed to connect PLCs together, but cybersecurity networking in general wasn’t part of the curriculum.

Paul Ducklin (02:39)

So security by design, as I guess we refer to it today, had a little bit of a back seat button.

Andrew McPhee (02:46)

It probably still does. If I went back and repeated my course today, I’m sure there’s still no security modules built in. When you put an interesting use case behind security, suddenly it starts to matter.

Shane Fry (02:58)

Yeah, especially when that use case is saving the business money because you don’t have cyber outages in your manufacturing plant.

Andrew McPhee (03:05)

For sure.

Paul Ducklin (03:06)

And I guess with increasing regulation and liability rules, particularly thinking of the EU Cyber Resilience Act, that certainly focuses the mind from a financial point of view. But I guess we’re all starting to feel the true physical dangers of the world.

Andrew McPhee (03:25)

When it comes to OT, safety is a huge concern, and a lot of times we focus on manufacturing because there’s the safety aspects there, but you think of like electric utilities and all the concerns we have of losing our power. You think of water utilities. There’s so many industries that are at risk of being exploited, and they all follow the same frailties of not having security in there by design.

Paul Ducklin (03:40)

Yes.

Another significant problem is that, particularly in very distributed industrial control systems, let’s think of the electricity grid, a lot of this infrastructure has actually been there for ages. And it may have hardware devices that are fifteen, twenty, twenty-five, maybe even thirty years old. It’s very hard to find out where all of it is. So how do you find out what you’ve got in an industrial control system?

Before you think about how you’re going to secure it.

Andrew McPhee (04:21)

One of the mistakes we tend to make whenever we try to protect the OT is we try to build this wall around OT and say, we’re just going to control what comes in and out. What we need to start doing is really putting visibility deeper into the OT to truly understand well, what is actually down there, what is connected. And the only way to do that reliably is with automated visibility tools. As other examples would show Shodan being the search engine for all of the things connected to the internet, and there’s a dedicated set of tools for looking at SCADA devices. Do you know if your devices have public IP addresses or not? You might think that they don’t, but if you’re not the one in control with them, how do you know what the engineers who put it in place did?

Paul Ducklin (05:01)

For attackers setting out and finding some potential targets, they don’t even need to do their own research, do they? They just look it up online.

Andrew McPhee (05:09)

If you go to Shodan and search for a vendor called Unitronix, you will find devices out there that are still available on the public internet. And CISA has advisories for that vendor specifically because they were used in water and wastewater treatment facilities. And one of the deployment methods was basically putting them online for remote monitoring. So CISA put out this warning, basically urging people to stop putting these devices out on the public internet. And the last time I ran it, there were still over six hundred instances of Unitronics alone.

That’s the issue to your point. There’s no research involved. I wanna attack wastewater treatment. What’s the prominent vendors there? Let me see if I can find one. Trying to find this country, let me go there.

Shane Fry (05:48)

And the hard part is the utility is gonna say, Well, hey, you know, let’s use Texas as an example. We don’t want to drive three hours, and so we need some remote mechanism, but everybody’s afraid to put IT-type security solutions in front of that OT device. So hey, let’s just make it easily accessible.

Paul Ducklin (06:05)

I guess another significant problem there is that inside the OT network, the world isn’t all TCP and UDP and Ethernet packets. There’s a lot of historical baggage. So there are lots of different buses, lots of different protocols, lots of things that an IT person wouldn’t know how to scan for. How do you build that asset discovery inside a network where some of the parts may be using legacy protocols that there aren’t that many people left who remember?

Andrew McPhee (06:39)

There’s two ways to do visibility of these OT protocols. There’s the passive way, where we just kind of listen to what’s happening, and then there’s the active way, where we get involved and we actually query those devices. 

An active does get a bad rep because we typically fall on words like port scan or network scanners that are traditional IT tools. And the reason they get a bad rep is if you do it incorrectly, you will, or you may, bring down OT devices because those legacy devices, they don’t really handle a lot of that querying in that way very well. But active discovery for OT is a whole different set of principles. You use the OT protocols themselves in order to do the discovery. If you’re a Rockwell shop, you’ll use Ethernet IP discovery messages. If you’re a Siemens shop, you’ll use S7 or ProfiNet. Every vendor has their own communication stack and we can leverage that stack in order to discover those protocols.

Paul Ducklin (07:09)

Yes.

Andrew McPhee (07:31)

Investing in an active discovery tool, just make sure it’s dedicated OT. Don’t use IT tools for your OT processes. You don’t really get a whole lot of information about what a device is just by knowing what ports are open. Take Siemens as the example. They have an application called TIA Portal. You use TIA portal to program all of the logic on those devices. When you open TIA portal, it does a discovery message. It looks for the Siemens PLCs in the network and gives you that inventory within the tool.

Paul Ducklin (07:44)

Yeah.

Andrew McPhee (07:59)

The thing, though, that we lose out on active discovery only, is you get an inventory. You know what exists on your network, you know the properties of that device. You don’t necessarily know the behavior. You don’t know what it’s doing on your network. That’s where the passive discovery comes in. That’s where I listen to the traffic. And that’s one of the biggest challenges we have in OT, is how do you get deep enough to actually listen to what these OT devices are doing? Because if you’re doing passive discovery up at a core switch or somewhere too high in a manufacturing plant, you don’t really see anything. So the biggest challenge we have is how do you get visibility deeper in that number?

Shane Fry (08:35)

Some of these old PLCs may not even be on a network switch, serial cables and other connectors. Yes. That I think makes it even harder.

Paul Ducklin (08:45)

 HPIB if if anyone remembers that.

Andrew McPhee (08:50)

Security is one of those things that we’re just trying to manage and mitigate risk. Yes, there’s risk of devices being on a serial bus, but in order to exploit an asset that’s using serial or on some sort of bus network, because you have been on the bus network. The reason why OT networks are being exploited is because those same legacy protocols that were once running on serial and they’re just running over IP. So you take something like a modbus and there’s no authentication on it. If I can reach a device over a modbus, I can tell it what to do.

If I tell a PLC to shut down, it’ll shut down. If I tell you to crank the temperature up to point of explosion, you’re gonna explode. There’s no protections really in place unless you engineer the protections as part of the solution. While yes, we’ll struggle on things like our bus networks, making sure that we know every OT device that’s on an IP network and understanding the behavior of that on its IP network, I think that gives us a great starting point to make sure that we’re secure.

Paul Ducklin (09:44)

How do you go about patching those devices in the way that we’ve got used to being able to do for things like laptops and servers on conventional networks? For a lot of these, particularly legacy devices, even small changes can have a disastrous effect, particularly if they have some real-time component that requires them to respond reliably within a specific time. How do you deal with that in a world where we’ve come to expect devices to take patches readily, quickly, easily, with web apps, could be several times a day. How do you deal with that with a twenty-five-year-old OT device?

Andrew McPhee (10:24)

It’s a difficult problem for sure. But when we look at, you know, why we patch vulnerabilities is we’re is we’re s trying to again mitigate the risk that that thing could be exploited. So what we have to look at outside of patching is what are the other risk mitigation techniques that we can put in place. Right. Let’s say a PLC has a a vulnerability to its web server. Well, if you’re not using the web server, just disable the web server, then you can’t exploit that vulnerability in the first place.

So it’s not patched. The firmware version it’s running still has a web server exploit, but the web server is not running. The other way, then, is just making sure that an exploit can’t be accessed over the internet. Let’s say you have an exploit that have happens over a very specific port. Well, is that port being used? No. Well, can we then just disable anyone from communicating to this device over that network port?

And if it is used and it is used in a critical process, just make sure only the trusted assets can actually communicate on those communication channels. The risk is still there, but it’s massively reduced. What we see all too often in OT is because there’s no controls in place and there’s no segmentation, I just need to exploit one of your vulnerabilities and then I can access everything.

Shane Fry (11:35)

That’s something Andy Kling at Schneider Electric talks about in some of his public talks is about how these devices don’t have authentication on who’s sending the messages. You might get a legitimate message from an illegitimate source or somebody is in between where that message came from and they modified it. He’s been on a quest for I don’t know how many years now trying to get device manufacturers to authenticate messages coming over Modbus and it’s just not picking up traction.

On the one end, it’s understandable. These devices aren’t patched a lot. They don’t have a lot of extra compute cycles. At the same time, like you said, once you’re in, it’s really easy to just send what you want, even if the device on the other end isn’t expecting that from you.

Paul Ducklin (12:18)

Shane, I guess another problem with devices of that sort is that particularly if the software that they’re running was developed, say a decade or more ago, you can understand why the developers would have focused on safety. If a well-formed message is received, then it will be processed promptly and correctly. For programmers of limited power devices, their job is process the messages correctly.

Somebody else’s job is make sure that only legitimate messages get sent. But the world’s not quite that simple, is it?

Shane Fry (12:54)

Nope, not at all.

Paul Ducklin (12:56)

For the foreseeable future, what can you do to increase the resilience of those devices without adding so much extra threat detection software or threat prevention software to them that they stop performing correctly? How can you build self-protection into them without changing their behavior or their memory usage significantly?

Andrew McPhee (13:20)

By basically not putting the onus on the endpoints themselves, but making sure that the network that they’re connected to is protected. One of the things that I think everyone in industry advocates for is the IEC 5243 zones and conduits model, and taking that approach of rather than one large OT network, it’s a lot of mini OT networks that you link together with conduit devices, so switches, routers, firewalls, whatever it is that you’re using to network between them.

And what that does is just kind of limits the blast radius if an attack does happen. There was some rogue backdoor that you didn’t realize came in the one of those machines that isn’t really controlled by you, but the OEN that you bought it for, and they were to be exploited, well, at least it’s contained to that one machine. They didn’t have a path to jump through your entire network or if your IT was exploited, are you protected from them pivoting into your OT, and the network is the reason why we’re vulnerable in the first place. And I think sometimes we overlook our responsibility to make sure that that network is secure, not just functional.

Paul Ducklin (14:20)

So you’re alluding to the concept of segmentation. That sounds easy in theory, but in practice, even in IT networks where we’ve known how to do this for years, people often find it very difficult, not least because they start with good intentions and then find that all the doors they closed end up getting propped open again. How do you solve that problem in a much more variegated, perhaps much less well-documented OT network if we can’t even get it right in IT network?

Andrew McPhee (14:53)

I think that’s part of the problem. And that was one of the questions you asked me when we first started. You asked how I got into OT security. And the reason you asked that question is not everyone gets into the OT security space because it is very different than traditional IT security. So if you take a technology or a capability like segmentation, like we’re already struggling with in IT, and then just try to apply it.

Paul Ducklin (15:10)

Yes.

Andrew McPhee (15:18)

The same principles and the same way you’re trying to do it in IT and do it in OT, well, you’re going to fail even worse in OT because of how different it is. And one of the biggest mistakes that I see with segmentation of OT networks is not even a technology problem. It’s not like we don’t have the right tools to do it. It’s the way we approach segmentation in this industry. So when we all move to the zero trust model, where nothing can connect to my network until you verify the identity, that doesn’t work enough.

The reason we’ve done it in IT is because I’m now working from Ireland. I’m going to be in Cisco Live in Vegas in a few weeks. I might be somewhere else in Europe a couple of weeks later. It doesn’t matter where I am in the world, my identity and my privileges follow me on my identity. In an OT network, a production line doesn’t work from a coffee shop from one week and then come back to the factory the next week. Should the HMI in production line one control the PLCs in production line two? And the answer is always no.

So then why would we create segmentation based on device type? We can’t say all HMIs have one relationship and all PLCs have another.

Paul Ducklin (16:26)

So the HMI, that’s the human machine interface, that’s the device that has the buttons on it that somebody can press that actually say the valve should actuate, the welding machine should start or stop, the temperature should be increased or decreased. And somehow that has to get a message to the device that will respond to the button.

Andrew McPhee (16:46)

Exactly, yes. And it’s purposely put in that area because it controls the function of that area. You take an automotive, you have a paint shop, you have a body shop, an assembly workshop, you have all of these different shops, they might be the exact same devices in them. They might all be using Rockwell automation, or they might all be using, say, FANUC robot arms somewhere in their line. But it doesn’t mean that all of these device types have the same privileges. So when we talk about the zones and the conduits model, and we always go back to

Paul Ducklin (17:11)

Yes.

Andrew McPhee (17:16)

This is why it becomes such a relevant standard for us. A zone can be a physical zone. So when you think of segmentation, think of the physical zones that you want to create. Sometimes we just need to go backwards without thinking a bit for OT. OT still runs legacy, so how would we have done segmentation in a legacy world? We would have done segmentation based on the office you’re in or the location you’re in.

Paul Ducklin (17:37)

Exactly, yes. When you try and divide up your network so that not everything can automatically see everything else, which clearly sounds as though it ought to be both safer and more secure, much, much less to go wrong, how do you A) measure that you’ve done it well, so that you don’t think, well I put in some firewalls, that should be enough and B) avoid being so zealous that you actually break something in a way that’s hard to predict in advance.

Andrew McPhee (18:08)

Great question. Yeah. So when we talk about security, I always love bringing it back to the concept of risk, of managing risk. There’s three terms that I really like that I think grounds a lot of the security principles that we have. You have this concept of a risk assessment, and that’s where the visibility comes in near risk assessment is just let me understand the threats I currently have to my network and what would happen if those threats were to be exploited. And then you look at two very similar terms, but they’re slightly different, if you have risk tolerance, which is how much risk are you essentially willing to manage if something was to be exploited. What is your downtime criteria? Are you okay with 24 hours of downtime, or can you only deal with five minutes? How much financial loss are you willing to accept? What’s a major incident? What’s a minor incident? And the goal of cybersecurity, when you think of the North Star, when are we done? It’s when the risk assessment aligns with your risk tolerances. You’re basically saying I’m willing to accept that if this machine was compromised, I would have be back up and running or I’ve minimize my financial loss to the point that it’s okay that it got exploited. So the last term then is risk acceptance, which is I’m willing to accept the level of risk because either I’m within my tolerances or that the mitigation is not cost-effective. And risk acceptance is not a one-time thing, just like risk assessment is not a one-time thing. The level of risk I’m willing to accept today does not have to be the same as I’m willing to accept 12 months from now.

So when it comes to security, it’s really how much risk are you willing to accept? That’s going to then determine everything. What’s the size of your zone? Are you willing to let an entire production line go down in case of a breach? Or do you have to limit it to a cell? That is what’s going to determine how much effort you put into your segmentation project.

Paul Ducklin (19:54)

I guess you also have to be aware of the fact that there may be regulatory changes that come upon you that alter the risk acceptance algorithms, if you like, that you’re allowed to use. It might become impossible to accept certain levels of risk in two years’ time that may have been considered reasonable enough two years ago. Is that correct?

Andrew McPhee (20:17)

A lot of these standards give you your own kind of leeway into the size of the zone, or your own judgment into how you’re gonna go about security. What they’re looking for, though, is that you’re thinking about it, you document it, that these things are auditable. I don’t foresee a regulatory body coming down and saying a zone has to be X size or this part of your network. I think they just want people to be doing something rather than what they’re doing today, which is very, very little.

Shane Fry (20:46)

Andrew, if I think about risk tolerance and risk acceptance. One of the things that I’ve heard a lot from OT manufacturing and other facilities is I don’t know what all I have. There was a really great talk at S4 in Miami back in twenty-four where Southern Company went to a small substation and said, Hey, we don’t even know what devices we have in our substation. So how do you think about risk tolerance and acceptance when there’s unknown amount of risk, you may not be able to see it or know what it is?

Andrew McPhee (21:20)

That’s why I said that risk acceptance isn’t a static thing, and how it will change over time. So if you take a substation, you’re accepting the risk that if that one substation goes down, I probably have to assume the entire substation is compromised. That well, I’m not able to contain it into smaller zones. But how can I make sure that other substations, other parts of the grid, are not affected by it? I’m willing to accept that I’m gonna treat that substation as one single entity. It’s either been compromised or not compromised.

And that’s the reason we also see NERC SIP bringing in new regulations around  the visibility within a substation. SIP fifteen internal network security monitoring, INSM is the acronym that they use, where they’re now mandating that you have visibility within that perimeter to understand if something is going wrong before it hits your actual perimeter device. We need to take the same logic in other verticals. 

The worst scenario I’ve ever heard is after a Cisco Live presentation I did on segmentation, a customer came up to me and said that if they walked into any OT facility anywhere in the world, that they could reach any other OT facility anywhere in the world. They weren’t even segmenting based on Geo, never mind IT OT boundaries. So everyone’s on a different stage of their journey. You have to accept whatever risk you have now today and create a project that allows you to reduce your risk acceptance over time. Because if you take on too much from the start, we’ll have the same conversation four years later and you won’t have done it.

Paul Ducklin (22:50)

In the UK where I live, the biggest industrial control incident in history was, of course, the Jaguar Land Rover outage. Their OT systems, their entire production line, as I understand it, essentially froze. And they couldn’t restart it because they no longer had a record of which vehicles on the production line were at what state of production. And that didn’t happen because of an OT vulnerability or an OT specific attack. It was actually a side effect of ransomware on the IT network. There was no part of the IT network that was invulnerable enough to a ransomware attack to allow the OT network to continue functioning at all.

Andrew McPhee (23:34)

You look at a modern OT network now, it’s it’s hard to distinguish well what exactly is OT because we tend to think of OT as the PLCs and the control systems that we have on the production floor, but we’re modernizing so much of our networks that we’re relying more and more on virtual space and even SAS. Virtual PLCs is now a thing where you can actually just run software to replace what that PLC was once doing for you.

Paul Ducklin (23:58)

So that’s like a Game Boy emulator for a PLC.

Andrew McPhee (24:02)

You’re just giving yourself the capability of being able to use modern software stacks in order to be able to control the inputs and outputs within a factory floor. We had a customer that had fifty thousand industrial PCs, and it’s the same customers also looking at virtual PLCs, but just the PCs alone. They had fifty thousand of them across their the manufacturing footprint. And they spent millions of euros with the European customer to upgrade from Windows seven to Windows ten.

And then six months later they announced Windows eleven and they said, well, we’re not doing that again.

And we underestimate just how much things outside of the control devices actually contribute to the OT itself. There was another example I have where the printers would have actually been a bottleneck for this OT. Because at the end of the production line, after they went and shrink-wrapped the pallet, they had to print the shipping label on where that pallet needed to go. And if those shipping labels don’t get printed, that pallet stays at the end of the production line because they don’t know where to put it.

And they only have about five or six pallets worth of buffer room before they’d have to shut down a production line. So if you compromise the printer, that OT network now shuts down. And there’s a lot of overlook things that we have within our processes. And this customer specifically, when I was talking to them about it, they realized, well, their printers are on their IT network. So if they had to cut off their IT from their OT, well, that means no printers. That means production line shuts down.

It comes back to visibility again, but visibility isn’t just what’s on my production line. It’s what is the behaviors, what’s the relationships, and end-to-end processes look like.

Paul Ducklin (25:42)

Yes, that word relationships, I think, is very, very important there, isn’t it? My production line’s working fine, but I actually can’t move this stuff that’s blocking up the exit to my factory because I can’t print a single jolly label. Yes. It seems crazy that you could fall into that, but it’s easy why you would imagine that these are two separate problems if you haven’t mapped that relationship out.

Shane Fry (26:05)

Sure.

And I think, Andrew, that’s a really interesting point too, blurring the lines of IT and OT. Most people think OT is just the PLCs, the HMIs, the servos, the motors, you know, the robotic arms, but can’t just clean cut all the time between IT and OT. Even if you’re looking at OT security, you’ve got to have some thought for what are my IT assets that are inside that OT perimeter too.

Andrew McPhee (26:30)

Yeah, unfortunately, it’s no one size fits all when it comes to how we deal with this. You need the visibility to get started, and once you figure out what your unique challenges are, then you have to find a way to mitigate those challenges.

Paul Ducklin (26:41)

Andrew and Shane, I’m conscious of time, so perhaps to finish up, I’ll give either or both of you a chance to say something to people in organizations that have OT networks that maybe are thinking, you know, I really need to take this whole cybersecurity thing more seriously. Where would you advise someone to start? What are the first steps if you haven’t taken any already?

Andrew McPhee (27:08)

Visibility for me is always the first step. Yes, there’s always things you can do without visibility, of course. You can create bigger barriers and bigger points of control. Sooner or later, that’s gonna bite you. Really get in and understand your OT processes with tools that help you do that, but also build that relationship with the OT team, figure out what the dependencies are, and help OT with their day-to-day job, because they will put things in like remote access tools without letting you know unless you do it for them securely. You just need to get involved with the OT teams, make sure that you are there working with them, helping them.

Shane Fry (27:44)

I think Andrew covered that super well. Visibility is key. I think I’ll maybe take it a step further of I think you need a physical asset inventory, but also a digital software asset inventory too, right? A lot of OT/ICS devices have a much longer support tail. It’s not like your mobile phone that maybe stops getting updates after two, three years. And EU CRA is even pushing some of that support lifecycle even further. Knowing what you physically have is great, being able to box that in and create those zones and segments properly is good, but also getting a good digital inventory so that you can plan out patches. When can we turn this off to actually do an update? And if you can’t, that gets back to Andrew’s points about accepting risk, and maybe you put some extra security controls around that to try and detect issues with those devices that you can’t patch just yet.

Paul Ducklin (28:32)

To finish up, it sounds as though that old maxim in the IT world that if you can’t measure it, you can’t manage it is extended in OT by the fact that if you don’t know it’s there in the first place, then you can’t measure it, and so you certainly can’t manage it. So gentlemen, thank you so much for that. Thanks for your thoughts on how people can actually make a good start to making sure that they’re not just in OT, but they are in cybersecurity as well. 

So that is a wrap for this episode of Exploited: The Cyber Truth. If you enjoy this podcast and find it useful, please like and share us on social media. So thanks to everybody who tuned in and listened. And remember, stay ahead of the threat. See you next time.