Exploited: The Cyber Truth,  a podcast by RunSafe Security. 

Paul Ducklin (00:05)

Welcome back everybody to Exploited the Cyber Truth. I am Paul Ducklin, joined as usual by Joe Saunders, CEO and founder of RunSafe Security. Hello Joe.

Joe Saunders (00:19)

Greetings, Paul. Great to be here. Look forward very much to this conversation today with DJ.

Paul Ducklin (00:24)

Yes. The guest we have today is Daniel Jablansky, who goes by DJ. Daniel, I’ll leave you to introduce yourself. You have quite a complicated job title and you have an interesting background that I think our listeners would like to know more about. Now it’s quite a mouthful of a topic today, but it is very, very important. The cyber physical truth, what we get wrong about attacks on critical infrastructure.

So DJ, why don’t you start by introducing yourself and your background and how you came to be so passionate about cybersecurity?

Danielle Jablanski (01:01)

Sure, I’ll try to keep it pretty brief. I work at an engineering firm. A lot of people call them EPC firms, engineering procurement and construction. We’re kind of EPA, engineering procurement and architecture. We do a lot of the design work. We actually partner with a lot of the builders on the construction side, just depending on which sector we’re working in. We focus primarily on rail. We have a buildings practice as well as water. We have our own integration service. And then we have standalone services in physical and cybersecurity, which is where I sit. Before I came to STV to kickstart this program, I was the OT strategist in the office of the technical director at CISA, the Cybersecurity Infrastructure Security Agency, under DHS, Department of Homeland Security. Before that, I was at a solution provider that does continuous monitoring in the OT space. And before that, I was actually an industry analyst. And my expertise was intrusion detection systems for industrial control systems, which has evolved into the continuous monitoring ecosystem today. Before that, I was in academia. I looked at cyber policy a lot.

And even before that I was a policy analyst looking at nuclear weapons technology and use and really focusing on the cybersecurity impacts from upgrading and modernizing nuclear weapon systems.

Paul Ducklin (02:03)

So DJ, how does risk differ between IT, where we’re concerned about dear, I couldn’t get into a Zoom meeting today and OT or Industrial Systems, where you might be a little bit more concerned about will my welding robot run a mut?

Danielle Jablanski (02:21)

When we talk about risk, it’s really interesting how this breaks down in cyber physical because there’s this age-old debate between assets and functions. And this is something CISA tried years ago to discern. Critical functions are things that we rely on for the economy, for national security, the delivery of safe water, those kinds of things. Assets, of course, are the industrial control systems, the PLCs, the DCS systems, the motor that’s running the baggage claim, a piece of equipment that’s operating based on computer inputs that has a physical action in the real world.

The reason that cyber physical is different, of course, from IT is two things. The assets themselves are built differently, and then their use in the world is different than computers, software, and email. So there is a physical risk to safety.

Paul Ducklin (03:03)

Yes, if you think about a train hammering down a railway track with fifty goods wagons behind it, that’s a little bit different from, dear, I had a five-minute outage during which I couldn’t make phone calls.

Danielle Jablanski (03:19)

Yeah, and trains are actually modernizing a little slower. Which part of a sensor actually knows where a train is is actually evolving now. It’s not part of the industry 4.0 conversations we have in sectors like manufacturing. Different sectors are actually on a different journey. Of course, cyber physical has risks to safety, the environment. There’s this potential for loss of life, especially in the oil and gas industry and manufacturing settings and chemicals. We have another set of risks in productivity, efficiency, and just in time manufacturing. And then you also have intersector transactions.

A huge amount of electricity goes to water. So water requires electricity. A huge amount of water goes to healthcare. Healthcare requires water, clean water. And so you have that intra-sector transactions that is not as complex when we just think about IT. So again, back to the functions and assets. It’s difficult to fully protect and prioritize functions and assets everywhere all the time, given those complexities, I would say, from a sector perspective, but also from that iceberg model. Everyone who might touch, deploy, access.

And ultimately be responsible for the maintenance of these systems at an asset level.

Paul Ducklin (04:19)

I guess if you think about an iceberg vertically, given how far they go down below the surface, the bit that you can’t see, that raises an enormous issue in cyber physical and OT systems for supply chain risks, doesn’t it? So I guess we’ll come back to the supply chain later, but it’s very much longer and more involved and much more variegated than it has really become in IT. Would you agree with that?

Danielle Jablanski (04:49)

I wouldn’t say it’s more variated. I just think it’s a little bit more complex. You actually have this marriage of communications data points and process data points or process variables. And those process variables actually dictate the logic in the process for whatever system. And what sensor we’re talking about matters. So there’s process anomalies like set points within the logic that can actually be implemented or implanted depending on who’s doing it, and a few different areas. So you have control logic.

Which is historically based on relay logic, which we’ve borrowed as a deterministic way to program control systems from electrical relays built on electromagnetic fields. These types of systems are what underpins the control systems that we talk about as PLCs and these different types of assets. Typically, when we talk about communications risks to these assets, we just talk about access, lack of encryption, passwords. We don’t actually get to the engineered portion of these systems, which is that control logic, again, built on relay logic.

There is the potential to have a tailored  attack that actually targets the programming function and variables that are defined within these systems. And so you can change the setting as a command to the system. You can actually change the parameters of the setting, and then you can otherwise manipulate the environment to evade detection. So say you were able to change those parameters in a way that made something that historically was not an acceptable set point acceptable because you’ve changed the parameters within the logic.

You can then evade detection by also targeting something like a safety instrumented system.

Paul Ducklin (06:15)

So would an example of that be say that if you’re an attacker, you might override some pressure limit in a boiler somewhere? Yes. But also manipulate the sensor so that it continues to say, yes, it’s one point four megapascals when in fact it’s now three point six or something dramatic.

Danielle Jablanski (06:35)

Right. So you could do either. So the Stuxnet example, the actual set points were changing so that the centrifuges were spinning at the speed that they weren’t supposed to spin at. But you also have the trisis example where the safety instrumented system was targeted, it tripped multiple times, and so they were able to figure out that something was going wrong. But in that scenario, you wouldn’t even detect some type of physical environmental temperature setting, something like that that is unsafe. And so there’s this other ability to slightly alter things over time without being discovered, which could impact manufacturing.

Paul Ducklin (07:04)

I guess there’s also the issue that these specialized devices, many of them may be five, ten, fifteen, twenty years old, even older, and they may have evolved and have been introduced from different vendors at different times for different purposes. Yes. There’s not necessarily the integration you might expect between, say, the pressure regulator and the pressure sensor.

Danielle Jablanski (07:30)

The interoperability between the systems themselves in a deterministic nature is actually pretty well designed and defined in terms of like the standards we have, OPC UA is a good example. But some control logic is actually throwing out data for a historical trend analysis in milliseconds. So you have tons of data points, but if you’re not actually analyzing that data in a comprehensive way, you’re not gonna see changes that would alter things if they’re not being reviewed.

So you have essentially these safe thresholds for your set points that dictate the command and control of the control logic itself, but then you have monitoring aspects on top of that. You want to know if a new device is on the network communicating, you want to know the average device to device communications, you wanna know if there’s large bandwidth spikes, things like that. But you also wanna know if there are process anomalies detected within those variables. And of course, in these heterogeneous environments, every piece of equipment actually comes with its own proprietary software as well.

And so there’s a number of touch points, but it does get very convoluted. We say that there’s an average of twelve vendors per environment. That’s hard to make specific because we’re talking about different different sectors, of course.

Paul Ducklin (08:34)

When you say environment, would an environment be a pump room or a whole factory?

Danielle Jablanski (08:39)

Yeah, a whole factory, a plant, electric utility potentially. And so when we look at like supply chain type incidents, we’re thinking what is the potential solar winds type impact we might see from a supply chain software position?

Paul Ducklin (08:53)

I’m trying not to smile and laugh, because that is not funny, but it should bring a sort of I guess a weary smile to everybody’s lips of just how dramatic the effect was.

Danielle Jablanski (09:06)

So I don’t want to call it security by obscurity because that’s not quite the right label, but having these multiple vendor systems in an environment does mean that a vulnerability or even a zero day in one of those vendors should not take down your entire environment because that vendor is only one component. There’s a debate actually within the industry on whether or not we should reduce the number of vendors or make these environments more homogeneous. I actually don’t think that that’s a great idea.

I think that consolidation and interoperability across industrial environments increases interdependence and creates more single points of failure.

Paul Ducklin (09:38)

You’re against the hey, let’s have all our eggs in one basket, you know.

Danielle Jablanski (09:43)

Exactly. I mean, if you think about cloud providers, there are three major cloud providers. So if you have mission critical functions that rely on one of the three cloud providers and that cloud service is no longer available and you have a twenty four seven operation that requires uptime, ac requires availability, that’s an obvious single point of failure. So why would we want to create more of those?

Paul Ducklin (10:02)

I am an Outlook user for my sins. And as we’re recording this, it’s a day after an outage that in my life was, I must admit, essentially insignificant. But the problem was that Microsoft kept saying, We think we found the problem, we think we’ve fixed it, we haven’t got it right,  well let’s start again. And you think that’s just fixing their email system.

Danielle Jablanski (10:04)

Yeah. CrowdStrike is a great example. So it wasn’t a targeted incident, but I was at CISA at the time and I couldn’t access my computer. And so that outage impacted government affairs.

Paul Ducklin (10:33)

That caused a boot loop, didn’t it?  The crash happened so soon in a kernel driver that had not been tested in the real world at all.  The weird thing there was that one of the ways to fix it apparently was just to wait and wait and wait and eventually you might get lucky and you might get the update before it crashed again. Which is not a great way to recover something like a railroad network or a factory or a hospital.

Danielle Jablanski (11:02)

Yes.

Paul Ducklin (11:03)

The next twenty operations might turn out really badly, but then the surgery will be restored to normal. That’s just not acceptable.

Danielle Jablanski (11:10)

You bring up something else that’s important to mention. So we talked about safety systems that are supposed to alert on unsafe parameters or environmental factors. Yes. There are also fail-safes, right? fail-safe procedures and processes that are also programmed into these. A very responsible asset owner will always have some type of sandbox or testing environment before they push anything. I was alluding to the third party reliance. And you see this across telcos and other kind of third party providers, especially in the MSSP space. The other risk is centralization risk, and we talk about that either with one type of product or with an entire sector relying on the same third party provider. So those are two things that have not been widely analyzed, I don’t think, across all of these different OT sectors. Of course, we have sixteen critical infrastructure sectors in the US. Years ago when I was a market analyst, I actually did some numbers with a building stock database. I was able to estimate that within four sectors energy, water, critical manufacturing, and food warehouses, so not farming, not agriculture, eight million facilities worldwide.

Paul Ducklin (12:07)

And that’s from a quarter of the sectors.

Danielle Jablanski (12:12)

Exactly. And it’s a subset as well. So critical manufacturing was estimated to be a certain percentage of overall manufacturing, but the centralization risk given that the attack surface is very, very difficult to study and analyze. And then when you get back to the OEM kind of version of this, understanding where in the world all of these OEM systems from each of the vendors has been deployed, you can probably count that pretty well based on, you know, what they’ve sold and some of that historical data.

But because the chain of custody is so broken, which is not the OEM’s fault, they sell it and people buy it and use it. And so in that middle ground of being bought, used, potentially resold, different updates, different integrators, different configurations, different security applied, understanding the provenance issue is something that I’ve been really passionate about in my own research because it really muddies the water. But I think the centralization risk is unpredictable. That goes back to my heterogeneous versus homogeneous strategy approach. I don’t think we should consolidate for the sake of consolidation.

I think it should be strategic and we should understand our interdependencies before we create accidental single points of failure.

Joe Saunders (13:12)

I’m struck just by all the layers that we’re talking about that DJ you’ve pointed out and that you even can extend into the software supply chain. As we say, there’s the OEM, then there’s their own developers, there’s third-party developers, the open source software. And if you think about 16 sectors, you think about just in those four that you mentioned, there’s eight million plants, you think about the layers within their OT infrastructure itself, and then you think about all the layers in the software supply chain, and you keep using a word that is.

Obviously very, very important the determinism in these systems so that they operate reliably as expected. It’s quite compelling, I think, to think about the software problem these organizations have. I agree with you, it shouldn’t be a homogenous environment. Imagine with all eight million manufacturing plants being identical. That’d be a scary, scary thought. But at the same time, when you have a very heterogeneous environment, then what you have is a distribution problem with software.

Paul Ducklin (14:02)

Yes.

Joe Saunders (14:10)

And updates getting applied to all these devices that do then stand the test of time far beyond what a normal cloud workload would. These gadgets, these devices, these cyber physical items, they are out there five, ten, fifteen, twenty, thirty years maybe in some cases. As you said up front, DJ, the consequence if something goes wrong is quite high. It could be an economic loss, but it certainly could be life and death as you suggested. I’m just struck by all the layers in all the directions and thinking about what the software update challenge is given a heterogeneous environment.

Danielle Jablanski (14:45)

That’s what kind of made me raise that assets versus functions. We can all kind of figure out deterministically which functions are so important, but then we can’t just go patch all the assets that those functions rely on. So it’s not that easy. In my own work and especially when I was doing strategy work at Sissa, I really focused on network security because network security is something that the asset owner can fully own, but also there’s so many partnerships involved.

My focus was always on legacy systems and effect space analysis. And so effect space analysis was always looking at an environment or a context and saying, What is the worst effect that could ever take place here? What is the worst case thing somebody can do? Increase the chemical levels in water, make something blow up in oil and gas. Many processes and tools are involved in network segmentation. I for years I’ve said people act like segmentation is like flipping on a switch. It’s not. But I do know that there are good indicators out there that show proper segmentation does actually make adversaries give up sooner. Here at STV, I’ve really pushed a lot of our clients to not jump to purchasing capabilities, but really focusing on building competence and capacity before deploying really expensive tools. Build your own baseline and invest intelligently in competence, capacity, and capabilities to verify that your controls are working for you and not that you’re just gonna build a a list of controls on paper.

Paul Ducklin (15:59)

DJ, we’ve alluded to segmentation, which is where you keep things that are supposed to be separate. But it seems that in the industrial control world, we’re also running headlong towards what you might call de-segmentation, with this thrust to go, hey, let’s just connect the IT and the OT systems together so that I can sit remotely on my laptop and I can have a web app and I can go and look at what the human interface devices in some far-flung pump room look like. Maybe even I can press a button in my browser that actually causes the effect that would have required someone to go into the pump room and press a physical button in the past.

Danielle Jablanski (16:42)

That’s my point on network focus versus assets. Yes. Your point on remote access is just one part of the picture. So there are all sorts of good reasons for data to flow in a network. Not just remote access and limiting the need to go places to do things to touch systems. For manufacturing, I have pulled up here the top five forecasting considerations. I think it was Toyota and Europe was able to actually increase their efficiency looking at tons of data points around how much paint they were wasting in their paint shop. We don’t talk about forecasting enough. So a lot of this these business use cases, just pulling from manufacturing, you’ve got economic forecasts, sales trends, supplier forecasts, seasonal changes and weather, and then business constraints. That’s just for manufacturing. When you look at controlling the grid, interoperability of the grid, you have all these other complexities of the way we need data. And so I think we’re looking at security from a network way.

But we’re not actually applying enough discretion around frivolous data and user access. Because we think of that access as being remote control or having one use case. Back to the capacity portion of my program, I’ve seen really well-resourced organizations deploy hundreds of thousands of dollars in security tools in their OT networks, and then their firewall rules are any, any.

Joe Saunders (17:54)

Ha ha ha.

Paul Ducklin (17:55)

People tend to like the idea that they can buy one tool or one new product and install it and it will magically fix the problem. Right. That was probably the reason they bought the firewall in the first place. And it probably started quite strictly at lockdown and then it just evolved into being weaker and weaker to allow more and more functionality.

Danielle Jablanski (18:17)

That’s human nature, right? We want silver bullets and we know they don’t exist, but we want to get as close as possible. The other thing I’ve seen is again, human nature, we have very static policies and procedures. They have to be dynamic. So we have to evolve with trends, both offensive and defensive. And when I say something is static, that’s not necessarily a dig at someone. These security teams are like overwhelmed. They’re very busy. They’ve got all of these different priorities. Security is a function of a business typically, so there’s all of these different factors at play.

The other issue is that nobody knows exactly what good enough looks like. There’s a lot of places that tell you where to start, but there’s nobody out there that says this is what good enough looks like and you will never be impacted again. And so that need for dynamic policies, procedures, reviews, updates, I think that that’s really overlooked from a program management standpoint.

Paul Ducklin (19:05)

Maybe this is a good time to talk about something that I know Joe and I are very passionate about, culturally, socially and technically, and that is the notion of secure by design. How can that help? And who should be responsible and how in a typical cyber physical network?

Danielle Jablanski (19:25)

There’s actually a document called Secure by Demand for OT. I’m not sure if you’ve seen it, but it’s priority considerations for vendors that’s been published for I think two years now, that is the secure by design equivalent for OT systems. Things like memory safe languages.

Paul Ducklin (19:38)

That’s the market end of it, isn’t it? Yes. Where the purchaser says, You know what? I’m not going to buy your system because I just don’t think it’s safe or secure enough.

Danielle Jablanski (19:47)

Or when you go to issue an RFP or bring in an integrator to select solutions, you can say, I want to cross-reference this. And there are some sector-specific examples of this as well. MTA in New York, the Transit Authority, actually has a consortium that they run with tons of disparate transit agencies across the country. And they’ve actually written a rail-specific procurement guidance because they’re spending tons of money on these systems. So even though it’s not a regulatory requirement to purchase X, Y, and Z based on the TSA directives, they can say if enough of us dictate that this is a requirement, then it’s a requirement. There’s power in numbers. We want this as part of our products.

Paul Ducklin (20:23)

Now Joe, you have some stats from surveys that RunSafe did in the past twelve months or so from healthcare and from automotive asking people how has your own attitude to cybersecurity influenced your purchasing? Do you want to say something about what you measured and what people said?

Joe Saunders (20:42)

This was a survey in the healthcare industry. Part of the finding was that I think it was forty six percent of hospital systems asking for and perhaps even making a no decision if there wasn’t sufficient security in place. And so the secure by design nature and being able to ask for that is becoming an important thing. Now, with that said, I’m gonna tease it out because then the new studies came out.

The early results are that that number has increased even significantly in the past year. It’s gone up. And I’m intrigued by the example that DJ offers in the transit consortium where people are developing a collective voice to advocate for security. I think security by demand enables a level playing field of how to talk about some of these issues, but having a common framework that is documented that people can talk about and get their arms around all the different aspects of security that could be demanded is very, very powerful coming from CISA out to all the organizations across the country. So I think it all works together in that sense.

Danielle Jablanski (21:47)

In the pseudo-construction world, sometimes we’re hired just to help organizations build the RFP. And so I can come in and say, hey, you really want to build cybersecurity into this RFP. Here’s why. If you’re doing it up front, you’re probably reducing your costs down the road. The more negative, I think, trend is that because there’s a lack of sector specific regulation at the federal level and not likely to be more in the near future, we’re seeing states now produce their own industry-specific cybersecurity requirements. So New York has it for water and wastewater. New Jersey has water and wastewater. Texas has some different ones for military and political infrastructure. Texas just launched its own cyber command. I know Nevada is doing some really interesting things around potentially mandating FedRAMP approved software. We’re gonna see these states have all these different specific and regional requirements for procurement, for design, for build, for operation, and regulations.

Joe Saunders (22:38)

You’ve brought up rail a couple of times, and I know you and your firm do work in the rail industry. And it’s another example if you think about all the different jurisdictions in which there is state run rail or rail infrastructure. Every jurisdiction has its own set of requirements in terms of how the car communicates with the network and what are the security requirements and expectations for delivery. And interestingly enough, in the US, the strategic rail network is vital for even the US military.

I think at least two thirds of that is used by the US government to transport munitions and other things to different bases around the country.

Danielle Jablanski (23:16)

And there’s different requirements for freight versus public transit as well.

Joe Saunders (23:20)

A hundred percent. And I do think when there’s large stakeholders, perhaps the US military in that sense, there can be a leader who can step in and try to drive those things, but there’s also a need then to organize and harmonize some of those standards across jurisdictions.

Danielle Jablanski (23:36)

The difficult thing though when we’re looking at standards, again, I know we’re on a tangent here, but I think it’s an important one is when we go in as a design firm and we’re saying, Hey, we’re gonna upgrade these DCS systems in this water utility and they’re super old, where does the upgrade end and the connection with the rest of the infrastructure start?

Joe Saunders (23:52)

I’m really curious about  your thoughts on this. And that is within the aviation industry, the FAA made cybersecurity obviously  a component of airworthiness. And so you think about the whole ecosystem to deliver a plane, and the manufacturer says, I’ve delivered you an airworthy plane from a cyber perspective. And what then, if a vulnerability does occur once that airliner has taken over that component, there has to be still ongoing relationship between supplier or manufacturer and the airliner in that case. Yeah. But it’s an interesting fact that they elevated cybersecurity as an airworthiness element. And in some ways that gives a North Star for everyone to point to in terms of what the expectations are.

Danielle Jablanski (24:36)

We used to say when I was in consulting the first time around that there was like a sixty forty when it came to unregulated markets adopting new safety or any other kinds of options. And so sixty percent would wait around and see what the other forty percent did in terms of North Star leadership. And then I would say in cyber, I would actually maybe it’s eighty twenty or ninety ten. If it’s not mandated, it’s not enforced, then everyone’s gonna kinda wait around and see if it become a market point of competition that I need to invest in or not. And then won’t invest in it until they see how other people sink or swim in that.

Paul Ducklin (25:05)

We started out talking about secure by design, and it became quite clear that secure by demand, where people just say I’m not going to buy the stuff that is insecure, is at least as important a driver. To finish up, Joe and DJ, if you could advise our listeners on one change they could make right now that would help them do better both to provide and to prefer to acquire devices and systems that are secure, what would that advice be?

Danielle Jablanski (25:40)

The procurement guidance and purchasing all of those decisions have to follow a risk analysis. Risk profile and perspective changes all the time. And that has to be what leads us into our procurement decisions. For me, when I do risk assessment, like I said, it’s always effects-based. So my key takeaway is understanding interoperability, third-party risk, and the security activities and tooling that you have that are going to maximize your defense in depth. How do I build these constants that will actually enhance my security controls in a way that I have evidence, I am reducing third party risk, I’m reducing single points of failure, and I understand my interdependencies in a way that will prevent the worst case scenario in my environment.

Paul Ducklin (26:19)

DJ, could I summarize that, do you think, realistically by saying very, very loosely and briefly, make sure you do the right thing for the right reasons. Don’t try and do everything in the hope that you pick one that works.

Danielle Jablanski (26:34)

Absolutely. Don’t boil the ocean and then Joe, I’m sure you’ve got something brilliant to say.

Joe Saunders (26:39)

Yeah, on demand, I have something brilliant to say, which is listen to DJ. I think she’s very, very smart. And this has been an excellent episode. Joking aside, although that’s true, I do think that defense in depth is key and finding ways to work with your vendors, with your suppliers. Take it as a holistic view. So if you are doing effects analysis, then share that and use it to help justify why you’re asking for things.

And don’t just demand them, even though everyone should adopt secure by demand.

Paul Ducklin (27:12)

And Joe, would you agree that one of the key things that we all need to do is to try and build a world in which we get ourselves off this basically patch everything patch often marathon treadmill.

Joe Saunders (27:27)

Yeah, I think it’s impossible to keep up with the patching requirements. And certainly in this distributed infrastructure, as we highlighted in this podcast, it’s almost impossible to ensure that patches will be there. So we do need to think about how to add in security so that you’re not relying on chasing vulnerabilities and patches, but you’re staying ahead of the threat.

Paul Ducklin (27:47)

Joe and DJ, thank you so much. I wish we didn’t have an end time that we had to hit. It feels like we’ve just got going. So maybe there’s an opportunity to dig into some of these issues in future episodes. Thanks so much for your passion and your intelligence and your excellent advice. That is a wrap for this episode of Exploited: The Cyber Truth. If you like this podcast and find it useful.

Please don’t forget to subscribe so you know when each new episode drops. Please like and share us on social media. If you listen to a podcast feed, please write us a nice review. That really helps us get the message out. I’m sure you can see from this episode that this podcast series is not a sales spiel, it is a community engagement podcast. So thanks to everybody who tuned in and listened. And don’t forget, stay ahead of the threat.

See you next time.