AI Wrote the Code—Who Owns the Risk?

March 12, 2026
 
 

In the Season 2 premiere of Exploited: The Cyber Truth, host Paul Ducklin is joined by RunSafe Security CEO Joseph M. Saunders  and embedded systems expert Jacob Beningo to examine how AI is changing embedded development and the new risks it introduces.

While AI can accelerate engineering productivity, it can also create hidden vulnerabilities, increase system complexity, and produce code that looks polished but hasn’t been fully validated. As discussed in the episode, AI-generated output should be treated as untrusted code that requires the same rigorous review, testing, and validation as human-written software. 

In this episode, they explore:

  • Why AI-generated code can introduce hidden vulnerabilities and complexity
  • The accountability challenge: who owns the risk when AI writes the code?
  • How AI output should be treated as untrusted code by default
  • Why rigorous testing, validation, and security reviews still matter
  • Practical ways engineering teams can use AI responsibly in embedded development

For engineers, security leaders, and product teams adopting AI-assisted development, this episode provides practical guidance on how to move faster with AI without compromising system trust or security.

Speakers: 

Paul Ducklin: Paul Ducklin is a computer scientist who has been in cybersecurity since the early days of computer viruses, always at the pointy end, variously working as a specialist programmer, malware reverse-engineer, threat researcher, public speaker, and community educator.

His special skill is explaining even the most complex technical matters in plain English, blasting through the smoke-and-mirror hype that often surrounds cybersecurity topics, and  helping all of us to raise the bar collectively against cyberattackers.

LinkedIn 


Joseph M. Saunders:
Joe Saunders is the founder and CEO of RunSafe Security, a pioneer in cyberhardening technology for embedded systems and industrial control systems, currently leading a team of former U.S. government cybersecurity specialists with deep knowledge of how attackers operate. With 25 years of experience in national security and cybersecurity, Joe aims to transform the field by challenging outdated assumptions and disrupting hacker economics. He has built and scaled technology for both private and public sector security needs. Joe has advised and supported multiple security companies, including Kaprica Security, Sovereign Intelligence, Distil Networks, and Analyze Corp. He founded Children’s Voice International, a non-profit aiding displaced, abandoned, and trafficked children.

LinkedIn

Guest Speaker –  Jacob Beningo: Embedded Systems Expert

Jacob Beningo is an embedded software consultant, founder of Beningo Embedded Group, creator of the Embedded Software Academy, and co-founder of the Embedded Online Conference. He helps developers and teams maintain and modernize their embedded software skills, architectures, and processes so they can deliver better firmware faster. 

LinkedIn

 

Watch the Full Episode

Episode Transcript

Exploited: The Cyber Truth,  a podcast by RunSafe Security. 

Paul Ducklin (00:06)

Welcome back everybody to Exploited the Cyber Truth. I am Paul Ducklin joined as usual by Joe Saunders, CEO and founder of RunSafe Security. Hello, Joe.

Joe Saunders (00:19)

Greetings, Paul. Great to be here.

Paul Ducklin (00:21)

And our special guest for this episode is Jacob Beningo, CEO of Beningo Embedded.

Jacob Beningo (00:29)

Yeah, thanks for the invitation.

Paul Ducklin (00:31)

Well, we have a very exciting and important topic to talk about, which is if you like one of the elephants in the room in all sorts of software development, but particularly in embedded security. our title for this episode is AI wrote the code, but who bears the risk?

As intriguing as that question is, Jacob, maybe because of your background in embedded programming and the fact that you’re deeply involved with the Zephyr RTOS real-time operating system. Maybe we can open up by you just explaining to our listeners the difference between the real-time in real-time operating system and what a lot of people think of as real-time, which just means, Hey, I had a team’s call and there was no lag. It’s a very different sort of real in real time operating system, isn’t it?

Jacob Beningo (01:29)

It definitely is. Yeah. When we think about real time embedded systems, this is where you’re thinking about bounded response times. Yes. So you want something that when you press your brake pedal in your car, for example.

Paul Ducklin (01:41)

It doesn’t stutter and go, hold on, buffering! 

Jacob Beningo (01:45)

Exactly. You want to respond within, let’s say, 10 milliseconds of a press, right? And so the real time there is there’s bounded responses. It’s deterministic. ⁓ There’s guarantees about how quickly it will respond. In the old days on a vehicle, you’d press your brake pedal, right? And it was connected to mechanical pieces. Today, there’s software in between. So there’s no physical connection between, for example, your brake and the actual brake pads on your vehicle. When we think about real time, this is where we have systems that they could be safety critical systems. They could just be normal devices, but they have bounded response times, have bounded periods of times where things have to happen. Otherwise you find either a degradation in the way that the system behaves. Like if you were to push a touch screen and nothing happens or you get a delay, let’s say for two seconds. And then suddenly, you you move the next menu, you get a decrease in the value of the actual product.

Paul Ducklin (02:34)

It can cause problems can’t it because people might press two or three times and they might inadvertently approve a transaction they didn’t intend to. Very different from pressing your brake pedal, sailing through a red light and then having the brake slam on when you’re on the other side of the intersection.

Jacob Beningo (02:41)

Exactly. So you can see there’s variations in degrees to real time based on the type of system you’re developing. Yes. From consumer electronics through medical transportation type systems and so on. But generally we look at it as that boundary context, that determinism, basically time guarantees that the operating system will, and even your code, because the RTOS itself, even though it’s real time, it’s up to us as a developer to make sure that we’re paying attention to those times and those responses and that we don’t break something on the RTOS itself.

Paul Ducklin (03:20)

So it’s not necessarily about being fast. You could have a response time that’s allowed to be 10 seconds, but not 10.001 seconds. Like you say, it’s all about the determinism rather than being mostly good enough.

Jacob Beningo (03:30)

Exactly. Oftentimes when we develop requirements for an embedded system, we talk about soft requirements and hard requirements. And those are soft real time where the deadline’s kind of loose. If you miss it, no big deal. Then there’s the hard real time, which is if you miss that deadline, well, system degrades or bad things could happen to the user.

Paul Ducklin (03:54)

And part of the compliance that you would have to go through and part of the verification for your software to be accepted will be to show A, in testing, you do meet those real time requirements and B, that the system has been built in a way that that is likely to be the case indefinitely.

Jacob Beningo (04:14)

Yeah, absolutely.

Paul Ducklin (04:16)

So when AI starts generating code that goes into firmware, how do things change?

Jacob Beningo (04:24)

I would say everything changes.

Paul Ducklin (04:27)

Now change is not necessarily bad, no. It’s the unpredictable side effects of change that are the problem.

Jacob Beningo (04:33)

Exactly. It’s the unintended consequences. It’s the changes to processes. It’s the changes to the way that we think about developing the systems in general. Even the fact that you don’t want to become complacent with the way that you’re developing that system. There’s all kinds of things that change from education and so on and so forth.

Paul Ducklin (04:51)

So do you think that there’s a risk other than the mere fact that AI is known to make mistakes, sometimes egregious ones that are non-obvious? Do you think there’s also a risk that instead of having to sit down and think carefully about code and then work on it and maybe get your peers to review it, that you just throw it at some statistical inferencing engine and back comes something that looks great? Do you think there’s a psychological or a social risk, if you like, that breeds overconfidence?

Jacob Beningo (05:21)

think that it can breed overconfidence if you’re not careful. If you’re thinking about the tool as something that I ask for a solution and it gives it to me and that’s the end of the day. But if you think of it as not necessarily better at doing it than you are, it’s actually a tool that can help you accelerate what you’re doing. But at the same time, it’s going to make mistakes just like anybody else. I’ve made some fantastic coding errors in my days. When we say, the AI hallucinated or it generated some bad output, I kind of say, well, you know, I’m just as guilty myself. And so how do I prevent myself from making those same mistakes? Yes. I put the right processes in place, right? I use the right tools and we go through a rigorous process to make sure that the output is what we actually expect it to be.

Paul Ducklin (06:07)

Who hasn’t coded a condition the wrong way around by mistake? Perhaps because somebody else chose a weird variable name, like disable this feature equals true. Yes. Which is bound to lead to semantic confusion later. Yeah. Now an AI can’t step beyond those sorts of mistakes if it’s basing its code on stuff that you’ve pointed at that already contain those mistakes.

Jacob Beningo (06:38)

That to some degree is where some things I think actually change because you go in using these tools thinking, well, it should be perfect, right? But what we’ve trained a lot of these code generators on is open source code that may be high quality. I don’t want to say open source is bad. There’s a lot of libraries that I use that are actually really good, but there’s also a lot of lesser quality code bases out there that I’m sure we’re still fed to AI. And so we kind of have this engineering 101 problem of garbage in, garbage out.

We haven’t given it all garbage in, but you have to tailor also your own requirements and inputs to it so that you actually generate the expected outputs.

Paul Ducklin (07:16)

Programmers learned that lesson of not being overconfident in code they’d copied and pasted. But with AI, things are in presentation terms a little bit different, aren’t they? Because the code often looks as though it was created from scratch. It doesn’t have all the comments left in saying, I wasn’t quite sure about this, you need to check it. So the code could, if you don’t know what you’re looking at, lead you to a false sense of security or make it look as though it needs review and testing and vulnerability assessment to a lesser degree than it really does. Would you agree with that?

Jacob Beningo (07:55)

Absolutely. You can get some code that looks fantastically polished out of it. You glance at it and you start doing a review and your mind goes, this looks great. Which is why I think when you do use AI tools, you really need to have a good checklist, a code review checklist, just like we do with our peer reviews. Look for vulnerabilities, run your stack analysis tools, run it on your intended target and actually do dynamic analysis on it as well. Pen testing, all of those types of things that we should be doing anyways.

We should also be making sure that we do that with our AI generated code as well.

Paul Ducklin (08:26)

This is not a new problem, is it? If you go back 30 or 40 or even 50 years in programming, when all you could really do was either get the manufacturer’s programming guide in a great big ring bound document, or you could get the textbook that was the flavor of the month or year at big universities. And you could look at code in there. But then firstly, you had to type it back in yourself. You couldn’t copy and paste it.

So you could introduce errors by mistake or that code might not have been correct. If it was in a book that was published five years ago, ideas might have changed. Better techniques might have come about. When it comes to something like real time performance predictability or determinism, it might not quite live up to its reputation.

Jacob Beningo (09:15)

Oh, absolutely. I’ve actually done the experiment where we had entry level engineers, they wrote some code for a system. The system as it turned out was not, let’s say running to the real time level that it should. It was using like 85 % of the CPU. We took an AI and had it run through the code and it actually took it from about 85 % CPU utilization down to about, I’ll say it was like 52%, drastic improvement. And then we took a senior level engineer like myself to come in and rewrite that code. And we got it down to like 40%.

That kind of gives you a little bit of a feel from a real time perspective of where these tools currently are at. AI doesn’t really understand how to fully optimize for the end targets. It just doesn’t have that experience. It’s mostly trained on code that’s open source. Maybe it’s C language being used for some other app, a type of application on a desktop or C plus plus for data science or something where they’re not as concerned with efficiency and performance.

Paul Ducklin (10:08)

You have a certain amount of memory. It comes with, the system on chip and that’s that. So how do you look for mistakes like that in AI generated code? Certainly as a writer, I noticed that a lot of what the AI writing bots do, they often write rather nice prose, trying to say what you want to hear, even if with the best one in the world, it’s told you the opposite of what is true because it just started from a false premise.

How do you deal with what you might call the devil in the details with AI generated code, particularly when it comes to real time operating systems, things like memory consumption, power usage, response time, determinism.

Jacob Beningo (10:53)

We’ll start with the first part that you mentioned there, which was how do we recognize and look at the code and kind of make sure that it’s meeting what we want it to do. One of the things that I have done in my development is I don’t allow AI to just run off and do a whole bunch of stuff at once. You’ll find that a lot of tools out there, they’re very much trying to push agentic workflows. Yes. Where you tell it a little bit of information and it just runs and just goes, goes, goes, goes, goes. I’d very much constrain it to, I want to do this one little thing and just let it generate a little bit of code.

And then I review that code. I’ll use it to potentially generate test cases for the code that I’m writing. I’d like to use test driven development where we write the test first, watch the test fail, write some code to make it pass. And then we refactor after that. So that it’s almost like I’m writing the code, but I’m pair programming with the AI. I keep it in my editor. see what it just generated. It’s not just letting the AI write the code for us. It’s understanding the tools, its capabilities. If I tell it this, what does it do?

If I give these inputs, how does that change the outputs and trying to get constraint around how do we actually use this? Where is this stuff actually going in the future?

Paul Ducklin (11:59)

So you use it as a tool, not as a magic wand.

Jacob Beningo (12:03)

Exactly.

We’re not vibe coding. Build me a weather station or build me a brake controller and just let it go and chug away for two hours and come back.

Paul Ducklin (12:05)

Yeah.

Do you think that the media are making things a bit more dangerous than perhaps they need to be with AI by talking up some of the stories that you’ve seen lately? Like, hey, this guy told AI to write a whole browser from scratch or hey, these guys spent $20,000 of credits and they got a very low quality C compiler. Yeah.

My understanding is that the researchers who were doing that, weren’t trying to build a browser or build a C compiler. If you want to see a compiler, go and download one that’s known to be good and don’t try and knit your own out of rotten string. The idea was to see what happens if you let a whole load of agents work together. But those stories have sort of been pitched as, Hey, like you just said, write me a brake controller for front disc brakes on a vehicle of this mass. Click.

That’s never gonna work, is it?

Jacob Beningo (13:07)

No, it really isn’t. There’s a whole bunch of process that needs to go into it. I recently put together a simple push button example. I call it a universal push button. Everybody in embedded systems at some point, they have a button that needs to be read. And all it does is it debounces the switch and it tells you, it pressed? Is it not pressed? Did someone double press it? That kind of thing. And I allowed AI to generate that code, but I didn’t tell it, build me a universal GPIO controller. What I first did was say, Hey, let’s generate some use cases around a push button.

Here’s a page long product description of a push button module and what it should do. And then I said, build me some user stories around this. And it generated like 10. And then I go into the use cases and I look at them. say, oh, this one’s kind of off. No, I don’t like this. Let’s delete this one. Let’s add this one. And I end up with good user stories. Then I tell it, let’s build requirements from that. And it generates requirements. And then you go in and you look at those requirements, you remove some, you added them following a process to where you have real requirements. And then you say, okay, off of this, let’s try to draw a software architecture and it draws something that’s horrible and you go and you fix the architecture. You’re following your normal development workflow, but you’re using AI to help accelerate pieces of it. A lot of times as an example, embedded developers start projects without a full set of requirements. They get a little description and AI becomes a tool that you can say, well, fill in the blanks and give me an idea of what’s here. And then if you take that, you might end up with a bunch of user stories that you could then go to your customer and say, Hey, is this what you want built? And they might say, no, no, no, I don’t want this and this and that. Okay, cool.

And this is just a single module. So you might say, okay, well now that I have user stories or requirements, I have an architecture. Go ahead and build that for me. I actually had a really interesting experience when I did this because I had a test-driven development agent I had created. You tell AI to follow test-driven development, use the TDD agent, write a test, watch it fail, write the production code for it, watch it pass, then move on to the next one. And I told it to go write the code. And after about, I’ll say 30 seconds, it had finished writing all the code, all the tests, everything. And I was like, my gosh, like that was extraordinarily fast. And then I noticed the output of the AI was, would you like to run the tests? And I was like, well, that’s interesting. And I said, didn’t you already run those tests and follow TDD? And it responded back. No, that would take too long.

Paul Ducklin (15:29)

I guess what that means is that given that the AI does not have and is not supposed to have this reasoning and intent that a human creator would, you have to take that into account in how you build AI into your workflow. It may be a partner, it may be a productive partner, but you’re not saying go and make all the decisions for me all the way along and

Jacob Beningo (15:48)

Exactly.

Paul Ducklin (15:59)

Wake me up when we reach the destination.

Jacob Beningo (16:01)

Exactly. You’re using it as part of your normal workflow to accelerate it, to do parallel processing for you. Maybe while I’m writing code, I’ll have it off doing some research, looking through a 3000 page data sheet or, Hey, we have this problem. Go search the web and find potential solutions for this. There’s a way to multitask in a way that we maybe weren’t able to before, or a way to process data even in a way that we haven’t been able to before. And so I really see this as a tool to help you accelerate.

It’s not the magic wand that you see in the media where they’re really trying to push it to make this thing happen. And it runs off and with a team of 30 other agents solves the problem for you. This is really a tool. think that helps us automate away the things that we do on a day-to-day basis that get in the way of us actually being creative of us actually solving the problems that we should be solving.

Joe Saunders (16:50)

And I think another example gets to fixing either vulnerabilities or bugs and things like that. And in the exact same way, prompting specific tasks or prompting the AI engine to maybe resolve a certain vulnerability and see what it comes up with, but not necessarily then offer, here’s my list of vulnerabilities, go fix everything for me. If you do it in a prompted way, one by one, and you can review those and maybe fine tune some of them.

Not only are you more efficient in fixing your vulnerabilities, but you also are then including that same level of control that Jacob’s advocating earlier in the process when you’re developing new features or testing out concepts. I do think part of the opportunity here though is to know when AI is being used and when it’s not being used. And I think that kind of metadata at some point is going to be useful. Wouldn’t we all love to know if a component was written by AI or not?

If we’re looking to assess even an open source component, whether it’s trustworthy or not. I do think it’s changing workflows and in the embedded software world, especially where there are mission critical systems, let’s think about aviation or autonomous driving, or even industrial controls and industrial automation where there’s blades and other things in facilities, there is a significant consequence if something goes wrong. So I think elevating your productivity is of course great and welcome.

And maintaining that level of discipline and control and checks and balances in the development process is also absolutely essential for any of this to make sense in the first place.

Jacob Beningo (18:28)

Absolutely.

Paul Ducklin (18:29)

Now, Joe, you raised what I think of as a vital background issue there, not just about the correctness of the code and its real-time performance and the accountability you have to what the code does, but also the accountability that you have, particularly if you’re using open source software, for correctly licensing the code. That if you have code that came from an open source project, but someone along the way, perhaps not with evil intent or not trying to violate any legal norms, but just because they thought it might be a good idea at the time. And therefore, if the standard version has a vulnerability, you might not realize that this machine rewritten version also has that vulnerability, but kind of invisibly. Do you think that’s a problem waiting to happen? Or is it something that we’re seeing already?

Joe Saunders (19:22)

I think it does happen. If you rely on SBOMs to identify maybe the CPE or the PURL associated with the component and that no longer gets identified, then you can’t link to the vulnerabilities. I do think that’s one example then of how metadata about when AI is used could be useful. I would like to show a positive view on or an optimistic view on this. And as a quick story, for years, I played in this Thanksgiving Day turkey bowl among friends and family and neighborhood football game.

Paul Ducklin (19:54)

Well, mind boggling, Joe, in a very pleasant way.

Joe Saunders (19:57)

It used to be that the rules, the kids always had to, the little kids always had to touch the ball once per possession. Otherwise the adults would just dominate the game. As the kids got older and became teenagers in high school football players and the parents got older and started to slow down a bit, the rule flipped. The parents had to touch the ball once per possession. And my point is in this world where we are relying on AI to help build certain components or solve certain vulnerabilities or fix a bug or what have you. There is a need Paul to your point that we do need to kind of assess trustworthiness of components. And I would venture to say that even to what Jacob alluded to earlier, he has made mistakes in his day, maybe one or two. But I think over time, we’re going to find that AI generated code could be more trustworthy than human generated code. And so that is some distant out in the future. There’s too many smart people that know how to build code. And to Jacob’s point, the engines haven’t quite caught up to know which one is great quality code and which one isn’t, although we’re making strides there. But I would venture to say, at some point, we’re going to trust AI code just as we realize that those young kids grew up and became great football players and the parents became outdated.

Paul Ducklin (21:20)

So that raises an important issue about timing and about trying new stuff. If you look at some of the success stories with AI generated code, it’s that, we can throw some AI code into this web app and dear, that didn’t work too well. Well, live and learn tomorrow. We’ll fix it. The day after we’ll have three more updates next week. Version 24 will come out. You don’t have that luxury with most embedded devices. Either the older ones cannot update over the air, or there are regulations that prohibit them from updating over the air. So how do you balance that? How do you balance the longevity of embedded devices? You wouldn’t expect to be using the laptop you have today in 25 years time. Glad I’m not using my 25 year old laptop, but that’s not true with embedded devices. So how do you get AI that’s very forward looking to think about what will be ancient past in the not too distant future.

Jacob Beningo (22:23)

It’s certainly a challenge. I think in general, what you have to do is just look at modern day best practices for developing embedded systems. And ultimately, if you’re using AI to generate that code, you need to design your systems in such a way that it’s scalable so that it can be broken up into pieces so that you can update in the field, small pieces of your system. So it can be compiled separately. It can be deployed separately in bed systems. tend to still generate more monolithic type of systems, at least in the microcontroller space where you might find you got to flash the entire device, but certainly things are coming along and things are changing. So some of the new parts that are out there are, mean, I don’t look at them almost as resource constrained devices anymore. They are extremely powerful devices. And then AI I found does have at least a fundamental understanding of software architecture and how to layer things. makes a lot of mistakes, but over time that will improve. Oftentimes if I, if I want to try to design a full thing, one of the things that’s in my prompts or in my agent files is that I tell it, Hey, this is a resource constraint device. Assume it’ll be used in the field for X amount of time. You have to give that background to it so that it understands the context of what’s being designed. That it is real time. That it’s something that’s been in the field for a long time. Maybe it needs to be updated. Maybe it needs to scale. Maybe it needs to support multiple customers and that sort of thing. Today in the embedded systems industry, I’m seeing a lot of people excited about AI, very hesitant to use it. We tend to be late adopters of technologies. Let me put it that way, just because of the constraints that we tend to be on.

I may be digressing there a little bit.

Paul Ducklin (23:53)

I guess it’s don’t throw the baby with the bath water. Yes. And definitely don’t throw the bath along with the baby in the bath water. This is a tool that you can bring incrementally into workflows that you’ve learned and developed and that you can trust. I think what I’m trying to say is you can probably do very well with AI, even in resource-contained embedded spaces, if you treat it as an evolution that helps you do software better or faster or test more or try more options rather than a revolution that basically lets you press a button and have no human input coding.

Jacob Beningo (24:37)

Exactly. One of the things that embedded systems developers are traditionally kind of, I’ll say not the greatest at is actually having a lot of unit tests and being really good at testing overall and building that test framework from unit tests, integration tests to full system tests.

Paul Ducklin (24:52)

Do think that’s because often the systems are monolithic? Yeah. So you kind of think, well, I can test it all at the end because the app and the operating system are kind of one big lump.

Jacob Beningo (25:01)

Yeah, we tend to think of it’s embedded. And so we want to couple ourselves tightly to the hardware. And so if you’re tightly coupled to the hardware, it becomes a lot harder to unit test things as you try to move up the stack because you have dependencies that, well, I can’t test that because it’s connected to the hardware.

Paul Ducklin (25:16)

It becomes a lot harder to adopt the latest generation of that hardware, which may not merely be faster and cheaper, it may actually be significantly safer because it might now have memory protection, or it might have better message passing, or it might have built-in floating point that saves you time that you were wasting before. So it makes you a bit more future-proof if you design more layered systems. absolutely. And AI can really help with that, can’t it? Because while you’re thinking of

Jacob Beningo (25:34)

Exactly.

Paul Ducklin (25:44)

The module you’re going to write tomorrow, your AI agents, rather than writing the code, they could simply be beavering away looking for problems in the code that you’ve had for years. You may be interested to find things that you simply wouldn’t have had the time or the energy or the will to look for otherwise.

Jacob Beningo (26:01)

Yeah, you could create an agent that when you check in your code looks for tight coupling between your application code and the low level drivers and just looks to see if it’s abstracted out or not. And as it runs, it can give out a report that says this module here is directly calling into the registers over here. Flag it as a technical debt to improve because you’re tightly coupled.

Paul Ducklin (26:19)

Joe and Jacob, to finish up because I’m conscious of time, I’d like to put a question that either or both of you can answer. For engineers, particularly embedded software engineers who want to use AI, maybe they’ve only been toying with it so far, or they’ve been unsure of it because they don’t know how they can fit it into their workflow, but they’re going to have to cross that bridge sooner or later because someone in their team is going to do it even if they don’t.

What things can they start doing right away that will let them enjoy the benefits while dealing with that problem that we mentioned at the start, namely that if AI wrote the code, who bears the risk? How do they keep all of that under control while enjoying the benefits?

Joe Saunders (27:08)

I think there’s a lot of things people can do. And if someone is wondering if they should start using AI, I would encourage them to jump in and start doing it one way, a super powerful tool, although it might be a little scary to people as OpenClaw. And it becomes a pretty amazing assistant to you, whether that’s taking care of daily tasks or even coming up with simple code to incorporate at some point. More generally, I think getting started in doing it and following the notion of having good guardrails, good practices. You’re not changing the way you’re testing. You’re ensuring you still do all the right testing and compliance items that you would otherwise do for safety critical systems. Then I would start incrementally and see some of the benefits in the return. Of course, you’ll always have people like Paul who want to write in assembly itself and may want to do everything by hand. And I’m sure he’s 100 % correct when he does that, but maybe not.

Maybe not even Paul.

Paul Ducklin (28:09)

100 % correct 94 % of the time.

Joe Saunders (28:12)

I’ve always been told that statistics never lie, but liars use statistics. With that, I would encourage people to get started and certainly be mindful of some of the risks that we talked about today. Don’t throw your process out just because you have a great new tool or gadget.

Jacob Beningo (28:31)

Yeah, I’ll maybe just encourage some ideas. First of all, this is a technology that I think is going to take over the world. So you can’t ignore it. You at least have to be trying out, trying to understand what it’s capable of, what it can do, where it can make your life easier in a safer way as you write your code. And so look at things that are maybe low-hanging fruit that you do on a daily basis, things that you could automate and kind of toss over the fence. As I mentioned earlier, we don’t like writing tests a lot as embedded software engineers. Yes. Ask AI to make a list of tests that you need for a module that you’re going to develop. Ask it to improve your build system, things like that. Things that are very low hanging fruit that may not be code generation itself, but eventually as you move towards using code generation, make sure that you are using static analyzers, dynamic analyzers. When you write code, ask it, hey, how could this be improved for XYZ vulnerabilities? Again, don’t treat it like it’s a hundred percent right, but look and see what it gives you and say, that’s interesting. That could be useful. You should at least be exercising the tool and exercising your own critical thinking skills and how do I use this tool? And again, I think it’s actually important that we continue to code. But I actually have a concern that in the future we’ll forget how to program. Hopefully programming will become like chess. We know that machines are better at chess than we are, but we still enjoy doing it. And so we continue to play chess.

Paul Ducklin (29:45)

That’s a great analogy because chess has never been more popular than after Deeper Blue proved that computers could always win if they really wanted to. And I guess the advent of the pocket calculator did not make mathematicians redundant. It didn’t make the intellectual exercise of coming up with proofs easier, but it did make doing things like validating proofs, looking for counter examples and testing your intuition in ways that you wouldn’t have time to do yourself perfectly possible. So that seems like a fantastic place to start if you haven’t already. Rather than saying, I’ll let the AI do all the things I want to do, how about letting it help you with all the things that you know you should have done, but technical debt, for want of a better term, has stopped you from doing so. You might be surprised at some of the horrors that you find that you can fix, and that would have improved things greatly, won’t it?

Jacob Beningo (30:42)

Exactly. There’s all those things you never had time to do that suddenly, potentially, you now have time to do.

Paul Ducklin (30:47)

Yes, no more excuses. Gentlemen, thank you so much for your time. It’s great to talk to people who are passionate about using new technology in exciting ways while keeping the risk tightly under control. So thanks to everybody who tuned in and listened. If you find this podcast insightful, please don’t forget to subscribe so you know when each new episode drops. Please like and share with us on social media. That really helps us a lot.

So once again, thanks to everybody who tuned in and listened. And remember, stay ahead of the threat. See you next time.