Exploited: The Cyber Truth,  a podcast by RunSafe Security. 

[Paul] (00:01)

Welcome back everybody to Exploited: The Cyber Truth. I am Paul Ducklin, joined as usual by Joe Saunders, CEO and Founder of RunSafe Security. Hello, Joe.

[Joe] (00:19)

Greetings, Paul.

[Paul] (00:21)

We have a special guest this week, Salim Blume, Director of Security Applications at RunSafe. Hello and welcome, Salim.

[Salim] (00:32)

Hello, Paul. Thanks for having me.

[Paul] (00:34)

We have an intriguing title this week, namely, what if open source gets you into hot water? And that’s source as in source code, not as in tomato sauce. And although we’re all techies talking here on this podcast, the licensing aspects relating to the technology you produce can be a terribly complicated part of making, supplying, selling, and supporting software, can’t it?

[Salim] (01:09)

Absolutely. It becomes so complex. It’s like most people just try and ignore it, but it’s something you absolutely cannot ignore. It doesn’t matter how big or small your company is. At one point, your software is being consumed by a customer of yours. And if it contains the wrong licenses, you’re putting yourself and, at times, your customers at risk.

[Paul] (01:29)

Now, a lot of people misunderstand what is meant by the very general term open source, don’t they? You can have free software, which is closed source; in other words, the source code is never revealed at all. Free has two legal senses: free as in freedom to use it, and also free as in not having to pay for it.

[Salim] (01:53)

The way that you can make that determination is by looking at the license of the software. If it is open source that you can go and look at, maybe it’s on GitHub, and it has a license file, you can see this is how you are allowed to consume it. And that’s the only way to know which free you’re dealing with. And it ends up being this huge legalese contracting. And if you’re just one guy trying to get started, how are you supposed to parse all this? Or, at the other scale, if you have thousands of repos, consuming tens of thousands of components, how are you supposed to handle every single license file that you’re putting into your own software?

[Paul] (02:30)

And I guess the other problem is if you’re consuming software that has come floating to you down the supply chain, A, how do you know what licenses really apply there? And B, how do you know whether the people upstream of you have made any effort at all to comply properly? Because in the end, you carry the can when your software goes to market.

[Salim] (02:54)

You’re right, and there are different combinations of those licenses. Somebody may write software and license it to say, if you’re non-commercial, it’s licensed this way. And if you are commercial, you are licensed this way.

[Paul] (03:06)

Wow, so in other words, it’s free and open source if you’re a hobbyist, or perhaps if you’re some certain type of organization, but not if you’re considered commercial, then you’re faced with how do you decide which side of that boundary you land?

[Salim] (03:23)

And it may be once you hit a certain revenue amount. And who’s keeping track of that? And you may be in compliance as long as you’re shipping a small quantity of devices. But as soon as you hit it big, who’s keeping track of, are we still compliant with this license that we were last month?

[Paul] (03:42)

So do you want to give a description of some of the main sorts of open source licenses that are out there?

[Salim] (03:49)

Yeah, sure. So there are hundreds, if not thousands, of licenses out there. They do fall into some broad categories. You see restrictive licenses, permissive licenses. There are different types of restrictive licenses. If you are poking around software licenses, you’re going to see GPL, you’re going to see MIT, you’re going to see A-GPL, and Apache, and BSD, and there are tons of them. MIT is a very popular permissive license because a license isn’t just for the person consuming the software, it’s for the person creating the software. And so an MIT license, it’s helpful for someone who writes, we’ll say a tool, put it on the internet, and say, I’m proud of this. I think people should use it, but I don’t want to be held liable for any bugs that you may find if you do put this into production. But I don’t care if you do use it in commercial software. So that’s, that’s more of a permissive license. Whereas a more restrictive license, GPL being again, probably the most popular one of those.

[Paul] (04:46)

GPL is short for General Public License, not GNU Public License, although it comes from the GNU Software Foundation originally, doesn’t it?

[Salim] (04:57)

There are also multiple versions of GPL and that is sort of the tip of the spear of this concept of Copyleft, which a lot of people have probably heard about. If they’re listening to this podcast, they’re probably interested in how am going to defend myself from this boogieman of copyleft?

[Paul] (05:14)

Now, copyleft, if I remember correctly, it actually started as a joke back in the 70s. Copyleft, all wrongs reserved and it was meant as a joke. But it was taken in a sort of periodical way to mean that this software is not in the public domain. You can’t do whatever you want with it and the legalities turn out to be quite different, don’t they, in terms of your liabilities if you use GPL software?

[Salim] (05:19)

Yes. It was codified by GPL releasing for the first time as an actual license. So going from a joke to actually this is something that we want to hold people accountable to. That idea being if I write software and release it under a license that could be considered copyleft and then you use my software, you are obligated to license your software under an equivalent copyleft license. Which is to say, if mine is from the corporate sense, a very restrictive license where you must open source all of your code and you use it. Now you must license your commercial code in the same way and potentially release all of your software.

[Paul] (06:28)

So in theory, you could have a giant product that has millions of lines of code in it, and you import into your code base one tiny little implementation, say of a cryptographic algorithm that might be 20 or 30 lines that is under a GPL, a copyleft style license. And if you then publish your program with that code in it, your entire code base essentially should become subject to that copyleft agreement as well and you have to publish all of your secrets for everyone to see. Is that right?

[Salim] (07:03)

Not only are you on the hook, there are organizations out there that believe so strongly in that, that they will come and look for violators of licenses such as GPL and take them to court to force them to open source their software. Vizio, the TV manufacturer, is currently on the hook for exactly that thing. Most famously would be Cisco who had a router family, a WRT 54G.

In 2003, it was found that the software on these routers was using the GPL license. But it’s a lot more complicated than that. Cisco had purchased Linksys for half a billion dollars. Linksys was making those routers. Linksys didn’t put the software there either. They got it from Broadcom with the Wi-Fi chip. Broadcom didn’t put it there either. They used a third-party contractor who wrote software that used a GPL license tool for the 802.11g driver, and that made it through four levels. This contractor went into Broadcom, into Linksys, through a merger and acquisition into Cisco. Cisco’s on the hook for that and did end up having to release all the source code for that particular router.

[Paul] (08:19)

Sort of happy ending there wasn’t there in that they decided well if we have to do this we’ll create a version of the router that you can basically flash with your own version of Linux. It worked as very positive marketing for them in the end didn’t it.

[Salim] (08:33)

It did. And it became, for a long time, the best-selling router because it was so easily modifiable. And there was such a community around doing that.

[Paul] (08:42)

Now in that case, you imagine a company like Cisco, you think, well, they’ll just get the checkbook out and they’ll just write a giant check and say, okay, well, we’ll pay you back licensing fees. But in most cases with GPL infringement, copyleft infringements, that doesn’t work, does it? Because the organization that’s going after you doesn’t want your money, it wants you to comply with the open source, the freedom side of things. Either you reveal all your source code, even the stuff you wanted to keep proprietary, or you stop selling the product entirely with that code in it and start again from scratch. Even if you have deep pockets, you can’t necessarily buy your way out.

[Salim] (09:25)

That’s what Cisco saw. That’s what Vizio is in the middle of fighting. You’re absolutely right. And not only are the organizations that are going after companies interested in that, the licenses are written in such a way that there is no backdoor. Even if the organization suing was interested in a monetary gain, that’s not how the license is written. We’re kind of abbreviating copyleft into GPL because it’s the most popular. There are a whole bunch of others, but when talking GPL, that’s not how it’s written.

[Paul] (09:49)

Yes.

[Salim] (09:55)

You can’t skirt out by cutting a check, as you say.

[Joe] (09:58)

And I might add right there, one of the standards or tests had been, it’s not meant if you’re just using the software internally for your own development purposes, but if, in fac,t you distribute the software. And there’s been kind of an evolution there where a SaaS enabled service under an AGPL license is intended to consider that that SaaS service is a form of distribution of software. And so even though you’re not shipping software to third parties, you’re delivering a service over a network. It also can constitute a software distribution and under the AGPL variation that would be obligated to copyleft as well.

[Paul] (10:40)

A is for Affero, isn’t it? Which I believe is the company that first came up with it. I’m understanding that was intended if you like to fight back against industry giants like Google, who would run a million Linux servers and make billions of dollars out of it. But you couldn’t get hold of their software because technically they’d say, well, it’s actually part of our network. Even ISPs, who provide routers where you don’t own the router but they lease it to you and you plug it in in your household, they’d say now that’s part of our network so that’s not covered by an open source license. But in the modern era, it might be because of the existence of licenses like AGPL. Have I got that correct, Joe?

[Joe] (11:28)

Yeah, one hundred percent. Just as Salim identified the Cisco case, famously Oracle was found in violation of an AGPL license from Qualcomm. Oracle had thought because it’s a network service they were offering, a SaaS-enabled offering through their cloud hosting. It wouldn’t be subject to it, but just as you say, they’re making a lot of money using a component that was intended to be copyleft.

[Paul] (11:53)

And am I right that for many of the fans of open source software, it isn’t really about the money at all. Oracle might be in their sights because they are just so wealthy, but even if you’re a tiny company, for many open source fans, it’s the principle that’s at stake. If you’re a tiny business, if you suddenly have to stop selling a product, that could have a disastrous effect upon you.

[Salim] (12:17)

Even as a small company, you may be forking an open source project on GitHub. If you’re just getting started, you may not realize the mistake you’re making and you’re doing so very openly. If you are working on a tool that is licensed with a copy of license, you make a fork. It’s very obvious to everybody out there that you’ve just taken that action. And so you put a target on your own back.

[Paul] (12:38)

And it’s my understanding that if you do rush into building software and you embrace open source or you think you have, but you don’t think about complying with all the necessary licenses, whether they’re copyleft or MIT style, whatever they might be, if you don’t think about that from the start, then if you do get into trouble later, even if you’re quite happy to comply, you say, yes, we’ll release all the source code, we’ll get everything together, that itself can be an enormous task, can’t it? Getting that right in a hurry is very difficult indeed.

[Salim] (13:15)

Yeah, I would say especially so for companies that release software on like embedded devices where it’s not just a website that’s out there accessible on a single domain name. As soon as you catch the issue, you can push a new version. Everything’s fine. If you instead are like a device manufacturer, you’re building robots for a factory and you have many versions of software and you have no way to really know what versions of your software exist out in the world. For instance, in GPLV3 section eight, have a grace period where you can have, depending on the violation, 60 or 30 days to cure your license violation. That’s a lot easier, like I say, if it’s just pushing a new dependency to a website. It’s monumentally more difficult on wide distributions of software running on end devices.

[Paul] (14:10)

It doesn’t matter how nice you’re trying to be about it, just finding out what you need to do to complete your compliance could take you a lot more than that 30 or 60 day grace period.

[Salim] (14:22)

In this perfect world, you may have a perfect understanding of all of the software you’ve ever shipped, and you may still have a customer who says, I’m not going to upgrade. There’s nothing you do about it. You can’t then cure the situation.

[Paul] (14:34)

Or you might not realise that the offending code was in one of your older products because you weren’t keeping track of things back then. Yeah. How do you go retrospectively and find it before somebody else decompiles your firmware and goes, ha ha ha, I’ve caught you out?

[Salim] (14:50)

I think what Cisco decided was that it’s far cheaper just to open source it.

[Paul] (14:54)

Yes, that can be difficult if you can’t get your hands on the source code that you had back then because maybe you weren’t quite so disciplined about version control and stuff like that. So, Salim, that raises the intriguing question. How do you keep track of all of this? How do you make sure that you haven’t accidentally included something that you did not expect? That one little super-efficient cryptographic algorithm that turns your MIT-licensed project into a GPL-licensed one.

[Salim] (15:29)

It starts with accurate SBOMs.

[Paul] (15:32)

That’s Software Bill of Materials.

[Salim] (15:35)

Yes, you are definitively stating this is what my software is. And that is how you can then say, okay, if I know that I was using this third-party software, now I can recognize, at that time on that version, that was GPL licensed. Gosh, I should do something about that. You can’t do that without an accurate assessment of your own software. 

And that also means the third-party software that you’re pulling into your own. That does mean that this is where it gets more difficult. If you’re pulling in a binary from somebody else, they had better be including their own SBOM where they attest this is what was in this binary before you use it. You want that for security reasons as well as for future vulnerability scanning. Now I know that this software that I’m putting into my system and then selling as a more valuable package, it is vulnerable to X, Y, Z in the future. Just as with the license problem of knowing do I need to remove dependencies that violate licenses? Now I need to upgrade or change dependencies for vulnerabilities. It all starts with an accurate SBOM.

[Paul] (16:43)

So it sounds as though there are some very good reasons just from a cybersecurity point of view, let alone a legal correctness licensing and community centric point of view, to know what’s in your software anyway. If you can’t tell what license applies to your program, how do you know what now well-known buffer overflows are latent inside it as well?

[Salim] (17:07)

Think you have stacked it well. You can solve the license problem. You can solve the vulnerability problem.

[Paul] (17:12)

So how do you make sure that your Software Bill of Materials really does reflect reality? We talked with Kelli in a recent podcast about, if you like, the difference between just opening up your cupboard and saying, well, I’ve only got this giant list of ingredients, so it’s probably going to be some or all of those, which A, may over-include things and B, you may accidentally have used one that wasn’t in the cupboard. Yep. Or you wait till the cake’s baked and then you have a taste afterwards and you go, yeah, it’s probably got sugar, bicarb, and a bit of salt. That’ll do.

[Salim] (17:48)

To watch as the cake is made, as the software is built, and write down every single thing. And it helps when you have a recipe. And then you can say, yes, I did just put two eggs in there. Did just put a cup of flour. You have to monitor every step of the process. The only way to do that is to automate it. Then you need to keep track of what you wrote down. A good way to do that is through a central repository of all of your SBOM information, then it’s very easy to say, now in these same automated processes, especially built on CI/CD pipelines, let’s look at what went in and now what needs to be remediated?

[Paul] (18:29)

And there are some standardized formats for how you record that bill of materials data, aren’t there? Yep. If you follow one of those well-known standards, it makes it much easier for the people who are upstream and downstream of you to consume that data automatically, instead of having to read through 17 different ways of writing a list of ingredients and possibly getting caught up in ambiguities.

[Salim] (18:56)

Biggest one is Cyclone DX. There’s SPDX, SWID. Cyclone DX is huge.

[Paul] (19:03)

Do have to put cryptographic hashes in there so you can identify the exact files that you use?

[Salim] (19:10)

You don’t have to, but the spec does allow for that. And it’s a great thing to do, especially again, when you’re consuming third-party software that does then allow you to put those into your license and vulnerability management tools. And then later look backwards if necessary and say, Hey, third-party supplier, this is something you may need to take a look at. Or I think I’m now at risk because of something you’ve introduced. This is something you have to deal with.

[Paul] (19:34)

Now Joe, you in previous podcasts have made some intriguing remarks about the potential liabilities of, if you like, trying too hard, or perhaps I even mean too casually, with generative AI by taking, say, a well-known open source tool, throwing it into an AI, generative AI system and saying, hey, can you improve this a bit? And then you end up with something that is slightly different, that doesn’t match. the source code has a different hash and then you do run the risk that you will fall out of visibility both to licensing checks and to vulnerability disclosures. So that’s maybe a little bit of a problem waiting to happen isn’t it?

[Joe] (20:21)

It is a problem that’s waiting to happen in part because you just may not know there’s vulnerabilities that otherwise you would be aware of had you used the original binary or component or library in that case.

[Paul] (20:34)

And this is not somebody being malevolent, saying, rewrite this so that it doesn’t look like the original so I can deliberately pretend that it wasn’t licensed. This could happen just with the best one in the world. You think, well, let’s see if we can make the code a bit shorter, a bit faster, a bit cooler.

[Joe] (20:49)

Right. But it also raises the question, there are really good reasons why you do want to use open source software. You don’t really want to recreate all these different components because that becomes inefficient. There’s a lot to gain from leveraging open source software, in part sharing the vulnerabilities so that you can ensure that your software is protected, but also to ensure that you don’t have to rebuild something and then maintain something that would otherwise be supported and generally available and used by thousands of other software companies or products or people or teams.

[Paul] (21:24)

In other words, if you make some tweaks that just happen to work for you, then when you go back to the community and say, hey, look, I made these changes, you should use them too. Not everyone else might agree with you and you might end up with something that you have to maintain completely independently with no help for years and years and years in the future. Whereas if you’ve been a little more agreeable to the community, then you’d have the community on your side forever more.

[Joe] (21:49)

And to Salim’s point earlier, than checking these things and doing it right from the first place, you get the benefit of the open source community and then therefore support it going forward. And as a result of that, it does raise some questions about different use cases. So we talked about earlier when a company acquires another software company, they do need to check for compliance to these open source licenses. But also your product team should be checking this before they release code.

And there’s all sorts of good ways to enforce that teams do check, so enforcing license checks like that at build time before you release software is one of those great opportunities to ensure you don’t fall into the cycle of having to catch up later after your customers find out the software could be in violation.

[Paul] (22:35)

So, Salim, for those of our listeners who aren’t developers themselves, do want to say a little bit about CI/CD? That’s continuous integration, continuous deployment or continuous delivery. And the idea of a CI/CD pipeline and how that influences the way that most people build software these days.

[Salim] (22:59)

Sure. At a high level, CI/CD pipeline is allowing you to say, I think this version of software is ready to go to a customer. Generally, what people want their pipelines to do is test the software, build it in a final form, and deliver it where it needs to go. Before you deliver your software anywhere, that is the best time to make sure you haven’t introduced additional vulnerabilities. haven’t introduced additional license problems. Do that in your pipeline when you’re saying, Hey, I think this neat little widget is going to be something cool that customers will like. Let me pull it in from this open source library. Yep. It works as I expected. Let’s ship it. Come to find out that was a GPL widget that somebody built in their spare time that they feel very strongly about. They are able to determine that you consumed that, that software there is, and they come after you and that cool widget. If you did not have a license check in place, your pipeline is now downstream and it’s too late.

[Paul] (24:01)

Now the continuous delivery side, I guess that’s much more relevant for things like web apps, where you can update them every afternoon if you want, than it is with embedded devices where you may deliberately want to minimize the number of times that you have to do an update because it’s harder and takes a lot longer to apply them for small devices which are distributed globally. So what are the emerging tools or practices other than just saying CI/CD that companies that make and sell embedded devices ought to be looking out for if they haven’t already crossed this licensing correctness bridge?

[Salim] (24:48)

We’ve talked about license enforcement. We’ve talked about vulnerability management. That’s important. But to reiterate a point that I think we’ve made throughout for embedded software, it is more critical than otherwise to catch things early. A twist on that, though, is what can you integrate early that will protect you later? So, yeah, if you can integrate a license compliance check, if you can integrate a vulnerability check, great. But what else can you do to protect yourself later. There are embedded RASP solutions where you can introduce security at your build that once you’ve shipped it, even if a vulnerability is detected, you can still know that you are protected from that vulnerability.

[Paul] (25:32)

Joe, do you just want to say what is meant by that term, RASP?

[Joe] (25:36)

RASP stands for runtime application self-protection. And there’s different variations of it. From the RunSafe perspective, what we consider a great runtime protection is the ability to ensure that you can prevent exploitation at runtime by implementing RunSafe memory safety protections. And in that case, what we do is we relocate where functions load into memory uniquely every time the software loads, making it very difficult, if not nearly impossible, for the attacker to know where a weakness or a vulnerability is in the first place.

[Paul] (26:14)

So the idea there is that it is an active defense that happens every time the software runs, rather than merely a build-time precaution that makes sure you don’t put the wrong stuff in there.

[Joe] (26:27)

It is an active defense. I think ultimately the biggest difference RunSafe brings to the table is other organizations would have put some other kind of software agent or wrapper around a solution to create that runtime defense. With RunSafe, you can achieve the same outcome without adding any new software on the target system.

[Paul] (26:48)

So presumably the idea there is that given that it often is very difficult and time consuming to update embedded devices, you don’t want the situation that if some modest vulnerability appears that would otherwise mean there’s little you can do except to push out patches possibly to the chagrin of your customers, that you may be able to say, look, okay, proactively protected against that if we only patch it later on when we do a whole load of other changes, we can show you that that is good enough.

[Joe] (27:23)

And imagine the value if you can generate a Software Bill of Materials, ensure license checks are made before you distribute software, and you can protect against vulnerabilities, whether a patch is available or not, all at the same time, leveraging RunSafe. So that’s my little plug for RunSafe this week. You are able to do all of those things with RunSafe Security.

[Salim] (27:47)

The whole point is to do it early.

[Paul] (27:49)

It sounds like Mr. Miyagi in the Karate Kid. Best way to avoid punch, no be there. It is very hard to fix these things retrospectively. So as far as you can, you want to make sure that you’re compliant right from the get-go. Even if that adds a tiny little bit of complexity and time to that build and ship process that you have. It will work well for licensing, and as Joe says, it will also help you with knowing what vulnerabilities you and your customers are exposed to in the future. Because if you can’t answer the licensing questions, you probably can’t answer the vulnerability questions either, can you?

[Salim] (28:27)

I think that captures it really well. You can’t put yourself in that position to find out.

[Paul] (28:31)

There isn’t a magic bullet that can just fix these things retrospectively. So with that, let me just say thank you so much, Joe and Salim, for talking about what is a thorny problem that is actually very critical in software engineering, even though it’s legal rather than technical. So thank you so much for your thoughtful insights. Thanks to everybody who tuned in and listened. That is a wrap for this episode of Exploited: The Cyber Truth.

If you enjoy this podcast, please don’t forget to subscribe so you know when each new episode drops. Please like and share us on social media too. And don’t forget to tell everyone in your team about us. Once again, thanks for listening. And remember, stay ahead of the threat. See you next time!