Exploited: The Cyber Truth, a podcast by RunSafe Security. 

[Paul] Welcome back to Exploited: The Cyber Truth. I am Paul Ducklin, and today, I’m joined by Joe Saunders, CEO and founder of RunSafe Security. 

[Paul] Hello, Joe. 

[Joe] Hey, Paul.

[Paul] It’s great to be back. Joe, this week, our topic is US telecoms under fire, implications of the Salt Typhoon campaign. So let’s get straight into it with a short recap because, last week, we talked about the Volt Typhoon threat group. The family name Typhoon is Microsoft’s meteorological nickname, if you like, that means it is associated with China. And just in case anyone’s worrying, all the perpetrator nicknames these days are bad weather conditions, so you won’t find a sunshine threat group or a gentle breeze.

[Paul] But also like last week when we had the Volt part of the name not implying that they were focusing on electricity companies, so the “Salt” in Salt Typhoon is not down to their predilection for hacking into food companies or the chemical industry. In fact, their primary notoriety is that we now know that they ended up all over the place in very many, very large telephone companies, primarily in the US. And these attacks go back at least two years. So, Joe, how did we come to this? 

[Joe] I think it’s a continuing theme that China plays a long game in its cyber espionage and cyber attacks.

[Joe] And they do it with purpose. They do it with strategy. They do it with well funded actor groups. And Salt Typhoon is another example of that is working overtime to achieve intelligence objectives and advance the cause of PRC in general. So I think this is another example of a well funded nation state looking at ways to influence the world, and the way to influence and understand what’s happening, and inserting themselves into intelligence collection, and this is another form of that kind of action.

[Paul] Yes. Because although this was a very much a get in and lie low sort of campaign, given how long it went on and how much information was collected, it wasn’t super carefully targeted or minimized, was it? There were hundreds of thousands of routers that were co-opted into the attack. And last week, we spoke about bugs in Ivanti products that were supposed to be there for security but let the attackers in. This time seems mainly to have been Cisco products with a pair of vulnerabilities that go back to at least 2023 when they were found and patched.

[Paul] The first one of those meant that the attackers didn’t even need to figure out passwords or buy in passwords from the cyber criminal underworld. This bug basically allowed them to create their own user with their own password on hundreds of thousands of devices at the same time. So what do you do about that? 

[Joe] Well, obviously, that’s a pernicious problem to solve. But when you have such broad adoption of software, when a persistent attacker gets on device, it is a very pernicious problem, especially one that’s well suited to stay on device and evade detection.

[Joe] And I think part of the reason it really came to bear last fall when the US government started to report this, and that’s the fall of 2024. I think there were concerns that this actor group, which is a large team, maybe multiple teams working globally, had focused on some, I would consider, high profile people, people involved in the presidential campaign, whether it’s staffers or others. And so when the US government finds out that there’s people listening in to either conversations or gaining access to messages, then it becomes a national security issue. And then when you start to peel back the onion to say that everybody’s exposed to it, that it doesn’t necessarily discriminate who could be targeted, it really creates concerns. It creates concerns that how does a threat actor like Salt Typhoon get such widespread access?

[Joe] When this first came out, I think part of the issue was that collecting this information and having access to possible insights from political leaders meant that it was a national security issue. And then when we didn’t want to necessarily disclose in the US the extent of the problem, it became a bigger and bigger issue, and suddenly we realized everyone was affected. So it’s caught up to us, because the actor was able to evade detection for a long time. And then when it was found out and we peeled back the onion, we were like we thought, oh my gosh. How did it come to this?

[Joe] And that’s how it is. It’s a well funded actor who thinks long term and knows when to strike when there’s an opportunity to get sense of information.

[Paul] So Joe, do we know or can we guess how many people would be in a team like Salt Typhoon? Do they form that team and work in that specific team for years? Or do some of the operatives move around so that they might do some Volt Typhoon work this month and some Salt Typhoon next month? Or is it more a case of all of the above? 

[Joe] Yeah. I think it’s all of the above, but I would say the primary thing to think about is this is a team that’s working globally, not just in The United States. Other parts of the world are affected. Multiple telecom organizations are affected.

[Joe] Five, ten, 20 telecom companies, don’t quite know the exact number, but let’s say at least 10, major ones. In multiple geographies around the world, that’s not a small team. That’s a team that is organized to achieve objectives in multiple different, like, subprojects within the overall effort. Right?

[Joe] So I’m thinking dozens of people on the team. In some cases, it can take six months for a small team to even get on one device inside one system. We’re talking 10 telecoms. We’re talking global in nature. We’re talking evading detection for multiple years. This is a well funded team that knows how to operate. So it’s a substantial team with China as the ultimate organization behind it. 

[Paul] Indeed, I was reading through a report earlier where the author of the report had, in my opinion, somewhat redundantly written, Salt Typhoon’s primary targets have been within The United States, Southeast Asia, and Africa, which is sort of a bit like saying, well, they’ve mainly focused on the planet Earth, but who knows where else they’ve been? It’s dug in deep, spread wide, and stay there quietly and effectively for as long as you possibly can. 

[Joe] Exactly right.

[Joe] When I think of China’s cyber operations, The US is outnumbered pretty significantly 50 to one. With a well funded, persistent focus, China has an advantage. They think long term. They think over decades, not just what can I figure out next week to get one device and then and then burn the system? Right?

[Joe] So if you’re in intelligence operations like we think Salt Typhoon must be, a part of their foreign intelligence service in China is trying to collect information over time, not just strike once and have an impact. Now we do know China does have a desire at some point at a time of their choosing to disrupt critical infrastructure from operating, let’s say, delivering water or delivering energy. But in this case, this is an intelligence collection operation. Salt Typhoon does not want to get detected. They want to evade detection.

And what’s ironic about this is that they were able to do this inside existing processes that even the US might want to know what’s going on with certain phone calls. So, China was able to turn this on its head and do some damage in their own way. 

[Paul] I guess there you are alluding to a US law called CALEA, the Communications Assistance for Law Enforcement Act, Lawful interception, as it’s generally known in Western democracies, where if you want to snoop on someone you can, but you go and get a warrant, and then there are provisions whereby you can record what they say in future calls and store that, and keep it for later either to gather intelligence or because you might need it in a lawsuit. In this case, the Salt Typhoon actors didn’t have to actually do the snooping. They basically snooped on the snoopers and stole data that had already been collected.

[Paul] They got hold of transcripts of important telephone conversations without needing to go near any mobile phones or the actual physical telephone network. 

[Joe] Absolutely right. And I think an important concept from a cybersecurity perspective is if you do create processes that enable backdoor entry, if you will, you have to think about how can a bad actor turn that process into their advantage. And that’s exactly what I think happened in this case. So lawful intercept, obviously, has been around for decades.

[Joe] We have, in The United States, a structured process as you described. And if you have demonstrated proof, you can go to a court and seek that warrant and then start to listen in. So that’s a structured process that’s meant to defend the US, and not to violate people’s privacy, but to look for threats that are legitimately identified. With that said, China has flipped that on its head in this case, leveraging the lawful in intercept concept and using it for its own means where there is no court to decide, do you have legitimate purpose to to listen in? So when you create these opportunities to enhance things from a law enforcement perspective or a counterterrorism perspective, you have to think about how someone might leverage that themselves.

[Joe] And in this case, I think ultimately when we get to how do you solve this problem, there are some technical things in that arena that we can do. The real trick here then for them is their technical prowess to leverage all that and evade detection. And so I think Salt Typhoon was well equipped to kinda hide and persist on devices and really figure out when it was to their advantage to listen in and get information they wanted. 

[Paul] In fact, it seems that in this case, as in the case of Volt Typhoon, as we discussed in the last podcast, the initial access via buggy Cisco routers was again one of those let’s chain two exploits back to back. My understanding is that the first bug was one that simply let you become a regular user, which is bad, but not super bad.

[Paul] And then immediately, having knitted their own password, they could then exploit the second bug, which was an old school command injection vulnerability, where you basically trick the device into running a command you chose, get hold of root access, implant your malware, and you don’t need to go in via the vulnerabilities anymore, because you’ve left a secret remote backdoor of your own. It’s very hard to fix that when those devices are essentially built with air quotes, no user serviceable parts inside. 

[Joe] 100% right. I mean, the complexity of this goes way up. When you describe it the way you do, you think about, okay, what’s the soft underbelly of technology?

[Joe] And it really is finding those technologies that are pervasive, they’re used in multiple contexts, and in areas, you know, like routers, where if you gain access, you can really do some key listening in. With that said, it’s not a simple step to say, oh, there’s one easy to exploit entry. In this case, there are many tools involved to ultimately get access to the Windows kernel. When CISA first came out, they came out with a laundry list of everything an enterprise, a telecom organization, or a product manufacturer has to do to slow down Salt Typhoon. The problem with that is we all want a simple answer to say, hey do x and you’re good. The trick is you’re working in all these different environments at different levels, the consumer level with the router in the house, routers inside Internet service providers, Windows applications or software inside telecom infrastructure in general. Which hole do you plug, and who plugs it, and how do you do that? So it’s a complex problem. And in this case, Salt Typhoon probably used 10, 12, 15 different tools throughout the process to gain access in different ways, and I’m sure we’ll talk about demo decks and other things.

[Joe] My thought is there are many tools used to gain access. And to your point, once they gain access, they’re sort of sitting there waiting to figure out when to operate and what to collect from there. 

[Paul] Absolutely. And if you have a hundred thousand compromised routers at your disposal, then it’s not a question that if somebody actually disconnects or replaces or patches one of them, that your access goes away. You’ve got 99,999 still to go.

[Paul] And since you mentioned Demodex, I would actually like to go there now. That was one of the Windows components that I believe Salt Typhoon was known for using. That name comes from Demodex, which is a skin mite. They burrow into your hair follicles and live off you parasitically, where you can’t easily see them or get rid of them. This was a Windows rootkit, malware that hides the presence of other malware. In other words, it goes out of its way to make sure that whatever you’re doing is even harder than ever to detect. And my understanding is that the Salt Typhoon actors actually used a video game cheat engine, because obviously video games, there’s a lot of money in that these days. So they went to what you might call the cyber underworld, got hold of a freely circulating tool that video game hackers used, and figured, hey, this is a great place to start. So, while your regular cyber criminals probably don’t know or use everything the state sponsored actors are using, You should certainly assume that everything you’ve ever heard of, read about, or seen in things like ransomware gangs that are just in it for the money, all of that technology is well known to the state sponsored groups. So they got all of that, plus their own secret sauce to go with it.

[Joe] 100% right. And I love the origin of the words, in the research you’ve done on Demodex. I think that implies the goal of those root kits is to gain persistent access and evade detection. And in this case, I think what they were trying to do was collect information, even metadata, so that they would know who’s talking to who or who’s communicating with whom. And I do like the idea of talking about Cheat Engine in this case.

[Joe] A lot of innovation happens in the cyber world, in the video gaming area, and those two areas really cross over nicely. And with that said, Cheat Engine then becomes a useful tool. Someone has to be creative enough and persistent enough to take a tool from it that’s used in one way to then reapply it for communications in a completely different domain. 

[Paul] And as you mentioned last week when we were talking about Volt Typhoon, one of the other deals with Demodex is it tries to do everything in memory. So it’s not sitting around on disk where you might trip over it and go, oh, dear, that shouldn’t be there. The bit of the kernel that is supposed to protect the kernel from being subverted is itself subverted, evading the detection of the operating system, of endpoint detection response software. So like you say, there are a lot of moving parts here. And they’re not just moving parts, they all, unfortunately, mesh together in a gearbox of evil, don’t they? 

[Joe] And when you look at it from a nation state perspective, it’s a complex enterprise with sophisticated technical expertise operating a bunch of tools in a way that avoids detection. And, of course, these fileless attacks resulting in compromising memory is generally a big problem in all embedded systems. It’s not unique to telecom. It’s true in all areas of critical infrastructure. And I think that’s why, you know, the US the NSA issued guidance on achieving memory safety, preventing different forms of memory based attacks and techniques used to compromise memory. The NSA issued guidance going back to 2022, November of 2022.

[Joe] And, ultimately, there is, what do I do specifically for Salt Typhoon and the myriad of tools that they’re using, which systems have to issue patches to to which devices, where, you know, critical infrastructure comes into play and where, you know, national interest comes into play and where business interest comes into play. We do need, as a nation and as a society, ways to attack these problems systemically, not just chasing them after we find out what they’re doing and trying to patch them. I do think that we have to address the systemic nature of the problem. And certainly, CISA, over time, has tried to identify memory safety as a core issue to solve for because it represents about 70% of vulnerabilities that do get exploited in critical infrastructure. And CISA has tried to bring education and awareness to that through its Secure by Design programs, has done a good job of that. And also, I think it all really got focus from the NSA in November of 2022 to solve this problem more generally.

[Paul] Joe, before we get on to how we normally like to end the podcast, which is a, you know, what to do, what next, I would like you to say a little bit about something you alluded to earlier, and that is metadata. Now metadata is a fancy way of saying data about data. And in the context of telephone calls, and messages, and location information, It is, if you like, who called whom, from where, at what time, without knowing a single word that they said. 

[Joe] Why is the metadata so useful? It is to connect the dots. It’s also used in the identifying which files are important, going forward. And so if you can start to develop patterns, and really identify individuals, And so there’s many reasons that that metadata is useful. It’s almost like getting the blueprint of what’s going on. 

[Paul] The way I like to put it is that when it comes to cybersecurity, a tiny bit of inconvenience can go an awful long way. It just gets you into a habit of putting small barriers up everywhere, which in many cases are better than having a giant barrier in one place that you can just walk around because it’s not protected on the side.

[Paul] A good example is that on Apple iPhones, you have a lock code to let you into the phone. But when you want to go into the passwords app to recover an MFA code, even if your phone’s already unlocked, you have to unlock it again. And that just means that if someone does steal your phone, there’s a lot they can do with it, but they can’t automatically and instantly just do everything. And sometimes, those small barriers can have a very big impact if everyone takes them seriously. 

[Joe] 100%. And I think in general, for an enterprise, Defense in depth is key. And so, if you think about the systems that operate, we wanna make sure that people don’t get access, but they can get access at many points. They can get access in the supply chain. They can get access into the build systems. They can then get access into over the air updates. They can gain access inside the infrastructure. They might have an insider. They might have a different entry point and get on other systems and then move across networks. And so I think one of the truths in cyber defense is that there will be ways that bad actors get on devices. And ultimately, we need those kinds of, defense in-depth measures that prevent their ability from executing whatever they wish to try to accomplish.

[Joe] Defense-in-depth in an embedded system then, we want to, you know, make sure that we have assurance that the software that the manufacturer ships is the software that loads on a device. We want to make sure that that software that loads on a device only has the components that were originally intended. And then when that software is running on a device, you want to shore up the memory and protect the memory so that runtime attacks can’t be perpetuated or propagated across systems without detection. So there’s many factors to cyber defense in general, and that doesn’t mean we don’t protect our networks and screen our systems and our people. Of course, we still need to do all of those things.

[Joe] It’s as much a software supply chain issue from source to run time as it is a network defense, and the integration thereof is what ultimately consumers need to be or buyers of technology need to be looking for from their providers. 

[Paul] So, Joe, if we crystallize those what to do answers now, just to conclude, it sounds as though really what you’re saying, if I can use a rather old fashioned way of describing it, is that sometimes even if we’ve spent good money on something that we thought would last for years, we just have to vote with our checkbooks. And if there’s a device where the vendor seems unwilling to patch or where there seem to be continual vulnerabilities that keep appearing, it may be necessary to actually get rid of a device, or get rid of a class of device, or get rid of a particular piece of software that’s in your supply chain, and replace it with something which is measurably better. 

[Joe] Right. And I think doing that in a systematic way. And what I mean by that is the full discipline of your software development process needs to incorporate Secure by Design methods and integrate cyber protections throughout the process. You build it in. You don’t just wait for it to be defended at the network downstream where the software gets deployed. So, ultimately, I would say it’s not about one specific patch or one specific vulnerability. It’s about the overall process.

[Joe] Sure. When we do find a weakness, then we have to patch it. But systemically, we need to educate and elevate our software development processes so they incorporate a comprehensive set of security practices into the process. And that includes protecting memory on devices and incorporating protections in there, but it also includes proper testing, proper patching, proper disclosing of vulnerabilities. And I’m an advocate of disclose as quickly as possible when something’s found because that helps elevate everybody to understand what the vulnerability is.

[Joe] And, you know, with that comes a very disciplined software development process so you can react when vulnerabilities are found down the road. 

[Paul] And as you said last week, the forthcoming Cyber Resilience Act, CRA, in the European Union is definitely something for people all around the world to watch. Because that is a sort of combination carrot and stick, isn’t it? It may change our attitudes to liability that means that we’ll get security by design, but also security by demand. People will be saying, I won’t buy your stuff unless it fulfills the design correctness.

[Joe] Absolutely right. And I think the Cyber Resilience Act is a great topic, so I look forward to talking about that in detail. 

[Paul] Hint, hint, folks. It will be coming up in a future episode of the podcast. So thank you so much for your time, Joe.

[Paul] Your passion flowing forth as usual, and thanks to everybody who tuned in. That’s a wrap for this episode of Exploited: the Cyber Truth. If you found this podcast insightful, please be sure to subscribe and share it with everyone else in your team. Stay ahead of the threat. See you next time.