Deep Dive into XZ Utils Backdoor [video](youtube.com)
youtube.com
Deep Dive into XZ Utils Backdoor [video]
https://www.youtube.com/watch?v=Q6ovtLdSbEA
101 comments
>nation-state-backed
Is that actually known?
Is that actually known?
From what I have read so far: speculative, at best.
It’s all based on the sophistication, the level of psychological manipulation, years of effort, multiple actors, and a working theory based on commit timings.
It’s all based on the sophistication, the level of psychological manipulation, years of effort, multiple actors, and a working theory based on commit timings.
Generally speaking, with attacks of this sophistication, if it's not a state-backed APT, it's someone who operates with the unofficial backing of a government to give plausible deniability.
I suppose it could be a few amateurs who took on a passion project, but... it's unlikely.
I suppose it could be a few amateurs who took on a passion project, but... it's unlikely.
Could be someone at a company that hacks into stuff and sells it to governments too. They're always looking for backdoors and I assume would do stuff like this.
So do we know what the cause of the 500 ms slowdown was?
It seems crazy to me that they spend so much time keeping this hidden and then never check for performance degradation.
It seems crazy to me that they spend so much time keeping this hidden and then never check for performance degradation.
AIUI, it was the dynamic loader audit hook checking symbol names against strings stored in a trie as a way to hide them. Tons of symbols, and slow process for each one.
AIUI?
As I Understand It
As I Understand I ? (First time i seethis abbreviation)
My working theory is they got sloppy. A patch was ready to be introduced into Linux to prevent this type of attack (dynamically loading unneeded libraries).
They were rushing to get it out into the wild and thus made many mistakes, including the performance degradation.
From what I recall, some distros were reporting build errors in bleeding edge versions. Actors were essentially debugging issues in production, and recommending to distro maintainers ways to “fix” (really hide) the build issue with xz
They were rushing to get it out into the wild and thus made many mistakes, including the performance degradation.
From what I recall, some distros were reporting build errors in bleeding edge versions. Actors were essentially debugging issues in production, and recommending to distro maintainers ways to “fix” (really hide) the build issue with xz
I've worked on long projects for a while, you can be relatively invisible for a year and let them know you're getting close (meaning a couple months) and the management will freak out and want it yesterday if it's near the end of a quarter and they think it will look good on quarterly reports. I assume the same can happen with government actors and "security companies".
The postgres developer had disabled a bunch of optimization in order to create a low noise floor on his test machine that increased the delay too.
Normally it would be much quicker but potentially could have been picked up downstream by other developers when the code became more mainstream.
He did an interview for Risky Business #743.
Normally it would be much quicker but potentially could have been picked up downstream by other developers when the code became more mainstream.
He did an interview for Risky Business #743.
Amazing lecture. Really awesome to hear more about the nitty gritty of this whole debacle.
I wonder if the certificate has shown up anywhere.
looks like this is now the next-stuxnet
More like a wannabe stuxnet.
Whoever was trying to get this into the wild was really sloppy in execution of an otherwise “beautiful” attack.
Stuxnet effectively put a stopgap in Iranian’s nuclear program. This backdoor could barely make it out of the alpha phase of development.
Whoever was trying to get this into the wild was really sloppy in execution of an otherwise “beautiful” attack.
Stuxnet effectively put a stopgap in Iranian’s nuclear program. This backdoor could barely make it out of the alpha phase of development.
They may have been rushed. Elsewhere itt:
> A patch was ready to be introduced into Linux to prevent this type of attack (dynamically loading unneeded libraries).
> A patch was ready to be introduced into Linux to prevent this type of attack (dynamically loading unneeded libraries).
[deleted]
A reasonable person would expect many more open source projects have been infiltrated and have similar exploits waiting to be found. Hard to prove until the next one is found, but if a second one like this is discovered we probably have a huge problem on our hands.
Since the part that takes longest is getting commit rights and there's a lot of waiting to do, what are the chances the attackers put all their eggs on infiltrating one specific project? I'd be trying to infiltrate at least 10 or 20 even if I were alone, to increase my chances. If you dedicate one or two years to build reputation with the current maintainers by doing a PR here and there, you can do that for a few projects in parallel.
Since the part that takes longest is getting commit rights and there's a lot of waiting to do, what are the chances the attackers put all their eggs on infiltrating one specific project? I'd be trying to infiltrate at least 10 or 20 even if I were alone, to increase my chances. If you dedicate one or two years to build reputation with the current maintainers by doing a PR here and there, you can do that for a few projects in parallel.
Assuming this would be smart to do. But xz is a very good target. It seems a lot of care went into selecting it. Consider:
* It is extremely critical. Used at multiple places in the boot process, with root or kernel mode
* Nevertheless an unnoticeable dependency. Nobody thougt about its security implications
* Stable and trustworthy for years (decades?)
* Managed mostly by 1 person
* That person was more than average vulnerable for pressure, but not in a very visible way.
* It is extremely critical. Used at multiple places in the boot process, with root or kernel mode
* Nevertheless an unnoticeable dependency. Nobody thougt about its security implications
* Stable and trustworthy for years (decades?)
* Managed mostly by 1 person
* That person was more than average vulnerable for pressure, but not in a very visible way.
I'm not saying it was a bad target. It was a perfect target. How many more would fullfil these criteria? Those would be the type of repositories I'd go and investigate.
For there to be another repository affected by the same attack you don't even need to think this specific attacker did it twice. You can just consider that multiple attackers had the same idea. I think it's way more likely that there's more repositories affected by the same type of attack than not.
I'd go as far as saying this is an issue of national / global security and there's space for an agency like the IMF but for software security that would be funded by all the countries of the world to bring more resources and assuredness to this type of dependencies.
Because even if this attacker didn't do more than N=1 and other attackers didn't have the idea before, they surely have it now.
For there to be another repository affected by the same attack you don't even need to think this specific attacker did it twice. You can just consider that multiple attackers had the same idea. I think it's way more likely that there's more repositories affected by the same type of attack than not.
I'd go as far as saying this is an issue of national / global security and there's space for an agency like the IMF but for software security that would be funded by all the countries of the world to bring more resources and assuredness to this type of dependencies.
Because even if this attacker didn't do more than N=1 and other attackers didn't have the idea before, they surely have it now.
About "there's space for an agency like the IMF but for software security"
See Tim Bray's proposal about "OSQI": https://www.tbray.org/ongoing/When/202x/2024/04/01/OSQI
See Tim Bray's proposal about "OSQI": https://www.tbray.org/ongoing/When/202x/2024/04/01/OSQI
I would add to that list:
* Has random binary test files in its repo, ideal for hiding exploit payloads.
* Has random binary test files in its repo, ideal for hiding exploit payloads.
Yes, this has happened. See this example from OpenJSF:
"The OpenJS Foundation Cross Project Council received a suspicious series of emails with similar messages, bearing different names and overlapping GitHub-associated emails. These emails implored OpenJS to take action to update one of its popular JavaScript projects to “address any critical vulnerabilities,” yet cited no specifics" [0,1].
[0] https://openjsf.org/blog/openssf-openjs-alert-social-enginee... [1] https://www.schneier.com/blog/archives/2024/04/other-attempt...
"The OpenJS Foundation Cross Project Council received a suspicious series of emails with similar messages, bearing different names and overlapping GitHub-associated emails. These emails implored OpenJS to take action to update one of its popular JavaScript projects to “address any critical vulnerabilities,” yet cited no specifics" [0,1].
[0] https://openjsf.org/blog/openssf-openjs-alert-social-enginee... [1] https://www.schneier.com/blog/archives/2024/04/other-attempt...
I watched the TC interview with Durov (Telegram) and apart from it being a big Telegram ad what struck me is that he told a story about his employee being approached by "an intelligence agency" and asked to reveal information about what open-source libraries they use.
It is kind of strange since their apps are supposed to be open source, maybe he meant the backend? Nonetheless, it seems this has been their modus operandi for a long time.
It is kind of strange since their apps are supposed to be open source, maybe he meant the backend? Nonetheless, it seems this has been their modus operandi for a long time.
Remember that NSA is openly interested in systemd and how it works. It's a double edged sword. They wanted to be sure that it's hardened as they like it, and note any "useful features" that might come handy later.
The thing is, as computers proliferate and we start to use them in more places, the effects of possible holes moves closer to our homes. From distant infra to near infra; from borderlines to our homes and transportation we use everyday. Even to our pockets via smartphones and other smart devices we host in our homes.
The thing is, as computers proliferate and we start to use them in more places, the effects of possible holes moves closer to our homes. From distant infra to near infra; from borderlines to our homes and transportation we use everyday. Even to our pockets via smartphones and other smart devices we host in our homes.
Hey, got a link handy for this claim?
Thanks
Thanks
Rabbit hole starts with [0], and goes to [1], which arrives to [2].
Stephen Smalley in question works at NSA [3].
[0]: https://news.ycombinator.com/item?id=9863896
[1]: https://www.phoronix.com/news/NSA-KDBUS-Credentials
[2]: https://lkml.iu.edu/hypermail/linux/kernel/1507.1/01758.html
[3]: https://www.linkedin.com/in/stephen-smalley
Stephen Smalley in question works at NSA [3].
[0]: https://news.ycombinator.com/item?id=9863896
[1]: https://www.phoronix.com/news/NSA-KDBUS-Credentials
[2]: https://lkml.iu.edu/hypermail/linux/kernel/1507.1/01758.html
[3]: https://www.linkedin.com/in/stephen-smalley
I would start with projects where the maintainer(s) suddenly got a very helpful contributor. I am sure there's a typical pattern. Then if a maintainer seems suspect investigate a little bit more. Here some balance is needed because probably many maintainers are not Mallory.
[deleted]
[deleted]
I would have paid money to see the look on the team’s face behind this back door as they realize the years of effort and planning go down the drain because of 1 person doing due diligence.
Speaking of that. Any updates on finding the person(s) behind this attack?
CISA (https://cisa.gov) are investigating. They got the emails between me and "Jia Tan" and other information we have. They can subpoena Google to get IP addresses behind the emails, but it depends on whether and where a VPN was used if they'll get any further than that. If we will ever see a fully public report is anyone's guess.
I doubt much will end up public, as most vpns have a zero retention policy, so that leaves the NSA being able to track things at a lower level.
If it's a state actor, it's not unreasonable to think they'd spin up their own VPN
All you need a Raspberry Pi and some public Wi-Fi network to create a jump point and hide among the 100s of devices going on and off that one public IP. With projects like TailScale you could set it up and plant it somewhere in a matter of hours
All you need a Raspberry Pi and some public Wi-Fi network to create a jump point and hide among the 100s of devices going on and off that one public IP. With projects like TailScale you could set it up and plant it somewhere in a matter of hours
If it is a state actor like China or North Korea, they own the gateways and firewalls and can ensure any traffic they do not want to be identified ever will be.
[deleted]
> most vpns have a zero retention policy
I thought most VPNs sold usage data to ad companies?
I thought most VPNs sold usage data to ad companies?
You would think they’d choose a good one.
I also recently wrote a single ephemeral socks5 proxy over a hidden service in Rust. Since they’ve probably compromised other machines in the past, they could’ve easily used something similar to proxy their connection through tor and to some random computer (access some vulnerable router through tor, proxy through it, etc). Hiding your tracks aren’t hard
I also recently wrote a single ephemeral socks5 proxy over a hidden service in Rust. Since they’ve probably compromised other machines in the past, they could’ve easily used something similar to proxy their connection through tor and to some random computer (access some vulnerable router through tor, proxy through it, etc). Hiding your tracks aren’t hard
I think we can safely assume they did not use one of those.
Nothing but speculation so far based on the times they were active. No real signatures left behind.
All we know is they were active during business hours of the UTC+1/3 timezones. They used a chinese sounding name as one of their front figures, the one who made most of the commits of malicious code. They wrote english messages that some people claim appear to be a Russian trying to write in English. But that last argument is pretty weak as they could have just masked their origin using Google translate, or even ChatGPT.
One of the biggest clues is actually one little commit that they let slip with a middle name to the chinese persona. And some people claim the names don't match up with different chinese cultures.
After that you can use all your psych 101 experience to try and analyze them but you'll just be guessing.
Personally I think using a chinese front name is a very interesting detail. They could have used any name in the world, there are open source contributors from all over the world.
Using a name in a country where most people have a very little insight is a good way of not being uncovered as fake of course. But why not use a Ugandan name? Also using a Chinese name can be perceived as pointing the blame at China, which might be sensitive geopolitically.
All we know is they were active during business hours of the UTC+1/3 timezones. They used a chinese sounding name as one of their front figures, the one who made most of the commits of malicious code. They wrote english messages that some people claim appear to be a Russian trying to write in English. But that last argument is pretty weak as they could have just masked their origin using Google translate, or even ChatGPT.
One of the biggest clues is actually one little commit that they let slip with a middle name to the chinese persona. And some people claim the names don't match up with different chinese cultures.
After that you can use all your psych 101 experience to try and analyze them but you'll just be guessing.
Personally I think using a chinese front name is a very interesting detail. They could have used any name in the world, there are open source contributors from all over the world.
Using a name in a country where most people have a very little insight is a good way of not being uncovered as fake of course. But why not use a Ugandan name? Also using a Chinese name can be perceived as pointing the blame at China, which might be sensitive geopolitically.
They also used a bunch of other fake names (Jigar Kumar, Dennis Ens) for the personas that put social pressure on the original maintainer. And the ifunc changes were done by one "Hans Jansen" who also later opened merge request to Debian. These are fairly eclectic names with no pattern, which was the point I guess. We don't know what level of misdirection is going on with the names, so "Jia Tan" is interesting but not particularly meaningful either way.
I would argue that Hans Jansen and Dennis Ens (and Jigar Kumar? I don't know about Indian perceptions of names) are a a bit off as well. They contain rhymes that make them sound slightly silly. Most parents would want to avoid these. Maybe chosen to avoid matching real people while still seeming commonplace.
So these names seem somewhat similar to "Jia Tan" in their almost-but-not-quite real quality.
So these names seem somewhat similar to "Jia Tan" in their almost-but-not-quite real quality.
Jia Cheong Tan is anagram for CIA Agent John. If you consider how much went into planning the backdoor operation, it seems virtually certain it's an internal joke.
And why an Asian name? It's certainly exploiting a psychological bias. Apart from what you noted, the names are much more generic than Ugandan names and IMO it's virtually impossible to track a legend down. And the number of OSS contributors from East Asia is much larger than from Africa. Hence a more normalized/frequent occurrence. And lastly if things start going south, some play on the r-card is always up the sleeve of PsyOps personnel.
And why an Asian name? It's certainly exploiting a psychological bias. Apart from what you noted, the names are much more generic than Ugandan names and IMO it's virtually impossible to track a legend down. And the number of OSS contributors from East Asia is much larger than from Africa. Hence a more normalized/frequent occurrence. And lastly if things start going south, some play on the r-card is always up the sleeve of PsyOps personnel.
Nice try, but you’ve decoded the alias incorrectly. Jia Cheong Tan is distinctly an anagram of Gotcha in Jean, which points to a French operation.
Nice try, but you've decoded ze alias incorrectly. Jia Cheong Tan is distinctly a anagram from Tanga Jochen I, which shows to a German operation.
A Chinese name is consistent with the timezone - alternatively Australia, Russia etc.
Here is my speculation.
I would look at the test framework - the attacker needs the complexity of it to hide away the attack and it is first thing he publishes - probably has been planned from the very beginning.
This is a homemade thing based on something called "Seatest".
How common is Seatest? Not that common as far as I can tell - maybe someone can correct me?
Here is my speculation.
I would look at the test framework - the attacker needs the complexity of it to hide away the attack and it is first thing he publishes - probably has been planned from the very beginning.
This is a homemade thing based on something called "Seatest".
How common is Seatest? Not that common as far as I can tell - maybe someone can correct me?
UTC+1/3 is NOT consistent with China's timezone.
I was referring to the time zone of +8 used in the git commits mentioned here:
https://www.wired.com/story/jia-tan-xz-backdoor
The Mystery of ‘Jia Tan,’ the XZ Backdoor Mastermind
The thwarted XZ Utils supply chain attack was years in the making. Now, clues suggest nation-state hackers were behind the persona that inserted the malicious code.
...
At a glance, Jia Tan certainly looks East Asian—or is meant to. The time zone of Jia Tan’s commits are UTC+8: That’s China’s time zone, and only an hour off from North Korea’s.
https://www.wired.com/story/jia-tan-xz-backdoor
The Mystery of ‘Jia Tan,’ the XZ Backdoor Mastermind
The thwarted XZ Utils supply chain attack was years in the making. Now, clues suggest nation-state hackers were behind the persona that inserted the malicious code.
...
At a glance, Jia Tan certainly looks East Asian—or is meant to. The time zone of Jia Tan’s commits are UTC+8: That’s China’s time zone, and only an hour off from North Korea’s.
> or is meant to.
Indeed. If it's nation state and years in the making then the idea of all the commit times being faked isn't a stretch.
Hell, even five-eyes (UK+AU+US+etc) could seek a linux backdoor and use Perth, Australia (GMT+8) timezones or have a submission bot in the UK that just commits at preset time from preset IP proxy.
Indeed. If it's nation state and years in the making then the idea of all the commit times being faked isn't a stretch.
Hell, even five-eyes (UK+AU+US+etc) could seek a linux backdoor and use Perth, Australia (GMT+8) timezones or have a submission bot in the UK that just commits at preset time from preset IP proxy.
I would guess the guy that maintained the Jia Tan identity was working out of Australia.
The time zone is consistent with Western Australia and the work starts by "branching out" a not widely used Australian test framework.
The actual attack is probably done by someone else.
The time zone is consistent with Western Australia and the work starts by "branching out" a not widely used Australian test framework.
The actual attack is probably done by someone else.
What is seatest? Can you elaborate? That's the first time I heard that mentioned
https://github.com/keithn/seatest
Which Jia Tan turns into "s-test".
Which Jia Tan turns into "s-test".
They used test driven development for the backdoor development? Kind of hilarious ngl
Yes. Test is a perfect place to hide an attack - reviewers pay far less attention to test code than the (what they perceive to be) production code.
UTC+1-3 are west to middle European time zones.
Even if the real origin is identified, that news piece will have much less circulation that the initial suspicion that it's Chinese.
So there are also PR reasons to create a fake trail.
The best lie confirms your confirmation bias. Everybody heard about the Chinese hackers attacking US, Ugandan hackers however would raise a few eyebrows.
So there are also PR reasons to create a fake trail.
The best lie confirms your confirmation bias. Everybody heard about the Chinese hackers attacking US, Ugandan hackers however would raise a few eyebrows.
Have there been articles with that headline? Or even suspicion?
And from Microsoft / GitHub - who would have a lot additional information (logs, ip-adresses, use of two-factor auth etc.)? Have they made a statement?
Usually Microsoft etc. don't hold back identifying "threat actors".
Is there a US police investigation ongoing that could ask Microsoft? The target of this attack has been US firms / persons so if they report it, I assume a US police investigation would be required.
Usually Microsoft etc. don't hold back identifying "threat actors".
Is there a US police investigation ongoing that could ask Microsoft? The target of this attack has been US firms / persons so if they report it, I assume a US police investigation would be required.
> And from Microsoft / GitHub - who would have a lot additional information (logs, ip-adresses, use of two-factor auth etc.)? Have they made a statement?
Based on a HN comment from a couple of weeks ago, by analyzing the attackers IP addresses from IRC chat logins, it seems they used a VPN service. If you think about it, it makes sense to always use VPN when doing an operation like this. So I think the ip addresses won't be of much use.
Based on a HN comment from a couple of weeks ago, by analyzing the attackers IP addresses from IRC chat logins, it seems they used a VPN service. If you think about it, it makes sense to always use VPN when doing an operation like this. So I think the ip addresses won't be of much use.
Might be able to tell which VPN service they used and can then subpoena it.
Based on the sophistication we've seen, they probably used Mullvad for their VPN. In that case, a subpoena wouldn't turn up anything.
I don’t know if it’s been tried, but if not it would be in everyone’s interest to see what Mullvad will actually cough up.
I have seen NordVPN’s response to a subpoena. Their response was that they had no records connecting an IP address at a specific date/time to any particular person.
it would be the end of their business if they did, as they have a strict no retention policy. This would mean they are lying to all their customers, so it is not going to happen.
Not that I know of. I'm sure that statement will be reserved for law enforcement.
> using a chinese front name is a very interesting detail
I have recently read that book https://www.ifitssmartitsvulnerable.com/ The author mentions a malware which at the first glance looked to originate from China (e.g. saved with Mandarin-localized MS Office) but was actually developed in Russia.
I have recently read that book https://www.ifitssmartitsvulnerable.com/ The author mentions a malware which at the first glance looked to originate from China (e.g. saved with Mandarin-localized MS Office) but was actually developed in Russia.
One of the fake names "Jigar Kumar" is Indian (Hindi/Urdu)
Is there anyone who have made complication on the most famous (searched by the whole world) anonymous people. No 1 must be Satoshi Nakamoto, and No 2 maybe this one? Is there any others?
Satoshi disappeared one day before Gavin went to the CIA for a Bitcoin presentation. He left the CAlert key and disappeared.
I learned about Bitcoin either days prior or days later. As a newbie I did not care who created Bitcoin, but now I regret not knowing and speaking with him directly.
I learned about Bitcoin either days prior or days later. As a newbie I did not care who created Bitcoin, but now I regret not knowing and speaking with him directly.
I thought it is established by now Paul Le Roux is almost certainly Satoshi. He had the motivation (money laundering), the cryptography knowledge and him never moving his coins after they became quite valuable is easily explained by the fact he is in prison. As far as I am aware there is no one else for whom all three would be present.
Whoever Satoshi Nakamoto is, once he (she?) moves the coins they become the target of lots of criminal enterprises. Those bitcoins make them a VERY rich person and draws a very bright target on their back. Maybe they just prefer not having to deal with this?
I remember reading the article proposing this and I seem to remember some pretty big differences in coding style (TrueCrypt vs. Bitcoin) and assumed development platform. I'm not convinced tbh
no, it's not "established". it's just another suspicion
DB Cooper
[deleted]
"Aw fuck. Oh well, the Chrono, Boost, libcurl and libsodium backdoors are still active. Can't believe they found them lol"
While libcurl is an alluring target it is really ill suited for this, Daniel Stenberg is paid to work on it full time and his track record speaks. The situation is really different from xz.
I am sure they are happy this one was noticed instead of one of the dozen worse ones they got in years ago.
[deleted]
`sudo ldd /usr/sbin/sshd` why would you put `sudo ldd` on a slide.
The lecturer did state this was very dangerous and should never be done in a real machine.
ldd has an unfortunate side effect of actually executing code within some libraries (the exact details, I don’t know. Maybe it’s library specific)
ldd has an unfortunate side effect of actually executing code within some libraries (the exact details, I don’t know. Maybe it’s library specific)
On very old glibc ldd used to set an env variable and run the executable. The assumption being that PT_INTERP is set to glibcs dynamic linker. The dynamic linker would dump resolved libraries and stop execution if the variable was set. My guess is they wanted one and only one code path, whereas a separate resolver in `ldd` could get out of sync with the dynamic linker and therefore report incorrect results. But obviously binaries do not have to have an interpreter set at all, or they may point it to any other executable that's not glibcs dynamic linker, and the result is it would just execute like normal.
The fact that the lecturer is aware of it makes `sudo ldd` look even more odd.
The fact that the lecturer is aware of it makes `sudo ldd` look even more odd.
[deleted]
[deleted]
bolangi(3)
I hope that one positive development from this saga will be that open source users do not want to been as the next Jugar Kumar and choose to act less entitled and demanding towards maintainers.
Is it really hard to just ignore those types of people? I would say it’s a 1 in 200 interaction (unless the project was recently hyped up somewhere)
I have just closed issues where the reporter or “user”, as you put it, is just unhelpful, rude, or non responsive.
No response needed most of the time.
I have just closed issues where the reporter or “user”, as you put it, is just unhelpful, rude, or non responsive.
No response needed most of the time.
I've never minded ignoring -anyone- who wasn't paying me if they're being disrespectful. I try my best generally, so I don't feel guilty, and if that's not good enough, they can go pound sand and cry about it; a good cry can definitely help you when you're at peak emotions is my thinking. A lot of people have other things going on and just snap at the smallest thing when they get frustrated.
Its a year from now, 2025, you open Hacker News and see on the front page another article or lecture about XZ. Maybe 2 years from now who knows :_)
Better to get this information out there to as many people. Heightened awareness of the problem is always a good thing.
We now know the answer to the question of "who would win?"
1) A years-long nation-state-backed hacking effort to infiltrate a software project and compromise most servers in the Western world
or
2) A German's obsession with efficiency and precision in engineering