Recently, The cybersecurity, development and technology communities were taken aback when it was disclosed that a backdoor (CVE-2024-3094) had been uncovered in XZ, a widely employed open-source data compression utility found on most Linux systems. This vulnerability enabled threat actors to establish remote connections and execute commands on affected systems with administrative privileges. Given the critical reliance of many internet infrastructures on Linux, the discovery of this backdoor was of significant concern and could have led to a major security breach if not detected promptly.
I will discuss the backdoor's impact, exploit details and mitigation strategies in this article.
Vulnerability Unveiled
On March 29, Microsoft engineer Andres Freund published the details regarding the Backdoor affecting. xz/liblzma.
Further disclosure detail can be found here.
Initially, the Backdoor was believed to be an SSH authentication bypass. However, further analysis revealed a more severe threat: Remote Code Execution (RCE). This grants attackers complete control over the affected system, potentially compromising sensitive data and disrupting operations.
How the exploit works:
This extremely sophisticated exploit has multiple stages and involves multiple steps chaining throughout the process.
- A series of malicious code disguised as test files would be invoked at some stage of the build process by a script
m4/build-to-host.m4. The output of this script de-obfuscates the cleverly hidden malicious code and executes it.
- The above code, once executed, extracts hidden data from a compressed file
good-large_compressed.lzmaand prepares it for execution by applying specific steps to decode and decompress it. - The resulting decompressed data is an elaborate bash script (infected.txt as attached in original disclosure), which checks for a few conditions, i.e. whether the system is
x86-64 Linuxor not, whether the build process is manual or a part of package build (Debian or RPM), etc. If conditions are met, the compilation process is modified to include the Backdoor; if not, the backdoor installation process is simply skipped. - The Backdoor hooks the PLT (Procedure Linkage Table), specifically the one used for SSH login
RSA_public_decrypt@....plt, to point to its own exploit code. This allows the Backdoor to run its malicious code whenever SSH authentication is attempted. - The Backdoor only activates if the login attempt includes specific data sets signed and encrypted by a private ED448 key controlled by the attacker. Once the payload is decrypted, it is executed with root privileges in the host system.
For detailed technical analysis, refer to the following resources
- https://gynvael.coldwind.pl/?lang=en&id=782
The Heist: How It Was Pulled Off
Now that we know how the exploit works let's discuss how it was pulled off on one of the most popular open-source projects. If not for its early discovery, this could easily be one of history's most devastating Supply Chain attacks. This attack was planned and executed over years of meticulous planning and deception and involved multiple steps.
Originally the concerned project was being maintained solely by Lasse Collin. Being the sole maintainer of such a huge project (unpaid hobby) for years understandably burned him out. Taking this as an opportunity, the attacker, with Jigar Kumar and Dennis Ans persona, began pressuring the maintainer into accepting another person as the maintainer, presumably Jia Tan, to reduce the burden and further boost the project's development.
The whole thread for this mail interaction can be found here.
Here, in these emails by Lasse, we can also see Jia Tan being referred to as a possible maintainer in the future.
After these discussions and pressure, Jia Tan was finally appointed maintainer in early 2023, and he merged his first PR on January 6, 2023.
Outside of the above XZ discussion, these two persona i.e Jigar and Dennis is never seen again.
After gaining trust and gaining privilege, Jia Tan started slowly unloading the exploit infrastructure required for the Backdoor.
As seen above, the final piece required for the Backdoor was added to the repository on March 9, 2024. Then, the poisoned tarball was released as versions 5.6.0 and 5.6.1.
Timeline of Events
An attacker, operating under the alias Jia Tan (Jia Cheong Tan or JiaT75) on GitHub, gained the community's trust. Here's the timeline:
- Aug 2021: Jia Tan creates a GitHub account.
- Late 2021/Early 2022: Jia Tan contributes to related open-source projects, building credibility.
- Early 2022: Jia Tan joins the XZ project mailing list and contributes minor code.
- Jan 2023: Jia Tan eventually earns maintainer privileges for XZ Utils.
- Jun 2023 - Mar 2024: Jia Tan begins unloading the exploit infrastructure disguised as test files.
- Feb 2024, Mar 9, 2024: The malicious code is introduced into versions 5.6.0 and 5.6.1.
- Mar 29, 2024: An engineer stumbles upon the exploit while benchmarking Postgres and notices an abnormal spike in CPU usage caused by SSHD processes.
For detailed timeline of events refer to this great post.
Checking Exposure and Mitigation
The impact of this vulnerability was not widespread and is mostly limited to bleeding age and experimental systems. The following table shows the affected systems.
| Distribution | Advisory |
|---|---|
| Fedora Rawhide | https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users |
| Fedora 41 | https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users |
| Debian testing, unstable and experimental distributions versions 5.5.1alpha-0.1 to 5.6.1-1. | https://lists.debian.org/debian-security-announce/2024/msg00057.html |
| openSUSE Tumbleweed and openSUSE MicroOS | https://news.opensuse.org/2024/03/29/xz-backdoor/ |
| Kali Linux | https://www.kali.org/blog/about-the-xz-backdoor/ |
| Arch Linux | https://archlinux.org/news/the-xz-package-has-been-backdoored/ |
You can also check your installed version by running the command.
strings which xz` | egrep '\(XZ Utils\)If your installed XZ util is version 5.6.0/5.6.1 then immediately downgrade to a known safe version, i.e 5.4.6.
Conclusion and Lessons Learned
While the Backdoor itself was sophisticated and involved many techniques, this supply chain attack was ultimately possible due to the one overworked developer maintaining this critical infrastructure, whom the threat actors could manipulate into giving them the required privileges for this attack.
This incident serves as a reminder of the complex challenges faced in maintaining the security of open-source software - especially those widely used in critical infrastructures like Linux systems, and the community's collective responsibility to uphold and improve security practices. The trust placed in such tools requires continuous vigilance and scrutiny to prevent vulnerabilities and attacks. Let this incident be a lesson for the open-source community as a whole. As software users, we must actively support projects we rely on, not just through bug reports and feature requests but also by contributing code, documentation, and financial resources.
Further Reading and References
- https://openwall.com/lists/oss-security/2024/03/29/4
- https://gynvael.coldwind.pl/?lang=en&id=782
- https://www.mail-archive.com/xz-devel@tukaani.org/msg00562.html
- https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
- https://git.tukaani.org/?p=xz.git;a=summary
Thank you for reading this article. Stay vigilant, stay safe, stay curious!