Scary Stories about Software Supply Chains
Critical lessons from software supply chain failures.
In the light of the recent Crowdstrike outage, it’s crucial to review how organizations receive their software updates and how they mitigate related issues. The verification procedures and update channels for Falcon — the software that caused all the chaos — were broken.
Let's explore some other incidents that might be relevant in this context.
xz
How much time can a hacker devote to compromising a system?
As we learned from the xz incident earlier this year (2024), the answer is quite a lot.
The xz library is widely used for compression, including in SSH traffic, which is often the primary way to access Linux machines. Imagine the potential consequences of embedding malicious code into this critical library.
Whether it was the work of a single person or a group remains unknown. However, Jia Tan spent several years building trust within the xz community by contributing useful code to the repository. They also manipulated the author by creating fake accounts to generate social pressure, pushing for certain changes. Eventually, Jia Tan was promoted to co-maintainer and managed to submit a disguised backdoor into the library.
Given that this library is used by virtually all Linux distributions, the potential for widespread compromise was enormous.
Fortunately, the backdoor was discovered before updates hit the disks.
Mitigation
What can we do?
It's currently too easy to create an account on GitHub. I believe Microsoft, along with other service providers with significant social impact, should strengthen identity verification processes. For instance, implementing a KYC (know-your-customer) process that includes credit card or identity document verification would be a good start. Software security is simply too important to leave unchecked.
Additionally, major vendors should increase their support for critical library authors through sponsorship programs. While some already do, it's not enough.
Linux OS distributors (e.g., RedHat, Canonical) need to invest more heavily in regression testing and thorough investigations. Yes, they do this already, but more can be done.
Organizations that rely on external software — which is nearly everyone — should definitely adopt data collection and anomaly detection strategies. With thousands of dependencies, you never know which might harbor a zero-day vulnerability or malicious code.
sunburst
Solarwinds is the company providing monitoring software. The hacker had access to their release system for more than a year. They managed to install malicious code into the update package that was automatically distributed to thousands of companies. The incident affected many government organizations.
Mitigation
The breach was eventually detected due to anomaly detection mechanisms and accidental observations on the client side. This underscores an important rule:
Never ignore unusual data points.
Another helpful measure is artifact validation. At a minimum, verifying the checksum of any package obtained from external channels is essential.
Additionally, most ecosystems support package signing, including:
This should be a de facto standard for any company serious about security. Package signing is a simple yet effective method to detect corrupted artifacts. This leads to another simple rule:
Only use signed packages; fail deployment if the signature mismatches.
Finally, don’t underestimate the importance of network perimeter protection — yes, the old reliable firewall.
browsealoud
Have you heard of crypto-jacking? It’s a type of cyberattack where compromised infrastructure is used to mine cryptocurrency without the user’s knowledge. This can even happen directly in your browser, whether you're on a desktop or mobile device.
A notable incident occurred in 2018 when hackers exploited release infrastructure of BrowseAloud, a popular JavaScript library designed to read website text aloud. Thousands of websites, many of them government-related, included this library as part of their functionality. The hackers injected malicious code into BrowseAloud, turning every device that loaded these websites into unwilling participants in a cryptocurrency mining operation.
What’s even more alarming is that crypto-jacking might have been just the tip of the iceberg. The attackers could have used the same method to steal personal information, including credit card details.
Mitigation
Most of the users of this library were linking directly to the latest version hosted (and hacked) on AWS. The same link was used by all — convenient, but very dangerous!
The key advice here is to avoid using the "latest" version of dependencies and always specify the exact version. Additionally, always verify the authenticity and integrity of the artifacts you use.
In fact, the HTML standard allows specifying the checksum for remote artifacts:
<script
src="https://code.jquery.com/jquery-3.7.0.slim.min.js"
integrity="sha256-tG5mcZUtJsZvyKAxYLVXrmjKBVLd6VpVccqz/r4ypFE="
crossorigin="anonymous">
</script>Log4Shell
Log4Shell is a classic zero-day vulnerability. By providing a maliciously crafted URL or HTTP header, a hacker can trick an application into making a callback to a remote server, which can then deliver and execute malicious Java bytecode on the victim’s server.
This vulnerability had a massive impact on the Java ecosystem. Although the exact numbers are unclear, it's estimated that over 80% of applications were using a vulnerable version of the log4j library at the time of discovery.
The impact was so significant in the enterprise world that even unrelated Docker images on Docker Hub had to be marked as log4shell-free:
Mitigation
In reality, most applications don't need to make external calls to the Internet. Default network policies should simply block all communication with external entities.
Log4Shell exploited JNDI and RMI, protocols that are rarely used by modern Java applications. This raises an important question for both library authors and users: Why keep outdated, unused code in your applications? Old code is a ticking time bomb.
Another critical aspect of handling such vulnerabilities is having a fast and efficient deployment process to apply fixes as soon as they become available. I've encountered companies that didn’t even have the original source code or CI/CD pipelines to redeploy their production applications with updated code.
The ability to rebuild and redeploy code at any time is crucial for security.
You’ve probably heard the saying, “If it works, don’t touch it.” This mindset is dangerous. We should continuously verify that critical elements work and remain adaptable to change.
leftPad
The leftPad incident in 2016 caused a significant disruption in the Node.js ecosystem, making most applications unbuildable for several hours.
Azer Koçulu, the author of several libraries published on npmjs.com, had his "kik" library forcibly removed by the governing organization due to a legal complaint. In protest, Azer removed all of his libraries from the repository, including leftPad — a dependency for many Node.js applications.
Mitigation
I sympathize with Azer's stance, as the legal complaint seemed insufficient to justify enforcement. I might have reacted similarly in protest.
However, this incident raised two critical issues:
How much can we trust public repositories or the companies managing them?
Why did the world come to depend so heavily on an 11-line code library like LeftPad?
Typically, I follow this motto:
The code you don’t need to write is the code you don’t need to test.
However, in light of this incident, insourcing a simple 11-line code block doesn’t seem unreasonable. Any developer can write it quickly, including unit tests.
Since public repositories can’t be trusted to maintain packages indefinitely, caching packages within your organization through public repository mirroring is also a prudent strategy.
Dependencies often have their own dependencies, leading deep down a rabbit hole. Ultimately, you need to understand what your applications rely on. Dependency tree analysis is a crucial aspect of an organization’s security procedures. The Software Bill of Materials (SBOM) has existed for more than a decade, and tools like Dependency-Track can help build a comprehensive overview.
Conclusion
We can't fix the world completely — vulnerabilities, compromised delivery channels, and hacks will continue to occur. However, from a business perspective, we can minimize risk by implementing:
Stricter default network policies
Comprehensive audit data collection, analysis, and anomaly detection
Use of signed packages and artifact validation
Public repository mirroring and critical evaluation of dependency sources
Dependency on specific versions of external OSS components
Continuous dependency tree and unused code analysis
Fast delivery procedures and effective patch management
Always functional CI/CD pipelines
Let’s strengthen our supply chains!