The xz Problem and a Look at our Supply Chain Strategy
Hardly a day goes by these days without IT news reports informing us about security vulnerabilities and urgent patches for well-known software products. And so many of us often accept security issues in software as a fact of life.
However, a recent incident has sent shockwaves through the security research community: the injection of an extremely well-disguised and deliberately placed backdoor into the xz compression library, which is used by many software projects. The insidious part: the backdoor was targeted at a completely different software component—namely Secure Shell (openssh) in conjunction with systemd—and only took effect in this specific combination.
The attack, which was evidently prepared with precision over a long period of time, exploited the project maintainer’s heavy workload and patiently gained commit rights to the xz repository.
Had a resulting side effect not been noticed rather by chance during performance tests, this vulnerability would certainly have left millions of Linux systems vulnerable to the group behind it over time. A disaster of epic proportions was narrowly averted.
This incident once again highlights the particular importance of the “supply chain”—that is, the dependencies used by a software product, often consisting of hundreds or thousands of separately managed libraries and toolkits, some of which are even used only indirectly across multiple layers.
How do we handle this for our own products? OpenXPKI also uses a large number of libraries. Several years ago, we decided to deliver the Enterprise Edition of OpenXPKI as an all-in-one bundle and to avoid using external dependencies managed by the operating system whenever possible. Thus, the build process for the Enterprise packages uses a manually curated list of dependencies to be packaged, combined with a local mirror of these software components on our build systems.
Before our releases, we take a critical look at these dependencies and their release notes to decide whether to include an upgrade in our build process. This allows us to precisely define which software versions of the dependencies we package and distribute.
Overall, however, supply chain attacks are difficult to manage. No one can perform comprehensive source code audits for a complex software product covering every integrated library and every single change made by the maintainers. And so we all have to rely a little on the fact that the open-source community’s protective mechanisms continue to function as they have in the past. Fortunately, this has worked well so far with xz and Heartbleed. Let’s hope the community remains just as vigilant!
