Observations from the XZ Utils Backdoor

The XZ UTILS breach

A backdoor was recently discovered in a critical open-source utility used by the two major Linux distributions which, had it gone undetected, could have caused immense damage. The people or entity behind the backdoor patiently waited years to create the right circumstances before inserting the vulnerability. Larger questions have been raised about securing software supply chains that will take years to address, but in the meantime, there are lessons all can learn from the sequence of events. The purpose of this post is to discuss three general observations that are as applicable to maintaining effective internal controls in association with SOC audits as they are to secure software development.

Understanding the Linux Backdoor

To set the stage, some brief background is helpful. The backdoor was introduced into the XZ Utils package, which is a data compression utility (similar to Zip in Windows) used across essentially all open-source operating systems. The backdoor was crafted in such a way that it allowed an attacker to remotely gain unauthorized access to a Linux computer running the malicious version of the utility that was also reachable using SSH. Fortunately, the vulnerable versions of XZ Utils didn’t make their way into official releases of the most popular Linux distributions, and they existed mostly in development/pre-release versions.

With an overall understanding of the situation, here are three of my observations as a SOC auditor.

 

Safeguards

Don’t Disable Safeguards

About six months before the backdoor was introduced, the malicious developer requested changes to disable a security check in the software test suite that would have detected the backdoor during the software build process. While security researchers are still debating the impact of this, it seems at the very least like efforts were successfully made to circumvent the process that would have prevented the backdoor from being incorporated.

Care should be taken when disabling or overriding controls and safeguards, even ones that may seem minor or redundant. From a SOC audit perspective, SOC reports revolve around controls at a service organization. If a company stops executing the control without the proper oversight it may result in not only an exception for that control in the next SOC report but could result in failing to meet key Trust Service Criteria and impacting the overall report opinion.

 

Social engineering

Beware of Social Engineering

One of the upsides and downsides of open-source software is that projects usually rely on volunteers to maintain the code, and often the maintainers of the project aren’t being paid directly for their efforts. Therefore, contributions from other volunteers are often happily accepted as they help spread the load of developing and maintaining the project. The user who created the backdoor, Jia Tan, was most likely not a real person at all, and there were probably several people working together on the backdoor who were masquerading as him.

Social engineering was used to initially develop trust with the XZ Utils maintainers when the user account started making contributions, and this resulted in Jia Tan becoming a co-maintainer of the project. Once the backdoor was ready to be inserted, more social engineering took place with additional dubious developers creating pressure on the primary maintainer to expedite releasing the versions of the utility with the backdoor.

A lesson to learn here is to know who you’re dealing with and not be pressured into premature actions. From a SOC audit perspective, it’s important to demonstrate that due diligence and vetting are performed where warranted. For example, hiring decisions should involve background or reference checks, and sufficient validation and review should be performed when establishing and continuing vendor and client relationships.

 

Vigilance is key

Be Vigilant

The backdoor was so well hidden that it went undetected within the XZ Utils project itself. It wasn’t identified until an observant developer at Microsoft, who was working on database code totally unrelated to XZ Utils or SSH, noticed some unusually high system usage and memory debugging errors on his system. Not only was this developer observant, but he was also diligent in tracking down the root cause and ultimately discovering the backdoor. Without his efforts, it’s hard to know how much damage would have taken place before the backdoor was found.

This shows how sometimes you or I can be the last line of defense before something bad happens. We need to be vigilant for things that don’t look right and willing to raise our hand when we do. I’ve seen excellent examples of observant people at my client organizations who have identified a failure or gap in a control and have taken prompt action to reduce risk to their organization and help their next SOC audit pass.

Conclusion

While the technical details of the XZ Utils hack are complicated, the non-technical aspects appear simple in hindsight: keep safeguards in place, don’t be fooled, and take action when something looks wrong.

Please don’t hesitate to reach out to the Linford team for guidance on your upcoming audit, or to help answer any questions you may have about the security of your organization.