Equifax + 3rd party code != security
Equifax + 3rd party code != security
It appears the Equifax breach hinged on an unupdated Apache Struts vulnerability. Lots of security people are talking about lots of different dimensions of this breach but one portion is the (in)secure use of 3rd party code. The security of 3rd party code is an area of application security that doesn’t get much attention so I want to highlight it.
Background
Reasonable engineers rely on the work of others. This work comes from other parts of the company, the open source community or a commercial product. Once used this code is part of the surface area of your codebase and from a security perspective must be treated just like homegrown code.
The problem is when security vulnerabilities come to light in this code, they don’t get updated. It is not top of mind that this stuff requires ownership, updating and care.
This can be especially important when, by lines of code, 30, 50, or 80% of your codebase is actually 3rd party code.
This is basic patching. OWASP calls it insecure components but the generalized form of it goes by the name “vulnerability management”. Vulnerability management exists across many areas: servers, routers, laptops, ipads on the conference rooms, etc.
How to fix
I’ve found this breaks down into 3 problems:
- Inventory — knowing all your inventory of code, systems, etc for which security vulns might be found and need updates.
- Vulnfeed — A feed of security updates against your inventory.
- Remediation — A process to actually patch those things. Usually requires human intervention but may be as easy as apt-get update.
My first cut of solving this problem was writing a script to guesstimate if something was a 3rd party package. If it was, get its name and version and stuff it in a csv (inventory). Then download the cvedb, compare (vulnfeed) and if there was a hit send an email to whomever checked the package and ask them to update (remediate). Then auto-nag them if they didn’t within a few days.
The biggest variable here is how difficult it is to inventory your code: sometimes 3rd party code is packaged and nicely versioned, other times its a tarball of code checked into source control. The trickiest case is when someone downloads the source code to openssl, pulls out range of lines from a file and copy/paste it into your codebase. There are commercial solutions that can help if the code is nicely packaged, I’ve had luck with src:clr.
To operationalize this system involves stuffing it in a script, automatically rebuilding the inventory index and most importantly figuring out how to effectively motivate others to fix the findings in a way that scales.
On motivation one trick that helped was careful wording. Our first few waves of emails about updating were ignored. We eventually found success with the wording: “you are currently in the very small minority of people who have not updated yet”. I stole this idea from Italian tax collectors who after lots of a/b testing found it to be the most successful.
Prioritization
Starting from scratch on this problem can be intimidating. There are so many things to patch but what to go for first? Some ideas here: CVE score, the closeness of the system/code to the edge/internet, the areas that have owners, the areas that touch sensitive data.
Conclusion
Like with all security work the biggest obstacle is apathy. Pointing to this Equifax breach should help motivate other to help fix this grungy but important problem.