Protecting from vulnerabilities in Java: How we managed the log4j crisis
You may have heard about the critical log4j vulnerability that set the web ablaze mid December 2021. Nicknamed ‘log4Shell’, or more officially ‘CVE-2021–44228’, this vulnerability was scored 10, which is the most critical. Thanks to this vulnerability on the most common logging library in the Java ecosystem, attackers could execute malicious code on a remote server. It was affecting versions 2.0-beta9 to 2.14.1, versions that contain JNDI and LDAP features, thanks to which the exploit could be performed. Apache released a patch in version 2.15, but that was not enough and another CVE was reported (CVE-2021–45046, critical), corrected in 2.16, quickly followed by CVE-2021–4104 and CVE-2021–45105 (high), patched in 2.17 and finally CVE-2021–44832 (medium) corrected in 2.17.1. You can get the full story on the log4j security bulletin.
Here at I Love My Local Farmer, we have several Java applications, Lambda functions and libraries, some of them using log4j, some others not. So we had to take measures to avoid these vulnerabilities. Based on this real event, this blog post will describe:
1/ How to detect vulnerabilities:
- Analysing dependencies with Maven and Gradle
- Using dedicated tools
2/ How to fix this vulnerabilities
Disclaimer
I Love My Local Farmer is a fictional company inspired by customer interactions with AWS Solutions Architects. Any stories told in this blog are not related to a specific customer. Similarities with any real companies, people, or situations are purely coincidental. Stories in this blog represent the views of the authors and are not endorsed by AWS.
Disclaimer #2
Event if I Love My Local Farmer is fictional, this blog post is based on a real story and real events. We indeed had to patch several examples on Github and other repositories, just like everyone on Earth.
Finding where we use log4j
The first step was to find all the projects where we use this library and which version we were using. We have projects on Github, on an internal Gitlab, and on another internal git repository. So we asked for each development team to take care of its projects on its repositories.
Analysing dependencies
The task was quite explorative and manual, taking from the memory of tenured engineers and looking at Maven pom.xml files or Gradle build.gradle files.
There are two commands we used massively in December:
$ mvn dependency:tree | grep log4j$ gradle dependencies | grep log4j
These commands give you all dependencies and transitive dependencies used by your Maven or Gradle project. It’s important to go beyond a simple Ctrl+f ‘log4j’ as there may be transitive dependencies (dependencies of your dependencies).
If you get any output with version less than 2.17.1, you have this vulnerability:
[INFO] | +- org.apache.logging.log4j:log4j-core:jar:2.14.1:compile
[INFO] | +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.14.1:compile
[INFO] | \- org.apache.logging.log4j:log4j-api:jar:2.14.1:compileUsing the same command without the grep will tell you where does it come from: direct or transitive dependencies.
For example in all our Java Lambda functions, we are using lambda-powertools to simplify structured logging:
[INFO] +- software.amazon.lambda:powertools-logging:jar:1.5.0:compile
[INFO] | +- org.apache.logging.log4j:log4j-core:jar:2.14.1:compile
[INFO] | +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.14.1:compile
[INFO] | \- org.apache.logging.log4j:log4j-api:jar:2.14.1:compileUsing OWASP Dependency-check
Another tool, available for Maven and Gradle is the OWASP Dependency-Check. Instead of looking for a specific version of a dependency, it analyses all the dependencies and generate an HTML report. Here’s an example for Maven:
$ mvn org.owasp:dependency-check-maven:6.5.3:checkOne or more dependencies were identified with known vulnerabilities in Todo:log4j-core-2.14.1.jar (pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1, cpe:2.3:a:apache:log4j:2.14.1:::::::) : CVE-2021-44228, CVE-2021-44832, CVE-2021-45046, CVE-2021-45105, CWE-502: Deserialization of Untrusted Data
log4j-slf4j-impl-2.14.1.jar (pkg:maven/org.apache.logging.log4j/log4j-slf4j-impl@2.14.1, cpe:2.3:a:apache:log4j:2.14.1:::::::) : CVE-2021-44228, CVE-2021-44832, CVE-2021-45046, CVE-2021-45105
See the dependency-check report for more details.
It can also be configured to break the build if a specific severity is reached (see example 3 in the documentation), which is pretty neat in a CI/CD pipeline. We will certainly adopt this in our projects.
Using snyk
Looking at Github, we have Dependabot activated on our projects but we did not get any alerts about log4j. This is because it only works with direct dependencies declared in Maven’s pom.xml. Based on this observation, we looked at code scanning tools proposed by Github and decided to give a try to snyk. In addition to code vulnerabilities detection and correction, it can also analyse docker images and Terraform, Kubernetes and CloudFormation templates.
It is really easy to set up. You choose the repositories you want to scan and in just a few seconds you get the result:
You can then try to fix the vulnerability automatically, but I’ll come back to this in the next part.
AWS also announced new checks in Inspector and GuardDuty to identify the existence of this vulnerability on EC2 instances and images stored in ECR (Elastic Container Registry). We don’t use these services yet. If you do, have a look at this blog post.
Fixing the vulnerability
Patching our applications
Once you know where you have an old version of log4j, it is quite easy to update it. Easy but when you have to do it 3 times, it becomes a bit boring. I remember sharing this blog post to my colleagues: “Upgraded to log4j 2.16? Surprise, there’s a 2.17 fixing DoS” a few days before Christmas. It was not the best present for developers. Anyway, we had to to do it…
Let’s start with the easiest: when you have a direct dependency to log4j. In that case simply update to the latest version. Be sure to do the same with other dependencies like log4j-slf4j-impl for example.
With Maven:
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.17.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.17.1</version>
</dependency>And Gradle:
implementation 'org.apache.logging.log4j:log4j-core:2.17.1'
implementation 'org.apache.logging.log4j:log4j-slf4j-impl:2.17.1'If the log4j dependency is transitive, we have 2 options:
1/ If the dependency that needs log4j has been updated by the maintainer, you can update it. For example, in the example above, powertools-logging 1.10.1 contains this latest version of log4j:
2/ If the dependency is not patched, then you need to force the version of log4j in your project.
- With Maven, simply declare an explicit dependency to log4j 2.17.1 (or newer) as it will override transitive ones. From the documentation:
Forcing a version
A version will always be honoured if it is declared in the current POM with a particular version — however, it should be noted that this will also affect other poms downstream if it is itself depended on using transitive dependencies.
- With Gradle, you can add a constraint that will enforce the use of a specific version (see this blog post from Gradle for more details) :
dependencies {
constraints {
implementation("org.apache.logging.log4j:log4j-core") {
version {
strictly("[2.17.1, 3[")
prefer("2.17.1")
}
because("CVE-2021-44228, CVE-2021-45046, CVE-2021-45105: Log4j vulnerable to remote code execution and other critical security vulnerabilities")
}
}
}Coming back to Github and snyk, I mentioned earlier that it could fix (or try to fix) the vulnerability automatically. Clicking on the green button we come to the following screen that proposes to open a PR and to fix some of the issues:
It will directly open a PR in Github with the modification of the pom.xml (upgrading powertools-logging to version 1.10.1 in that case). It looks pretty neat:
Patching our applications was actually the easiest part. The harder ones are validation and deployment. Fortunately, we don’t rely much on log4j and just use it for simple logging tasks (we are not using JNDI, LDAP or all the impacted stuff). Fortunately again, we have a decent unit test coverage plus a few end-to-end tests, and CI/CD pipelines. So we decided to let them roll as the change was really minor. We also performed some checks in the different environments: verify we still have logs, validate the happy paths and some critical features. But we did not have a proper UAT campaign: too long and expensive for such a small change. And everything is fine as of now.
What else can we do?
GovCERT.ch, the Computer Emergency Response Team (GovCERT) of the Swiss government has published an great infographics (below), describing how the vulnerability could be exploited and how to prevent it:
Looking at this, I noticed the ability to block an attack even before it reaches our servers, thanks to a Web Application Firewall (WAF). At I Love My Local Farmer, we are using the AWS WAF for some of our most critical applications, so I went to the documentation and discovered that AWS had already added a rule (Log4JRCE) to a group we already use (AWSManagedRulesKnownBadInputsRuleSet).
And according to this security bulletin and blog post, AWS made other changes to protect us from this issue. For example on Lambda:
For cases where a customer function includes an impacted Log4j2 version, we have applied a change to the Lambda Java managed runtimes and base container images (Java 8, Java 8 on AL2, and Java 11) that helps to mitigate the issues in CVE-2021–44228 and CVE-2021–45046. Customers using managed runtimes will have the change applied automatically. Customers using container images will need to rebuild from the latest base container image, and redeploy.
Conclusion
I won’t blame Apache and the log4j maintainers, I can only imagine how hard it is to maintain a library used by millions of developers. But I find it really crazy and scary that such a library, intended for logging, made so much noise and was a door for such a critical vulnerability. Like Murphy’s law says “everything that can happen will happen”. So we need to be prepared for this kind of event, and automate to detect and correct issues. You should implement simple things like the OWASP Dependency checker in the build pipeline, and eventually invest in a a code scanning tool like snyk. Also be aware of what happens, follow the news and blog posts from your service providers, it often contains valuable information.
