Log4j2 - Quality and Evolution

Key quality attributes and current status

The key quality attributes discussed in previous essays are performance and security.

Performance

Peak throughput is one of the critical indexes of a logging system. It measures the maximum throughput over a short period and describes the system’s ability to handle bursts. Log4j2 featuring using an Asynchronous logger can provide the best performances compared to the major counterparts and is even capable of offering larger throughput with more threads, while others remain the same or decrease ¹.

Security

As a widely used Java environment component that could access the details and execution codes in the system, the security of Log4j2 could have a major impact on the security of the whole system. But there had been a critical security compromise last year that caused widespread influence. Apache Log4j 2.0-beta7 through 2.17.0 using the JNDI feature does not protect against attackers controlling certain components, allowing attackers to execute arbitrary code loaded remotely when message lookup substitution is enabled². Later, it was discovered that the fix for CVE-2021-44228 was incomplete in certain configurations, marked as CVE-2021-45046³. These bugs were categorized as critical in the national vulnerability database, with scored 10(highest) and 9. Although solved in later versions, these bugs introduced severe vulnerability of system integrity and authorization. The project will be continuously fixing future bugs to improve system security.

Overall software quality processes

The overall quality process of Log4j2 can be further divided into 3 aspects: issue management, commitment and releasing, and contributor guidance.

Issue management

Troubleshooting and improvements are critical in the software quality process. Log4j2 project uses the JIRA system for proposing bugs, questions, and improvements⁴. Developers can contribute to Log4j2 by creating tickets and with findings or ideas on JIRA. Tickets can be classified as bugs, improvements, questions, and tasks. Reporters also need to provide the priority, version, component, and description to assist the project team in locating and analyzing the. The team can respond to the tickets, and discuss possible actions. JIRA serves as a platform where problems are accurately reported, and issues are organized systematically. In this way, the project could react and solve the issues exhaustively and promptly.

Commitment and releasing

To resolve conflicts over changes and continuously produce a quality system, Log4j2 has a complete and systematic releasing process. Before creating a release package, the contributors need to add necessary tests and check component integrity⁵, and have the actions censored by the project team⁴. After the package is staged in the repository, voting for deciding whether to release it⁶ is required. A binding release vote of the PMC is the critical gating step in releasing. After the voting, it becomes a formal offering of the ASF⁷. With such process and critical stages controlled by the project team, the software quality is guaranteed.

Contributor guidance

As an open-source software project, Log4j2 relies on external contributors to improve the quality. The project guides the contributors to improve the quality of this process. Log4j2 project provides a complete definition of the project’s coding standards for source codes. It covers not only conventions or coding standards, but aesthetic issues as well⁴. It covers the source file structure, class declarations, formatting, naming rules, and so on. This gives comprehensive help to the contributors to generate high-quality codes, which is a critical element in software quality.

Continuous Integration using GitHub Actions

“Continuous Integrations doesn’t get rid of bugs, but it does make them dramatically easier to find and remove,” said Martin Fowler⁸. Log4j2 is an open-source project hosted on GitHub, contributors create pull requests and push code every day, therefore, in order to allow the project to iterate quickly while still maintaining high quality, Log4j2 introduces the Continuous Integration (CI) pipeline.

Before July 2020, the Log4j2 CI pipeline relied on the Travis CI and Jenkins. However, as GitHub Actions is gaining popularity, the PMC members Log4j2 proposed to migrate the CI pipeline to GitHub Actions because they want Log4j2 a more GitHub-friendly project⁹. So far, Log4j2 has set two workflows on GitHub Actions as its CI pipeline, CodeQL and build¹⁰, when a contributor creates a new pull request or pushes a commit to the working branch, these workflows will be automatically executed. The detailed GitHub Actions diagram is shown below.

• CodeQL: Log4j2 uses CodeQL to analyze the code in its GitHub repository to find security vulnerabilities and coding errors, and the results are shown as code scanning alerts in GitHub¹¹.
• build: The purpose of the build is to run maven verify on different operating systems to make sure nothing else was accidentally broken by the latest code changes, so this workflow has 3 jobs for ubuntu-latest, windows-latest, and macos-latest separately.

“Head first” Log4j2 testing

To improve code quality and maintainability, Log4j2 has a large amount of test code. The importance of testing has been clearly stated in CONTRIBUTING.md⁵:

• Make sure you have added the necessary tests for your changes.
• Run all the tests with mvn clean verify to assure nothing else was accidentally broken.

Since Log4j2 is a multi-module project with Maven, and each module has a relatively independent feature, so almost all tests are unit tests. These unit tests are mainly built based on Mockito¹² and JUnit¹³, and most of the test code has been upgraded from JUnit4 to JUnit5¹⁴.

It is easy to run the tests for all modules in Log4j2 using the Maven command mvn verify, but the testing procedure can take a lot of time, for example, the time of mvn verify for a single pull request or commit in Log4j2 GitHub Actions pipeline is 22:35 min¹⁵. After a careful analysis of the testing procedure, we found that the main reasons for the long time spent are:

• The number of modules involved in testing is large: 46¹⁵.
• There are many asynchronous waits (seconds level) in the test code for log4j-core.

To further analyze the testing procedure, we ran the tests locally and used maven-surefire-report-plugin to check the test results and Jacoco¹⁶ to get the test coverage data. We only list the results of log4j-api and log4j-core because they are the core module of Log4j2.

Package	Tests	Errors	Failures	Skipped	Success Rate	Time(sec)	Test Coverage (Class), %
log4j-api	659	0	0	2	99.697%	25.005	86
log4j-core	2341	0	0	12	99.487%	449.314	81

As you can see from the table, most of the tests are running properly and the test coverage is high, everything looks good so far. The only thing that Log4j2 didn’t do so well is that the test coverage information was not mentioned in the development process.

Hotspot components

Hotspot components are the modules in a codebase where most development changes have happened in the recent year. By narrowing down the refactorings to a small part of the system, hotspot analysis can guide the improvement of the code.

We used CodeScene¹⁷ to analyze the hotspot of the Log4j2 source code. The result is shown in the graph below.

The most significant hotspot is the PropertiesConfiguration.java and its related test file. Most other changes are also relevant to the log4j-1.2-api module, which is consistent with the roadmap analysis in our first essay: The changes are mostly trying to make Log4j2 more compatible with the old version by solving the current issues¹⁸. Other amendments are designed to make the system more robust by, e.g., allowing unwanted attempts ¹⁹.

In the future, other than general upgrades like changing to Java 11, we assume the hotspots will still evolve around the plugins to improve their performance and compatibility with other platforms to extend the capability of Log4j2.

Code quality

CodeScene’s Code Health parameter provides the users with a straightforward way of measuring code quality based on known patterns that increase the maintenance costs²⁰. It is also proved that low code health means more potential security errors²¹. So code health analysis focuses on the efficiency and security of the codebase.

When developers reflect on their project goals, hotspot components with low code health should always be prioritized because the complicated code that changes often is usually a more urgent problem than others²².

As for Log4j2, AbstractConfiguration.java scores the lowest code health among all the hotspots. The main issue is the low cohesion of the class design: The module has at least two different responsibilities amongst its 66 functions. Low cohesion is problematic since it means that the module contains multiple behaviors and leads to harder-to-understand code requiring more tests.

The below figure shows the code health score of AbstractConfiguration.java:

Overall, no major refactoring is needed for Log4j2. The code health of the hotspots scored an average of 8.41 out of 10.

Quality Culture

As an open-source framework, Log4j2 continues to evolve with the help of its users. Apache encourages users to help improve Log4j2. To ensure the consistency of the Code, it provides Code Style Guidelines on its official website. Like other programming style guides, the issues covered span not only aesthetic issues of formatting but other types of conventions or coding standards as well ²³.

Log4j2 uses Github to manage PR and JIRA to manage issues. So before developers can contribute, they need to sign up for a JIRA account. In its Github repository, there is a dedicated document describing how to contribute to the project. Users need to clearly describe the issue including steps to reproduce when it is a bug and fill in the earliest version that has the issue ⁵. The Log4j2 community is quite active, with almost every new issue being raised. Administrators are also quick to process issues, and usually, issues can be answered within a day.

Issues are usually problems that people encounter when using Log4j2. When an issue is caused by a bug in Log4j2, it is converted to a PR. For example, LOG4J2-3345 says message Parameters use the configured default MessageFactory, but the JPL API requires MessageFormat style formatting. A day later in PR#713, this bug was fixed.

However, not all issues require code modification to resolve. For example, the problem presented by LOG4J2-3431 is repeated from LOG4J2-3419 a few days ago and is resolved in PR#789. Some issues are caused by improper use of the framework (LOG4J2-3399), in which case the administrator will suggest how to deal with the problem and mark it as “Information Provided”. Or as in LOG4J2-3397, marked as “Not A Bug”.

Also, not all issues are related to the functionality of Log4j2. For example, LOG4J2-3285 was about tooling configuration and process for community management. It comes from a discussion in Log4J2-3284(which is fixed in PR#653 and PR#698) about whether the current JIRA configuration allows assigning tickets to folks who aren’t committers.

Some issues are classified as improvements, and they come from the user’s expectation of future functionality(LOG4J2-1449, LOG4J2-3308). Such problems often take a long time to resolve, and some(LOG4J2-3337) may be considered invalid.

Technical Debt

Technical debt ²⁴ is a metaphor proposed by Ward Cunningham. He likens the internal quality defects that make it harder to modify and expand the system further to financial debt. And the extra effort required to add new features is the interest paid on the debt.

We used Better Code Hub to analyze the code quality of Log4j2, trying to find potential technical Debt in the code. We can see that Log4J2 performs well in “Keep Your Codebase Small”, “Write Clean Code”, and “Write Short Units of Code”, which are conducive to subsequent development and maintenance. The disadvantage of “keeping Architecture Components Balanced” can be ignored because Log4j2’s nature causes most of the code to be in ’log4j-core’. However, having a large unit interface in Log4j2 does make the unit difficult to understand. For example, ‘JeroMqAppender createAppender()’ has 25 parameters. In addition, the tight coupling of the codebase can cause the consequences of a change in one piece of code to be magnified.

As a widely used logging framework, the Log4j2’s security vulnerability could cause losses to a large number of users. Unfortunately, vulnerability is one of its technical debt. In December 2021, details of a remote code execution vulnerability in Log4j2 were made public. An attacker can exploit this vulnerability to execute arbitrary code on the target server. Because of its simple triggering mode and wide range of use, the vulnerability is extremely harmful.

---

1. Apache Log4j2. Performance. Retrieved 19 March 2022 from:https://logging.apache.org/log4j/2.x/performance.html ↩︎

2. Information Technology Laboratory. CVE-2021-44228 Detail. Retrieved 19 March 2022 from https://nvd.nist.gov/vuln/detail/CVE-2021-44228 ↩︎

3. Information Technology Laboratory. CVE-2021-45046 Detail. Retrieved 19 March 2022 from https://nvd.nist.gov/vuln/detail/CVE-2021-45046 ↩︎

4. Apache Log4j2. Apache Log4j Project Guidelines. Retrieved 19 March 2022 from: https://logging.apache.org/log4j/log4j-2.3.2/guidelines.html ↩︎

5. Apache Log4j2. Contributing to Apache Log4j 2. Retrieved 18 March 2022 from: https://github.com/apache/logging-log4j2/blob/release-2.x/CONTRIBUTING.md ↩︎

6. Apache Log4j2. Log4j 2 Release Process. Retrieved 19 March 2022 from: https://cwiki.apache.org/confluence/display/LOGGING/Log4j+2+Release+Process ↩︎

7. Apache. Release Creation Process. Retrieved 19 March 2022 from: https://infra.apache.org/release-publishing.html ↩︎

8. Martin Fowler. Continuous Integration. Retrieved 17 March 2022 from: https://martinfowler.com/articles/continuousIntegration.html ↩︎

9. Volkan Yazici. Implement CI pipeline via GitHub Actions. Retrieved 17 March 2022 from: https://issues.apache.org/jira/browse/LOG4J2-2764 ↩︎

10. Apache Log4j2. GitHub Actions workflows. Retrieved 17 March 2022 from: https://github.com/apache/logging-log4j2/actions ↩︎

11. GitHub Docs. About code scanning with CodeQL. Retrieved 17 March 2022 from: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql ↩︎

12. Mockito team. Mockito. Retrieved 18 March 2022 from: https://site.mockito.org/ ↩︎

13. JUnit team. JUnit 5. Retrieved 18 March 2022 from: https://junit.org/junit5/ ↩︎

14. Matt Sicker. Migrate to JUnit 5. Retrieved 18 March 2022 from: https://issues.apache.org/jira/browse/LOG4J2-2653 ↩︎

15. Apache Log4j2. build (ubuntu-latest). Retrieved 18 March 2022 from: https://github.com/apache/logging-log4j2/runs/5605493715?check_suite_focus=true#step:8:13983 ↩︎

16. Mountainminds GmbH & Co. KG. JaCoCo - Java Code Coverage Library. Retrieved 18 March 2022 from: https://www.eclemma.org/jacoco/trunk/index.html ↩︎

17. CodeScene. CodeScene Use Cases and Roles. Retrieved 18 March 2022 from: https://codescene.com/docs/CodeSceneUseCasesAndRoles.pdf ↩︎

18. Apache Log4j2 Github. Pull request:Log4j 1.2 bridge issues with filters. Retrieved 19 March 2022 from: https://github.com/apache/logging-log4j2/pull/753 ↩︎

19. Apache Log4j2 Github. Pull request:LOG4J2-3407 Log4j 1.2 bridge Check for non-existent appender when parsing properties. Retrieved 19 March 2022 from: https://github.com/apache/logging-log4j2/pull/761 ↩︎

20. CodeScene Guides. CODE HEALTH – HOW EASY IS YOUR CODE TO MAINTAIN AND EVOLVE?. Retrieved 19 March 2022 from: https://codescene.io/docs/guides/technical/code-health.html ↩︎

21. Adam Tornhill. Predicting Security Vulnerabilities with Behavioral Code Analysis. From: https://codescene.com/blog/predicting-security-vulnerabilities-with-behavioral-code-analysis/ ↩︎

22. CodeScene Guides. HOTSPOTS. Retrieved 19 March 2022 from: https://codescene.io/docs/guides/technical/hotspots.html ↩︎

23. The Apache Software Foundation. Apache Log4j Code Style Guidelines. https://logging.apache.org/log4j/2.x/javastyle.html ↩︎

24. Martin Fowler. TechnicalDebt. https://www.martinfowler.com/bliki/TechnicalDebt.html ↩︎