In the previous post, we went through the architectural design of the Moby project and its main part - the Docker daemon. We also discussed multiple views and how it relates to other parts of the modern container ecosystem. A question that remains is how the key quality attributes that are identified in our first essay (modularity, composability, ease of integration, usability, security, and reusability) are ensured. Some of the modularity and security features can generally be tested automatically. However, the maintainers also need to manually track the other quality attributes. In this essay we will discuss how the quality of the Moby project is ensured as well as how technical debt has accumulated since the beginning of the project and the plans to refactor it.

Satisfying system’s key quality attributes

The modularity of the system is satisfied by splitting the Moby projects into different repositories. Additionally, the Go modules in the project can also be used as dedicated libraries in different projects.

Over the last years, the maintainers externalized some components from the Moby core project. For example, this has been done with containerd. Separating these components and defining a standard for the communication between them and Moby helps with the composability of the system.

By keeping the API predictable and stable, the end product is easy to integrate. The API is defined in OpenAPI, from which it is possible to generate API calls and payload code representation in arbitrary programming languages. The changes in the OpenAPI schema allow users to determine changes in the API and react to them in the client applications, ensuring that after changes, they still work.

The Moby project stated that they want to ensure security with the help of sane defaults to all the configuration options. The maintainers actively safeguard the configuration so that all of the components are reasonably secure when incorporated and have well-defined behavior that is understandable for the engineers that work with it.

Software quality processes

The whole codebase is written in the Go programming language. This language was designed in 2007 by Google. It is based around productivity, and its features are designed for larger codebases¹. Unlike traditional languages like C/C++ or Java, Go pre-defines the syntax style in which it should be programmed, and tools for testing, linting and downloading packages are included. Moby maintainers employ these tools to ensure the quality of the code contributions.

The whole process of contributing is managed and checked to ensure the stability and quality of the system. The Moby repository includes a CONTRIBUTION.md file, which describes the contribution process. Issues can be created on Github and are required to follow a certain structure. This structure provides the contributors with a description of the problem and reproduction steps in case of the bug or - for the feature requests - with the vision of a new feature and an example of use. These issues can be linked to pull requests, which a Jenkins pipeline will check. Finally, maintainers can accept the pull request and merge the changes.

Test processes’ rigor

In the case of the main Moby repository, contributors are encouraged to run the entire Moby Engine test suite locally before submitting a pull request with a code change. The test suite contains different targets for testing (specified in the image below), allowing to check how the changes made affect the code behavior granularly. This is important since running the entire test suite can take over half an hour.

A unit test is a piece of code that invokes a single, small piece of code (unit of work) to verify the unit works as expected. The unit tests either use the Go standard testing package or gocheck². Their purpose is to test a feature without having to spin up all the project’s dependencies - for example to test Dockerfile parsing.

The integration tests combine multiple units to test their communication. For example, by running the Docker daemon, calling its API and verifying the result. They use the gocheck testing library, since it’s more feature rich than the internal one.

The tests can be run locally on the contributor’s host or within a development container. The latter way provides an environment for deterministic execution of the tests without polluting the host operating system and having Docker daemon under test using the host Docker instance’s store directory.

When building the binaries for different architectures like x86 and ARM are assembled. Tests are dynamically skipped depending on the architecture and operating system that runs them. For some situations, this makes sense. However, some of the test files include conditions to skip certain tests whenever there is already a Docker daemon running on the machine. When used improperly, this can introduce uncertainty and non-determinism into the test suite.

For the rigorousness of the test processes, a contribution should first be locally tested; when passed, it can be merged and run against the continuous integration pipeline.

Continuous integration processes

Continuous integration - often referred to as CI - is the practice of merging code changes from multiple contributors into a single software project in an automated fashion. It is considered good practice and allows developers to merge code changes into the central repository frequently. Moby uses the Jenkins automation server to take on these functionalities. Moby’s Jenkins is maintained and managed by Docker Inc. However, the infrastructure itself is provided by CloudBees, which offers commercial-grade Jenkins instances and supports open-source projects like Moby³.

The illustration below shows an example pipeline execution⁴. Here we see different tests for each of the different architectures like x86 and ARM, as mentioned earlier. The integration tests against Windows machines assure the correct behavior of Windows containerization. Other pipeline branches tackle the process of building and testing specific flavors of Docker daemon, for example, the rootless mode. Also, there is a dedicated branch for static analysis, which makes sure that all the code, script and markdown files comply with the predefined set of rules, ensuring consistency.

**Figure:** Example CI pipeline in Docker’s Jenkins

Hotspot components from the past and the future

Hotspots were analyzed with the use of git effort⁵ from the git-extras package. It scans the repository regarding the number of commits per file and the active days - the number of days that contributed modifications to a particular file. In the figure below, there is a list of files modified more than 200 times since the beginning of the project.

**Figure:** Files from the Moby repository with more than 100 commits made to them since the beginning of the project

The most commonly modified file is the daemon.go, which contains the platform-agnostic code responsible for running the Docker daemon. With almost 1500 lines of code, it serves as a typical example of a project hotspot. Interestingly, the Moby team is aware of that and mentions that file in its roadmap⁶ as a place that is in particular need of internal decoupling.

Another place where the files are commonly modified is CLI integration tests in the integration-cli directory.The project’s roadmap mentions the necessity of migrating the tests from integration-cli to integration by putting them into separate files and replacing the CLI invocations with direct API calls. The CLI integration tests are meant to be moved to a separate Docker CLI repository instead.

Analysis of the codebase changes made last year shows that although the previously identified hotspots are still there, specific changes correspond to the project roadmap’s points.

**Figure:** Files from the Moby repository that were modified most often in the last year

We can see most of the development being done around the daemon in terms of the code. Unlike in the previous analysis, this time, we can see that much more development was done in the platform-specific implementations (daemon_unix.go and daemon_windows.go). This shows that the objective of decoupling the daemon is in process.

The amount of changes in the integration-cli directory seems to have slowed down in the last few years. Further inspection with git effort shows that the tests that were migrated to the integration directory do not experience any code churn and were modified at most a few times since their creation.

The code quality, with a focus on hotspot components

Hotspot components, just like the parts of Moby’s repository that are less contributed to, are maintained to relatively high code quality due to the workflows happening within most open-source repositories. Here we refer to the well-known stages of a new contribution, which is that all new contributions and code additions have to be approved by repository maintainers and other relevant people before getting merged into the main branch. This allows people to check whether the contribution maintains the guideline to be followed as outlined in CONTRIBUTING.md. This is the last step in the rigorous code quality check process, involving manual checks of idiomatic and effective Go code, proper documentation and comments, and other aspects that any tool suite cannot automatically check. Moreover, as can be seen in the CI configuration file⁷ at the time of writing this, the validation procedure of each commit includes a rich series of scripts which makes sure that the following are respected:

No changes to the default seccomp⁸ profile;
Maintaining unspecific code and no internal imports in the utility packages folder pkg/*;
YAML files syntax validity and correct format;
API definitions are up-to-date;
TOML files syntax validity and correct format;
The changelog is well-formed;
No new component tests should be added to the integration-cli folder;
Go files syntax validity and correct format;
Shell script syntax validity and correct format;

Since these tests are run on each pushed commit, all the code currently in the repository maintains these rules. It is impossible to merge a new change to the main branch if the CI pipeline fails; thus, no new code quality issues are introduced to the repository.

The quality culture

The project uses labels for communicating the state of the pull requests which are added by the maintainers. status/0-triage, status/1-design-review, status/2-code-review, status/3-docs-review, and status/4-merge⁹ are used for the different stages that the code can be in. However, it should be noted that not all of these stages are used consistently. For example, there are many pull requests that have been merged without the status/4-merge label¹⁰ ¹¹ ¹² ¹³ ¹⁴.

In addition to this, there is a list of labels that forbid a given pull request to be merged as this PR for different reasons reasons: status/failing-ci, status/more-info-needed, status/needs-attention, and status/needs-vendoring. Before making a pull request, contributors are asked¹⁵ to run formatting tools and run the tests locally to ensure that their changes maintain the required coding standard and no other systems/modules are broken.

Overall, a positive code culture is present in the Moby repository. Most maintainers and reviewers provide valuable feedback. The discussions around issues and high-impact pull requests have only one thing in mind: maintaining a high standard and good code quality.

Technical debt assessment

Technical debt is the term given to the cost of refactoring code so that it is up to the standard it aims for¹⁶. Most of the time, technical debt is caused by the choice of faster implementation over the long-term maintainability of the code. To analyze the technical debt of the Moby project, we will look at TODO and FIXME comments, as well as analyzing the code with the SonarQube tool.

A metric that could be used is the number of TODOs and FIXMEs comments in the source code. Developers often write these when they foresee a problem that should be fixed in the near future. However, they are notorious for sticking around. The Moby project currently contains 578 TODOs and 140 FIXMEs. Although this seems like a lot in absolute numbers, given that the project consists of more than 1900 files, in perspective, it is not too large.

A tool to look at a system’s maintainability and technical debt is SonarQube. This open-source utility uses static analysis to reflect on the coding style, code complexity, code smells, and other metrics. These are all combined to give an estimate of the project’s technical debt. SonarQube scores the maintainability of the moby project with an A¹⁷.

**Figure:** Maintainability overview of the Moby project, as computed by SonarQube

The above diagram shows the number of code lines with the code smells and the technical debt they introduce. According to it, the vast majority of the code smells have a low impact and would be fast to fix.

The Technical Debt Ratio, the remediation cost over the development cost, is around 0.4%. Given the number of lines of code in the project and an estimate of 0.06 days to develop a single line of code, remediating the whole project is estimated to take around 63 workdays. This shows that the code of the Moby project is well maintained.

Conclusion

The overall quality of the Moby project is in good shape. Although the project is open for contributions from anyone, sufficient quality checks are in place to ensure the maintainability and reliability of the system. The continuous integration pipeline is helping the maintainers to review pull requests quickly. The Moby maintainers are also aware of the current code hotspots and technical debt. They included necessary refactors in the roadmap, as well as introduced the CI pipeline tasks that limit contribution to the parts under refactor, like the integration-cli tests.

Moby - Quality and Evolution