Quality and Evolution

Introduction

The Quod Libet project has been under development since 2004 ¹. Over nearly two decades, the project has evolved significantly. The project has developed processes which help it satisfy its most important quality attributes, but it has also accumulated technical debt which can make it difficult to satisfy others.

Satisfaction of Quality Attributes

Quod Libet has placed a strong focus on certain quality attributes, such as performance, maintainability, and modularity. The projects official processes reflect these values, with an emphasis placed on avoiding performance regression, and on ensuring a standardized API for plugins. This effort has paid off, and the results are clear.

• Quod Libet’s Performance is very good. Not only can the program scale up to massive music collections, the program also starts up quickly, and maintains low latency during use.
• Maintainability is high. Quod Libet has made the build process straightforward, so it’s easy to begin developing for the project. As a result, the program receives commits from a long list of contributors ².
• The Modularity of the program makes development of plugins simple. Success in this regard has allowed Quod Libet to accumulate nearly 100 unique plugins.

Other quality attributes have received less attention, and outcomes have been less strong in these regions. For example, the project’s prioritization of power-users and lack of interface testing can cause Usability to suffer.

Quality Processes

The Quod Libet team has the goal to ensure that all the development versions are stable. The team assesses that the contributors’ patches respond to the specified coding guidelines. The guidelines regulate how the external contributor is supposed to introduce new features or patches, and mandate the contributor to comment the code in addition to passing all the existing tests that come with Quod Libet. The tests offer practical quality assessing and extensively cover most of Quod Libet’s aspects and features including plugins, audio formats, browser windows and connection to external services. Tests are further detailed in the dedicated section of this Essay. The Quod Libet team assesses the code so to ensure that no tests are skipped, and that any exception is thoroughly justified.

Testing and Continuous Integration

Quod Libet maintains 164 automated test files which can be found in quodlibet/tests in the source code. Quod Libet uses the CPython unittest framework, and pytest using python 3.7 or greater as a test runner ³, which allows the developer to write small, scalable and easy to read tests. Another advantage is that aspiring developers do not need to learn a new syntax or programming language for the testing framework, as the tests are written in python like the rest of Quod Libet’s code.

Executing the teste suite locally is as simple as running:

./setup.py
test

while the coverage can be checked by running:

./setup.py
coverage

Quod Libet does not accept any patches which break the provided unit tests. These tests are the bare minimum, however developers are encouraged to write their own tests for the functionality which they want to implement. It is also not acceptable to remove tests or change tests such that they pass, unless sufficient explanation is provided. When testing on the main branch, the feedback received is: 4482 tests passed, 52 skipped and 1 warning. The time it takes to run the full suit of tests is 138.86 seconds on a quad core CPU. Quod Libet also encourages the users of Quod Libet to test the latest code and provide feedback to the developers.

Despite its slow release cycle, Quod Libet maintains a healthy practice of Continuous Integration (but not Continuous Deployment). As mentioned above, Quod Libet comes with a set of integration tests that all must pass before a patch is accepted. This allows new patches to be easily integrated into the existing codebase without causing regressions.

When a merge request is submitted, several jobs are run, these are fedora, msys2, flatpak, macos, ubuntu-20.04, ubuntu-poetry. These ensure that Quod Libet runs on all the supported platforms, including Windows, Mac OS and the various Linux distributions. Only if all jobs pass, can a pull request be merged.

Code Quality

To measure the code quality of Quod Libet, we used Sonar Qube with Sonar Scanner. The image below shows our results:

According to these tools, Quod Libet has a technical debt of 22 days, which is the number of days it would approximately take to fix all of the code smells. A code smell is not a bug or an error that decreases the usability of the project itself, however it does reduce the quality of the code itself which could lead to problems further down the line, leading to difficulties in maintaining the project. Quod Libet was given a maintainability rating of A, which means that it is easy to maintain as the number of code smells is quite low, however the majority of these are classified as critical or major, as shown in the figure below.

Quod Libet was also reported to have 57 bugs, of which 48 are major, 2 are critical and 2 are blockers, with the 2 blocker bugs being 7 and 12 years old.

Hotspots

The most effective way of evaluating Quod Libet’s code quality is by looking at its hotspots. We can determine the project’s hotspots by examining its history of git commits. By this metric, Quod Libet’s top 10 most frequently changed files are:

1. quodlibet/widgets.py
2. quodlibet/quodlibet.py
3. quodlibet/NEWS
4. quodlibet/quodlibet/qltk/quodlibetwindow.py
5. quodlibet/quodlibet/__init__.py
6. quodlibet/library.py
7. quodlibet/quodlibet/qltk/songlist.py
8. quodlibet/quodlibet/formats/_audio.py
9. quodlibet/quodlibet/const.py
10. quodlibet/player.py

The purpose of these most changed files is mostly attributed to the core of the music player; GUI, library, songlist, audio, player.

• Widgets.py is currently located in the quodlibet/ext/songsmenu/brainz/widgets.py directory. The code is neat and not complex so easy readable. Variable names indicate a clear meaning and the code is well structured.
• Quodlibet.py is the startup script of Quodlibet and contains only 4 lines of code to start up the application.
• NEWS is the file that contains all the information on updates of Quod Libet and its newer versions.
• Quodlibetwindow.py is a GUI window that contains a classes with various functions and properties for the GUI. It is a long file with almost 1400 lines of code. Just as widgets.py, the code is not complex. Variable and function names give a clear indication of its purpose and the code is well structured.

This in general seems to be a core goal of Quod Libet: easy readable, non complex code.

It is interesting to see that for the last year, the Quod Libet code was probably already optimized to a certain extent and its developers were more focused on generalizing its use by working on different language implementations. The 10 most frequent changed files of the past year are primarily translations:

1. po/quodlibet.pot
2. po/es.po
3. po/fr.po
4. po/zh_CN.po
5. po/pt.po
6. po/nb.po
7. po/de.po
8. po/tr.po
9. po/en_GB.po
10. po/nl.po

Technical Debt

Technical debt is another lens through which we can examine Quod Libet’s code quality. Git-code-debt is used to assess the technical debt of Quod Libet. For the last year the code base increased but not by excessive amounts of lines of code. Also no decrease was noted so it seems there were no cases where lots of messy lines of code were dropped that were cleaned up after. When we go back to the merge that was done between end of april and beginning of may that caused the decrease in lines of code, https://github.com/quodlibet/quodlibet/commit/4e70c8189e7fbcd06f86d0d5b9b55cc2975eb1ae, it was an update of the PO files. The PO files are the language files. From the assesment of the results of git-code-debt it is not likely Quod Libet has a large technical debt.

Quality Culture

The discussions surrounding pull requests are an important piece of evidence which we can use to understand how Quod Libet arrived at its current level of code quality.

Quod Libet provides a template for new pull requests, and this can give us a clue as to exactly where the project’s stated values lie.

Check-list
----------

 * [ ] There is a linked issue discussing the motivations for this feature or bugfix
 * [ ] Unit tests have been added where possible
 * [ ] I've added / updated documentation for any user-facing features.
 * [ ] Performance seems to be comparable or better than current `master`


What this change is adding / fixing
-----------------------------------
<!-- A high-level description of the changes.
Hopefully the commits are atomic and well described, but this is the overall
summary of the change, to other developers / maintainers, plus any TODOs. -->

The top priority is that each PR is associated with a specific issue. This is important because it ensures that each PR has been discussed to some degree, and it discourages change for the sake of change.

The presence of unit tests for all new features or changes to behavior is important to ensure stability. The project’s CI pipeline is only as valuable as the tests it runs, so it’s good to keep the test suite up to date with the code being added.

Quod Libet’s documentation is a strong suit of the project, it makes sense that they would prioritize keeping this up to date in the same way as the tests. Moreover, documentation can ensure stability in ways unit tests cannot, it is one of only a few ways to ensure the consistency of the UI.

Finally, performance is given special attention, and measures should be taken to avoid regression in this regard. This makes sense, because the program’s performance is an especially important quality attribute of the project. After all, Quod Libet’s marketing copy specifically lists support for massive music collections as a key feature.

These are all noble goals, but to what degree are they actually achieved in practice? We can find out by looking at some of Quod Libet’s merged PRs. To obtain a representative sample of what the discussion surrounding the project looks like, the 10 most recent non-trivial Pull Requests were selected along with the issues they were intended to resolve.

This case study reveals several ways in which the project sometimes falls short of its stated goals.

Many of the issues on this list were created solely to justify the associated PR. This isn’t a bad practice on the face of it, because it encourages discussion before deciding if a PR is worthwhile. It is not quite ideal, because it would be better if most changes to Quod Libet were driven by actual requests from users.

PRs #3938, #3923, and #3918 are not associated with any issue at all. While #3938 references earlier discussion of the problem in an earlier PR, the other two make no mention of outside discussion at all.

Most of the merged PRs do not complete the checklist from the template before merging, and several omit it entirely. These PRs are mostly small changes in behavior, and none of them add to the unit test suite or to documentation. Even though this may seem less important for such small changes, it’s vital in order to prevent future regression.

Quod Libet benefits from its small scale, and these breaches in custom don’t appear to cause significant harm, but it’s clear that the project would benefit from following its processes more strictly.