In order to keep the quality of the architecture and code of Element high, a couple of measures are taken in the development process. The measures that need to be taken by the contributor are clearly explained in a contributing document ¹. Every contribution needs to be reviewed in order to get merged. This will mainly be done by the Element core team. In addition, each feature addition also means that tests must be added. This ensures that every feature has at least some tests. However, most repositories have no specified test coverage threshold. Also, bug fixes do not need additional test cases. Last but not least, there is an elaborate specification of the code style that needs to be used ². There are also measures that are automated to ensure a certain level of quality. These are executed by a continuous integration pipeline.

Key elements of continuous integration

As has been described in the previous essay, there are multiple repositories which are important for making Element work. This means that continuous integration happens at multiple places at once.

Anyone can contribute to the codebases. However, contributing to it means that a contributor agrees that their contribution will fall under the license of the respective codebase ¹. Contributions can only be added to one of the codebases through pull requests for the respective codebase. Each pull request needs to have a description about which changes have been made. This description needs to contain the following information:

Changelog entry
A reference to any bugs fixed by the pull request
What changes are made in the pull request and the reason for it
If it is something visual, it should contain before and after screenshots
Information about how the change can be tested
Comments to make it easier for the reviewer to understand what is going on

Each pull request for each codebase also has a different pipeline which consists of different jobs. These jobs can for example be used to run certain tests. The Codecov bot is used to easily display the difference in code coverage of the test suites before and after the change of the pull request ³. Furthermore, each pull request has to be reviewed by a member of the official core development team of Element.

For example, in the matrix-react-sdk repository each pull request is accompanied by around 10 jobs ⁴. Although, the number of jobs can vary depending on the repository. These jobs handle the following things:

Various kinds of testing, done through different jobs
Layered preview build
Notifying other repositories
Building the codebase

Testing

The codebases use a wide variety of different testing methods such as unit testing, end-to-end testing and integration testing. To get an idea of how extensively the codebases have been tested, we consider the repositories we have looked at before in the previous essay:

Element web, doesn’t contain any tests as it is just a skin for matrix-react-sdk.
Matrix-react-sdk contains 1257 tests with the following statistics ⁵:
- Statement coverage: 27.14%
- Branch coverage: 24.63%
- Function coverage: 24.52%
- Line coverage: 27.59%
Matrix-js-sdk, contains 704 tests with the following statistics ⁶:
- Statement coverage: 57.71%
- Branch coverage: 51.95%
- Function coverage: 57.67%
- Line coverage: 58.20%

It can be seen straight away that the codebases that actually implement the functionality of Element, are the only ones that are being tested. Element-web, which is a skin for matrix-react-sdk is not being tested at all.

For matrix-react-sdk, there are not a lot of requirements when it comes to testing. The only thing that is mentioned is that it needs comprehensive unit tests. However, when looking at the coverage reports for matrix-react-sdk, they seem to be rather low. Which does not seem like very comprehensive testing at all. The requirements for testing in matrix-react-sdk are loosely defined ¹. When looking at the test coverage of matrix-react-sdk it looks like as if they do not care about the test coverage.

Then for matrix-js-sdk, unit tests are required and a contributor should aim for at least 80% coverage in new code ¹. The effects of this statement are quite clear as the coverage in matrix-js-sdk is notably higher than the test coverage in matrix-react-sdk. However, the test coverage is still far below the mentioned 80%. This could be due to that it is not always possible to get a certain test coverage. It would make sense to not include code that cannot be tested in the test coverage analysis, otherwise you can not get a clear idea of how well the tests cover the code.

In general, the Element core team seems to take testing serious and have a clear framework setup to make testing easier. Although, they do not enforce their own testing requirements.

Hotspot components from the past and future

Analyzing Element’s repository ⁷ is easier said than done, since Element is built on Matrix. If we want a full overview of its components we need to have a look at Matrix’ components as well. There are two Matrix SDKs ⁸⁹ which Element uses. To generate these system level hotspots, we must therefore take Element’s code, as well as these two SDKs into account. We will be using CodeScene ¹⁰ to generate a hotspot overview.

element-web	matrix-js-sdk	matrix-react-sdk

For the standard element-web code, most activity is coming from the webpack.config.js file, with the rest of the system being quite timid. This is not exactly the case with both Matrix SDKs. Most activity in the React SDK can be found in views, structures and stores, whilst in the JS SDK most activity can be found primarily in the client.ts file. This file is over 5000 lines long and has more than 200 commits per year. Furthermore, this file has a very low code health score of 2 out of 10 (given by CodeScene). For the React SDK, the least healthy file is EventTile.tsx with over 1300 lines of code and also more than 200 commits per year. This file’s score is a bit higher than the React SDK’s client.ts file with a health score of 4 out of 10. For the future, a more general view would tell us that the most activity among these three repositories is mostly web related. So this will probably continue into the future. Element’s only activity can be found in webpack.config.js, which is also an indicator for future hotspots.

Code quality, focus on hotspot components

When redirecting our focus back to Element’s own repository, the before mentioned webpack.config.js file is the place to be. This file has some issues which are detected by the static analysis tool SonarLint ¹¹. Fortunately there is no reason for concern, as most of these issues are small code smells. Some of these code smells are: assigning to a variable instead of just returning the expression, name shadowing from outer scope, nested literals and some high cognitive complexity.

Let’s have a look at what SonarLint means with the high cognitive complexity. This is a snippet of a function which contains lots of conditionals, which could prove difficult to refactor and/or debug. Take note of the number of if/else statements as well as those present in assigning constants.

module.exports = (env, argv) => {
    const nodeEnv = argv.mode;
    const devMode = nodeEnv !== 'production';
    const useHMR = process.env.CSS_HOT_RELOAD === '1' && devMode;
    const fullPageErrors = process.env.FULL_PAGE_ERRORS === '1' && devMode;
    const enableMinification = !devMode && !process.env.CI_PACKAGE;

    const development = {};
    if (devMode) {
        development['devtool'] = "eval-source-map";
    } else {
        if (process.env.CI_PACKAGE) {
            development['devtool'] = 'source-map';
        } else {
            development['devtool'] = 'nosources-source-map';
        }
    }
    ...

Such code is not a major issue, but this is the file where SonarLint found the most issue. This along with CodeScene’s analysis of this file being a hotspot makes it somewhat noteworthy in this section.

Quality culture

To asses the quality culture of the Element project, issues and pull requests from the element-web ⁷ repository are analysed. This includes code for the web version Element. Although, issues and pull requests are handled in this repository, most of the actual changes are done in other repositories. For example: matrix-react-sdk ⁹ or matrix-js-sdk ⁸.

One thing that immediately catches your eye is the amount of issues open on this project. The issue count exceeds 5000+. Although it is not extraordinary for a large project such as Element to have a lot of issues open, it is unusual to still include issues from the year 2015 ¹². Since more important issues can be drowned by other less relevant ones, the moderators of Element might benefit by cleaning up the issues from time to time. This is however resolved by the excellent use of tags on the issues and pull requests. Taking the following issue¹³ as an example, it is clear that they use the S-Major tag for more important issues.

Further, Element is extremely specific in requirements for setting up a pull request. In their contributing document¹, they specify how to create a pull request and create issues. Almost all issues include steps to reproduce the issue, as well as images or videos, expected outcome and platform (for example see: ¹⁴). Though they specify strict regulations against code style, while analysing pull requests we could not find a single mention of it. Although, this could be due to developers adhering to these style conventions and the use of automated test cases to check for lint errors. See this issue for a failed pipeline due to lint errors ¹⁵. One thing the moderators do manually check for, is writing test cases. In this example ¹⁶, the pull request is blocked due to the changes not including test cases.

Finally, the interactions on issues and pull requests are active with moderators responding quick on issues. Often issues are also discussed in their own communication channel ¹⁷. Sometimes the interactions are really shallow, only including a LGTM. Other times they are lengthy and healthy discussions ¹⁸ ¹⁹.

Technical debt

Element definitely has its technical debt. One of the reasons our group decided to choose Element, is that it contains a lot of bugs and we are motivated to fix them. This is also visible on the issue board on GitHub ²⁰. Although most of these issues are minor bugs, it does show that there is a lot of inconsistency in the codebase.

Though the technical debt has not become that bad that new features are not being implemented. When looking at there roadmap ²¹ it is clear that the features they said out to do are now available on Element. With polls being the most recent example, which were added on 3 November 2021. Although one could provide a counter example to this, such as a conversation thread feature. This was created on 5 November 2020.

Since Element is developing at an extremely quick rate, adding as much features as possible, their technical debt might keep increasing. It is clearly visible that with each new future a ton of other features break ²². Therefore it is recommended that Element keeps their focus on implementing tests such that when a new feature is introduced, another one does not break. This should help to improve the internal quality of the overall codebase.

Quality attributes

As mentioned in the previous essays, the three main quality attributes that are important for Element are security, interoperability, and availability. Interoperability and availability are ensured by the decentralized architecture of Matrix. For availability, no additional measures are taken to ensure a high quality. For interoperability, a transpiler is used to ensure that the source code is runnable by as many people as possible.

Even though security is taken very serious in the design of Matrix, which is the underlying protocol used by Element, it is not considered in the design of the CI pipeline. For example, besides static analysis with ESLint, no security tests are done. A possible first improvement would be to use static analysis tools that look for known security vulnerabilities in dependencies ²³. This is easy to add to the project and can provide valuable insights. Besides the architecture and the CI pipeline, the Element project has clear guidelines when a vulnerability is found ²⁴.

References

M. (n.d.-c). matrix-js-sdk/CONTRIBUTING.md at develop · matrix-org/matrix-js-sdk. GitHub. Retrieved March 18, 2022, from https://github.com/matrix-org/matrix-js-sdk/blob/develop/CONTRIBUTING.md ↩︎
matrix-org. (n.d.-b). matrix-react-sdk/code_style.md. GitHub. Retrieved March 18, 2022, from https://github.com/matrix-org/matrix-react-sdk/blob/develop/code_style.md ↩︎
Codecov | Code Coverage - GitHub Marketplace. (n.d.). GitHub. Retrieved March 17, 2022, from https://github.com/marketplace/codecov ↩︎
Actions · matrix-org/matrix-react-sdk. (n.d.). GitHub. Retrieved March 17, 2022, from https://github.com/matrix-org/matrix-react-sdk/actions ↩︎
Fixes mx_MLocationBody_markerBorder · matrix-org/matrix-react-sdk@fe9ebe2. (n.d.). GitHub. Retrieved March 17, 2022, from https://github.com/matrix-org/matrix-react-sdk/runs/5583262820?check_suite_focus=true ↩︎
Apply redaction logic to threaded events · matrix-org/matrix-js-sdk@d2933d3. (n.d.). GitHub. Retrieved March 17, 2022, from https://github.com/matrix-org/matrix-js-sdk/runs/5585198718?check_suite_focus=true ↩︎
V. (n.d.). GitHub - vector-im/element-web: A glossy Matrix collaboration client for the web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web ↩︎
M. (n.d.-a). GitHub - matrix-org/matrix-js-sdk: Matrix Client-Server SDK for JavaScript. GitHub. https://github.com/matrix-org/matrix-js-sdk ↩︎
M. (n.d.-a). GitHub - matrix-org/matrix-react-sdk: Matrix SDK for React Javascript. GitHub. Retrieved March 18, 2022, from https://github.com/matrix-org/matrix-react-sdk ↩︎
Software Quality Visualization - Tech Debt｜CodeScene. (n.d.). Code Scene. From https://codescene.com/ ↩︎
Free and Open Source Code Quality & Security IDE Extension. (n.d.). SonarLint. From https://www.sonarlint.org/ ↩︎
V. (n.d.-l). We don’t handle 403s on rooms gracefully · Issue #18 · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/issues/18 ↩︎
V. (n.d.-d). Incorrect handling of deleted thread root messages · Issue #21466 · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/issues/21466 ↩︎
V. (n.d.-f). Long names are displayed inconsistently in thread summary · Issue #21472 · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/issues/21472 ↩︎
V. (n.d.-d). Auto-Element: Instant account registration and bookmark-based account login, for easier new user onboarding. by techmindful · Pull Request #18900 · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/pull/18900 ↩︎
V. (n.d.-h). Updated workflow to check if unlabelled issue is in X-Needs-Info column by ankur12-1610 · Pull Request #20528 · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/pull/20528 ↩︎
Element. (n.d.). Element | #element-dev. App.Element. Retrieved March 18, 2022, from https://app.element.io/#/room/#element-dev:matrix.org ↩︎
V. (n.d.-e). Auto-Element: Instant account registration and bookmark-based account login, for easier new user onboarding. by techmindful · Pull Request #18900 · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/pull/18900 ↩︎
V. (n.d.-i). Specify a local address when exposing ports with Docker by gibson042 · Pull Request #20891 · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/pull/20891 ↩︎
V. (n.d.-h). Issues · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/issues ↩︎
Element public roadmap · vector-im/roadmap. (n.d.). GitHub. Retrieved March 1, 2022, from https://github.com/vector-im/roadmap/projects/1 ↩︎
V. (n.d.-i). Plain text chat export fails for rooms with redacted polls · Issue #21337 · vector-im/element-web. GitHub. Retrieved March 18, 2022, from https://github.com/vector-im/element-web/issues/21337 ↩︎
GitHub. (n.d.). About Dependabot alerts. GitHub Docs. Retrieved March 18, 2022, from https://docs.github.com/en/code-security/dependabot/dependabot-alerts/about-dependabot-alerts ↩︎
Matrix. (n.d.). Security Disclosure Policy. Retrieved March 18, 2022, from https://www.matrix.org/security-disclosure-policy/ ↩︎

Element - Quality and Evolution