AssertJ Core: Product Vision & Problem Analysis

AssertJ is a testing library for Java which provides users with clear and specific assertions. It is used by testers to test production code, by developers to understand the bugs in their code, and (indirectly) by users of software which AssertJ tests. It is an open-source project used widely across the Java community and has many stakeholders and desired quality attributes. This essay focuses on determining the product vision and analysing the domain.

AssertJ Core is one of the six modules of AssertJ, a library whose goal is to “provide a rich and intuitive set of strongly-typed assertions to use for unit testing”¹. The library also promises useful error messages, improved test code readability, and easy integration with popular IDEs for better usability². AssertJ Core focuses on providing assertions for JDK types (such as String, File, or Map).

In this essay, we will be using the terms “developer” and “tester” with distinct meanings. “Developer” denotes users of AssertJ who need to understand the output that it provides to be able to correctly implement or fix their production code. “Tester” represents users who write AssertJ test code (which test what the developers wrote). In the real world, this will often be the same person, however, we believe that this distinction is important for a few key concepts in this essay.

Key Domain Concepts

Being a library with a very specific focus, AssertJ falls into an obvious key domain: software testing. While we think of this as a single domain, there are multiple important aspects to consider. Testing is an important part of software development³, yet it can become quite tedious. The tools that we use for it should be reliable and easy to use. Therefore, we have identified three key concepts that are necessary for the success of a testing library.

Firstly, AssertJ should be easy to use for testers. This encompasses everything that is related to writing tests. This concept is crucial, since a library that is not understood by its target users will probably not be used. A key feature here is providing a clear overview of what is available within the library. This can be done through rigorous documentation or consistent naming conventions such that IDE auto-completion provides a sufficient overview. Preferably, it is a combination of both of these things.

Secondly, it is important that developers are able to read tests which use AssertJ. Every test in a test suite should be there for a reason and a developer should be able to look at the test and understand what it does and why it’s there. This can be achieved by clear assertion syntax, tailored assertions with meaningful names, and good documentation.

Lastly, testers and developers should understand the output of the test cases written using AssertJ. While the two previous points focused on static features, this one comes into play during runtime, i.e. when the test suite is executed. If a test fails, the users need to be able to understand the reason behind the failure. Errors and assertion failures should thus be communicated clearly.

Use Cases

AssertJ has one primary behaviour: asserting the behaviour of software when testing. It provides many different assertions which are tied to the types of the asserted objects. Let’s illustrate a specific use case by showing AssertJ in practice when testing the following method:

/**
 * This method takes a map and key and returns the corresponding value
 * if the key is present in the map.
 */
public static <K,V> Optional<V> findInMap(Map<K,V> map, K key) {...}

AssertJ can very expressively test the results of this method using the following assertions:

@Test
void testNotPresent() {
  // map does NOT contain key
  assertThat(findInMap(map, key)).isEmpty();
}

@Test
void testPresent() {
  // map contains key
  var result = findInMap(map, key);
  assertThat(result).isPresent();
  assertThat(result).hasValue(expectedValue);
}

Additionally, we write tests to find mistakes in our code. When a test fails, a developer reads the output of the error message to find out where the mistake is. This is another interaction between the user and AssertJ. Thanks to the many strongly-typed assertions, AssertJ is able to provide detailed error messages. An example can be found in a failed assertion for Map:

java.lang.AssertionError:
Expecting actual:
  {"A1"=300, "B5"=250}
to contain exactly (and in same order):
  ["A1"=300, "B5"=200]
but some elements were not found:
  ["B5"=200]
and others were not expected:
  ["B5"=250]

Context

It is important that we also discuss the external context within which AssertJ operates. This includes the users who interact with and rely on the library, as well as its dependencies. We discuss these in two separate subsections. The following figure illustrates the relationship between all of them:

Users

We have already defined the first two types of users in the introduction: the “tester” and the “developer”. These are both direct users, who interact with the library itself and actively use it.

A third type of user that is important in the context of AssertJ is the user who uses software which is tested using AssertJ. Unlike the two previous ones, this type of user does not interact with the library directly, but they should be able to put trust in the software which AssertJ helps test. Thus, they are a key player in the reliability requirements.

Dependencies

Perhaps the most obvious dependency is the language for which the “J” in AssertJ stands for: Java. The library is both written in Java and provides assertions for its types. The various modules of AssertJ each provide strongly-typed assertions¹ for different Java libraries but AssertJ Core focuses on the basic JDK types. Thus, the library is closely tied to the evolution of Java.

Additionally, important dependencies are the testing frameworks within which AssertJ operates. An example is the commonly used JUnit 5 or TestNG, both of which provide the infrastructure for running tests on the production code. AssertJ has to fit into their design and be part of the testing toolchain.

Closely related to the testing frameworks, other testing libraries are an important dependency that AssertJ must be able to cooperate with. This includes assertions available from the frameworks themselves (which might overlap largely with the ideas of AssertJ), but also other libraries with different focuses such as Mockito. Being able to flexibly use all these libraries within one test suite is often beneficial or even necessary for the test code⁴.

Finally, the IDE through which users interact with AssertJ is also a dependency. While a bit more abstracted from the library itself due to the in-between layer of the testing framework, providing clear test output to the user is a job that AssertJ and the IDE have to collaborate on. Therefore, this interaction also has to be considered when making design choices within AssertJ.

Identifying Stakeholders

Next, we identify the different stakeholders involved in the AssertJ Core project and their respective expectations from the system. We classify the stakeholders into end users, maintainers, and businesses.

End Users

Firstly, we consider the developers and testers who use AssertJ to test production code. For them, it is crucial that assertions provide a guarantee of correctness. Therefore, it is important for the testers that the library itself is tested and that it provides the promised functionality with no major flaws. Additionally, these stakeholders also care about the ease-of-use of the library.

Similarly, we discuss the end users who use applications tested by AssertJ. These users care about the proper functioning of the applications they are using and their main interest will be the correctness of the library even though they might not be aware of it. They are what we classify as “silent stakeholders”⁵, i.e. that they do not really have a say in the development but their interests are still an important factor to consider.

Maintainers

With “maintainers” we refer to the people who contribute to AssertJ. They are a very important stakeholder, since they are the ones to directly decide what happens to the library. We can divide them into two groups: the contributors and owners of the project. The contributors write code and implement features they want and the owners make decisions about whether the contributions are of sufficient standard. Both groups are interested in the functionalities and quality of the library.

Businesses

Companies that rely on software tested with AssertJ form a group of important stakeholders. Generally, they could be considered silent, although in an open-source project such as AssertJ, they can have a say if they want to. They need to be sure that the software they use, whether it is internally in their organization or as a part of their product, has been properly tested and that they can trust AssertJ when it reports success.

Quality Attributes

The main quality characteristics defined by ISO 25000 Standards⁶ are summarised in the table below. It also shows the three most relevant quality attributes we identified for the different stakeholders within the system. (The businesses are left out of further analysis, since they do not provide a new perspective.)

We can easily see from the table above that functional suitability and reliability are the most important quality attributes for AssertJ Core as a whole. This makes sense as it is a testing library and making sure it can be relied upon is absolutely crucial. This is a common theme with most stakeholders of the system.

Unsurprisingly, maintainability is another crucial characteristic for maintainers of the system. This is because easily adapting the library due to changes in requirements makes their job easier. For developers, both the ease with which the system is used as well as the performance efficiency greatly influence their interactions with AssertJ. Testers writing tests using the library prefer it to be compatible with their favourite IDE, to provide intuitive code completion, and to be easily understood. Users, or people who implicitly trust that AssertJ sufficiently tests the software they are using, are not interested in the actual codebase but care about the reliability and security properties of the system as a whole.

Product Roadmap

Going back to AssertJ’s ambition of providing “a rich and intuitive set of strongly-typed assertions for unit testing”¹, one can argue that this has been mostly achieved already. There is no next big milestone for AssertJ Core except for keeping up with changes in dependencies, resolving any bugs that may arise, and accepting contributions. Additionally, the whole testing library might need to be extended in the future to provide assertions for newly created types. There is no clear roadmap written down by the maintainers, but there are currently two milestones defined in the repository and the next major version bump contains issues that focus on refactoring the codebase.

Ethics

The role of software testing within society is becoming more and more crucial as software slowly grows into almost every aspect of our lives. While some bugs in software do not make that big an impact, “the failure of some software could cause catastrophic consequences for individuals and even society”⁷. It thus becomes crucial that the software we write is tested to the best possible extent, using tools that we trust.

This is also where a very important ethical consideration regarding AssertJ comes into play, since there is a chance that it will be used to test crucial software. This means that if there is a bug in the library, it might lead testers into a false sense of security, potentially resulting in terrible consequences. The question of responsibility is a common issue in open-source libraries: users should be aware of this and use the libraries “at their own risk”, but the maintainers should acknowledge the role of their projects in the software ecosystem.

As a small side note on the ethics of construction, we should also mention that AssertJ is an open-source Java library which an Apache 2.0 license, allowing developers to contribute to and use it almost freely. Their Code of Conduct asks participants to respect each other and meet the community standards.

References