Mattermost is a collaboration platform for teams of any size. The system needs to be equipped to handle communication in large teams. The local mobile app will only be used by one person (namely the user of that specific phone) at a time, but there are still scalability challenges. In this essay, we outline a few challenges related to performance and user experience and propose a few solutions.
Key Challenges
The number of active users in a Mattermost channel might be very large. The mobile app needs to be able to cope with a high frequency of messages and a large message history in channels. This large history can be particularly challenging when the search bar is used to find a common word. Smart decisions need to be made here to keep the application running smoothly.
Aside from those performance issues, there is also the possible effect on user experience when the number of participants in a channel increases. The UI needs to be designed so that a large number of new messages from different conversations are still quick and easy to read. It is our opinion that the threads used to organize message responses can be hard to read at times. Mattermost is already working on a new1 to improve this.
When Mattermost gains a larger audience it is likely going to consist of less experienced or tech-friendly users. The system needs to guard itself against possible abuse by these users. Mattermost mobile has a few pending issues related to this scalability challenge:
- Very long posts crash/freeze the mobile app2
- Large Image File/Size Upload Fails - Small Image File/Size Upload Success - iOS or Android3
- Cannot upload high resolution pictures in Android app4
- Android app freezes when too many gifs in one message5
Since such scalability issues seem to occur frequently, we have decided to perform our analysis of the robustness of the iOS app when having to send and load large messages.
Empirical Quantitative analysis
For our quantitative analysis, we focus on the scalability of the key feature of Mattermost: messaging. While exploring the app and putting its performance to the test, we discovered that after sending multiple large messages to a channel, the message loading of the channel stalled and the user interface froze, making the app unusable.
The problem manifests itself when the app loads in messages. This happens when the user refreshes the specific channel, directly causing the loading of the messages, or by opening the drawer from another channel, which causes the app to start loading the new messages in the background. In both scenarios the user interface freezes and the app becomes unusable.
Experiment Setup
To gain insight in this problem we performed the following empirical quantitative analysis. We performed two different experiments in which we studied the impact of the size and the volume of messages on the performance of the app, which is measured in the loading time of the messages.
Our experiments were conducted on an iPhone 12 mini, with iOS version 15.1.1, Mattermost Beta version 1.50.0, and server version 6.3.2. The Mattermost Beta app was built using Xcode and the results were analyzed using Xcode Instruments.
For our experiment, we focused on text-only messages, consisting of only ‘x’ characters. Mattermost puts the character limit of a text image on 16,383 (2¹⁴ - 1) characters, so each message consists of maximally 16,383 ‘x’ characters.
The first experiment we conducted studied the impact of the size of messages on the loading time. We sent out 8 different sizes of messages, from 128 (2⁷) to 16,383 (2¹⁴ - 1) characters. For each size, we sent 10 messages and took the average loading time.
The second experiment we conducted studied the impact of the volume of messages on the loading time. When loading messages, Mattermost loads them all at once, with some irregularities where it shows parts of the messages sent while it is still loading. We sent out 10 different volumes, from 1 message at a time, up to 10 messages sent at a time. For each volume, we sent out 3 batches and took the average loading time.
Results
The first experiment resulted in the following data:
Figure: Impact of the message size.
We see an increasing growth of the loading time when the image size is increased to its maximum limit. The app deals fairly well with the smaller-sized messages but starts to slow down significantly for the larger sizes, with an almost fourfold increase in loading time with only a twofold increase in message size for the two biggest message sizes.
As a hypothesis of why the average loading time grows exponentially, we note that Mattermost uses regular expressions on message parsing. According to SonarQube, in the worst case, the complexity of a regular expression is exponential in the size of the input6.
The second experiment resulted in the following data:
Figure: Impact of the message volume.
We see that as the message volume increases, the average loading time also seems to increase. An important side-effect of increasing the message volume is that the performance of the app seems to become very unstable. When repeating our experiments, we found that for some measurements the app kept loading without stopping, rendering the app unusable for an indefinite amount of time. These anomalies were omitted from the average loading times shown above.
Causes
We attribute the decline in performance to thermal throttling. When the app tries to download and process large amounts of data, the CPU temperature increases, leading to throttling and bad performance. When we push this even further, we reach a point where the app freezes and no longer seems to be able to finish the download.
The thermal throttling can be seen in our second experiment where we send multiple batches of four maximum-length messages:
Figure: Impact of thermal throttling.
After four batches, the thermal state of the CPU is increased from the “nominal” state to the “serious” state, leading to the performance drop seen in the CPU profiler’s fifth measurement.
Architectural decisions
Mattermost messages and media are stored both on the server and on the client. They are stored on the server so that if a user joins a channel, they can see the older messages. They are stored on the client so that when reopening the app, the messages do not have to be loaded from the server again.
A Mattermost server imposes various limits on media transfer to its clients. For example, a message cannot be longer than 16,383 (2¹⁴ - 1) characters and 65,535 (2¹⁶ - 1) bytes, presumably because messages with a large number of characters slow down the app. Also, a media file cannot be larger than 50 MB to save data bandwidth.
Mattermost customers that host their own server are able to change these limits: the file build/docker/nginx/default.conf defines the client_max_body_size as 50M. The maximum amount of characters of a message cannot be changed, however.
Mattermost has search functionality to search for messages7. Certain queries can take a long time to return results: searching a trivial query like “and” on the Mattermost server for this course, for instance, takes a significant amount of time (> 5 seconds). Mattermost does, however, allow their customers to use Elasticsearch, which aims to optimize search queries8.
Finally, a large list like the display of messages in a channel could cause performance issues. A device can only store so many messages in memory, and having too many messages stored could make the app slow. To combat this, Mattermost Mobile uses RecyclerViews and UITableViews for Android and iOS, respectively. These tools load the cells lazily and aggressively drop its content out of memory when the cells disappear from the screen.
Proposed architectural changes
In the previous section, we looked at architecture choices in the Mattermost app that affect its performance, reliability, and ultimately, user experience. We identified two main areas of interest that affect these qualities, namely: CPU usage and memory pressure. In this section, we look at possible solutions to these problems. While these changes are too wide-ranging for us to implement them for an experiment, we can reason why these changes would work by analyzing the current code.
Memory for large messages
One of the main issues Mattermost Mobile needs to deal with is the fact that users may want to upload very long and memory-intensive content to a channel. Mattermost has a character limit of 16,383 (2¹⁴ - 1) characters, but on memory-constrained devices like mobile devices, this can prove to be too much, as was shown by the experiments. Furthermore, users may send large multimedia files, such as GIFs, which are currently stored in memory in their entirety to be rendered inline.
Solution one: aggressively drop cells from screen
As discussed in a previous section, Mattermost Mobile uses optimized list views to manage which messages stay in memory. However, if only the limited number of messages on screen are already memory-intensive enough to cause memory to run out, we’re in trouble: this solution cannot help anymore. We will need the other tools in our toolbox, which we discuss below.
Solution two: load one message at a time
A large number of messages can crash the app, as we showed before. If we only load one message at a time and keep track of memory pressure, we can solve this issue. If pressure rises too high, we don’t render new messages anymore. However, this has two significant drawbacks. The first is that rendering multiple messages will be slower, because we don’t make full use of the parallelization the GPU has to offer. Secondly, the user experience is still not good: although it’s better than freezing the app, the user will still be annoyed that their messages are not loading.
Solution three: convert long messages to a file
This is a solution that can be quite helpful and does not impact user experience that badly: if we detect that the content we’re about to render is very large, and we suspect we may run into memory problems, don’t load the content inline but render the content as a file, which the user can download by tapping on it. This allows the user to save the file to their phone without running into memory issues.
Solution four: don’t load large messages
As a last resort, the server or app could detect large messages and simply refuse to load them. While this is of course bad for the user experience, a freezing app is even worse. It should therefore be considered in case the solutions proposed above are not feasible.
Conclusion
In this essay, we discussed the performance of the app, viewed as a scalability challenge, and ways we could combat occurrences of lagging and freezing of the mobile app without disrupting the user experience more than necessary. We focused on the influence of the size and volume of messages on the message loading times and showed that these times increased rapidly to the point of freezing the app. We proposed four possible solutions to this problem.
Sources
-
Mattermost: Collapsed Reply Threads. https://mattermost.com/blog/dev-sneak-peek-collapsed-reply-threads/#:~:text=Collapsed%20Threads,in%20the%20right%2Dhand%20side. Accessed: 27-Mar-22 ↩︎
-
Github issue: Very long posts crashes/freezes the mobile app. https://github.com/mattermost/mattermost-mobile/issues/5956 Accessed: 22-Mar-22 ↩︎
-
Github issue: Large Image File/Size Upload Fails - Small Image File/Size Upload Success - iOS or Android. https://github.com/mattermost/mattermost-mobile/issues/5818 Accessed: 22-Mar-22 ↩︎
-
Github issue: cannot upload high resolution pictures in android app. https://github.com/mattermost/mattermost-mobile/issues/5080 Accessed: 22-Mar-22 ↩︎
-
Github issue: Android app freezes when too many gifs in one message. https://github.com/mattermost/mattermost-mobile/issues/4421 Accessed: 22-Mar-22 ↩︎
-
SonarQube: Regular expressions should not be vulnerable to Denial of Service attacks. https://rules.sonarsource.com/java/RSPEC-2631 Accessed: 24-Mar-22 ↩︎
-
Mattermost: Search for Messages. https://docs.mattermost.com/channels/search-for-messages.html Accessed: 20-Mar-22 ↩︎
-
Mattermost Elasticsearch. https://docs.mattermost.com/scale/elasticsearch.html Accessed: 20-Mar-22 ↩︎