Self-hosting a Matrix server for 5 years
Experiences with the Matrix protocol, Matrix Synapse server, bridges, and Element mobile apps.
I have been hosting a Matrix server for about five years now, mostly for text chats between a few relatives and close friends, and a bridge to WhatsApp for a few more people. These are my experiences.
Matrix protocol
I don't have many thoughts on the protocol itself.
The only thing that I don't really understand is the decision on data replication. If a user on server A joins a room on server B, recent room data is copied from server B to server A and then kept in sync on both servers. I suppose this reduces the load on the original server at the expense of federation overhead and space on other servers. However, this also creates a situation where anything said across federation cannot be unsaid, which is an ironic situation for a protocol/system that often comes up when talking about privacy.
IIRC, fediverse/ActivityPub uses a similar approach.
Synapse server
Synapse is the only choice that supports bridges, which was why I wanted to try Matrix in the first place. And back in 2019-2020 this was the only choice anyway.
As of right now, I run Synapse, PostgreSQL, and coturn directly, without containerization, on a small VPS.
Works well
Works fairly reliably, supports bridges, and is more efficient that it was in 2020.
API is well documented, and allows authenticating and sending (unencrypted) messages via simple HTTP calls. At some point in time, I wanted to write a simple shell client to use with SXMO and such.
Does not have an admin panel
There is no admin page or panel. There was a third-party admin site, but it's an entire site just for making HTTP calls. So I ended up writing my own.
(Nowadays, the ESS deployment includes developer-made admin, see Future section)
Requires PostgreSQL
While technically, Synapse can work with a sqlite database (and which at first seems like an OK choice for having <10 users on the server), it WILL become corrupted. So PostgreSQL is de-facto mandatory.
(Already a part of new ESS)
Requires federation
Initial setup presumes that the server is going to be federated, and there is no good way to turn it off. The best workaround involves a blank whitelist of federated servers.
GitHub issue: Single config option to disable federation
I don't know the implications of disabling it.
Needs constant cleanup
Message retention policy can be set up server-wide, but also per-room. There are specific lines in the configuration that need to be set to actually enable a service that runs the cleanup.
Synapse keeps the room even after all of the members leave it, including federated rooms. This results in many (sometimes large) rooms without local members orphaned on the server, taking up database space.
Deleting messages (events) with attachments does not delete the attachment (because another message might refer to it?), which means that the sent files continue existing on the server indefinitely. Another privacy implication. A simple "delete all files older than X" script works great until it deletes avatars. So yeah, seems like this is something that should be handled by the Synapse server instead of cobbled-together scripts.
Even after extensive cleanup, PostgreSQL database might need to be vacuumed to reduce the disk space it takes up.
Database grows out of control
Even for my small server with <10 active users, database size reached several gigabytes.
Synapse keeps track of room states in an append-only (!) table named state_groups_state. Deleting a room does not delete the state_groups_state records. So it is never automatically cleaned up, and grows in size infinitely. It is possible to delete many of those records from the database directly, and Element (the company) provides some tool to "compress" those records, but again, something that should be handled by the server.
Good article about state_groups_state
Users cannot be deleted
This is simply not an option in the API. Server admin can perform a "deactivate" (disable login) and "erase" (remove related data, which claims to be GDPR-compliant) on user accounts, but the accounts themselves stay on the server forever.
Wait, what? Why?
How this not considered a GDPR violation is a mystery to me. Even on my tiny server, I have users who use their first name as their ID and bridged WhatsApp users that use phone numbers as IDs.
Future
While Matrix-Element ecosystem has been catering towards government and corporate entities for some time, there have been multiple recent announcements about its future.
Specifically, Element (the company) is now providing an all-in-one Element Server Suite (ESS) to replace the current setup, including
It is intended for non-professional use, evaluations, and small to mid-sized deployments (1–100 users).
ESS Community includes 7 components/services, now requires a minimum of 2 CPUs, 2GB of RAM, and runs using... Kubernetes? IMO, this is an overkill for dozen users.
For comparison, Snikket, an all-in-one solution with similar functionality using XMPP, requires a single CPU and 128MB (!) RAM for 10 or so users.
Yes, I have seen the ansible setup script setup recommended, but at this point, it looks like it is intended for the previous version(s).
Matrix server setup using Ansible and Docker
Also, the ESS handles account creation and calls in an entirely different way, more on that later.
Matrix-WhatsApp bridge
Pretty great. Easy to install and set up, works really well, and needs only occasional (semi-yearly or so) updates when WhatsApp changes their web API. Does not support calls.
Element Classic
Same on all platforms
Element exists and looks consistent on Android, iOS, and web, making it easier for regular users and for troubleshooting.
No image captions
This is silly, but while (official?) bridges support image captions, official Element app does not. The answer in the FAQ? Get a better app. Well, OK.
No image caption in Element Classic.
Image with a caption in SchildiChat Classic (the better app).
Slow notifications
Sometimes it can take up to a few minutes to get a message, even between two Android clients using Google Cloud Messaging. Sometimes it is nearly instant. Still unsure of the cause.
No offline indication
One unreliable way to tell that the server is unreachable is the endless loading bar. But even then, it eventually goes away without indicating any errors.
Then, when sending a message, the user receives "Unable to send message". Frustration ensues.
But I know the app is trying to call the /sync endpoint. Why doesn't it show any errors when that fails?
Security key and device verification
IIRC the first thing the app does is ask user to back up their signing keys and enter the key password, without a simple explanation. Not a great experience for regular users.
Some people reported issues with Element losing its keys or frequently requesting to be re-verified. Thankfully I have not encountered these.
Third-party services
Even if you connect to a self-hosted server, Element Classic could attempt to connect to vector.im integration server and matrix.org key backup server.
Element X
Element X is now recommended as the new and better client. It is not.
Slower
Somehow, it is slower. Clicking on a conversation takes 0.5-1.0 seconds to load it, compared to almost instant load on Classic.
Perhaps it does work better for accounts with many large rooms, but that is not my case.
Sorting
Conversations are sorted by... who knows. It is not recent nor alphabetical.
No background sync
Element X does not support periodic background sync, so you need to set up ntfy or something similar to use Element X on a de-googled device. Seems like a simple enough fail-safe (even WhatsApp does this), but it was dropped for some reason.
Requires "sliding sync" option on the server
This "sliding sync" option is available only for newer Synapse versions, and only if running with PostgreSQL database (which should already be the case - see above). Probably not an issue unless the user tries to connect Element X to an outdated Synapse.
Calls are not backward compatible
Calling with Element X requires Element Call (part of ESS). This supports group calls, but... only video calls at the moment.
You also might be asked to tell your contact to install the new app:
I don't regularly use calls, but some people I would like to invite to my server would want to use them.
Onboarding is bad
A few years ago, I ended up either temporarily enabling unrestricted registration (a terrible idea), or creating my users' accounts manually, because the "invite" matrix.to link was broken, and registration tokens did not work correctly in mobile apps.
So let's see how it works now. Keep in mind, I am still on standalone Synapse, not ESS.
Element X onboarding
I am a user, and I was to register an account on my friend's server. I see that Element X is now a recommended app, so let's try that.
Click "Create account" (which is a different style that does not look like a button for some reason).
But I want an account on a different server. Click "Change account provider".
Click "Other".
Now I can search for the server my friend is hosting, and it should appear in the list below the search.
As server admin: I do not remember if Synapse server has to enable/keep federation for this to work.
Yes! That is what I want, why is this so verbose?
WTF. So Element X cannot create even the simplest username+password account. That is all I want, I don't want to sign in with Google, Apple, or any other form of third-party authentication.
Element Classic onboarding
I was unable to register an account using Element X, so Element Classic should work better.
Ok, "CREATE ACCOUNT".
What difference does this make? Skip.
The current official app is telling me to use Element X. Just tried that. Click "EDIT" where it says "matrix.org" (which does not say "server", actually) and enter the server name.
Why not? No explanation. Sure, I'll use a web client.
Well, fuck me, I guess. Why can't I just create an account?
As a server admin: Synapse is set to allow registrations via registration tokens, because unrestricted registration is a bad idea. I did not find where the /static/client/register path is set.
IIRC it is possible to register an account by going to a web-hosted Element app, such as app.element.io, which will allow to register an account using a registration token. But then the user has to deal with the headache of cross-verifying their mobile device to the web app (which they might never use).
So now what?
Matrix-Element is growing, building new features, and acquiring large customers (mostly government entities AFAIK). However, the new corporatesque ESS Community is not worth it in my opinion. I don't need fancy auth, third-party IDs, group video conferencing, or even federation for that matter. But it is clear that Synapse and Element X are severely crippled and are not designed to work without these services.
I will probably switch to Snikket, which is more efficient, has timely notifications, and very smooth onboarding.
Who cares?
¯_(ツ)_/¯