total descendants::0 total children::0 |
Let's start with a high-level summary. The matrix.org homeserver is backed by a large PostgreSQL database instance. Parts of an index on one of tables in this database had become corrupted. We are unsure exactly what caused this corruption, but believe it happened at least a year ago, and likely significantly longer. The nature of this corruption was such that it had little or no effect at first. However, a background maintenance task which removes old, unreferenced data from this table recently started working on the corrupted region. Due to the corrupt index, the maintenance task incorrectly removed active data from the table, in effect corrupting rooms. Having identified the problem, we rebuilt the corrupted index, and then restored the data that had been incorrectly removed, from database backups. https://matrix.org/blog/2025/07/postgres-corruption-postmortem/ |
| |||||||||||||||||||||||