New SSD for mirrorbrain database
Thanks to our monitoring, we were notified soon enough that one of the (standard) SSDs we use in our database backend for download.opensuse.org was about to die...
The problem was fixed by
- starting our hot-standby database as new master
- replacing the half broken SSD with a new one
- re-syncing from the current master ...and while we are on it, we took the chance to let PostgreSQL do some cleanups via the "vacuum" and "reindex" commands: the database shrunk from ~30GB to ~10GB, but this took nearly 2 hours!
During the whole time, download.opensuse.org was still reachable - just a bit slow for 1 hour during the vacuuming (sorry for that). Now everything is back to normal, but we will try to figure out how the autovacuum can be tuned, so the database cleanup can be done automatically.