QA (public) » openQA Project (public) » openQA Infrastructure (public)

Motivation¶

We always keep resolving storage host alert getting over more than 85% and while doing so we always scratch our head what data to delete.
Instead we should come-up with some data backup and retention policy for OSD and if possible for O3 as well, such that we should never have to be worried about low storage space for automatic data backup, unless there are some unavoidable circumstances.

Acceptance Criteria¶

• AC1: We have a properly documented backup and retention policy for storage.qe.prg2.suse.org (e.g. on qe-infra wiki page https://gitlab.suse.de/suse/wiki/-/blob/main/qe_infrastructure.md)
• AC2: We don't get the regular alert that the storage is running full
• AC3: We still keep data that is relevant for users

Suggestions¶

• ask on slack in #eng-testing and if people don't speak up it's their fault
• Save less snapshots
• exclude certain data
• enter filenames of old assets at the search at https://openqa.suse.de/admin/assets and remove them if they're not used anymore
• Discuss within tools team about backup and retention policy and come-up with an optimal backup proposal (keeping the motivation in mind)
• Discuss and present the proposal to other teams to bring everyone on the same page, if required re-iterate the proposal from AC1
• Cleanup old assets/data/logs from OSD and if required from O3 as well, implement the proposal (approved from AC2)

Further details¶

storage.qe.prg2.suse.org via rsnapshot in /home/rsnapshot
 - backup of openqa data (test result files without assets - "test result archive" - e.g. screenshots, video, serial log)
   - archive
   - fixed isos
   - fixed hdd images

 backup-vm via rsnapshot /home/rsnapshot
- osd database + /etc