action #96269
closed
openQA Project - coordination #103947: [saga][epic] Scale up: Future proof backup of o3+osd
Define what a "complete OSD backup" should or can include
Added by okurz about 3 years ago.
Updated over 2 years ago.
Description
Motivation¶
We want to have a complete OSD backup but for that first we need to understand what can be included in "complete", depending on importance, restorability, expected sizes and available space
Acceptance criteria¶
- AC1: We know for which data from OSD we have enough space to save at least one copy on storage.qa.suse.de
Suggestions¶
- DONE: Find out currently available space on storage.qa.suse.de -> as of 2021-07-29 we have 11TB free
- Get an overview of which components of storage and which folders take how much space, e.g. based on
df -h
Filesystem Size Used Avail Use% Mounted on
…
/dev/vda1 20G 6.8G 12G 37% /
/dev/vdb 100G 74G 27G 74% /srv
…
/dev/vdc 7.0T 5.9T 1.2T 84% /assets
/dev/vde 5.5T 2.0T 3.6T 37% /space-slow
/dev/vdd 5.5T 4.8T 809G 86% /results
- Update #88546 with the result of what should/can be included in a complete backup
- Copied from action #88546: Make use of the new "Storage Server", e.g. complete OSD backup added
- Related to action #44078: Implement proper backups for o3 size:M added
- Status changed from New to Blocked
- Assignee set to okurz
- Parent task changed from #80546 to #103947
Based on our experiences that we have now since recently with backup of o3 to storage.qa and the available space on storage.qa I would say the following rsnapshot based backup should be feasible:
backup root@openqa.suse.de:/var/lib/openqa/images openqa.suse.de/root
backup root@openqa.suse.de:/var/lib/openqa/archive openqa.suse.de/root
backup root@openqa.suse.de:/var/lib/openqa/share/factory/iso/fixed openqa.suse.de/root +rsync_long_args=--hard-links
backup root@openqa.suse.de:/var/lib/openqa/share/factory/hdd/fixed openqa.suse.de/root +rsync_long_args=--hard-links
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/633
- Status changed from Blocked to Feedback
Both merged. Monitoring on storage.qa. Currently rsnapshot is already running and busy with syncing from o3. Decided with mkittler to leave out "images" from backup as it seems to take too long. Created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/637 , merged and triggered rsnapshot manually in a screen session. Monitoring.
- Status changed from Feedback to Resolved
backup looks good the past days. df
shows enough free space. Paths like /storage/rsnapshot/alpha.0/openqa.suse.de/root/var/lib/openqa/share/factory/hdd/fixed/ look good.
Also available in: Atom
PDF