action #96269
closedopenQA Project - coordination #103947: [saga][epic] Scale up: Future proof backup of o3+osd
Define what a "complete OSD backup" should or can include
0%
Description
Motivation¶
We want to have a complete OSD backup but for that first we need to understand what can be included in "complete", depending on importance, restorability, expected sizes and available space
Acceptance criteria¶
- AC1: We know for which data from OSD we have enough space to save at least one copy on storage.qa.suse.de
Suggestions¶
- DONE: Find out currently available space on storage.qa.suse.de -> as of 2021-07-29 we have 11TB free
- Get an overview of which components of storage and which folders take how much space, e.g. based on
df -h
Filesystem Size Used Avail Use% Mounted on
…
/dev/vda1 20G 6.8G 12G 37% /
/dev/vdb 100G 74G 27G 74% /srv
…
/dev/vdc 7.0T 5.9T 1.2T 84% /assets
/dev/vde 5.5T 2.0T 3.6T 37% /space-slow
/dev/vdd 5.5T 4.8T 809G 86% /results
- Update #88546 with the result of what should/can be included in a complete backup
Updated by okurz over 2 years ago
- Copied from action #88546: Make use of the new "Storage Server", e.g. complete OSD backup added
Updated by okurz over 2 years ago
- Related to action #44078: Implement proper backups for o3 size:M added
Updated by okurz over 2 years ago
- Status changed from New to Blocked
- Assignee set to okurz
#44078 first should help
Updated by okurz over 2 years ago
Based on our experiences that we have now since recently with backup of o3 to storage.qa and the available space on storage.qa I would say the following rsnapshot based backup should be feasible:
backup root@openqa.suse.de:/var/lib/openqa/images openqa.suse.de/root
backup root@openqa.suse.de:/var/lib/openqa/archive openqa.suse.de/root
backup root@openqa.suse.de:/var/lib/openqa/share/factory/iso/fixed openqa.suse.de/root +rsync_long_args=--hard-links
backup root@openqa.suse.de:/var/lib/openqa/share/factory/hdd/fixed openqa.suse.de/root +rsync_long_args=--hard-links
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/633
Updated by okurz over 2 years ago
Missing ssh keys
On storage.qa called
cd /root/.ssh
ssh-keygen -t ed25519 -N '' -C 'root@storage.qa.suse.de, backup OSD' -f id_ed25519.backup_osd
then copied to pillars:
ssh storage.qa "sudo cat /root/.ssh/id_ed25519.backup_osd" > id_ed25519.backup_osd
ssh storage.qa "sudo cat /root/.ssh/id_ed25519.backup_osd.pub" > id_ed25519.backup_osd.pub
Created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/376 and https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/636
Updated by okurz over 2 years ago
Both merged. Monitoring on storage.qa. Currently rsnapshot is already running and busy with syncing from o3. Decided with mkittler to leave out "images" from backup as it seems to take too long. Created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/637 , merged and triggered rsnapshot manually in a screen session. Monitoring.
Updated by okurz over 2 years ago
- Status changed from Feedback to Resolved
backup looks good the past days. df shows enough free space. Paths like /storage/rsnapshot/alpha.0/openqa.suse.de/root/var/lib/openqa/share/factory/hdd/fixed/ look good.