Project

General

Profile

action #96269

openQA Project - coordination #103947: [saga][epic] Scale up: Future proof backup of o3+osd

Define what a "complete OSD backup" should or can include

Added by okurz 10 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2021-07-29
Due date:
% Done:

0%

Estimated time:

Description

Motivation

We want to have a complete OSD backup but for that first we need to understand what can be included in "complete", depending on importance, restorability, expected sizes and available space

Acceptance criteria

  • AC1: We know for which data from OSD we have enough space to save at least one copy on storage.qa.suse.de

Suggestions

  • DONE: Find out currently available space on storage.qa.suse.de -> as of 2021-07-29 we have 11TB free
  • Get an overview of which components of storage and which folders take how much space, e.g. based on
df -h
Filesystem      Size  Used Avail Use% Mounted on
…
/dev/vda1        20G  6.8G   12G  37% /
/dev/vdb        100G   74G   27G  74% /srv
…
/dev/vdc        7.0T  5.9T  1.2T  84% /assets
/dev/vde        5.5T  2.0T  3.6T  37% /space-slow
/dev/vdd        5.5T  4.8T  809G  86% /results
  • Update #88546 with the result of what should/can be included in a complete backup

Related issues

Related to openQA Infrastructure - action #44078: Implement proper backups for o3 size:MResolved2018-11-20

Copied from openQA Infrastructure - action #88546: Make use of the new "Storage Server", e.g. complete OSD backupResolved

History

#1 Updated by okurz 10 months ago

  • Copied from action #88546: Make use of the new "Storage Server", e.g. complete OSD backup added

#2 Updated by okurz 10 months ago

  • Related to action #44078: Implement proper backups for o3 size:M added

#3 Updated by okurz 10 months ago

  • Status changed from New to Blocked
  • Assignee set to okurz

#44078 first should help

#4 Updated by okurz 5 months ago

  • Parent task changed from #80546 to #103947

#5 Updated by okurz 5 months ago

Based on our experiences that we have now since recently with backup of o3 to storage.qa and the available space on storage.qa I would say the following rsnapshot based backup should be feasible:

backup  root@openqa.suse.de:/var/lib/openqa/images openqa.suse.de/root
backup  root@openqa.suse.de:/var/lib/openqa/archive        openqa.suse.de/root
backup  root@openqa.suse.de:/var/lib/openqa/share/factory/iso/fixed        openqa.suse.de/root        +rsync_long_args=--hard-links
backup  root@openqa.suse.de:/var/lib/openqa/share/factory/hdd/fixed        openqa.suse.de/root        +rsync_long_args=--hard-links

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/633

#6 Updated by okurz 5 months ago

  • Status changed from Blocked to Feedback

#7 Updated by okurz 5 months ago

Missing ssh keys

On storage.qa called

cd  /root/.ssh
ssh-keygen -t ed25519 -N '' -C 'root@storage.qa.suse.de, backup OSD' -f id_ed25519.backup_osd

then copied to pillars:

ssh storage.qa "sudo cat /root/.ssh/id_ed25519.backup_osd" > id_ed25519.backup_osd
ssh storage.qa "sudo cat /root/.ssh/id_ed25519.backup_osd.pub" > id_ed25519.backup_osd.pub

Created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/376 and https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/636

#8 Updated by okurz 5 months ago

Both merged. Monitoring on storage.qa. Currently rsnapshot is already running and busy with syncing from o3. Decided with mkittler to leave out "images" from backup as it seems to take too long. Created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/637 , merged and triggered rsnapshot manually in a screen session. Monitoring.

#9 Updated by okurz 4 months ago

  • Status changed from Feedback to Resolved

backup looks good the past days. df shows enough free space. Paths like /storage/rsnapshot/alpha.0/openqa.suse.de/root/var/lib/openqa/share/factory/hdd/fixed/ look good.

Also available in: Atom PDF