Project

General

Profile

action #44078

Implement proper backups for o3 size:M

Added by okurz almost 3 years ago. Updated about 2 months ago.

Status:
Workable
Priority:
Low
Assignee:
-
Target version:
Start date:
2018-11-20
Due date:
% Done:

0%

Estimated time:

Description

Motivation

We should find a backup space for o3, e.g. 4TB for test data/assets

Acceptance criteria

  • AC1: Test assets are backed up and can be restored
  • AC2: Test results are backed up and can be restored
  • AC3: Screenshots are backed up and can be restored
  • AC4: List of installed packages is backed up for reference

Suggestions

  • Use storage.qa.suse.de
  • We already have database backups elsewhere
  • Consider snapshots (if available) or rsync

Out of scope

Further details


Related issues

Related to openQA Infrastructure - action #69577: Handle installation of the new "Storage Server"Resolved2020-08-04

Related to openQA Infrastructure - action #88546: Make use of the new "Storage Server", e.g. complete OSD backupBlocked

Related to openQA Infrastructure - action #96269: Define what a "complete OSD backup" should or can includeBlocked2021-07-29

Copied to openQA Infrastructure - action #94015: proper backup for osdNew

History

#2 Updated by okurz over 2 years ago

As o3 is in a special network I recommend rsnapshot on backup.qa.suse.de syncing the data from o3.

#3 Updated by okurz over 2 years ago

#4 Updated by okurz over 2 years ago

  • Status changed from New to In Progress
  • Assignee set to okurz

Talked with tbro. 4TB is a problem. Anything in the range of up to 100GB is no problem to have, using space on old netapp, on request. However, I think then we can just as easily go with backup.qa which is easier for us given that we have full control of that machine and can trigger backups from there as suggested in #44078#note-2

#5 Updated by okurz over 2 years ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

Unassigning again as preparation for longer absence. I should really not leave tickets assigned to me dangling "In Progress" :)

#7 Updated by okurz over 2 years ago

Automatic backup for the o3 webui host introduced with https://gitlab.suse.de/okurz/backup-server-salt/tree/master/rsnapshot covering so far /etc and the SQL database dumps. As next steps I recommend to save a bit more if feasible from /var/lib/openqa as well as from the workers which are transactional-servers. Maybe as well just /etc plus an autoyast profile.

#8 Updated by okurz about 1 year ago

  • Priority changed from Normal to Low
  • Target version set to Ready

#9 Updated by okurz 12 months ago

  • Related to action #69577: Handle installation of the new "Storage Server" added

#10 Updated by okurz 12 months ago

  • Status changed from Workable to Blocked
  • Assignee set to okurz

waiting for #69577 or #41918 first

#11 Updated by okurz 4 months ago

#12 Updated by okurz 4 months ago

  • Status changed from Blocked to New
  • Assignee deleted (okurz)

storage.qa.suse.de in place and usable

#13 Updated by cdywan 3 months ago

  • Subject changed from proper backup for o3 to Implement proper backups for o3 size:M
  • Description updated (diff)
  • Status changed from New to Workable

#14 Updated by okurz 3 months ago

  • Related to action #88546: Make use of the new "Storage Server", e.g. complete OSD backup added

#15 Updated by okurz 3 months ago

  • Related to action #96269: Define what a "complete OSD backup" should or can include added

#16 Updated by okurz about 2 months ago

  • Description updated (diff)

#17 Updated by okurz about 2 months ago

  • Description updated (diff)

#18 Updated by okurz about 2 months ago

  • Description updated (diff)

#19 Updated by cdywan about 2 months ago

I'm taking it now. Discussing it in the weekly I feel like it might be easier to treat it as a spike and propose something, and then see if we're happy with that :-D

#20 Updated by cdywan about 2 months ago

  • Assignee set to cdywan

#21 Updated by cdywan about 2 months ago

  • AC4: List of installed packages is backed up for reference
$ rpm -qa
[...]
$ zypper in gnome-maps-40.1-1.2.x86_64
[...]
Loading repository data...
Reading installed packages...
'gnome-maps.x86_64 = 40.1-1.2' is already installed.
There is an update candidate 'gnome-maps-40.4-1.1.x86_64' for 'gnome-maps-40.1-1.2.x86_64', but it does not match the specified version, architecture, or repository.
$ rpm -qa > packages.txt; sudo zypper in -D $(sed 's/-[a-z].*//' packages.txt)
[...]
$ zypper -x se --type=package -i
[...]
$ pip3 install --user yq
[...]
$ zypper -x se --type=package -i > packages.xml
$ sudo zypper in -D $(~/.local/bin/xq '.stream|."search-result"|."solvable-list"|.solvable|.[]|."@name"' packages.xml | sed 's/"//g')

Proof of concept using xq to process XML output from zypper seems to work for (re)storing the list of installed packages.

#22 Updated by okurz about 2 months ago

I wasn't aware about "yq", sounds great! But why do you care about this? rpm -qa should suffice and we don't want to recover systems from this list, just have it ready as a reference. I see now that in https://progress.opensuse.org/journals/427051/diff?detail_id=404851 you added the "list of packages". As this is a rather short document I suggest to just include it in the existing rsnapshot config for backup.qa.suse.de, see https://gitlab.suse.de/qa-sle/backup-server-salt/-/blob/master/rsnapshot/rsnapshot.conf#L27 for how we call commands to backup SQL. I suggest you exclude the part about "list of packages" into another ticket or just do it to get to the more demanding/interesting parts of screenshots+assets+results for which we should use storage.qa.suse.de as target

#23 Updated by cdywan about 2 months ago

  • Assignee deleted (cdywan)

okurz wrote:

I wasn't aware about "yq", sounds great! But why do you care about this? rpm -qa should suffice and we don't want to recover systems from this list, just have it ready as a reference.

Like I already said on the call, "proper backups" includes being able to recover from said backups in my book. And the other ACs mention that as well. I don't know what "a reference" means in this context. I'll unassign since it seems we're still unclear on the goal of the ticket.

Also available in: Atom PDF