Actions
action #130648
closedcoordination #130582: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui and production workloads, to openSUSE Leap 15.5
Upgrade all other LSG QE salt controlled machines to openSUSE Leap 15.5
Start date:
2023-06-09
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Motivation¶
- Need to upgrade machines before EOL of Leap 15.4 and have a consistent environment
Acceptance criteria¶
- AC1: all LSG QE salt controlled machines run a clean upgraded openSUSE Leap 15.5 (no failed systemd services, no left over .rpm-new files, etc.) except for OSD workers
Acceptance tests¶
- AT1-1:
sudo salt -C 'not G@roles:worker and not G@roles:webui' grains.get oscodename | grep -B1 'Leap 15.4'is empty
Suggestions¶
- read https://progress.opensuse.org/projects/openqav3/wiki#Distribution-upgrades
- Reserve some time when the related services are not heavily relied upon
- Keep IPMI interface ready and test that Serial-over-LAN works for potential recovery or for virtual machines virt-manager access
- After upgrade reboot and check everything working as expected, if not rollback, e.g. with
snapper rollback
Rollback actions¶
- Remove silence
alertname=Failed systemd services
Further details¶
- Don't worry, everything can be repaired :) If by any chance the machines gets misconfigured in many cases there are btrfs snapshots to recover, the IPMI Serial-over-LAN, etc.
Updated by okurz 11 months ago
- Copied from action #130588: Upgrade osd workers to openSUSE Leap 15.5 added
Updated by okurz 7 months ago
- Description updated (diff)
# salt --no-color -C 'not G@roles:worker and not G@roles:webui and not G@osrelease:15.5' grains.get oscodename
openqaw5-xen.qa.suse.de:
openSUSE Leap 15.4
qamaster.qe.nue2.suse.org:
openSUSE Leap 15.4
baremetal-support.qe.nue2.suse.org:
openSUSE Leap 15.4
tumblesle.qe.nue2.suse.org:
openSUSE Leap 15.4
jenkins.qe.nue2.suse.org:
openSUSE Leap 15.4
monitor.qe.nue2.suse.org:
openSUSE Leap 15.4
backup-vm.qe.nue2.suse.org:
openSUSE Leap 15.4
schort-server.qe.nue2.suse.org:
openSUSE Leap 15.4
Now I can call the https://progress.opensuse.org/projects/openqav3/wiki/#Distribution-upgrades with the above command.
salt --no-color -C 'not G@roles:worker and not G@roles:webui and not G@osrelease:15.5' cmd.run 'export new_version=15.5; zypper --releasever=$new_version ref && systemctl stop openqa-continuous-update.timer && zypper -n --releasever=$new_version dup --dry-run --auto-agree-with-licenses --replacefiles --download-in-advance'
to try, then
salt --no-color -C 'not G@roles:worker and not G@roles:webui and not G@osrelease:15.5' cmd.run 'export new_version=15.5; zypper --releasever=$new_version ref && systemctl stop openqa-continuous-update.timer && zypper -n --releasever=$new_version dup --auto-agree-with-licenses --replacefiles --download-in-advance && reboot'
the salt package locks get in the way. See #131249 . But we want to try out the new salt package anyway if it has a patch. So removing locks
salt --no-color -C 'not G@roles:worker and not G@roles:webui and not G@osrelease:15.5' cmd.run 'zypper rl salt salt-minion salt-bash-completion && zypper rl -t patch openSUSE-SLE-15.4-2023-2571 openSUSE-SLE-15.4-2023-3145 openSUSE-SLE-15.4-2023-3863 && export new_version=15.5; zypper --releasever=$new_version ref && systemctl stop openqa-continuous-update.timer ||: && zypper -n --releasever=$new_version dup --auto-agree-with-licenses --replacefiles --download-in-advance && reboot'
Updated by okurz 7 months ago
- Status changed from In Progress to Feedback
Machines weren't properly upgraded. on monitor I had to do rpm --rebuilddb and conducted upgrade manually.
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1014 to not add security:sensor repo where not required anymore.
Updated by okurz about 2 months ago
- Related to action #155179: Participate in alpha-testing of new version of velociraptor-client added
Updated by okurz about 1 month ago
- Copied to action #157996: Upgrade all other LSG QE salt controlled machines to openSUSE Leap 15.6 added
Actions