Project

General

Profile

Actions
Copy link

action #130648

closed

coordination #130582: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui and production workloads, to openSUSE Leap 15.5

Upgrade all other LSG QE salt controlled machines to openSUSE Leap 15.5

Added by okurz over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Category:
Organisational
Target version:
Ready
Start date:
2023-06-09
Due date:
% Done:

0%

Estimated time:
Tags:
infra

Description

Motivation

  • Need to upgrade machines before EOL of Leap 15.4 and have a consistent environment

Acceptance criteria

  • AC1: all LSG QE salt controlled machines run a clean upgraded openSUSE Leap 15.5 (no failed systemd services, no left over .rpm-new files, etc.) except for OSD workers

Acceptance tests

  • AT1-1: sudo salt -C 'not G@roles:worker and not G@roles:webui' grains.get oscodename | grep -B1 'Leap 15.4' is empty

Suggestions

  • read https://progress.opensuse.org/projects/openqav3/wiki#Distribution-upgrades
  • Reserve some time when the related services are not heavily relied upon
  • Keep IPMI interface ready and test that Serial-over-LAN works for potential recovery or for virtual machines virt-manager access
  • After upgrade reboot and check everything working as expected, if not rollback, e.g. with snapper rollback

Rollback actions

  • Remove silence alertname=Failed systemd services

Further details

  • Don't worry, everything can be repaired :) If by any chance the machines gets misconfigured in many cases there are btrfs snapshots to recover, the IPMI Serial-over-LAN, etc.

Related issues 3 (0 open — 3 closed)

Related to QA - action #155179: Participate in alpha-testing of new version of velociraptor-clientResolvedokurz2024-02-08

Actions
Copied from openQA Project - action #130588: Upgrade osd workers to openSUSE Leap 15.5Resolvedokurz

Actions
Copied to openQA Project - action #157996: Upgrade all other LSG QE salt controlled machines to openSUSE Leap 15.6Resolvedokurz

Actions
Actions
Copy link
 #1

Updated by okurz over 1 year ago

  • Copied from action #130588: Upgrade osd workers to openSUSE Leap 15.5 added
Actions
Copy link
 #2

Updated by okurz over 1 year ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by okurz about 1 year ago

  • Target version changed from future to Tools - Next
Actions
Copy link
 #4

Updated by okurz about 1 year ago

  • Target version changed from Tools - Next to Ready
Actions
Copy link
 #5

Updated by okurz about 1 year ago

  • Tags set to infra
  • Status changed from New to In Progress
  • Assignee set to okurz
Actions
Copy link
 #6

Updated by okurz about 1 year ago

  • Description updated (diff)
# salt --no-color -C 'not G@roles:worker and not G@roles:webui and not G@osrelease:15.5' grains.get oscodename
openqaw5-xen.qa.suse.de:
    openSUSE Leap 15.4
qamaster.qe.nue2.suse.org:
    openSUSE Leap 15.4
baremetal-support.qe.nue2.suse.org:
    openSUSE Leap 15.4
tumblesle.qe.nue2.suse.org:
    openSUSE Leap 15.4
jenkins.qe.nue2.suse.org:
    openSUSE Leap 15.4
monitor.qe.nue2.suse.org:
    openSUSE Leap 15.4
backup-vm.qe.nue2.suse.org:
    openSUSE Leap 15.4
schort-server.qe.nue2.suse.org:
    openSUSE Leap 15.4

Now I can call the https://progress.opensuse.org/projects/openqav3/wiki/#Distribution-upgrades with the above command.

salt --no-color -C 'not G@roles:worker and not G@roles:webui and not G@osrelease:15.5' cmd.run 'export new_version=15.5; zypper --releasever=$new_version ref && systemctl stop openqa-continuous-update.timer && zypper -n --releasever=$new_version dup --dry-run --auto-agree-with-licenses --replacefiles --download-in-advance'

to try, then

salt --no-color -C 'not G@roles:worker and not G@roles:webui and not G@osrelease:15.5' cmd.run 'export new_version=15.5; zypper --releasever=$new_version ref && systemctl stop openqa-continuous-update.timer && zypper -n --releasever=$new_version dup --auto-agree-with-licenses --replacefiles --download-in-advance && reboot'

the salt package locks get in the way. See #131249 . But we want to try out the new salt package anyway if it has a patch. So removing locks

salt --no-color -C 'not G@roles:worker and not G@roles:webui and not G@osrelease:15.5' cmd.run 'zypper rl salt salt-minion salt-bash-completion && zypper rl -t patch openSUSE-SLE-15.4-2023-2571 openSUSE-SLE-15.4-2023-3145 openSUSE-SLE-15.4-2023-3863 && export new_version=15.5; zypper --releasever=$new_version ref && systemctl stop openqa-continuous-update.timer ||: && zypper -n --releasever=$new_version dup --auto-agree-with-licenses --replacefiles --download-in-advance && reboot'
Actions
Copy link
 #7

Updated by okurz about 1 year ago

  • Description updated (diff)
Actions
Copy link
 #8

Updated by okurz about 1 year ago

  • Status changed from In Progress to Feedback

Machines weren't properly upgraded. on monitor I had to do rpm --rebuilddb and conducted upgrade manually.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1014 to not add security:sensor repo where not required anymore.

Actions
Copy link
 #9

Updated by okurz about 1 year ago

  • Status changed from Feedback to Resolved
Actions
Copy link
 #10

Updated by okurz 9 months ago

  • Related to action #155179: Participate in alpha-testing of new version of velociraptor-client added
Actions
Copy link
 #11

Updated by okurz 8 months ago

  • Copied to action #157996: Upgrade all other LSG QE salt controlled machines to openSUSE Leap 15.6 added
Actions
Copy link

Also available in: Atom PDF