QA » openQA Project » openQA Infrastructure

On o3 we've noticed problems (see #99189#note-15 and subsequent comments), so I suppose it makes sense to add a lock before upgrading:

zypper al qemu-ovmf-x86_64

A lock for qemu-seabios shouldn't be necessary (as of https://github.com/os-autoinst/os-autoinst/pull/1838).

The following commands can be run via salt i.e. sudo salt -C 'G@roles:worker' cmd.run '...' where ... is the command to execute on the machine:

• To confirm which workers still need upgrading: grep VERSION=\"15.2 /etc/os-release || true
• To download packages: zypper --releasever=15.3 ref && zypper --releasever=15.3 -n dup --allow-vendor-change --download-only
• To perform the upgrade zypper --no-refresh --releasever=15.3 -n dup --allow-vendor-change --auto-agree-with-licenses --replacefiles

To start with I'm only preparing the upgrade, which shouldn't cause much trouble

(Edit: Updated above commands after the fact in case they're copied and re-used in the future, but also mentioned below)

cdywan wrote:

• To perform the upgrade zypper --releasever=15.3 -n dup --allow-vendor-change --auto-agree-with-licenses --replacefiles --download-in-advance

To start with I'm only preparing the upgrade, which shouldn't cause much trouble

Much quicker than I thought. Since only 58 out of 248 are working I decided to go for it.

It looks like openqaworker3 remained unaffected (is still on Leap 15.2).

Note that #104016 was simply caused by a missing reboot. It would make sense to reboot the machines shortly after doing the upgrade.

mkittler wrote:

It looks like openqaworker3 remained unaffected (is still on Leap 15.2).

Note that #104016 was simply caused by a missing reboot. It would make sense to reboot the machines shortly after doing the upgrade.

Ack. I was apparently mistaken that the workers would reboot daily anyway, that's why I thought I could avoid an extra reboot

As suggested by @mkittler I'm now using sudo salt -C 'G@roles:worker' cmd.run 'needs-restarting --reboothint' to check if any machines still needed a reboot (and unlike sudo rebootmgrctl status it doesn't require priviledge escalation, and doesn't need me to check the running kernel)

So next step checking the remaining workers which show up as Reboot is suggested.

There were issues with 3 workers that I was still investigating (because, well, in alot of output it wasn't immediately clear) will update comment from notes in a minute

openqaworker-arm-1.suse.de

Rebooted twice since it was still flagged as Reboot is suggested

openqaworker-arm-2.suse.de

Rebooted, and now looking to run new packages

openqaworker3.suse.de

I couldn't identify any issues and solely re-ran dup and rebooted 🤷️

According to salt the machine is still running 15.2 despite my having completed the upgrade and ssh is unresponsive 🧐️

sudo salt -C 'G@roles:worker' cmd.run 'systemctl is-active --quiet sshd || systemctl restart sshd || systemctl status sshd'
Dec 17 13:49:22 openqaworker3 sshd-gen-keys-start[7203]: /usr/sbin/sshd-gen-keys-start: line 7: ssh-keygen: command not found

sudo salt -C 'G@roles:worker' cmd.run 'systemctl is-active --quiet sshd || zypper in -n openssh-common'
No update candidate for 'openssh-common-8.4p1-3.3.1.x86_64'. The highest available version is already installed.

So somehow ssh-keygen is unaccounted for 🤔️

Apparently we had inconsistencies because all repos weren't auto-refreshing: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/631

malbec.arch.suse.de
grenache-1.qa.suse.de

Rebooted, and now looking to run new packages

grenache also needed a little kick in the vine:

> systemctl --failed
os-autoinst-openvswitch.service loaded failed failed os-autoinst openvswitch helper
> sudo systemctl restart os-autoinst-openvswitch.service

• Should document in the wiki to check if zypper reports any orphans after upgrades (#104142) i.e. zypper packages --orphaned
• Consider re-installing via salt, after dup&other clean-ups i.e. salt state-apply so that we're in a state with all required packages but w/o manually installed or left-over packages that aren't in salt
• Use -q to avoid getting lost in logs -q is much too quiet 🤐️
• Perform the upgrade with --no-refresh to avoid race conditions after prior package refresh and download
• I didn't cover any inactive machines here

On openqaworker3 a problem was that zypper dup did not want to follow through originally so something was missed and we ended up with non-operative sshd after reboot.

The proper fix is in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/631 to make the SUSE_CA repo consistently auto-refresh, same as others.

We started with looking into rpm files and did salt '*' cmd.run 'test -e /etc/salt/minion.rpmnew && mv /etc/salt/minion{,.bak} && mv /etc/salt/minion{.rpmnew,}' && salt '*' state.sls_id /etc/salt/minion salt.minion

Fix for kvm udev rules deployment: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/632

see https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/750090#L8454

We seem to have extra packages on openqaworker3:

> sudo salt -C 'G@roles:worker' cmd.run 'zypper packages --orphaned'
openqaworker3.suse.de:
i | @System    | libply-boot-client4     | 0.9.4+git20190304.ed9f201-lp152.4.4 | x86_64
i | @System    | libply-splash-core4     | 0.9.4+git20190304.ed9f201-lp152.4.4 | x86_64
i | @System    | libply-splash-graphics4 | 0.9.4+git20190304.ed9f201-lp152.4.4 | x86_64
i | @System    | libply4                 | 0.9.4+git20190304.ed9f201-lp152.4.4 | x86_64
i | @System    | libyui-ncurses11        | 2.54.5-lp152.1.3                    | x86_64
i | @System    | libyui11                | 3.9.3-lp152.1.3                     | x86_64
> sudo salt -C 'G@roles:worker' cmd.run 'zypper rm -y -u $(zypper packages --orphaned | awk "/^i/{ print $5 }" ORS=" ") hello'
> Installation has completed with error

That last message doesn't seem to contradict the fact that the packages are gone 🤓️