QA (public) » openQA Project (public) » openQA Infrastructure (public)

#111992

Upgrading openqaworker11 and openqaworker12 manually as they are currently not in salt. openqaworker12 is still a Leap 15.2 so I will do a Leap15.2->15.4 direct upgrade for fun and because I am curious what happens.

Many repositories have changed the name format from "openSUSE_Leap_$releasever" to "$releasever" so we need to adapt for that.
Actually by now there are more machines controlled by our salt structure so let's upgrade them along the way

powerqaworker-qam-1.qa.suse.de has some conflicts about python2 packages, manually removed first.

sudo salt --no-color --state-output=changes -C 'powerqaworker-qam-1.qa.suse.de' cmd.run 'zypper -n rm -u python2-libxml2-python'

then all the machines:

sudo salt --no-color --state-output=changes -C 'not G@roles:webui' cmd.run '(rpm -q qemu-ovmf-x86_64 && zypper al qemu-ovmf-x86_64) ; zypper rr telegraf-monitoring && sed -i -e "s@/openSUSE_Leap_@/@g" /etc/zypp/repos.d/* && zypper -n --releasever=15.4 ref && zypper -n --releasever=15.4 dup --auto-agree-with-licenses --replacefiles --download-in-advance'

seems to have gone fine

$ salt \* grains.get oscodename
storage.qa.suse.de:
    openSUSE Leap 15.4
openqaworker2.suse.de:
    openSUSE Leap 15.4
openqaworker3.suse.de:
    openSUSE Leap 15.4
openqaworker9.suse.de:
    openSUSE Leap 15.4
openqaworker6.suse.de:
    openSUSE Leap 15.4
QA-Power8-5-kvm.qa.suse.de:
    openSUSE Leap 15.4
openqaworker5.suse.de:
    openSUSE Leap 15.4
openqaworker14.qa.suse.cz:
    openSUSE Leap 15.4
powerqaworker-qam-1.qa.suse.de:
    openSUSE Leap 15.4
openqa-monitor.qa.suse.de:
    openSUSE Leap 15.4
QA-Power8-4-kvm.qa.suse.de:
    openSUSE Leap 15.4
openqaworker13.suse.de:
    openSUSE Leap 15.4
grenache-1.qa.suse.de:
    openSUSE Leap 15.4
openqa.suse.de:
    openSUSE Leap 15.4
openqaworker-arm-2.suse.de:
    openSUSE Leap 15.4
openqaworker8.suse.de:
    openSUSE Leap 15.4
backup.qa.suse.de:
    openSUSE Leap 15.4
openqaworker10.suse.de:
    openSUSE Leap 15.4
openqaworker-arm-1.suse.de:
    openSUSE Leap 15.4

except for openqaworker-arm-3 that repeatedly crashed. Need to try harder. Triggered reboot for most workers now.

openqaworker8, openqaworker9 and openqaworker14 failed to come up yet, "maintenance" mode on openqaworker8 and 9 at least, failed "openqa_nvme_format". The actual command failing is mdadm --create /dev/md/openqa --level=0 --force --assume-clean --raid-devices=1 --run /dev/nvme0n1 with "mdadm: cannot open /dev/nvme0n1: Device or resource busy". Well, that's understandable as on openqaworker8+9 nvme0n1 has three partitions and is also used for the root filesystem hence it's already "busy". Only nvme0n1p3 should be used here.
The reason seems to be this:

├─nvme0n1p2 259:2    0   100G  0 part /var/tmp
│                                     /var/spool
│                                     /var/opt
│                                     /var/log
│                                     /var/lib/pgsql
│                                     /var/lib/named
│                                     /var/lib/mysql
│                                     /var/lib/mariadb
│                                     /var/lib/mailman
│                                     /var/lib/libvirt/images
│                                     /var/lib/machines
│                                     /var/crash
│                                     /var/cache
│                                     /usr/local
│                                     /tmp
│                                     /srv
│                                     /opt
│                                     /boot/grub2/x86_64-efi
│                                     /boot/grub2/i386-pc
│                                     /.snapshots
│                                     /
└─nvme0n1p3 259:3 bash -ex /usr/local/bin/openqa-establish-nvme-setup
# lsblk --noheadings | grep -v nvme | grep "/$"
│                                     /

maybe lsblk has changed it's format. Fixed in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/714, patched openqaworker8+9 manually.

Upgrade done. Some machines fail, specific tickets created to handle