action #157972
open
coordination #157969: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui and production workloads, to openSUSE Leap 15.6
Upgrade o3 workers to openSUSE Leap 15.6
Added by okurz 4 months ago.
Updated 16 days ago.
Description
Motivation¶
- Need to upgrade workers before EOL of Leap 15.5 and have a consistent environment
Acceptance criteria¶
- AC1: all o3 worker machines run a clean upgraded openSUSE Leap 15.6 (no failed systemd services, no left over .rpm-new files, etc.)
Suggestions¶
- read https://progress.opensuse.org/projects/openqav3/wiki#Distribution-upgrades
- Reserve some time when the workers are only executing a few or no openQA test jobs
- Keep IPMI interface ready and test that Serial-over-LAN works for potential recovery
- Use the instructions from above but use
transactional-update shell
for transactional update workers
- After upgrade reboot and check everything working as expected, if not rollback, e.g. with
transactional-update rollback
Further details¶
- Don't worry, everything can be repaired :) If by any chance the worker gets misconfigured there are btrfs snapshots to recover, the IPMI Serial-over-LAN, a reinstall is possible and not hard, there is no important data on the host (it's only an openQA worker) and there are also other machines that can jobs while one host might be down for a little bit longer. And okurz can hold your hand :)
- Copied from action #130585: Upgrade o3 workers to openSUSE Leap 15.5 added
- Subject changed from Upgrade o3 workers to openSUSE Leap 15.5 to Upgrade o3 workers to openSUSE Leap 15.6
- Description updated (diff)
- Assignee deleted (
okurz)
- Target version changed from Ready to future
- Target version changed from future to Tools - Next
- Target version changed from Tools - Next to Ready
- Target version changed from Ready to Tools - Next
- Target version changed from Tools - Next to Ready
- Target version changed from Ready to Tools - Next
- Related to action #162239: [s390x] test fails in bootloader_start due to slow response from z/VM hypervisor and/or changed response on "cp i cms" command added
- Status changed from New to In Progress
- Assignee set to okurz
- Target version changed from Tools - Next to Ready
Same as on OSD workers. System network does not come up. Connected over IPMI SoL and found from systemctl status wickedd-nanny
Jun 14 11:53:04 openqaworker21 wickedd-nanny[2755]: device br0: call to org.opensuse.Network.Firewall.firewallUp() failed: DBus method call timed out
…
same for other network devices. Rolling back.
From fvogt
org.opensuse.Network.Firewall is not firewalld, it's wicked. That it timed out means that the service exists and something is running. wicked provides this and firewallUp() just calls /etc/wicked/extensions/firewall up. And that basically just calls firewall-cmd to assign interfaces. wickedd-nanny is waiting for wicked which waits for firewalld
- Due date set to 2024-06-29
Setting due date based on mean cycle time of SUSE QE Tools
- Related to action #162320: multi-machine test failures 2024-06-14+, auto_review:"ping with packet size 100 failed.*can be GRE tunnel setup issue":retry added
- Due date deleted (
2024-06-29)
- Status changed from In Progress to New
- Assignee deleted (
okurz)
- Target version changed from Ready to Tools - Next
- Target version changed from Tools - Next to Ready
Moving this to the backlog because it's blocking #162239 which is already on the backlog - alternatively ofc you can re-consider the blocked ticket.
- Related to action #162683: s390x libvirt started kvm machines on Leap 15.6 fail with "unsupported configuration: machine type 's390-ccw-virtio-8.2' does not support ACPI" size:M added
- Status changed from New to Blocked
- Assignee set to okurz
- Target version changed from Ready to Tools - Next
- Copied to action #163469: Upgrade a single o3 worker to openSUSE Leap 15.6 added
- Related to action #163472: Upgrade a single osd worker to openSUSE Leap 15.6 added
Also available in: Atom
PDF