action #157972
closedcoordination #157969: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui and production workloads, to openSUSE Leap 15.6
Upgrade o3 workers to openSUSE Leap 15.6 size:S
Added by okurz 9 months ago. Updated about 1 month ago.
0%
Description
Motivation¶
- Need to upgrade workers before EOL of Leap 15.5 and have a consistent environment
Acceptance criteria¶
- AC1: all o3 worker machines run a clean upgraded openSUSE Leap 15.6 (no failed systemd services, no left over .rpm-new files, etc.)
Suggestions¶
- read https://progress.opensuse.org/projects/openqav3/wiki#Distribution-upgrades
- Reserve some time when the workers are only executing a few or no openQA test jobs
- Keep IPMI interface ready and test that Serial-over-LAN works for potential recovery
- Apply the workaround for #162296, i.e.
zypper al -m "boo#1227616" *firewall* - Use the instructions from above
- After upgrade reboot and check everything working as expected, if not rollback, e.g. with
transactional-update rollback
Further details¶
- Don't worry, everything can be repaired :) If by any chance the worker gets misconfigured there are btrfs snapshots to recover, the IPMI Serial-over-LAN, a reinstall is possible and not hard, there is no important data on the host (it's only an openQA worker) and there are also other machines that can jobs while one host might be down for a little bit longer. And okurz can hold your hand :)
Files
| pb-discover.log (11.4 KB) pb-discover.log | gpathak, 2024-10-25 09:21 | ||
| clipboard-202411051428-dg60d.png (52.6 KB) clipboard-202411051428-dg60d.png | gpathak, 2024-11-05 08:58 | ||
| clipboard-202411121229-elbgt.png (50 KB) clipboard-202411121229-elbgt.png | gpathak, 2024-11-12 06:59 |
Updated by okurz 9 months ago
- Copied from action #130585: Upgrade o3 workers to openSUSE Leap 15.5 added
Updated by okurz 6 months ago
- Related to action #162239: [s390x] test fails in bootloader_start due to slow response from z/VM hypervisor and/or changed response on "cp i cms" command added
Updated by okurz 6 months ago · Edited
Same as on OSD workers. System network does not come up. Connected over IPMI SoL and found from systemctl status wickedd-nanny
Jun 14 11:53:04 openqaworker21 wickedd-nanny[2755]: device br0: call to org.opensuse.Network.Firewall.firewallUp() failed: DBus method call timed out
…
same for other network devices. Rolling back.
From fvogt
org.opensuse.Network.Firewall is not firewalld, it's wicked. That it timed out means that the service exists and something is running. wicked provides this and firewallUp() just calls /etc/wicked/extensions/firewall up. And that basically just calls firewall-cmd to assign interfaces. wickedd-nanny is waiting for wicked which waits for firewalld
Updated by openqa_review 6 months ago
- Due date set to 2024-06-29
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz 6 months ago
- Related to action #162320: multi-machine test failures 2024-06-14+, auto_review:"ping with packet size 100 failed.*can be GRE tunnel setup issue":retry added
Updated by okurz 6 months ago
- Related to action #162683: s390x libvirt started kvm machines on Leap 15.6 fail with "unsupported configuration: machine type 's390-ccw-virtio-8.2' does not support ACPI" size:M added
Updated by okurz 6 months ago
- Copied to action #163469: Upgrade a single o3 worker to openSUSE Leap 15.6 added
Updated by okurz 6 months ago
- Related to action #163472: Upgrade a single osd worker to openSUSE Leap 15.6 added
Updated by okurz 3 months ago
- Related to action #162296: openQA workers crash with Linux 6.4 after upgrade openSUSE Leap 15.6 size:S added
Updated by gpathak about 2 months ago · Edited
openqaworker20 has been updated, some tests that were successful after upgrade
- https://openqa.opensuse.org/tests/4585980
- https://openqa.opensuse.org/tests/4585961
- https://openqa.opensuse.org/tests/4585948
Kerosene has been updated, but it is now stuck in Petitboot.
Workers to be updated
- aarch64-o3.qe.nue2.suse.org
- openqaworker22
- openqaworker23
- openqaworker24
- openqaworker25
- openqaworker26
- openqaworker28
- openqaworker-arm21
- openqaworker-arm22
- qa-power8-3
Updated by openqa_review about 2 months ago
- Due date set to 2024-11-06
Setting due date based on mean cycle time of SUSE QE Tools
Updated by gpathak about 2 months ago · Edited
Workers updated:
- aarch64-o3: https://openqa.opensuse.org/tests/4587863
- openqaworker20: https://openqa.opensuse.org/tests/4588779
- openqaworker21: https://openqa.opensuse.org/tests/4588803
- openqaworker22: https://openqa.opensuse.org/tests/4587687
- openqaworker23: https://openqa.opensuse.org/tests/4588354
- openqaworker24: https://openqa.opensuse.org/tests/4588778
- openqaworker25: https://openqa.opensuse.org/tests/4588814
- openqaworker26: https://openqa.opensuse.org/tests/4589711, https://openqa.opensuse.org/tests/4589787
- openqaworker28: https://openqa.opensuse.org/tests/4589708, https://openqa.opensuse.org/tests/4589704
- openqaworker-arm21: https://openqa.opensuse.org/tests/4588411
- openqaworker-arm22: https://openqa.opensuse.org/tests/4587879
kerosene.qe.nue2.suse.org is stuck in Petitboot
Workers to be updated:
- qa-power8-3
Updated by okurz about 2 months ago
- Copied to action #168916: find out what's the current state of openqaworker27 size:S added
Updated by gpathak about 2 months ago · Edited
- File pb-discover.log pb-discover.log added
Logged-in to IPMI, dropped to petitboot shell, sda3 and sdb1 partitions were mounted automatically, the contents of which are:
/ # ls
bin etc lib linuxrc mnt proc run sys usr
dev init lib64 media opt root sbin tmp var
/ #
/ #
/ # mount
rootfs on / type rootfs (rw,size=66787584k,nr_inodes=1043556)
devtmpfs on /dev type devtmpfs (rw,relatime,size=66787584k,nr_inodes=1043556,mode=755)
proc on /proc type proc (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777)
sysfs on /sys type sysfs (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,relatime)
/dev/sda3 on /var/petitboot/mnt/dev/sda3 type ext4 (ro,relatime,norecovery)
/dev/sdb1 on /var/petitboot/mnt/dev/sdb1 type ext4 (ro,relatime,norecovery)
/ #
/ #
/ # ls /var/petitboot/mnt/dev/sda3/boot
boot grub2 initrdbe vmlinuxbe
/ #
/ #
/ # ls /var/petitboot/mnt/dev/sdb1/backup/root-rsync/boot
System.map-4.4.104-18.44-default initrd-4.4.180-102-default
System.map-4.4.120-45-default initrdbe
System.map-4.4.159-73-default symvers-4.4.104-18.44-default.gz
System.map-4.4.165-81-default symvers-4.4.120-45-default.gz
System.map-4.4.179-99-default symvers-4.4.159-73-default.gz
System.map-4.4.180-102-default symvers-4.4.165-81-default.gz
boot symvers-4.4.179-99-default.gz
boot.readme symvers-4.4.180-102-default.gz
config-4.4.104-18.44-default sysctl.conf-4.4.104-18.44-default
config-4.4.120-45-default sysctl.conf-4.4.120-45-default
config-4.4.159-73-default sysctl.conf-4.4.159-73-default
config-4.4.165-81-default sysctl.conf-4.4.165-81-default
config-4.4.179-99-default sysctl.conf-4.4.179-99-default
config-4.4.180-102-default sysctl.conf-4.4.180-102-default
do_purge_kernels vmlinux
grub2 vmlinux-4.4.104-18.44-default
initrd vmlinux-4.4.120-45-default
initrd-4.4.104-18.44-default vmlinux-4.4.159-73-default
initrd-4.4.120-45-default vmlinux-4.4.165-81-default
initrd-4.4.159-73-default vmlinux-4.4.179-99-default
initrd-4.4.165-81-default vmlinux-4.4.180-102-default
initrd-4.4.179-99-default vmlinuxbe
/ #
/ #
/ #
Attempting to boot using kexec from sdb1/backup/root-rsync/boot while specifying the root filesystem with root=UUID=e0110d2d-84f3-4590-8d12-a472f82cac8b (which corresponds to /dev/sda3) successfully loads the kernel and initrd. However, the machine fails to connect to the network, rendering it unusable.
kexec --load /var/petitboot/mnt/dev/sdb1/backup/root-rsync/boot/vmlinux-4.4.180-102-default --initrd /var/petitboot/mnt/dev/sdb1/backup/root-rsync/boot/initrd --append "root=UUID=e0110d2d-84f3-4590-8d12-a472f82cac8b root=/dev/disk/by-id/scsi-1IBM_IPR-0_5DB3D30000000020-part3 disk=/dev/disk/by-id/scsi-1IBM_IPR-0_5DB3D30000000020 resume=/dev/disk/by-id/scsi-1IBM_IPR-0_5DB3D30000000020-part2 nospec kvm.nested=1 kvm_intel.nested=1 kvm_amd.nested=1 kvm-arm.nested=1 crashkernel=210M"
Looking at petitboot diagnostics, found out that it tries tftpboot and tries to fetch files from http://10.168.192.10/tftpboot/
Updated by gpathak about 2 months ago
After an update, the kerosene machine lost all the kernel modules in the /lib/modules/ directory, and the GRUB entries were not visible in Petitboot. I found a backup of the root filesystem at /var/lib/openqa/backup/root-rsync on /dev/sdb1. After copying the /lib/modules/ directory from the backup to the original root filesystem on /dev/sda3, I attempted to boot using the same kexec command as above. This time, the network came up, and the machine became accessible via SSH.
Next, I copied the /var/lib/openqa/backup/root-rsync/boot directory to the root filesystem on /dev/sda3. This made the boot entries visible in Petitboot, and the machine automatically booted after a soft reboot and also after an ipmi chassis power reset command.
After upgrading the kernel to 6.4.0-150600.23.25, the issue related to the failing task on o3 was resolved https://openqa.opensuse.org/tests/4617414#line-235.
- https://openqa.opensuse.org/tests/4619371
- https://openqa.opensuse.org/tests/4619374
- https://openqa.opensuse.org/tests/4619373
However, it appears that snapper is not configured correctly, and the root filesystem is using the ext4 format.
Updated by gpathak about 2 months ago · Edited
@okurz @nicksinger @jbaier_cz Can any one of you please review the state of kerosene?
I can then upgrade the qa-power8-3.
As suggested, I rebooted kerosene multiple times, I observed some (random) worker instances always fails to start after a reboot, but most of the workers starts automatically at boot.
Updated by gpathak about 2 months ago · Edited
Seems like Kerosene is facing some issue: https://suse.slack.com/archives/C02CANHLANP/p1730816673953399?thread_ts=1730299105.029509&cid=C02CANHLANP
After searching a bit about the issue, came across this https://bugzilla.redhat.com/show_bug.cgi?id=1215618
As suggested, I executed ppc64_cpu --smt=off on kerosene (not sure if the changes are persistent across reboots https://github.com/ibm-power-utilities/powerpc-utils/issues/11) and re-triggered the jobs:
Updated by gpathak about 2 months ago · Edited
Somehow, GRUB is no longer present on qa-power8-3. After a reboot, Petitboot didn't show any entries, and the machine got stuck at the Petitboot screen.
After dropping to the Petitboot shell and running kexec:
kexec --load /var/petitboot/mnt/dev/sda3/boot/vmlinux --initrd /var/petitboot/mnt/dev/sda3/boot/initrd --append "root=UUID=1cd396af-c783-43b9-92d4-210b129a5cdd nospec kvm.nested=1 kvm_intel.nested=1 kvm_amd.nested=1 kvm-arm.nested=1 crashkernel=210M"
The machine successfully booted, and the network came up. However, I discovered that the GRUB binary wasn't present in /boot/grub2/.
An attempt to reinstall GRUB using grub2-install /dev/sda did install grub but resulted in the following error:
grub2-install: error: the chosen partition is not a PReP partition.
The Petitboot still isn't displaying any boot entries.
Unlike kerosene, power8-3 does not have a PReP partition. Running fdisk -l listed all partitions (as shown), but there was no PReP partition found on power8-3.
qa-power8-3:~ # fdisk -l
Disk /dev/sda: 1.09 TiB, 1200243695616 bytes, 2344225968 sectors
Disk model: HUC101212CSS600
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: DAA364D6-D309-41A3-A07D-1BF8A366C008
Device Start End Sectors Size Type
/dev/sda1 2048 2344225934 2344223887 1.1T Linux filesystem
Disk /dev/sdc: 361.31 GiB, 387956342784 bytes, 757727232 sectors
Disk model: IPR-0 5EC32700
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 770DFCF6-D082-4E67-832A-7A7DA23297D9
Device Start End Sectors Size Type
/dev/sdc3 1067008 753530879 752463872 358.8G Linux filesystem
/dev/sdc4 753530880 757727198 4196319 2G Linux swap
qa-power8-3:~ #
Now, it seems only a re-installation can fix this issue.
@nicksinger @okurz Do we have any other approach? I did search for relevant tasks/tickets but couldn't find any helpful information.
Updated by gpathak about 1 month ago
kerosene and power-qa-8-3 needs a fresh installation with proper partitioning and brtfs filesystem.
kerosene doesn't have brtfs and power-8-3 doesn't have PReP Boot partition.
@okurz Should we create separate ticket for these two activities?
Updated by okurz about 1 month ago
gpathak wrote in #note-36:
kerosene and power-qa-8-3 needs a fresh installation with proper partitioning and brtfs filesystem.
kerosene doesn't have brtfs and power-8-3 doesn't have PReP Boot partition.
@okurz Should we create separate ticket for these two activities?
Hold on. I don't see where that conclusion comes from that a full reinstall would be necessary. Can you elaborate on what happened that caused this now? And please keep at least one machine operational until the other is fully back. For now due to those open points I suggest to continue in this ticket.
Updated by gpathak about 1 month ago · Edited
@okurz
Not sure about how it happened on power-8, but I figured that it doesn't have a PReP boot partition even before performing an upgrade when I issued a reboot from ssh session just to check if Petitboot lists all GRUB entries and it didn't. After that I tried reinstalling grub on /dev/sda and got grub2-install: error: the chosen partition is not a PReP partition.
Kerosene is good, but the only thing that's concerning is snapper isn't configured and is not taking snapshots (not sure if it works on ext4)
I will try to make kerosene stable first and then target qa-power-8 cause power-8 isn't even booting, manual kexec is working though, grub is gone.
Updated by gpathak about 1 month ago
The openqa-auto-update and openqa-continuous-update is causing kerosene workers to go offline, the machine also become unreachable after an update is performed via auto-update.
Masked following units for time being:
- openqa-auto-update.timer
- openqa-auto-update.service
- openqa-continuous-update.service
- openqa-continuous-update.timer
Updated by gpathak about 1 month ago
- Copied to action #169576: Recover qa-power8-3 power machine size:S added
Updated by okurz about 1 month ago
For kerosene please set a static hostname "kerosene" using hostnamectl
Updated by okurz about 1 month ago
- Due date changed from 2024-11-06 to 2024-11-15
We encountered some problems with the ppc64le setup. I am expecting other team members to help within this week to resolve this quickly, bumped due date accordingly.
Updated by gpathak about 1 month ago · Edited
Did a fresh installation of leap 15.5 on kerosene, it's more than 12 hours the machine is up and running openQA tests.
Earlier, the kernel was crashing causing the machine unreachable.
@nicksinger is working on recovering qa-power-8-3 machine in a separate ticket: #169576
@okurz Almost all the AC's are covered.
Updated by okurz about 1 month ago
okurz wrote in #note-41:
For kerosene please set a static hostname "kerosene" using
hostnamectl
Updated by okurz about 1 month ago
Please make sure worker instances show up with that name on o3 and reference successful openQA jobs for validation
Updated by gpathak about 1 month ago · Edited
okurz wrote in #note-46:
Please make sure worker instances show up with that name on o3 and reference successful openQA jobs for validation
Kerosene is showing up as kerosene-8, with the transient hostname maybe because of the DNS response received from https://gitlab.suse.de/OPS-Service/salt/-/blob/production/salt/profile/dns/files/nue2_suse_org/dns-qe.nue2.suse.org?ref_type=heads&plain=1#L397 (I am not sure)
The MAC Address entry for eth7 interface of kerosene in hosts.yaml has hostname set to kerosene-8 https://gitlab.suse.de/OPS-Service/salt/-/blob/production/pillar/domain/qe_nue2_suse_org/hosts.yaml?ref_type=heads#L622
Updated by gpathak about 1 month ago · Edited
kerosene workers were up and running tests:
- https://openqa.opensuse.org/admin/workers/1108
- https://openqa.opensuse.org/admin/workers/1112
- https://openqa.opensuse.org/admin/workers/1109
- https://openqa.opensuse.org/admin/workers/1111
- https://openqa.opensuse.org/admin/workers/1333
Right now running ./reboot-stability-check kerosene.qe.nue2.suse.org, will release Kerosene once the script completes, provided the machine remains stable after the check.
Updated by nicksinger about 1 month ago
- Copied to deleted (action #169576: Recover qa-power8-3 power machine size:S)
Updated by nicksinger about 1 month ago
- Blocks action #169576: Recover qa-power8-3 power machine size:S added
Updated by gpathak about 1 month ago
gpathak wrote in #note-48:
kerosene workers were up and running tests:
- https://openqa.opensuse.org/admin/workers/1108
- https://openqa.opensuse.org/admin/workers/1112
- https://openqa.opensuse.org/admin/workers/1109
- https://openqa.opensuse.org/admin/workers/1111
- https://openqa.opensuse.org/admin/workers/1333
Right now running
./reboot-stability-check kerosene.qe.nue2.suse.org, will release Kerosene once the script completes, provided the machine remains stable after the check.
The script finished around 22 iterations and everytime the machine booted successfully.
Kerosene is on Leap 15.5
Updated by gpathak about 1 month ago
- Status changed from In Progress to Feedback
Updated by gpathak about 1 month ago · Edited
Attempting to upgrade kerosene or qa-power8 from Leap 15.5 to 15.6 with zypper locks for kernel, util-linux and firewall results in package conflicts
Loading repository data...
Reading installed packages...
The following 23 items are locked and will not be changed by any action:
Available:
firewall-applet firewall-config firewalld firewalld-lang firewalld-prometheus-config firewalld-rpcbind-helper firewall-macros
kernel-default-base kernel-default-base-rebuild kernel-default-devel kernel-default-livepatch kernel-default-livepatch-devel keylime-firewalld
plasma5-firewall plasma5-firewall-lang SuSEfirewall2 susefirewall2-to-firewalld
Installed:
kernel-default kernel-default-extra kernel-default-optional python3-firewall util-linux yast2-firewall
The following 4 package updates will NOT be installed:
kernel-default kernel-default-extra kernel-default-optional util-linux
Nothing to do.
Warning: Enforced setting: $releasever=15.6
Retrieving repository 'devel_openQA' metadata ..............................................................................................[done]
Building repository 'devel_openQA' cache ...................................................................................................[done]
Retrieving repository 'devel_openQA_Leap' metadata .........................................................................................[done]
Building repository 'devel_openQA_Leap' cache ..............................................................................................[done]
Retrieving repository 'openSUSE-Leap-15.6-1' metadata -----------------------------------------------------------------------------------------[|]
Note: Received 1 new package signing key from repository "openSUSE-Leap-15.6-1":
Those additional keys are usually used to sign packages shipped by the repository. In order to
validate those packages upon download and installation the new keys will be imported into the rpm
database.
New:
Key Fingerprint: 7F00 9157 B127 B994 D5CF BE76 F74F 09BC 3FA1 D6CE
Key Name: SUSE Package Signing Key <build@suse.de>
Key Algorithm: RSA 4096
Key Created: Thu 19 Jan 2023 01:39:40 PM GMT
Key Expires: Mon 18 Jan 2027 01:39:40 PM GMT
Rpm Name: gpg-pubkey-3fa1d6ce-63c9481c
The repository metadata introducing the new keys have been signed and validated by the trusted
key:
Repository: openSUSE-Leap-15.6-1
Key Fingerprint: AD48 5664 E901 B867 051A B15F 35A2 F86E 29B7 00A4
Key Name: openSUSE Project Signing Key <opensuse@opensuse.org>
Key Algorithm: RSA 4096
Key Created: Mon 20 Jun 2022 02:03:14 PM GMT
Key Expires: Fri 19 Jun 2026 02:03:14 PM GMT
Rpm Name: gpg-pubkey-29b700a4-62b07e22
Retrieving repository 'openSUSE-Leap-15.6-1' metadata ......................................................................................[done]
Building repository 'openSUSE-Leap-15.6-1' cache ...........................................................................................[done]
Retrieving repository 'Update repository of openSUSE Backports' metadata ...................................................................[done]
Building repository 'Update repository of openSUSE Backports' cache ........................................................................[done]
Retrieving repository 'Non-OSS Repository' metadata ........................................................................................[done]
Building repository 'Non-OSS Repository' cache .............................................................................................[done]
Repository 'Open H.264 Codec (openSUSE Leap)' is up to date.
Retrieving repository 'Main Repository' metadata ...........................................................................................[done]
Building repository 'Main Repository' cache ................................................................................................[done]
Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata ..............................................[done]
Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ...................................................[done]
Retrieving repository 'Main Update Repository' metadata ....................................................................................[done]
Building repository 'Main Update Repository' cache .........................................................................................[done]
Retrieving repository 'Update Repository (Non-Oss)' metadata ...............................................................................[done]
Building repository 'Update Repository (Non-Oss)' cache ....................................................................................[done]
All repositories have been refreshed.
Warning: Enforced setting: $releasever=15.6
Loading repository data...
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...
Problem: 1: the to be installed util-linux-systemd-2.39.3-150600.2.1.ppc64le conflicts with 'kernel < 5.8' provided by the installed kernel-default-5.3.18-150300.59.93.1.ppc64le
Solution 1: Following actions will be done:
remove lock to allow removal of kernel-default-5.3.18-150300.59.93.1.ppc64le
remove lock to allow removal of util-linux-2.36.2-150300.4.23.1.ppc64le
remove lock to allow removal of kernel-default-extra-5.3.18-150300.59.93.1.ppc64le
remove lock to allow removal of kernel-default-optional-5.3.18-150300.59.93.1.ppc64le
Solution 2: Following actions will be done:
deinstallation of util-linux-systemd-2.37.4-150500.9.17.2.ppc64le
deinstallation of os-prober-1.76-150100.4.2.1.ppc64le
deinstallation of sysconfig-netconfig-0.85.9-150200.12.1.ppc64le
deinstallation of sysconfig-0.85.9-150200.12.1.ppc64le
deinstallation of syslog-service-2.0-11.2.noarch
deinstallation of yast2-4.5.27-150500.3.6.2.ppc64le
Solution 3: keep obsolete util-linux-systemd-2.37.4-150500.9.17.2.ppc64le
Choose from above solutions by number or cancel [1/2/3/c/d/?] (c): c
[screen is terminating]
kerosene-8:~ # zypper ll
# | Name | Type | Repository | Comment
--+------------------+---------+------------+------------------------------------------
1 | *firewall* | package | (any) | boo#1227616
2 | *kernel-default* | package | (any) | poo#119008, kernel regression boo#1202138
3 | util-linux | package | (any) | poo#119008, kernel regression boo#1202138
kerosene-8:~ #
Leaving kerosene-8 machine on Leap 15.5, created a new ticket for upgrading the ppc machines #169939
Updated by gpathak about 1 month ago
- Copied to action #169939: Upgrade Power8 o3 workers to openSUSE Leap 15.6 added
Updated by nicksinger about 1 month ago
- Blocks deleted (action #169576: Recover qa-power8-3 power machine size:S)
Updated by nicksinger about 1 month ago
- Related to action #169576: Recover qa-power8-3 power machine size:S added
Updated by okurz about 1 month ago
- Due date deleted (
2024-11-15) - Status changed from Feedback to Resolved
okurz@ariel:~> hosts="openqaworker21 openqaworker22 openqaworker23 openqaworker24 openqaworker25 openqaworker26 openqaworker27 openqaworker28 openqaworker-arm21 openqaworker-arm22 qa-power8-3"; for i in $hosts; do echo "### $i" && ssh root@$i "grep VERSION_ID /etc/os-release" ; done
### openqaworker21
VERSION_ID="15.6"
### openqaworker22
VERSION_ID="15.6"
### openqaworker23
VERSION_ID="15.6"
### openqaworker24
VERSION_ID="15.6"
### openqaworker25
VERSION_ID="15.6"
### openqaworker26
VERSION_ID="15.6"
### openqaworker27
VERSION_ID="15.6"
### openqaworker28
VERSION_ID="15.6"
### openqaworker-arm21
VERSION_ID="15.6"
### openqaworker-arm22
VERSION_ID="15.6"
### qa-power8-3
VERSION_ID="15.5"
same for aarch64-o3.qe.nue2.suse.org
shows we are all good with exception of ppc which is handled in separate ticket.