QA (public) » openQA Project (public)

Same as on OSD workers. System network does not come up. Connected over IPMI SoL and found from systemctl status wickedd-nanny

Jun 14 11:53:04 openqaworker21 wickedd-nanny[2755]: device br0: call to org.opensuse.Network.Firewall.firewallUp() failed: DBus method call timed out
…

same for other network devices. Rolling back.

From fvogt
org.opensuse.Network.Firewall is not firewalld, it's wicked. That it timed out means that the service exists and something is running. wicked provides this and firewallUp() just calls /etc/wicked/extensions/firewall up. And that basically just calls firewall-cmd to assign interfaces. wickedd-nanny is waiting for wicked which waits for firewalld

blocked on #162296

openqaworker20 has been updated, some tests that were successful after upgrade

• https://openqa.opensuse.org/tests/4585980
• https://openqa.opensuse.org/tests/4585961
• https://openqa.opensuse.org/tests/4585948

Kerosene has been updated, but it is now stuck in Petitboot.

Workers to be updated

• aarch64-o3.qe.nue2.suse.org
• openqaworker22
• openqaworker23
• openqaworker24
• openqaworker25
• openqaworker26
• openqaworker28
• openqaworker-arm21
• openqaworker-arm22
• qa-power8-3

Workers updated:

• aarch64-o3: https://openqa.opensuse.org/tests/4587863
• openqaworker20: https://openqa.opensuse.org/tests/4588779
• openqaworker21: https://openqa.opensuse.org/tests/4588803
• openqaworker22: https://openqa.opensuse.org/tests/4587687
• openqaworker23: https://openqa.opensuse.org/tests/4588354
• openqaworker24: https://openqa.opensuse.org/tests/4588778
• openqaworker25: https://openqa.opensuse.org/tests/4588814
• openqaworker26: https://openqa.opensuse.org/tests/4589711, https://openqa.opensuse.org/tests/4589787
• openqaworker28: https://openqa.opensuse.org/tests/4589708, https://openqa.opensuse.org/tests/4589704
• openqaworker-arm21: https://openqa.opensuse.org/tests/4588411
• openqaworker-arm22: https://openqa.opensuse.org/tests/4587879

kerosene.qe.nue2.suse.org is stuck in Petitboot

Workers to be updated:

• qa-power8-3

@okurz @nicksinger @jbaier_cz Can any one of you please review the state of kerosene?
I can then upgrade the qa-power8-3.

As suggested, I rebooted kerosene multiple times, I observed some (random) worker instances always fails to start after a reboot, but most of the workers starts automatically at boot.

Seems like Kerosene is facing some issue: https://suse.slack.com/archives/C02CANHLANP/p1730816673953399?thread_ts=1730299105.029509&cid=C02CANHLANP
After searching a bit about the issue, came across this https://bugzilla.redhat.com/show_bug.cgi?id=1215618

As suggested, I executed ppc64_cpu --smt=off on kerosene (not sure if the changes are persistent across reboots https://github.com/ibm-power-utilities/powerpc-utils/issues/11) and re-triggered the jobs:

• https://openqa.opensuse.org/tests/4622363
• https://openqa.opensuse.org/tests/4622364
• https://openqa.opensuse.org/tests/4622366
• https://openqa.opensuse.org/tests/4622367

Somehow, GRUB is no longer present on qa-power8-3. After a reboot, Petitboot didn't show any entries, and the machine got stuck at the Petitboot screen.
After dropping to the Petitboot shell and running kexec:

kexec --load /var/petitboot/mnt/dev/sda3/boot/vmlinux --initrd /var/petitboot/mnt/dev/sda3/boot/initrd --append "root=UUID=1cd396af-c783-43b9-92d4-210b129a5cdd nospec kvm.nested=1 kvm_intel.nested=1 kvm_amd.nested=1 kvm-arm.nested=1 crashkernel=210M"

The machine successfully booted, and the network came up. However, I discovered that the GRUB binary wasn't present in /boot/grub2/.

An attempt to reinstall GRUB using grub2-install /dev/sda did install grub but resulted in the following error:
grub2-install: error: the chosen partition is not a PReP partition.
The Petitboot still isn't displaying any boot entries.

Unlike kerosene, power8-3 does not have a PReP partition. Running fdisk -l listed all partitions (as shown), but there was no PReP partition found on power8-3.

qa-power8-3:~ # fdisk -l
Disk /dev/sda: 1.09 TiB, 1200243695616 bytes, 2344225968 sectors
Disk model: HUC101212CSS600 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: DAA364D6-D309-41A3-A07D-1BF8A366C008

Device     Start        End    Sectors  Size Type
/dev/sda1   2048 2344225934 2344223887  1.1T Linux filesystem


Disk /dev/sdc: 361.31 GiB, 387956342784 bytes, 757727232 sectors
Disk model: IPR-0   5EC32700
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 770DFCF6-D082-4E67-832A-7A7DA23297D9

Device         Start       End   Sectors   Size Type
/dev/sdc3    1067008 753530879 752463872 358.8G Linux filesystem
/dev/sdc4  753530880 757727198   4196319     2G Linux swap
qa-power8-3:~ # 

Now, it seems only a re-installation can fix this issue.

@nicksinger @okurz Do we have any other approach? I did search for relevant tasks/tickets but couldn't find any helpful information.

kerosene and power-qa-8-3 needs a fresh installation with proper partitioning and brtfs filesystem.
kerosene doesn't have brtfs and power-8-3 doesn't have PReP Boot partition.
@okurz Should we create separate ticket for these two activities?

gpathak wrote in #note-36:

kerosene and power-qa-8-3 needs a fresh installation with proper partitioning and brtfs filesystem.
kerosene doesn't have brtfs and power-8-3 doesn't have PReP Boot partition.
@okurz Should we create separate ticket for these two activities?

Hold on. I don't see where that conclusion comes from that a full reinstall would be necessary. Can you elaborate on what happened that caused this now? And please keep at least one machine operational until the other is fully back. For now due to those open points I suggest to continue in this ticket.

@okurz
Not sure about how it happened on power-8, but I figured that it doesn't have a PReP boot partition even before performing an upgrade when I issued a reboot from ssh session just to check if Petitboot lists all GRUB entries and it didn't. After that I tried reinstalling grub on /dev/sda and got grub2-install: error: the chosen partition is not a PReP partition.

Kerosene is good, but the only thing that's concerning is snapper isn't configured and is not taking snapshots (not sure if it works on ext4)
I will try to make kerosene stable first and then target qa-power-8 cause power-8 isn't even booting, manual kexec is working though, grub is gone.

The openqa-auto-update and openqa-continuous-update is causing kerosene workers to go offline, the machine also become unreachable after an update is performed via auto-update.
Masked following units for time being:

• openqa-auto-update.timer
• openqa-auto-update.service
• openqa-continuous-update.service
• openqa-continuous-update.timer

For kerosene please set a static hostname "kerosene" using hostnamectl

Did a fresh installation of leap 15.5 on kerosene, it's more than 12 hours the machine is up and running openQA tests.
Earlier, the kernel was crashing causing the machine unreachable.
@nicksinger is working on recovering qa-power-8-3 machine in a separate ticket: #169576

@okurz Almost all the AC's are covered.

okurz wrote in #note-41:

For kerosene please set a static hostname "kerosene" using hostnamectl

Please make sure worker instances show up with that name on o3 and reference successful openQA jobs for validation

okurz wrote in #note-46:

Please make sure worker instances show up with that name on o3 and reference successful openQA jobs for validation

Kerosene is showing up as kerosene-8, with the transient hostname maybe because of the DNS response received from https://gitlab.suse.de/OPS-Service/salt/-/blob/production/salt/profile/dns/files/nue2_suse_org/dns-qe.nue2.suse.org?ref_type=heads&plain=1#L397 (I am not sure)

The MAC Address entry for eth7 interface of kerosene in hosts.yaml has hostname set to kerosene-8 https://gitlab.suse.de/OPS-Service/salt/-/blob/production/pillar/domain/qe_nue2_suse_org/hosts.yaml?ref_type=heads#L622

kerosene workers were up and running tests:

• https://openqa.opensuse.org/admin/workers/1108
• https://openqa.opensuse.org/admin/workers/1112
• https://openqa.opensuse.org/admin/workers/1109
• https://openqa.opensuse.org/admin/workers/1111
• https://openqa.opensuse.org/admin/workers/1333

Right now running ./reboot-stability-check kerosene.qe.nue2.suse.org, will release Kerosene once the script completes, provided the machine remains stable after the check.

gpathak wrote in #note-48:

kerosene workers were up and running tests:

• https://openqa.opensuse.org/admin/workers/1108
• https://openqa.opensuse.org/admin/workers/1112
• https://openqa.opensuse.org/admin/workers/1109
• https://openqa.opensuse.org/admin/workers/1111
• https://openqa.opensuse.org/admin/workers/1333

Right now running ./reboot-stability-check kerosene.qe.nue2.suse.org, will release Kerosene once the script completes, provided the machine remains stable after the check.
The script finished around 22 iterations and everytime the machine booted successfully.
Kerosene is on Leap 15.5

Attempting to upgrade kerosene or qa-power8 from Leap 15.5 to 15.6 with zypper locks for kernel, util-linux and firewall results in package conflicts

Loading repository data...
Reading installed packages...

The following 23 items are locked and will not be changed by any action:
 Available:
  firewall-applet firewall-config firewalld firewalld-lang firewalld-prometheus-config firewalld-rpcbind-helper firewall-macros
  kernel-default-base kernel-default-base-rebuild kernel-default-devel kernel-default-livepatch kernel-default-livepatch-devel keylime-firewalld
  plasma5-firewall plasma5-firewall-lang SuSEfirewall2 susefirewall2-to-firewalld
 Installed:
  kernel-default kernel-default-extra kernel-default-optional python3-firewall util-linux yast2-firewall

The following 4 package updates will NOT be installed:
  kernel-default kernel-default-extra kernel-default-optional util-linux
Nothing to do.
Warning: Enforced setting: $releasever=15.6
Retrieving repository 'devel_openQA' metadata ..............................................................................................[done]
Building repository 'devel_openQA' cache ...................................................................................................[done]
Retrieving repository 'devel_openQA_Leap' metadata .........................................................................................[done]
Building repository 'devel_openQA_Leap' cache ..............................................................................................[done]
Retrieving repository 'openSUSE-Leap-15.6-1' metadata -----------------------------------------------------------------------------------------[|]
Note: Received 1 new package signing key from repository "openSUSE-Leap-15.6-1":

  Those additional keys are usually used to sign packages shipped by the repository. In order to
  validate those packages upon download and installation the new keys will be imported into the rpm
  database.

  New:
  Key Fingerprint:  7F00 9157 B127 B994 D5CF BE76 F74F 09BC 3FA1 D6CE
  Key Name:         SUSE Package Signing Key <build@suse.de>
  Key Algorithm:    RSA 4096
  Key Created:      Thu 19 Jan 2023 01:39:40 PM GMT
  Key Expires:      Mon 18 Jan 2027 01:39:40 PM GMT
  Rpm Name:         gpg-pubkey-3fa1d6ce-63c9481c

  The repository metadata introducing the new keys have been signed and validated by the trusted
  key:

  Repository:       openSUSE-Leap-15.6-1
  Key Fingerprint:  AD48 5664 E901 B867 051A B15F 35A2 F86E 29B7 00A4
  Key Name:         openSUSE Project Signing Key <opensuse@opensuse.org>
  Key Algorithm:    RSA 4096
  Key Created:      Mon 20 Jun 2022 02:03:14 PM GMT
  Key Expires:      Fri 19 Jun 2026 02:03:14 PM GMT
  Rpm Name:         gpg-pubkey-29b700a4-62b07e22

Retrieving repository 'openSUSE-Leap-15.6-1' metadata ......................................................................................[done]
Building repository 'openSUSE-Leap-15.6-1' cache ...........................................................................................[done]
Retrieving repository 'Update repository of openSUSE Backports' metadata ...................................................................[done]
Building repository 'Update repository of openSUSE Backports' cache ........................................................................[done]
Retrieving repository 'Non-OSS Repository' metadata ........................................................................................[done]
Building repository 'Non-OSS Repository' cache .............................................................................................[done]
Repository 'Open H.264 Codec (openSUSE Leap)' is up to date.                                                                                      
Retrieving repository 'Main Repository' metadata ...........................................................................................[done]
Building repository 'Main Repository' cache ................................................................................................[done]
Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata ..............................................[done]
Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ...................................................[done]
Retrieving repository 'Main Update Repository' metadata ....................................................................................[done]
Building repository 'Main Update Repository' cache .........................................................................................[done]
Retrieving repository 'Update Repository (Non-Oss)' metadata ...............................................................................[done]
Building repository 'Update Repository (Non-Oss)' cache ....................................................................................[done]
All repositories have been refreshed.
Warning: Enforced setting: $releasever=15.6
Loading repository data...
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...

Problem: 1: the to be installed util-linux-systemd-2.39.3-150600.2.1.ppc64le conflicts with 'kernel < 5.8' provided by the installed kernel-default-5.3.18-150300.59.93.1.ppc64le
 Solution 1: Following actions will be done:
  remove lock to allow removal of kernel-default-5.3.18-150300.59.93.1.ppc64le
  remove lock to allow removal of util-linux-2.36.2-150300.4.23.1.ppc64le
  remove lock to allow removal of kernel-default-extra-5.3.18-150300.59.93.1.ppc64le
  remove lock to allow removal of kernel-default-optional-5.3.18-150300.59.93.1.ppc64le
 Solution 2: Following actions will be done:
  deinstallation of util-linux-systemd-2.37.4-150500.9.17.2.ppc64le
  deinstallation of os-prober-1.76-150100.4.2.1.ppc64le
  deinstallation of sysconfig-netconfig-0.85.9-150200.12.1.ppc64le
  deinstallation of sysconfig-0.85.9-150200.12.1.ppc64le
  deinstallation of syslog-service-2.0-11.2.noarch
  deinstallation of yast2-4.5.27-150500.3.6.2.ppc64le
 Solution 3: keep obsolete util-linux-systemd-2.37.4-150500.9.17.2.ppc64le

Choose from above solutions by number or cancel [1/2/3/c/d/?] (c): c

[screen is terminating]
kerosene-8:~ # zypper ll

# | Name             | Type    | Repository | Comment
--+------------------+---------+------------+------------------------------------------
1 | *firewall*       | package | (any)      | boo#1227616
2 | *kernel-default* | package | (any)      | poo#119008, kernel regression boo#1202138
3 | util-linux       | package | (any)      | poo#119008, kernel regression boo#1202138

kerosene-8:~ # 

Leaving kerosene-8 machine on Leap 15.5, created a new ticket for upgrading the ppc machines #169939