QA (public) » openQA Project (public) » openQA Infrastructure (public)

After removing Kernel 6.4.0.150600.23.30, the machine is now able to boot properly without going into maintenance mode.
Triggered some jobs on o3 to check if tests pass:

• https://openqa.opensuse.org/tests/4761132
• https://openqa.opensuse.org/tests/4761134
• https://openqa.opensuse.org/tests/4761139

I was able to upgrade the qa-power8 machine to 15.6 with Kernel version 6.13.0-rc6-3.g25a2d29-default installed from https://download.opensuse.org/repositories/Kernel:/HEAD/PPC/
Have to replace suse module tools to non-standard repository https://download.opensuse.org/repositories/home:/mwilck:/suse-module-tools/PowerPC/
Installed filesystem-84.87-16.1.ppc64le package from https://build.opensuse.org/package/show/openSUSE:Factory/filesystem and changed vendor to obs://build.opensuse.org/Base:System

Tried cloning jobs to run to qa-power8, but the jobs are failing
Fatal error in command /usr/bin/chattr +C /var/lib/openqa/pool/4/raid: Can't locate object method "quirkiness" via package "Mojo::IOLoop::ReadWriteProcess" at /usr/lib/os-autoinst/osutils.pm line 63
https://openqa.opensuse.org/tests/4764546

other error was main: Can't locate object method "_fork_collect_status" via package "Mojo::IOLoop::ReadWriteProcess" at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess/Session.pm line 79
https://openqa.opensuse.org/tests/4765533

gpathak wrote in #note-19:

Not sure how to save it permanently.

Turned out that /etc/default/grub needs to have some extra configuration that wasn't present on qa-power8.
Checked petrol and diesel for grub configuration in /etc/default/grub, copied the common configuration on qa-power8 as well and the boot issue mentioned in my previous comment is resolved.

Right now running reboot-stability-check script https://github.com/os-autoinst/scripts/blob/master/reboot-stability-check.
Linux Kernel Version 6.4 is currently active and running on qa-power8

The jobs on qa-power8-3 were failing with error:
[2025-01-10T19:04:01.488855+01:00] [debug] [pid:29227] Fatal error in command/usr/bin/chattr +C /var/lib/openqa/pool/4/raid: Can't locate object method "quirkiness" via package "Mojo::IOLoop::ReadWriteProcess" at /usr/lib/os-autoinst/osutils.pm line 63.

Turned out that there were two ReadWriteProcess.pm files — one located in /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop and the other in /usr/lib/perl5/site_perl/5.26.1/Mojo/IOLoop. The latter did not have any quirkiness defined, such as:

has [qw(blocking_stop serialize quirkiness total_sleeptime_during_kill)] => 0;

This definition was present in the vendor_perl version.

To resolve the issue, I had to delete the ReadWriteProcess.pm file and ReadWriteProcess directory from site_perl, restarted the job, and isotovideo started successfully.

• https://openqa.opensuse.org/tests/4777953

I will do some more tests.

• https://openqa.opensuse.org/tests/4777971
• https://openqa.opensuse.org/tests/4777985

Executed some more jobs with Kernel 6.4:

• https://openqa.opensuse.org/tests/4780030
• https://openqa.opensuse.org/tests/4780019
• https://openqa.opensuse.org/tests/4780250
• https://openqa.opensuse.org/tests/4780251
• https://openqa.opensuse.org/tests/4780241

Seems the machine is stable now, will keep it under observation to see if it crashes again, if it doesn't crash then I will move to next machine and upgrade kerosene.

Disabled worker slots 11..30 on kerosene to redirect some job schedule to run on qa-power8.

Upgraded kerosene to Leap 15.6 as well, and is under observation.

gpathak wrote in #note-28:

qa-power8-3 crashed again

Installed old firewalld package
zypper in --oldpackage https://download.opensuse.org/update/leap/15.3/sle/noarch/firewalld-0.9.3-150300.3.3.1.noarch.rpm https://download.opensuse.org/update/leap/15.3/sle/noarch/python3-firewall-0.9.3-150300.3.3.1.noarch.rpm https://download.opensuse.org/update/leap/15.3/sle/noarch/firewalld-lang-0.9.3-150300.3.3.1.noarch.rpm

and added package lock zypper al -m "boo#1227616" *firewall*

Kerosene is also crashing but it seems unrelated to firewall (I am not sure at this point because I am unable to get any crash log yet)
The issue on kerosene seems to be related to corrupted filesystem of node /dev/sdb1

[  125.598476][ T4515] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282262: comm perl: deleted inode referenced: 22315870
[  125.861617][ T4515] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282262: comm perl: deleted inode referenced: 22315865
[  125.877535][ T4515] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282262: comm perl: deleted inode referenced: 22315864
[  140.606753][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  140.609626][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  140.622667][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  141.286896][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  141.295465][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  141.392507][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  141.395382][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  141.399007][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  141.403024][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  141.407136][ T4035] EXT4-fs error (device sdb1) in ext4_free_inode:362: Corrupt filesystem
[  174.313161][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22315988
[  174.327871][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22315987
[  174.339871][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22315986
[  174.343795][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22554957
[  174.349235][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22315988
[  174.351765][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22315987
[  174.355750][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22315986
[  174.359731][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22554957
[  174.368650][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22315988
[  174.371705][ T4455] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm perl: deleted inode referenced: 22315987
[  181.099461][ T4045] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm worker: deleted inode referenced: 22315988
[  181.145671][ T4045] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm worker: deleted inode referenced: 22315987
[  181.149643][ T4045] EXT4-fs error (device sdb1): ext4_lookup:1858: inode #22282268: comm worker: deleted inode referenced: 22315986

19-01-2025: Reverted kerosene to version 15.5 to enable the execution of scheduled/blocked PPC64 tests
19-01-2025: Applied the solution from #162296 on qa-power8-3 (ref https://progress.opensuse.org/issues/162296#note-35). The system remains stable, with all 8 slots successfully running tests for over 24 hours

Will be upgrading kerosene-8 as well.

Kerosene-8 is also upgraded to 15.6 running kernel 6.4.0-150600.23.33-default

okurz wrote in #note-33:

1. What about https://progress.opensuse.org/issues/169939#Rollback-Actions
2. Can you please comment on the bugzilla ticket about the kernel regression which prevented us kernel upgrades since Leap 15.4

The machines are still getting unreachable with kernel 6.4.0-150600.23.33-default, not immediately after boot-up as mentioned in https://bugzilla.suse.com/show_bug.cgi?id=1227616#c2, but now the machines are going offline after approximately 24 Hours.
I can't find a way to collect the reason of the machines going offline.

This happened on kerosene: https://bugzilla.suse.com/show_bug.cgi?id=1227616#c15
It is not happening consistently everytime.

@gpathak in your bug reports please add details about steps to reproduce and what statistics about failures you have. Also please reference the last good and the original bug report we had why we kept the kernel at the Leap 15.3 version

okurz wrote in #note-41:

@gpathak in your bug reports please add details about steps to reproduce and what statistics about failures you have. Also please reference the last good and the original bug report we had why we kept the kernel at the Leap 15.3 version

There are no clear steps to reproduce the issue at this time. The kernel on the Power8 machine occasionally crashes, causing the system to go offline while running an openQA test (though the specific test is unknown). This does not directly impact openQA results, as the job is simply marked as canceled due to the worker becoming unreachable.

The last known stable kernel version was 5.3.18.

Right now I am focusing on one power8 machine and I am currently utilising qa-power8 for the task
Going back to square-one, starting it all over and recording what I did/doing here in this ticket.

12-Feb-2025:

• Re-enabled firewalld-2.0.1-150600.3.5.1 from http://download.opensuse.org/update/leap/15.6/sle/
• Disabled custom nftables systemd service
• added all available tap devices to bridge br1 with command tapdevs=$(ip -o link show | awk -F': ' '{print $2}' | grep tap) && for i in $tapdevs; do echo $i; ovs-vsctl add-port br1 $i; done since some tests were failing https://openqa.opensuse.org/tests/4851764 with error 'tapXX' is not connected to bridge 'br1'

gpathak wrote in #note-45:

Right now I am focusing on one power8 machine and I am currently utilising qa-power8 for the task
Going back to square-one, starting it all over and recording what I did/doing here in this ticket.

12-Feb-2025:

• Re-enabled firewalld-2.0.1-150600.3.5.1 from http://download.opensuse.org/update/leap/15.6/sle/
• Disabled custom nftables systemd service
• added all available tap devices to bridge br1 with command tapdevs=$(ip -o link show | awk -F': ' '{print $2}' | grep tap) && for i in $tapdevs; do echo $i; ovs-vsctl add-port br1 $i; done since some tests were failing https://openqa.opensuse.org/tests/4851764 with error 'tapXX' is not connected to bridge 'br1'

It's been more than 24 Hours and there are significant number of tests ran on qa-power8-3, so far the machine looks stable.
Let's wait and watch the behaviour until end of this week.

spent time to setup kdump on qa-power8 since it went into the same state.
The kdump setup is not yet complete, will continue it later.

gpathak wrote in #note-53:

spent time to setup kdump on qa-power8 since it went into the same state.
The kdump setup is not yet complete, will continue it later.

Did kdump setup on kerosene as well as qa-power-8, hopefully it should now generate a kernel dump if kernel crashes again in future

• zypper in -y kexec-tools kdump-*
• added below contents to /etc/sysconfig/kdump

KDUMP_KERNELVER="/boot/vmlinux-6.4.0-150600.23.33-default"  
KDUMP_CPUS=2  
KDUMP_COMMANDLINE_APPEND=""  
KDUMP_AUTO_RESIZE="false"  
KDUMP_FADUMP="false"  
KDUMP_FADUMP_SHELL="false"  
KDUMP_IMMEDIATE_REBOOT="true"  
KDUMP_SAVEDIR="/var/crash"  
KDUMP_KEEP_OLD_DUMPS=2  
KDUMP_FREE_DISK_SIZE=64  
KDUMP_VERBOSE=0  
KDUMP_DUMPLEVEL=31  
KDUMP_DUMPFORMAT="compressed"  
KDUMP_CONTINUE_ON_ERROR="true"  
KDUMP_NETCONFIG="auto"  
KDUMP_NET_TIMEOUT=30  

• Symlinks for vmlinux and initrd in /boot/ directory should not be dangling
• Added GRUB_CMDLINE_LINUX_DEFAULT+=" crashkernel=1G" to /etc/default/grub and then update-bootloader
• Started kexec-load and kdump services systemctl enable --now kexec-load.service; systemctl enable --now kdump
• Rebooted the machine once
• Verified kdump by triggering a "kernel panic" with the procfs interface to Magic SysRq keys: echo c > /proc/sysrq-trigger

Discussed the options briefly:

• The best work-around idea currently is changing away from btrfs e.g. ext4 but that could obviously lead to corruption
• Automatic reboots like we do on some arm machines (which crashes less readily)
• Leave it for now. You made an amazing effort to find a working version. It is what it is. We voted in favor of this

livdywan wrote in #note-55:

Discussed the options briefly:

• The best work-around idea currently is changing away from btrfs e.g. ext4 but that could obviously lead to corruption
• Automatic reboots like we do on some arm machines (which crashes less readily)
• Leave it for now. You made an amazing effort to find a working version. It is what it is. We voted in favor of this

I just did a fresh Leap 15.6 installation on qa-power8-3, in my opinion the kernel in-stability seems to be ruled out because kerosene-8 is running flawlessly for more than 5 days.
I discovered some error message in boot logs (#177973) that can be a sign of dying storage medium, and I think it can be a cause of unstable behavior of qa-power8-3 machines, I am saying this because in most of the kernel crash logs of qa-power8, the stack trace points to kernel filesystem APIs for btrfs and in this case the rootfs is of type btrfs

gpathak wrote in #note-57:

livdywan wrote in #note-55:

Discussed the options briefly:

• The best work-around idea currently is changing away from btrfs e.g. ext4 but that could obviously lead to corruption
• Automatic reboots like we do on some arm machines (which crashes less readily)
• Leave it for now. You made an amazing effort to find a working version. It is what it is. We voted in favor of this

I just did a fresh Leap 15.6 installation on qa-power8-3, in my opinion the kernel in-stability seems to be ruled out because kerosene-8 is running flawlessly for more than 5 days.
I discovered some error message in boot logs (#177973) that can be a sign of dying storage medium, and I think it can be a cause of unstable behavior of qa-power8-3 machines, I am saying this because in most of the kernel crash logs of qa-power8, the stack trace points to kernel filesystem APIs for btrfs and in this case the rootfs is of type btrfs

Since, the crash was more frequent, I rolled back qa-power8-3 to Leap 15.5. If the machine crashes again because of filesystem or inode issue then I think it can be handled in #177973