QA » openQA Project » openQA Tests » qe-yam

opensusebasetest::wait_boot -> lib/opensusebasetest.pm:922 called opensusebasetest::wait_boot_past_bootloader -> lib/opensusebasetest.pm:778 called opensusebasetest::wait_boot_textmode -> lib/opensusebasetest.pm:714 called testapi::assert_screen
[2024-02-01T11:14:30.298319+01:00] [debug] [pid:75171] <<< testapi::assert_screen(mustmatch=[
    "linux-login",
    "emergency-shell",
    "emergency-mode"
  ], timeout=500)

• PR:
• From VR, this might be not a time issue. even in dev mode, the screen hang there. need to check the console output for more details.
• this case hang at the SUSE screen for 2+ hours
• now the VR run with QEMURAM=4096 and patch:

--- a/tests/console/validate_lvm_raid1.pm
+++ b/tests/console/validate_lvm_raid1.pm
@@ -27,7 +27,7 @@ sub run {
     _check_raid1_partitioning($config, $expected_num_devs);
     _remove_raid_disk($config, $expected_num_devs);
     _reboot();
-    $self->wait_boot;
+    $self->wait_boot(textmode => 1, ready_time => 600, bootloader_time => 300);
     _check_raid_disks_after_reboot($config, $expected_num_devs);
 }

@@ -69,7 +69,7 @@ sub _remove_raid_disk {

 sub _reboot {
     record_info('system reboots');
-    power_action('reboot', textmode => 'textmode');
+    power_action('reboot', textmode => 1);
 }

From the VR above, I think this may maybe a bug while not a timing issue from our test side, we can wait for the new build to verify it.

Also with last build 53.1 and 4GB RAM the failure rate is 2/10. See VRs

Similar result with 6GB RAM. See VRs

you can focus in x86_64, aarch64 is pretty slow in openQA to debug.

Issue still present on build 61.1

if you already detected a bug, this is not the place to report it :) but bugzilla, I assume you are blocked here not able to make it stable.

By increasing:

• bootloader_time => 400
• ready_time => 600

in $self->wait_boot(bootloader_time => 400, ready_time => 600); there is ~ 50% passed.
See VRs

Created WIP PR#18827

Slack thread with more info from previous errors:
https://suse.slack.com/archives/C02CLB2LB7Z/p1710139073566099

More info also here:

• bsc#1150370 raid1 does not recover faulty disk after reboot
• bsc#1172203 Inconsistent behavior of raid1 disk resync after reboot

Fresh system installation of SLES Text mode, with 4 hard disks.
/dev/md0 consists of 4 devices.
After first boot, one of the disk is set to faulty and removed:

mdadm --detail /dev/md0 # this will return "Active devices : 4" every time
mdadm --manage /dev/md0 --set-faulty /dev/vdd2
mdadm --manage /dev/md0 --remove /dev/vdd2
reboot
mdadm --detail /dev/md0 # this will return sometimes "Active devices: 3" and sometimes "Active Devices: 4". 

See comment:

The above assumption is not correct, md raid kernel and user space code don't provide such assurance.
The behavior varies depending on the timing, that is for different internal state the md raid is in when fail the component disk and reboot the system, the observed result varies.

By removing the call to sub _remove_raid_disk, now the VRs works fine.
Removed WIP for PR#18827. Ready to merge.

Merged PR#18827

Verified in build 64.2