Project

General

Profile

action #62240

[functional][u][s390x-kvm-sle12] test fails in reboot_gnome - timeout, SUT takes long time to reboot

Added by SLindoMansilla 6 months ago. Updated 2 months ago.

Status:
Rejected
Priority:
High
Assignee:
Category:
Bugs in existing tests
Target version:
-
Start date:
2020-01-17
Due date:
% Done:

0%

Estimated time:
Difficulty:
Duration:

Description

Observation

Maybe the timeout needs to be increased.

Reproducible

  • In scenario sle-15-SP2-Online-s390x-allpatterns@s390x-kvm-sle12
  • Fails since Build 122.1
  • Current occurrence: 126.1

Expected result

Last good: 108.1

Further details

Always latest result in this scenario: latest


Related issues

Has duplicate openQA Tests - action #65001: SUT will stay in shutdown after reboot command on s390 svirt backendRejected2020-03-30

History

#1 Updated by mgriessmeier 6 months ago

if it ever actually reboots and not just shut down, will investigate next week

#2 Updated by maritawerner 6 months ago

  • Priority changed from Normal to High

This Testcase fails twice in osd. Could we get that fixed until beta2 in two weeks?

#3 Updated by mgriessmeier 6 months ago

it looks like it is hanging in grub, and we dont press return: https://openqa.suse.de/tests/3796544#step/reboot_gnome/27

sorry my mistake, we already miss grub in serial, but in fact it is shown, just too late - so yeah most likely timeout issue

#4 Updated by mgriessmeier 6 months ago

  • Status changed from New to Feedback
  • Assignee set to mgriessmeier

#5 Updated by mgriessmeier 6 months ago

  • Status changed from Feedback to Resolved

180 seems to be fine: https://openqa.suse.de/tests/3831488/

will close for now

#6 Updated by zluo 4 months ago

  • Status changed from Resolved to Workable

https://openqa.suse.de/tests/3983008#step/reboot_gnome/18

it seems that this issue comes back.

#7 Updated by bchou 4 months ago

Hello mgriessmeier and zluo,

May I treat the s390x reboot problem from my test case are the related issue with this one?
https://openqa.nue.suse.com/tests/3987551#step/fips_setup/28

The issue was reproduced from new build (155.1) and it is also the problem about boot fail after rebooting s390x.

Thanks a lot for your information in advance.

#8 Updated by llzhao 4 months ago

I have tired "time_out=300s" security test case still failed on the same reason:
"echo >$pty" got "-bash:/dev/pts/5: permission denied"

FYI:
http://openqa.nue.suse.com/tests/4004166#
in lib/opensusebasetest.pm:
...

wait_serial('GNU GRUB', 180) || diag 'Could not find GRUB screen, continuing nevertheless, trying to boot';
wait_serial('GNU GRUB', 300) || diag 'Could not find GRUB screen, continuing nevertheless, trying to boot';

#9 Updated by llzhao 4 months ago

llzhao wrote:

I have tired "time_out=300s" security test case still failed on the same reason:
"echo >$pty" got "-bash:/dev/pts/5: permission denied"

FYI:
http://openqa.nue.suse.com/tests/4004166#
in lib/opensusebasetest.pm:
...

wait_serial('GNU GRUB', 180) || diag 'Could not find GRUB screen, continuing nevertheless, trying to boot';
wait_serial('GNU GRUB', 300) || diag 'Could not find GRUB screen, continuing nevertheless, trying to boot';

FYI:
Sometimes we got error "ambiguous redirect" too as the "$pty" became NULL after reboot, it seems the vm is not rebooted successfully even waited more than 300s.

#10 Updated by mgriessmeier 4 months ago

that is actually: https://bugzilla.suse.com/show_bug.cgi?id=1167210
but Yast team has a workaround with softfail implemented for AutoYaST tests, I will do the same for all the other tests

#11 Updated by zluo 4 months ago

the last good one is not available anymore, I would like to compare with successful run at first.

As I can remember that kvm was not stable on Power8 and it never does reboot successfully. zkvm could be different and stable. Question: reboot needs to be handed over to libvirt and host, right?

#12 Updated by rfan1 4 months ago

mgriessmeier wrote:

that is actually: https://bugzilla.suse.com/show_bug.cgi?id=1167210
but Yast team has a workaround with softfail implemented for AutoYaST tests, I will do the same for all the other tests

Thanks Matthias/Zaoliang, so it can explain that we hit "echo >$pty" got "-bash:/dev/pts/5: permission denied" during the fips tests which run in openQA.

However, the issue is gone with the latest RC1 drop. https://openqa.nue.suse.com/tests/4022816#step/fips_setup/31
So seems the bug 1167210 is fixed?

#13 Updated by bchou 4 months ago

Hello,
The issue is gone at build 162.1/163.1, but it can be reproduced again from 163.11 and 164.1 newer build ( Regression ). Base one the bsc#1167210 - [Build 160.1] System shuts down instead of reboot on zkvm, we think it could be a real product bug. In our test run in openQA, the reboot function block most of our s390x cases. Is there any workaround that can be applied for this point? Thanks a lot.

#14 Updated by pcervinka 3 months ago

  • Has duplicate action #65001: SUT will stay in shutdown after reboot command on s390 svirt backend added

#15 Updated by okurz 3 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: fips_env_mode_tests_crypt_core
https://openqa.suse.de/tests/4111367

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#16 Updated by rfan1 3 months ago

The issue can be seen again with drop 176.1, after reboot a vm. I saw qemu crash

2020-04-13 01:43:59.146+0000: shutting down, reason=crashed
2020-04-13T01:43:59.147443Z qemu-system-s390x: terminating on signal 15 from pid 1712 (/usr/sbin/libvirtd)

Most of fips cases are failed due to this:
https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=176.1&groupid=268

#17 Updated by rfan1 3 months ago

Thanks much Liang for providing the setup. it helped much.

The issue can not be seen on s390zp12 with different(newer) OS/qemu/libvirt version.

s390zp12:~ # uname -r
5.3.18-10-default
s390zp12:~ # cat /etc/*release
NAME="SLES"
VERSION="15-SP2"
VERSION_ID="15.2"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP2"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp2"
s390zp12:~ # virsh version
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0

After the os installation, vm can boot up successfully. and no issue seen if I reboot it again.

That means 2 issues mentioned in the bug is not seen on this hypervisor.

BR//Richard.

Also available in: Atom PDF