action #43880: [functional][u][s390x][sporadic] test fails in shutdown on s390x - openQA Tests - openSUSE Project Management Tool

Actions

#1

Updated by okurz over 5 years ago

Status changed from New to Workable
Priority changed from Normal to High

hm, there is check_shutdown(60) which again is not that long. I recommend to bump the timeout and crosscheck with the code branches which might already wait longer.

Actions

Copy link

#2

Updated by okurz over 5 years ago

Target version set to Milestone 22

Actions

Copy link

#3

Updated by mgriessmeier over 5 years ago

Status changed from Workable to In Progress
Assignee set to mgriessmeier

I will take a look into this,
though the 60s seems to have worked in the past - might be different issue - let's investigate

Actions

Copy link

#4

Updated by okurz over 5 years ago

try SCHEDULE=tests/boot/boot_to_desktop,tests/shutdown/shutdown, or add in "bootloader_zkvm" in there.

Actions

Copy link

#5

Updated by mgriessmeier over 5 years ago

okurz wrote:

try SCHEDULE=tests/boot/boot_to_desktop,tests/shutdown/shutdown, or add in "bootloader_zkvm" in there.

problem is that the only fails in shutdown I can find are on create_hdd jobs - so we might miss the issue if we just use a qcow image
so I will first schedule the whole job once and grab the image from there

Actions

Copy link

#6

Updated by okurz over 5 years ago

well, I thought the "boot+shutdown" test would be super-fast to finish so you can schedule 100 runs easily on production and reject the hypothesis that it can also appear in the image-booting scenarios.

Actions

Copy link

#7

Updated by mgriessmeier over 5 years ago

triggered 50 jobs with custom scheduled - so far 2/11 failed =)
http://opeth.suse.de/tests/overview?build=107.5&distri=sle&version=15-SP1

Actions

Copy link

#8

Updated by mgriessmeier over 5 years ago

mgriessmeier wrote:

triggered 50 jobs with custom scheduled - so far 2/11 failed =)
http://opeth.suse.de/tests/overview?build=107.5&distri=sle&version=15-SP1

so 8 out of 50 failed, I will investigate those and trigger 50 more with increased timeout to see if the failure rate is reduced or completely gone

Actions

Copy link

#9

Updated by mgriessmeier over 5 years ago

ok, with double timeout it's 1 fail out of 50...
will go with *3 which we also have for other cases...

Actions

Copy link

#10

Updated by okurz over 5 years ago

Related to coordination #35215: [functional][u][epic][medium] test fails on shutdown module added

Actions

Copy link

#11

Updated by okurz over 5 years ago

Related to action #43064: [functional][u] test fails in boot_into_snapshot and reboot_gnome with encrypted setup - test not waiting long enough for shutdown/reboot until we end up in grub which shows in post_fail_hook added

Actions

Copy link

#12

Updated by okurz over 5 years ago

Related to action #43616: [functional][u][sporadic] test fails in shutdown - SUT took longer than 60 seconds to shutdown, no logs available (non-s390x) added

Actions

Copy link

#13

Updated by okurz over 5 years ago

Related to action #41183: [functional][u] soft-fail in shutdown should have valid bugref added

Actions

Copy link

#14

Updated by okurz over 5 years ago

Related to action #42038: [sle][functional[u] test fails in shutdown - add post_fail_hook for shutdown module if possible added

Actions

Copy link

#15

Updated by okurz over 5 years ago

Related to action #38108: [openqa][kernel] Power down needs to have longer timeout due longer shutdown of BTRFS related service added

Actions

Copy link

#16

Updated by okurz over 5 years ago

Related to action #35892: [qe-core][functional][hard] test fails in kdump_and_crash - improve bootup/shutdown debugging approach added

Actions

Copy link

#17

Updated by okurz over 5 years ago

custom scheduling rocks, right? ;)

Good evaluation. Please see the related issues as well. We should on one hand fix the test failing which we can do by bumping the timeout but also we should provide bug reports or help with investigation on already existing ones. The approach you found might be especially helpful for that.

@oorlov, as the "shutdown specialist", what suggestions can you give how to investigate the issue further or how to track according bugs with soft-fails?

@mgriessmeier maybe just find corresponding bugs about "shutdown takes too long", comment in there with suggestions how to investigate, e.g. point to the openqa-clone-job commands you used to reproduce easily.

Actions

Copy link

#18

Updated by mgriessmeier over 5 years ago

Priority changed from High to Normal

triple timeout also fails 2 out of 50 - but since it is improving the situation significantly, here is the PR:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6374

reducing urgency therefore and continuing on suggestions from okurz

Actions

Copy link

#19

Updated by mgriessmeier over 5 years ago

Status changed from In Progress to Workable
Assignee deleted (~~mgriessmeier~~)

Urgency was removed - not working on the followup atm

Actions

Copy link

#20

Updated by okurz over 5 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: create_hdd_textmode@zkvm
https://openqa.suse.de/tests/2347165

Actions

Copy link

#21

Updated by okurz over 5 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: create_hdd_gnome
https://openqa.suse.de/tests/2369949

Actions

Copy link

#22

Updated by zluo about 5 years ago

My PR (https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6702 , see #42038 for details) might be helpful to check the cause of shutdown issue.

Actions

Copy link

#23

Updated by szarate about 5 years ago

I wonder if setting the $soft_fail_data hash to have a hard timeout of 180 and a soft timeout of 60 would work in this case, today we got 3 failures so far, most of them in the order of 2 minutes between the moment the poweroff command is sent and the moment where virsh actually reports that the SUT is actually shut off...

Actions

Copy link

#24

Updated by okurz about 5 years ago

Target version changed from Milestone 22 to Milestone 23

Actions

Copy link

#25

Updated by SLindoMansilla about 5 years ago

Subject changed from [functional][u][s390x] test fails in shutdown on s390x to [functional][u][s390x][sporadic] test fails in shutdown on s390x

New occurrence on OSD: https://openqa.suse.de/tests/2496334
It seems to be sporadic.

Actions

Copy link

#26

Updated by zluo about 5 years ago

Assignee set to zluo

I filed yesterday ticket #48545, maybe this is related issue, let me check this and give an update on this.

Actions

Copy link

#27

Updated by zluo about 5 years ago

Status changed from Workable to In Progress

Actions

Copy link

#28

Updated by zluo about 5 years ago

http://openqa.suse.de/tests/2508379#next_previous

100 test runs triggered now

Actions

Copy link

#29

Updated by zluo about 5 years ago

WIP PR:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6940

with extend timeout for check_shutdown:

Created job #2508629: sle-15-SP1-Installer-DVD-s390x-Build178.3-create_hdd_gnome@s390x-kvm-sle12 -> https://openqa.suse.de/t2508629

Actions

Copy link

#31

Updated by okurz about 5 years ago

Priority changed from Normal to High

back to "High" as still or again linked to currently failing tests.

Actions

Copy link

#32

Updated by okurz about 5 years ago

Related to action #46076: [sle][functional][u][medium] test fails in shutdown on minimalx added

Actions

Copy link

#33

Updated by zluo about 5 years ago

my WIP PR which got tested on osd https://openqa.suse.de/t2508629 shows no issue and on my local server I tested:

http://f40.suse.de/tests/1526#next_previous shows all successful test runs, 100 total, a couple of tests failed at very beginning bootloader_zkvm which is a different issue.

Actions

Copy link

#34