action #32371
closed[functional][sporadic][u][medium] snapper_nodbus_setup randomly fails to get into emergency shell
0%
Description
Observation¶
opensuse-15.0-DVD-x86_64-Build145.2-extra_tests_filesystem@64bit fails in snapper_undochange or snapper_thin_lvm
Acceptance criteria¶
- AC1: Test is stable and doesn't fail or bug is created (possibly with soft-failure), or separate ticket is created with investigation results
Tasks¶
- Investigate the failure, try to reproduce manually
- Come up with the possible solution based on results (e.g. increasing timeout)
- Verify solution by performing many runs locally
Reproducible¶
Fails since (at least) - Random
Check the history at https://openqa.opensuse.org/tests/620715#previous
The test randomly fails to get into emergency shell after executing systemctl rescue, eg. https://openqa.opensuse.org/tests/620715#step/snapper_undochange/23 and https://openqa.opensuse.org/tests/620298#step/snapper_thin_lvm/56
It might be an product issue, but since it was random happened, I can't say that, just record this issue here.
Updated by okurz almost 7 years ago
- Subject changed from snapper_nodbus_setup randomly fails to get into emergency shell to [functional][opensuse][leap][sporadic]snapper_nodbus_setup randomly fails to get into emergency shell
- Category set to Bugs in existing tests
Updated by okurz almost 7 years ago
- Subject changed from [functional][opensuse][leap][sporadic]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][opensuse][leap][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell
- Target version set to Milestone 18
Updated by okurz almost 7 years ago
- Due date set to 2018-04-10
- Assignee set to michalnowak
- Priority changed from Normal to High
- Target version changed from Milestone 18 to Milestone 15
Linked to current test failures in openSUSE Leap 15.0 and annoying because sporadic -> "high".
@michalnowak am I right to assume that you introduced the "snapper_nodbus" code and maintain it? Could you take a look? Feel free to unassign with a comment if I'm mistaken or better: Look for the test code maintainer and assign to that person.
Updated by okurz almost 7 years ago
- Has duplicate action #30189: [sle][functional][easy][u][userspace] test fails in snapper_thin_lvm - missing rescue mode message added
Updated by okurz almost 7 years ago
- Subject changed from [functional][opensuse][leap][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell
It seems #30189 describes the corresponding SLE problem so does not look like "openSUSE Leap" specific.
Updated by michalnowak almost 7 years ago
- Assignee deleted (
michalnowak)
Contributed the code though not maintaining it.
It seems to fail on systemctl rescue, the interesting part is behind the splash screen.
Updated by okurz almost 7 years ago
- Assignee set to riafarov
assigning to @riafarov as test maintainer for snapper_cleanup.pm then. Rodion, can you take a look please? Feel free to just make workable for next sprint, S14, and unassign again.
Updated by riafarov almost 7 years ago
- Description updated (diff)
- Status changed from New to Workable
- Assignee deleted (
riafarov)
Updated by okurz over 6 years ago
- Subject changed from [functional][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][sporadic][u][medium] snapper_nodbus_setup randomly fails to get into emergency shell
- Description updated (diff)
Updated by oorlov over 6 years ago
- Status changed from In Progress to Feedback
I was able to reproduce the issue on local machine
Tests were failed due to low timeout (10 sec) on waiting for emergency screen.
The timeout was increased to 30 sec.
After that, all of almost 40 runs were passed without failures. (http://10.160.64.32/tests/32, ... , http://10.160.64.32/tests/69).
Pull request: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4765
Updated by oorlov over 6 years ago
'snapper_undochange' is failed on the last build.
It seems like increasing timeout to 30 sec was not enough. The system shows splash screen a little bit longer.
Updated by michalnowak over 6 years ago
Looking at log-console https://openqa.opensuse.org/tests/652926#step/snapper_undochange/25 which got triggered via post_fail_hook() in btrfs_test.pm, the system actually got to the rescue mode. Maybe it's a good idea to have something like this in the test so we see what's below the plymouth splash on root-console:
sub post_fail_hook {
my ($self) = @_;
send_key 'esc';
wait_screen_change;
$self->SUPER::post_fail_hook;
}
Updated by mgriessmeier over 6 years ago
- Due date changed from 2018-04-10 to 2018-04-24
Updated by oorlov over 6 years ago
- Status changed from In Progress to Feedback
@michalnowak, I guess it is a good idea to have that part of code.
If dramatically increased timeout (up to 120sec) will not resolve the issue, then I'll use your advice.
Right now, after more investigation locally I assume that the issue belongs to machine performance, as locally I was able to reproduce the case, when 30 sec was not enough for rescue mode to be boot by lowering CPU performance and running several workers at the same time.
After increasing to 120 sec, and running 13 builds, the issue is not appeared for me anymore.
Pull request: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4820
Updated by michalnowak over 6 years ago
You can actually have them both: longer timeout & error-handling code when the former does not work.
Updated by oorlov over 6 years ago
- Status changed from Feedback to Resolved
snapper_undochange or snapper_thin_lvm are passed in the last 4 builds.
https://openqa.opensuse.org/tests/653771
https://openqa.opensuse.org/tests/654361
https://openqa.opensuse.org/tests/654820
https://openqa.opensuse.org/tests/655780