action #32371
closed
[functional][sporadic][u][medium] snapper_nodbus_setup randomly fails to get into emergency shell
Added by mlin7442 almost 7 years ago.
Updated over 6 years ago.
Category:
Bugs in existing tests
Description
Observation¶
opensuse-15.0-DVD-x86_64-Build145.2-extra_tests_filesystem@64bit fails in snapper_undochange or snapper_thin_lvm
Acceptance criteria¶
- AC1: Test is stable and doesn't fail or bug is created (possibly with soft-failure), or separate ticket is created with investigation results
Tasks¶
- Investigate the failure, try to reproduce manually
- Come up with the possible solution based on results (e.g. increasing timeout)
- Verify solution by performing many runs locally
Reproducible¶
Fails since (at least) - Random
Check the history at https://openqa.opensuse.org/tests/620715#previous
The test randomly fails to get into emergency shell after executing systemctl rescue, eg. https://openqa.opensuse.org/tests/620715#step/snapper_undochange/23 and https://openqa.opensuse.org/tests/620298#step/snapper_thin_lvm/56
It might be an product issue, but since it was random happened, I can't say that, just record this issue here.
- Subject changed from snapper_nodbus_setup randomly fails to get into emergency shell to [functional][opensuse][leap][sporadic]snapper_nodbus_setup randomly fails to get into emergency shell
- Category set to Bugs in existing tests
- Subject changed from [functional][opensuse][leap][sporadic]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][opensuse][leap][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell
- Target version set to Milestone 18
- Due date set to 2018-04-10
- Assignee set to michalnowak
- Priority changed from Normal to High
- Target version changed from Milestone 18 to Milestone 15
Linked to current test failures in openSUSE Leap 15.0 and annoying because sporadic -> "high".
@michalnowak am I right to assume that you introduced the "snapper_nodbus" code and maintain it? Could you take a look? Feel free to unassign with a comment if I'm mistaken or better: Look for the test code maintainer and assign to that person.
- Has duplicate action #30189: [sle][functional][easy][u][userspace] test fails in snapper_thin_lvm - missing rescue mode message added
- Subject changed from [functional][opensuse][leap][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell
It seems #30189 describes the corresponding SLE problem so does not look like "openSUSE Leap" specific.
- Assignee deleted (
michalnowak)
Contributed the code though not maintaining it.
It seems to fail on systemctl rescue, the interesting part is behind the splash screen.
assigning to @riafarov as test maintainer for snapper_cleanup.pm then. Rodion, can you take a look please? Feel free to just make workable for next sprint, S14, and unassign again.
- Description updated (diff)
- Status changed from New to Workable
- Assignee deleted (
riafarov)
- Subject changed from [functional][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][sporadic][u][medium] snapper_nodbus_setup randomly fails to get into emergency shell
- Description updated (diff)
- Status changed from Workable to In Progress
- Status changed from In Progress to Feedback
'snapper_undochange' is failed on the last build.
It seems like increasing timeout to 30 sec was not enough. The system shows splash screen a little bit longer.
Looking at log-console https://openqa.opensuse.org/tests/652926#step/snapper_undochange/25 which got triggered via post_fail_hook() in btrfs_test.pm, the system actually got to the rescue mode. Maybe it's a good idea to have something like this in the test so we see what's below the plymouth splash on root-console:
sub post_fail_hook {
my ($self) = @_;
send_key 'esc';
wait_screen_change;
$self->SUPER::post_fail_hook;
}
- Status changed from Feedback to In Progress
- Due date changed from 2018-04-10 to 2018-04-24
- Status changed from In Progress to Feedback
@michalnowak, I guess it is a good idea to have that part of code.
If dramatically increased timeout (up to 120sec) will not resolve the issue, then I'll use your advice.
Right now, after more investigation locally I assume that the issue belongs to machine performance, as locally I was able to reproduce the case, when 30 sec was not enough for rescue mode to be boot by lowering CPU performance and running several workers at the same time.
After increasing to 120 sec, and running 13 builds, the issue is not appeared for me anymore.
Pull request: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4820
You can actually have them both: longer timeout & error-handling code when the former does not work.
- Status changed from Feedback to Resolved
Also available in: Atom
PDF