Project

General

Profile

action #32371

[functional][sporadic][u][medium] snapper_nodbus_setup randomly fails to get into emergency shell

Added by mlin7442 over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Start date:
2018-02-27
Due date:
2018-04-24
% Done:

0%

Estimated time:
Difficulty:
medium

Description

Observation

opensuse-15.0-DVD-x86_64-Build145.2-extra_tests_filesystem@64bit fails in snapper_undochange or snapper_thin_lvm

Acceptance criteria

  • AC1: Test is stable and doesn't fail or bug is created (possibly with soft-failure), or separate ticket is created with investigation results

Tasks

  1. Investigate the failure, try to reproduce manually
  2. Come up with the possible solution based on results (e.g. increasing timeout)
  3. Verify solution by performing many runs locally

Reproducible

Fails since (at least) - Random

Check the history at https://openqa.opensuse.org/tests/620715#previous

The test randomly fails to get into emergency shell after executing systemctl rescue, eg. https://openqa.opensuse.org/tests/620715#step/snapper_undochange/23 and https://openqa.opensuse.org/tests/620298#step/snapper_thin_lvm/56

It might be an product issue, but since it was random happened, I can't say that, just record this issue here.


Related issues

Has duplicate openQA Tests - action #30189: [sle][functional][easy][u][userspace] test fails in snapper_thin_lvm - missing rescue mode messageRejected2018-01-11

History

#1 Updated by okurz over 4 years ago

  • Subject changed from snapper_nodbus_setup randomly fails to get into emergency shell to [functional][opensuse][leap][sporadic]snapper_nodbus_setup randomly fails to get into emergency shell
  • Category set to Bugs in existing tests

#2 Updated by okurz over 4 years ago

  • Subject changed from [functional][opensuse][leap][sporadic]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][opensuse][leap][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell
  • Target version set to Milestone 18

#3 Updated by okurz over 4 years ago

  • Due date set to 2018-04-10
  • Assignee set to michalnowak
  • Priority changed from Normal to High
  • Target version changed from Milestone 18 to Milestone 15

Linked to current test failures in openSUSE Leap 15.0 and annoying because sporadic -> "high".

@michalnowak am I right to assume that you introduced the "snapper_nodbus" code and maintain it? Could you take a look? Feel free to unassign with a comment if I'm mistaken or better: Look for the test code maintainer and assign to that person.

#4 Updated by okurz over 4 years ago

  • Has duplicate action #30189: [sle][functional][easy][u][userspace] test fails in snapper_thin_lvm - missing rescue mode message added

#5 Updated by okurz over 4 years ago

  • Subject changed from [functional][opensuse][leap][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell

It seems #30189 describes the corresponding SLE problem so does not look like "openSUSE Leap" specific.

#6 Updated by michalnowak over 4 years ago

  • Assignee deleted (michalnowak)

Contributed the code though not maintaining it.

It seems to fail on systemctl rescue, the interesting part is behind the splash screen.

#7 Updated by okurz over 4 years ago

  • Assignee set to riafarov

assigning to riafarov as test maintainer for snapper_cleanup.pm then. Rodion, can you take a look please? Feel free to just make workable for next sprint, S14, and unassign again.

#8 Updated by riafarov over 4 years ago

  • Description updated (diff)
  • Status changed from New to Workable
  • Assignee deleted (riafarov)

#9 Updated by okurz over 4 years ago

  • Subject changed from [functional][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][sporadic][u][medium] snapper_nodbus_setup randomly fails to get into emergency shell
  • Description updated (diff)

#10 Updated by cwh over 4 years ago

  • Difficulty set to medium

#11 Updated by oorlov over 4 years ago

  • Assignee set to oorlov

#12 Updated by oorlov over 4 years ago

  • Status changed from Workable to In Progress

#13 Updated by oorlov over 4 years ago

  • Status changed from In Progress to Feedback

I was able to reproduce the issue on local machine
Tests were failed due to low timeout (10 sec) on waiting for emergency screen.

The timeout was increased to 30 sec.
After that, all of almost 40 runs were passed without failures. (http://10.160.64.32/tests/32, ... , http://10.160.64.32/tests/69).

Pull request: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4765

#14 Updated by okurz over 4 years ago

PR merged

#15 Updated by oorlov over 4 years ago

'snapper_undochange' is failed on the last build.

It seems like increasing timeout to 30 sec was not enough. The system shows splash screen a little bit longer.

#16 Updated by michalnowak over 4 years ago

Looking at log-console https://openqa.opensuse.org/tests/652926#step/snapper_undochange/25 which got triggered via post_fail_hook() in btrfs_test.pm, the system actually got to the rescue mode. Maybe it's a good idea to have something like this in the test so we see what's below the plymouth splash on root-console:

sub post_fail_hook {
  my ($self) = @_;
  send_key 'esc';
  wait_screen_change;
  $self->SUPER::post_fail_hook;
}

#17 Updated by oorlov over 4 years ago

  • Status changed from Feedback to In Progress

#18 Updated by mgriessmeier over 4 years ago

  • Due date changed from 2018-04-10 to 2018-04-24

#19 Updated by oorlov over 4 years ago

  • Status changed from In Progress to Feedback

@michalnowak, I guess it is a good idea to have that part of code.
If dramatically increased timeout (up to 120sec) will not resolve the issue, then I'll use your advice.

Right now, after more investigation locally I assume that the issue belongs to machine performance, as locally I was able to reproduce the case, when 30 sec was not enough for rescue mode to be boot by lowering CPU performance and running several workers at the same time.

After increasing to 120 sec, and running 13 builds, the issue is not appeared for me anymore.

Pull request: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4820

#20 Updated by michalnowak over 4 years ago

You can actually have them both: longer timeout & error-handling code when the former does not work.

#21 Updated by oorlov over 4 years ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF