Project

General

Profile

Actions

action #32371

closed

[functional][sporadic][u][medium] snapper_nodbus_setup randomly fails to get into emergency shell

Added by mlin7442 almost 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Start date:
2018-02-27
Due date:
2018-04-24
% Done:

0%

Estimated time:
Difficulty:
medium

Description

Observation

opensuse-15.0-DVD-x86_64-Build145.2-extra_tests_filesystem@64bit fails in snapper_undochange or snapper_thin_lvm

Acceptance criteria

  • AC1: Test is stable and doesn't fail or bug is created (possibly with soft-failure), or separate ticket is created with investigation results

Tasks

  1. Investigate the failure, try to reproduce manually
  2. Come up with the possible solution based on results (e.g. increasing timeout)
  3. Verify solution by performing many runs locally

Reproducible

Fails since (at least) - Random

Check the history at https://openqa.opensuse.org/tests/620715#previous

The test randomly fails to get into emergency shell after executing systemctl rescue, eg. https://openqa.opensuse.org/tests/620715#step/snapper_undochange/23 and https://openqa.opensuse.org/tests/620298#step/snapper_thin_lvm/56

It might be an product issue, but since it was random happened, I can't say that, just record this issue here.


Related issues 1 (0 open1 closed)

Has duplicate openQA Tests - action #30189: [sle][functional][easy][u][userspace] test fails in snapper_thin_lvm - missing rescue mode messageRejectedokurz2018-01-11

Actions
Actions #1

Updated by okurz almost 6 years ago

  • Subject changed from snapper_nodbus_setup randomly fails to get into emergency shell to [functional][opensuse][leap][sporadic]snapper_nodbus_setup randomly fails to get into emergency shell
  • Category set to Bugs in existing tests
Actions #2

Updated by okurz almost 6 years ago

  • Subject changed from [functional][opensuse][leap][sporadic]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][opensuse][leap][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell
  • Target version set to Milestone 18
Actions #3

Updated by okurz almost 6 years ago

  • Due date set to 2018-04-10
  • Assignee set to michalnowak
  • Priority changed from Normal to High
  • Target version changed from Milestone 18 to Milestone 15

Linked to current test failures in openSUSE Leap 15.0 and annoying because sporadic -> "high".

@michalnowak am I right to assume that you introduced the "snapper_nodbus" code and maintain it? Could you take a look? Feel free to unassign with a comment if I'm mistaken or better: Look for the test code maintainer and assign to that person.

Actions #4

Updated by okurz almost 6 years ago

  • Has duplicate action #30189: [sle][functional][easy][u][userspace] test fails in snapper_thin_lvm - missing rescue mode message added
Actions #5

Updated by okurz almost 6 years ago

  • Subject changed from [functional][opensuse][leap][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell

It seems #30189 describes the corresponding SLE problem so does not look like "openSUSE Leap" specific.

Actions #6

Updated by michalnowak almost 6 years ago

  • Assignee deleted (michalnowak)

Contributed the code though not maintaining it.

It seems to fail on systemctl rescue, the interesting part is behind the splash screen.

Actions #7

Updated by okurz almost 6 years ago

  • Assignee set to riafarov

assigning to @riafarov as test maintainer for snapper_cleanup.pm then. Rodion, can you take a look please? Feel free to just make workable for next sprint, S14, and unassign again.

Actions #8

Updated by riafarov almost 6 years ago

  • Description updated (diff)
  • Status changed from New to Workable
  • Assignee deleted (riafarov)
Actions #9

Updated by okurz almost 6 years ago

  • Subject changed from [functional][sporadic][u]snapper_nodbus_setup randomly fails to get into emergency shell to [functional][sporadic][u][medium] snapper_nodbus_setup randomly fails to get into emergency shell
  • Description updated (diff)
Actions #10

Updated by cwh almost 6 years ago

  • Difficulty set to medium
Actions #11

Updated by oorlov almost 6 years ago

  • Assignee set to oorlov
Actions #12

Updated by oorlov almost 6 years ago

  • Status changed from Workable to In Progress
Actions #13

Updated by oorlov almost 6 years ago

  • Status changed from In Progress to Feedback

I was able to reproduce the issue on local machine
Tests were failed due to low timeout (10 sec) on waiting for emergency screen.

The timeout was increased to 30 sec.
After that, all of almost 40 runs were passed without failures. (http://10.160.64.32/tests/32, ... , http://10.160.64.32/tests/69).

Pull request: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4765

Actions #14

Updated by okurz almost 6 years ago

PR merged

Actions #15

Updated by oorlov almost 6 years ago

'snapper_undochange' is failed on the last build.

It seems like increasing timeout to 30 sec was not enough. The system shows splash screen a little bit longer.

Actions #16

Updated by michalnowak almost 6 years ago

Looking at log-console https://openqa.opensuse.org/tests/652926#step/snapper_undochange/25 which got triggered via post_fail_hook() in btrfs_test.pm, the system actually got to the rescue mode. Maybe it's a good idea to have something like this in the test so we see what's below the plymouth splash on root-console:

sub post_fail_hook {
  my ($self) = @_;
  send_key 'esc';
  wait_screen_change;
  $self->SUPER::post_fail_hook;
}
Actions #17

Updated by oorlov almost 6 years ago

  • Status changed from Feedback to In Progress
Actions #18

Updated by mgriessmeier almost 6 years ago

  • Due date changed from 2018-04-10 to 2018-04-24
Actions #19

Updated by oorlov almost 6 years ago

  • Status changed from In Progress to Feedback

@michalnowak, I guess it is a good idea to have that part of code.
If dramatically increased timeout (up to 120sec) will not resolve the issue, then I'll use your advice.

Right now, after more investigation locally I assume that the issue belongs to machine performance, as locally I was able to reproduce the case, when 30 sec was not enough for rescue mode to be boot by lowering CPU performance and running several workers at the same time.

After increasing to 120 sec, and running 13 builds, the issue is not appeared for me anymore.

Pull request: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4820

Actions #20

Updated by michalnowak almost 6 years ago

You can actually have them both: longer timeout & error-handling code when the former does not work.

Actions #21

Updated by oorlov almost 6 years ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF