Project

General

Profile

Actions

action #71227

open

coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA

coordination #102909: [epic] Prevent more incompletes already within os-autoinst or openQA

[ux] job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."

Added by Xiaojing_liu about 4 years ago. Updated over 2 years ago.

Status:
Workable
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
QA (public, currently private due to #173521) - future
Start date:
2020-09-11
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://openqa.suse.de/tests/4673747 in incomplete, reason is: "Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."
https://openqa.suse.de/tests/4673747/file/autoinst-log.txt shows:

[2020-09-11T02:59:50.763 CEST] [debug] Backend process died, backend errors are reported below in the following lines:
'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.
    backend::baseclass::wait_serial(backend::svirt=HASH(0x55a087239aa8), HASH(0x55a08804a420)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 89
    backend::baseclass::handle_command(backend::svirt=HASH(0x55a087239aa8), HASH(0x55a088c2c420)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 625
    backend::baseclass::check_socket(backend::svirt=HASH(0x55a087239aa8), IO::Handle=GLOB(0x55a087bb56f8), 0) called at /usr/lib/os-autoinst/backend/svirt.pm line 337
    backend::svirt::check_socket(backend::svirt=HASH(0x55a087239aa8), IO::Handle=GLOB(0x55a087bb56f8), 0) called at /usr/lib/os-autoinst/backend/baseclass.pm line 276
    eval {...} called at /usr/lib/os-autoinst/backend/baseclass.pm line 191
    backend::baseclass::run_capture_loop(backend::svirt=HASH(0x55a087239aa8)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 146
    backend::baseclass::run(backend::svirt=HASH(0x55a087239aa8), 13, 16) called at /usr/lib/os-autoinst/backend/driver.pm line 86
    backend::driver::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x55a087733d48)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
    eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
    Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x55a087733d48), CODE(0x55a088ef7228)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 477
    Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x55a087733d48)) called at /usr/lib/os-autoinst/backend/driver.pm line 88
    backend::driver::start(backend::driver=HASH(0x55a0884b1268)) called at /usr/lib/os-autoinst/backend/driver.pm line 52
    backend::driver::new("backend::driver", "svirt") called at /usr/bin/isotovideo line 222
    main::init_backend() called at /usr/bin/isotovideo line 273

Checking the git log of test distri between "first bad" and "last good" (when tests were green again) shows:

$ git log1 --no-merges de9fd2b5dcf5d9bc31807e29cd720ac80a1c7f9f..bb6dc4672fc0177c4fb20d7dfd08ddb824e0d0a4
f9f7351ab Remove /tmp as subvolume in profile validation
ee4098d8f Improve pre- and post-feature test
74cd182e8 Rename sap.slice to SAP.slice
f2d6865d7 Wait snapper to settle down to assert root-console

with some changes to "lib/virt_feature_test_base.pm" which might be related.

Problem

This can happen if the console is not properly initialized by test code.
From the error message it is unclear if test code is the culprit or some temporary, flaky error in the infrastructure. This could be cleaner.

Acceptance criteria

  • AC1: It is obvious to the test reviewer if this is a (temporary) infrastructure issue or a problem in test code that needs to be fixed

Suggestions

  • Crosscheck how "current_console" is used within os-autoinst
  • Improve error handling with explicit error messages on misuse
  • Optional: Apply something similar for "reset_consoles". For this also see #60437 and https://github.com/os-autoinst/os-autoinst/pull/1288

Related issues 1 (0 open1 closed)

Related to openQA Tests (public) - action #71230: [SLE] Test 'default_install_svirt' is incomplete in OSD with error: 'Backend process died'ResolvedXiaojing_liu2020-09-11

Actions
Actions #1

Updated by Xiaojing_liu about 4 years ago

  • Subject changed from job incompletes with auto_review: "Backend process died: ''current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'" to job incompletes with auto_review: "Backend process died: ''current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'"
Actions #2

Updated by Xiaojing_liu about 4 years ago

  • Subject changed from job incompletes with auto_review: "Backend process died: ''current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'" to job incompletes with auto_review: "Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'"
Actions #3

Updated by Xiaojing_liu about 4 years ago

  • Subject changed from job incompletes with auto_review: "Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'" to job incompletes with auto_review:"Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."
Actions #4

Updated by Xiaojing_liu about 4 years ago

  • Subject changed from job incompletes with auto_review:"Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932." to job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."
Actions #6

Updated by okurz about 4 years ago

  • Description updated (diff)
  • Category set to Feature requests
  • Status changed from New to Workable
  • Target version set to Ready

latest job in this scenario seems fine again: Following https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=svirt-vmware65&test=default_install_svirt&version=15-SP3 yields passed result https://openqa.suse.de/tests/4675996 . Unclear what fixed this but I assume it was someone from SUSE QA SLE Virtualization. Still we can improve UX here.

Actions #7

Updated by Xiaojing_liu about 4 years ago

  • Related to action #71230: [SLE] Test 'default_install_svirt' is incomplete in OSD with error: 'Backend process died' added
Actions #8

Updated by Xiaojing_liu about 4 years ago

okurz wrote:

latest job in this scenario seems fine again: Following https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=svirt-vmware65&test=default_install_svirt&version=15-SP3 yields passed result https://openqa.suse.de/tests/4675996 . Unclear what fixed this but I assume it was someone from SUSE QA SLE Virtualization. Still we can improve UX here.

I added the related ticket. The job was incomplete because tester did not specify ISO=xxx when triggering the job.

Actions #9

Updated by okurz about 4 years ago

  • Target version changed from Ready to future
Actions #10

Updated by okurz about 4 years ago

  • Parent task set to #62420
Actions #11

Updated by okurz about 3 years ago

  • Parent task changed from #62420 to #102909
Actions #12

Updated by okurz over 2 years ago

  • Subject changed from job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932." to [ux] job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."
Actions

Also available in: Atom PDF