action #71227
opencoordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA
coordination #102909: [epic] Prevent more incompletes already within os-autoinst or openQA
[ux] job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."
0%
Description
Observation¶
https://openqa.suse.de/tests/4673747 in incomplete, reason is: "Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."
https://openqa.suse.de/tests/4673747/file/autoinst-log.txt shows:
[37m[2020-09-11T02:59:50.763 CEST] [debug] Backend process died, backend errors are reported below in the following lines:
'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.
backend::baseclass::wait_serial(backend::svirt=HASH(0x55a087239aa8), HASH(0x55a08804a420)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 89
backend::baseclass::handle_command(backend::svirt=HASH(0x55a087239aa8), HASH(0x55a088c2c420)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 625
backend::baseclass::check_socket(backend::svirt=HASH(0x55a087239aa8), IO::Handle=GLOB(0x55a087bb56f8), 0) called at /usr/lib/os-autoinst/backend/svirt.pm line 337
backend::svirt::check_socket(backend::svirt=HASH(0x55a087239aa8), IO::Handle=GLOB(0x55a087bb56f8), 0) called at /usr/lib/os-autoinst/backend/baseclass.pm line 276
eval {...} called at /usr/lib/os-autoinst/backend/baseclass.pm line 191
backend::baseclass::run_capture_loop(backend::svirt=HASH(0x55a087239aa8)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 146
backend::baseclass::run(backend::svirt=HASH(0x55a087239aa8), 13, 16) called at /usr/lib/os-autoinst/backend/driver.pm line 86
backend::driver::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x55a087733d48)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x55a087733d48), CODE(0x55a088ef7228)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 477
Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x55a087733d48)) called at /usr/lib/os-autoinst/backend/driver.pm line 88
backend::driver::start(backend::driver=HASH(0x55a0884b1268)) called at /usr/lib/os-autoinst/backend/driver.pm line 52
backend::driver::new("backend::driver", "svirt") called at /usr/bin/isotovideo line 222
main::init_backend() called at /usr/bin/isotovideo line 273
Checking the git log of test distri between "first bad" and "last good" (when tests were green again) shows:
$ git log1 --no-merges de9fd2b5dcf5d9bc31807e29cd720ac80a1c7f9f..bb6dc4672fc0177c4fb20d7dfd08ddb824e0d0a4
f9f7351ab Remove /tmp as subvolume in profile validation
ee4098d8f Improve pre- and post-feature test
74cd182e8 Rename sap.slice to SAP.slice
f2d6865d7 Wait snapper to settle down to assert root-console
with some changes to "lib/virt_feature_test_base.pm" which might be related.
Problem¶
This can happen if the console is not properly initialized by test code.
From the error message it is unclear if test code is the culprit or some temporary, flaky error in the infrastructure. This could be cleaner.
Acceptance criteria¶
- AC1: It is obvious to the test reviewer if this is a (temporary) infrastructure issue or a problem in test code that needs to be fixed
Suggestions¶
- Crosscheck how "current_console" is used within os-autoinst
- Improve error handling with explicit error messages on misuse
- Optional: Apply something similar for "reset_consoles". For this also see #60437 and https://github.com/os-autoinst/os-autoinst/pull/1288
Updated by Xiaojing_liu about 4 years ago
- Subject changed from job incompletes with auto_review: "Backend process died: ''current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'" to job incompletes with auto_review: "Backend process died: ''current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'"
Updated by Xiaojing_liu about 4 years ago
- Subject changed from job incompletes with auto_review: "Backend process died: ''current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'" to job incompletes with auto_review: "Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'"
Updated by Xiaojing_liu about 4 years ago
- Subject changed from job incompletes with auto_review: "Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932.'" to job incompletes with auto_review:"Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."
Updated by Xiaojing_liu about 4 years ago
- Subject changed from job incompletes with auto_review:"Backend process died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932." to job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."
Updated by okurz about 4 years ago
- Description updated (diff)
- Category set to Feature requests
- Status changed from New to Workable
- Target version set to Ready
latest job in this scenario seems fine again: Following https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=svirt-vmware65&test=default_install_svirt&version=15-SP3 yields passed result https://openqa.suse.de/tests/4675996 . Unclear what fixed this but I assume it was someone from SUSE QA SLE Virtualization. Still we can improve UX here.
Updated by Xiaojing_liu about 4 years ago
- Related to action #71230: [SLE] Test 'default_install_svirt' is incomplete in OSD with error: 'Backend process died' added
Updated by Xiaojing_liu about 4 years ago
okurz wrote:
latest job in this scenario seems fine again: Following https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=svirt-vmware65&test=default_install_svirt&version=15-SP3 yields passed result https://openqa.suse.de/tests/4675996 . Unclear what fixed this but I assume it was someone from SUSE QA SLE Virtualization. Still we can improve UX here.
I added the related ticket. The job was incomplete because tester did not specify ISO=xxx
when triggering the job.
Updated by okurz over 2 years ago
- Subject changed from job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932." to [ux] job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."