Project

General

Profile

Actions

action #108452

closed

test incompletes in patch_and_reboot with 'Seems like os-autoinst has produced a result which openQA can not display.', reason auto_review:"backend died: encountered object.*consoles::VNC.*, but allow_blessed.*myjsonrpc":retry

Added by okurz over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2022-03-16
Due date:
% Done:

0%

Estimated time:

Description

Observation

openQA test in scenario sle-15-SP2-JeOS-for-kvm-and-xen-Updates-x86_64-jeos-filesystem@svirt-xen-hvm incompletes in
patch_and_reboot
with 'Seems like os-autoinst has produced a result which openQA can not display.' and reason

"backend died: encountered object.consoles::VNC., but allow_blessed.*myjsonrpc"

Reproducible

Fails since https://openqa.suse.de/tests/8333789

Expected result

Last good: https://openqa.suse.de/tests/8326263

Further details

Always latest result in this scenario: latest

I suspect a regression in os-autoinst. git log between last good and first bad of os-autonist git log1 --no-merges c69906cf..c2dc1343

1479faea Add checks for initialization of `consoles::VNC` accessors from args
7be44a78 Avoid warnings about redefinition of `has`
736006e4 (Martchus/vnc-timeout) Increase default VNC timeout for remote connections
12a23096 Reduce pipe size in test to make coverage less flaky
39669553 myjsonrpc: Don't limit how often select may return EINTR
31b46ec0 (okurz/enhance/simplify9_sigs) Use proper perl "signatures" in common files - distribution
392f8b0e (okurz/feature/snapshot_test) Simplify '$next_test' assignment in autotest
1513ae26 (okurz/feature/refactor_hyperv) backend: Harden hyperv|vmware regex vmm family check
f8ca4046 backend::svirt: Extract methods _is_hyperv/_is_vmware
11016926 backend::svirt: Extract method _vmm_family
fc6c3dc8 Record info in next test module after loading snapshot
28bf8c00 (okurz/feature/autotest_error_feedback) Add error handling for uninitialized isotovideo socket
9ce7a28e (okurz/feature/autotest_simplify) xt: Ensure all perl files using Mojo::Base as best practice
467d996e Use Mojo::Base in autotest as well
fdfad603 t: Ensure to show exception details when python test modules fail
8556e11a (okurz/feature/whitespace) Delete whitespace at end-of-line in all applicable files
28e66842 (okurz/fix/vnc) Fix off-by-one error in color calculation
2ea9b5df (okurz/enhance/simplify9_simplify) t: Simplify 07-commands.t
eba0ac51 Simplify osutils::find_bin with first
8ec50014 (okurz/feature/simplify2_simplify_vnc_console) Simplify consoles::VNC
c1a968a0 (Amrysliu/clean_tpm_files) Cleanup tpm*.permall before creating sock file
a7f03219 (okurz/feature/scripts_test_isotovideo2) Simplify usage check in check_qemu_oom
78a717ed Align isotovideo 'usage' method to others
7a04b230 (cfconrad/pr_fix_validate_script_output) t: Refactor 03-testapi.t
bfd13ccc (okurz/feature/scripts_test) Add proper command line parsing for all scripts
de3573f2 Add proper header to os-autoinst-openvswitch
aabd3ba6 Add dependency for crop.py
7e99d058 Remove obsolete python2 support in crop.py
3f920ef9 Remove obsolete autogen.sh
62f802e0 (Martchus/qemu-oom) Fix condition in script `check_qemu_oom`
4ebf9dbe testapi: Fix validate_script_output() argument parsing
Actions #1

Updated by okurz over 2 years ago

  • Project changed from openQA Tests (public) to openQA Project (public)
  • Category changed from Bugs in existing tests to Regressions/Crashes
Actions #2

Updated by okurz over 2 years ago

  • Due date set to 2022-03-30
  • Status changed from New to In Progress
  • Assignee set to okurz
Actions #3

Updated by okurz over 2 years ago

Triggered reproduction test runs with

for i in {1..10}; do openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/8333789 _GROUP=0 BUILD=okurz_poo108452 TEST=jeos-okurz_poo108452_$i SCHEDULE=tests/installation/bootloader_svirt.pm,tests/installation/bootloader_uefi.pm,tests/jeos/firstrun.pm,tests/jeos/record_machine_id.pm,tests/console/system_prepare.pm,tests/console/force_scheduled_tasks.pm,tests/jeos/grub2_gfxmode.pm,tests/jeos/diskusage.pm,tests/jeos/build_key.pm,tests/console/prjconf_excluded_rpms.pm,tests/console/journal_check.pm,tests/microos/libzypp_config.pm,tests/console/suseconnect_scc.pm,tests/qa_automation/patch_and_reboot.pm WORKER_CLASS=svirt-xen,openqaworker2; done

-> https://openqa.suse.de/tests/overview?distri=sle&version=15-SP2&build=okurz_poo108452

and investigation jobs with

echo https://openqa.suse.de/tests/8333789 | env host=openqa.suse.de openqa-investigate

-> https://openqa.suse.de/tests/8333789#comment-501892

EDIT: Wait, https://openqa.suse.de/tests/8332025#comment-501674 already shows four incompletes stopping in same step, so neither test nor product regression is the likely cause here. This seems to be a clear regression in the infrastructure, e.g. os-autoinst or openQA or dependencies. I downgraded openqaworker2 for checking with zypper -n in --oldpackage /var/cache/zypp/packages/devel_openQA/x86_64/os-autoinst-4.6.1646045612.c69906cf-lp153.1102.1.x86_64.rpm

https://openqa.suse.de/tests/overview?distri=sle&version=15-SP2&build=okurz_poo108452 has three passed, no failed, hypothesis confirm. Now bisecting. Installing intermediate package zypper -n in --oldpackage /var/cache/zypp/packages/devel_openQA/x86_64/os-autoinst-4.6.1646659002.ff0946af-lp153.1111.1.x86_64.rpm . Previously we tried build 1102, most recent is 1133. -> https://openqa.suse.de/tests/overview?distri=sle&build=okurz_poo108452_1111&version=15-SP2

That passed. Meaning that the error must be in git log1 --no-merges ff0946af..origin/master

1479faea Add checks for initialization of `consoles::VNC` accessors from args
7be44a78 Avoid warnings about redefinition of `has`
736006e4 (Martchus/vnc-timeout) Increase default VNC timeout for remote connections
12a23096 Reduce pipe size in test to make coverage less flaky
39669553 myjsonrpc: Don't limit how often select may return EINTR
31b46ec0 (okurz/enhance/simplify9_sigs) Use proper perl "signatures" in common files - distribution
392f8b0e (okurz/feature/snapshot_test) Simplify '$next_test' assignment in autotest
1513ae26 (okurz/feature/refactor_hyperv) backend: Harden hyperv|vmware regex vmm family check
f8ca4046 backend::svirt: Extract methods _is_hyperv/_is_vmware
11016926 backend::svirt: Extract method _vmm_family
fc6c3dc8 Record info in next test module after loading snapshot
28bf8c00 (okurz/feature/autotest_error_feedback) Add error handling for uninitialized isotovideo socket
9ce7a28e (okurz/feature/autotest_simplify) xt: Ensure all perl files using Mojo::Base as best practice
467d996e Use Mojo::Base in autotest as well
fdfad603 t: Ensure to show exception details when python test modules fail
8556e11a (okurz/feature/whitespace) Delete whitespace at end-of-line in all applicable files
2ea9b5df (okurz/enhance/simplify9_simplify) t: Simplify 07-commands.t
eba0ac51 Simplify osutils::find_bin with first

Turns out it was actually 7be44a78. mkittler fixed it already with https://github.com/os-autoinst/os-autoinst/pull/1994

Actions #4

Updated by mkittler over 2 years ago

The underlying problem should be fixed by: https://github.com/os-autoinst/os-autoinst/pull/1994

Actions #5

Updated by okurz over 2 years ago

  • Subject changed from test incompletes in patch_and_reboot with 'Seems like os-autoinst has produced a result which openQA can not display.', reason auto_review:"backend died: encountered object.*consoles::VNC.*, but allow_blessed.*myjsonrpc" to test incompletes in patch_and_reboot with 'Seems like os-autoinst has produced a result which openQA can not display.', reason auto_review:"backend died: encountered object.*consoles::VNC.*, but allow_blessed.*myjsonrpc":retry
Actions #6

Updated by okurz over 2 years ago

  • Due date deleted (2022-03-30)
  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF