Project

General

Profile

action #41330

[functional][y][s390x][investigation][timebox:4h] test fails in welcome - half-open socket in post_fail_hook causing incomplete job

Added by okurz over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
riafarov
Category:
Bugs in existing tests
Target version:
SUSE QA - Milestone 22
Start date:
2018-09-19
Due date:
2019-01-29
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-15-SP1-Installer-DVD-s390x-btrfs+low_ram@s390x-zVM-vswitch-l3 fails in
welcome

Reproducible

Fails since (at least) Build 19.3 reproducibly

Expected result

Job should not incomplete

Further details

Always latest result in this scenario: latest

Related issues

Related to openQA Tests - action #32746: [sle][tools][remote-backends][hard] Incomplete job because console isn't responding correctly. Half-open socket on IPMIResolved2018-03-05

History

#1 Updated by mloviska over 4 years ago

Same issue with sle-15-SP1-Installer-DVD-s390x-Build58.1-create_hdd_textmode@s390x-kvm-sle12

[2018-10-02T23:15:32.0330 CEST] [debug] <<< backend::console_proxy::__ANON__(wrapped_call={
  'function' => 'stop_serial_grab',
  'console' => 'svirt',
  'args' => []
})
DIE The console isn't responding correctly. Maybe half-open socket? at /usr/lib/os-autoinst/backend/baseclass.pm line 241.

 at /usr/lib/os-autoinst/backend/baseclass.pm line 80.
    backend::baseclass::die_handler('The console isn\'t responding correctly. Maybe half-open sock...') called at /usr/lib/os-autoinst/backend/baseclass.pm line 241
    eval {...} called at /usr/lib/os-autoinst/backend/baseclass.pm line 156
    backend::baseclass::run_capture_loop('backend::svirt=HASH(0x72aad18)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 129
    backend::baseclass::run('backend::svirt=HASH(0x72aad18)', 13, 16) called at /usr/lib/os-autoinst/backend/driver.pm line 77
    backend::driver::__ANON__('Mojo::IOLoop::ReadWriteProcess=HASH(0x74a5d70)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
    eval {...} called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
    Mojo::IOLoop::ReadWriteProcess::_fork('Mojo::IOLoop::ReadWriteProcess=HASH(0x74a5d70)', 'CODE(0x2e2b7f0)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 476
    Mojo::IOLoop::ReadWriteProcess::start('Mojo::IOLoop::ReadWriteProcess=HASH(0x74a5d70)') called at /usr/lib/os-autoinst/backend/driver.pm line 79
    backend::driver::start('backend::driver=HASH(0x71722b8)') called at /usr/lib/os-autoinst/backend/driver.pm line 50
    backend::driver::new('backend::driver', 'svirt') called at /usr/bin/isotovideo line 184
    main::init_backend() called at /usr/bin/isotovideo line 248
[2018-10-02T23:19:30.0705 CEST] [debug] Destroying openQA-SUT-3 virtual machine
[2018-10-02T23:19:30.0861 CEST] [debug] Connection to root@s390p8.suse.de established

#3 Updated by okurz over 4 years ago

  • Due date set to 2018-12-18
  • Target version changed from Milestone 20 to Milestone 21

#4 Updated by cachen over 4 years ago

  • Related to action #32746: [sle][tools][remote-backends][hard] Incomplete job because console isn't responding correctly. Half-open socket on IPMI added

#5 Updated by szarate over 4 years ago

#6 Updated by okurz over 4 years ago

  • Subject changed from [functional][y][s390x][zvm] test fails in welcome - half-open socket in post_fail_hook causing incomplete job to [functional][y][s390x] test fails in welcome - half-open socket in post_fail_hook causing incomplete job

most recent example: sle-15-SP1-Installer-DVD-s390x-Build107.5-textmode@s390x-kvm-sle12, autoinst-log.txt tells:

[2018-12-01T18:02:44.637 CET] [debug] /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/reboot_after_installation.pm:39 called power_action_utils::power_action
[2018-12-01T18:02:44.637 CET] [debug] <<< backend::console_proxy::__ANON__(wrapped_call={
  'args' => [],
  'console' => 'svirt',
  'function' => 'stop_serial_grab'
})
[2018-12-01T18:05:34.561 CET] [debug] Backend process died, backend errors are reported below in the following lines The console isn't responding correctly. Maybe half-open socket? at /usr/lib/os-autoinst/backend/baseclass.pm line 241.

#7 Updated by riafarov over 4 years ago

  • Status changed from New to Workable

#8 Updated by riafarov over 4 years ago

  • Subject changed from [functional][y][s390x] test fails in welcome - half-open socket in post_fail_hook causing incomplete job to [functional][y][s390x][investigation][timebox:4h] test fails in welcome - half-open socket in post_fail_hook causing incomplete job

#9 Updated by riafarov over 4 years ago

  • Due date changed from 2018-12-18 to 2019-01-29
  • Target version changed from Milestone 21 to Milestone 22

No capacity postponing.

#10 Updated by riafarov over 4 years ago

  • Assignee set to riafarov

#11 Updated by riafarov over 4 years ago

  • Status changed from Workable to Resolved

There was a patch provided by mgriessmeier which should resolve such issues in general and in case it reoccurs it should incomplete as is backend issue.
There were only 2 runs where we got incomplete during the postfailhook, which both somehow appeared during attempts of collecting strace and gdb output. I was not able to reproduce the issue manually though.

Also available in: Atom PDF