Project

General

Profile

action #106654

[ipmi][openqa][vnc] Massive test run failures with 'IO::Socket::INET: connect: Connection refused' due to "Use of uninitialized value.*connect_timeout in addition.*consoles/VNC.pm line 13.*":retry

Added by waynechen55 5 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2022-02-11
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test run failures are arising more and more due to the same error:

[2022-02-11T01:34:05.510069+01:00] [info] ::: basetest::runtest: # Test died: Error connecting to VNC server localhost:38869: IO::Socket::INET: connect: Connection refused at /usr/lib/os-autoinst/testapi.pm line 1762.

But all manual ssh attempts succeeded.

Please refer to all relevant test runs as below:
https://openqa.suse.de/tests/8140400
https://openqa.suse.de/tests/8140119
https://openqa.suse.de/tests/8139589
https://openqa.suse.de/tests/8137426
https://openqa.suse.de/tests/8137424

Steps to reproduce

Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
call openqa-query-for-job-label poo#106654


Related issues

Related to openQA Project - action #105882: Test using svirt backend fails with auto_review:"Error connecting to VNC server.*localhost.*Connection refused"Resolved2020-10-30

Related to openQA Project - action #99663: Use more perl signatures - os-autoinst size:MWorkable2021-10-01

Copied to openQA Tests - action #106681: [virtualization] warnings with stacktraces in test logs about "Use of uninitialized value in pattern match"New

History

#1 Updated by okurz 5 months ago

  • Project changed from openQA Infrastructure to openQA Project
  • Category set to Concrete Bugs
  • Status changed from New to Feedback
  • Assignee set to okurz
  • Target version set to Ready

please follow the ticket template from https://progress.opensuse.org/projects/openqav3/wiki/#Defects . I saw that the problems only appear with the new build. Did you crosscheck with an older build? Did you crosscheck with an older state of tests?

I triggered investigation runs now with echo https://openqa.suse.de/tests/8140400 | host=openqa.suse.de openqa-investigate, comment available in https://openqa.suse.de/tests/8140400#comment-485767 . Awaiting test results.

#2 Updated by okurz 5 months ago

  • Related to action #105882: Test using svirt backend fails with auto_review:"Error connecting to VNC server.*localhost.*Connection refused" added

#3 Updated by okurz 5 months ago

  • Status changed from Feedback to In Progress

so inconclusive results. I triggered build=okurz_investigation_poo106654; end=040 openqa-clone-set https://openqa.suse.de/tests/8140935 ${build}_uefi-gi-guest_sles12sp5-on-host_developing-xen:investigate:last_good_tests_and_build:78a8b9064552b95543ab89a27a5678bb7987bbc8+91.2 SCHEDULE=tests/boot/boot_from_pxe

so awaiting results from https://openqa.suse.de/tests/overview?build=okurz_investigation_poo106654_uefi-gi-guest_sles12sp5-on-host_developing-xen%3Ainvestigate%3Alast_good_tests_and_build%3A78a8b9064552b95543ab89a27a5678bb7987bbc8%2B91.2&distri=sle&version=15-SP4

#4 Updated by mkittler 5 months ago

This PR should fix the problem, I'm currently testing it on openqaworker2: https://github.com/os-autoinst/os-autoinst/pull/1953

The test job I've created is still scheduled: https://openqa.suse.de/tests/8141551

#5 Updated by okurz 5 months ago

  • Related to action #99663: Use more perl signatures - os-autoinst size:M added

#6 Updated by openqa_review 4 months ago

  • Due date set to 2022-02-26

Setting due date based on mean cycle time of SUSE QE Tools

#7 Updated by okurz 4 months ago

  • Copied to action #106681: [virtualization] warnings with stacktraces in test logs about "Use of uninitialized value in pattern match" added

#8 Updated by okurz 4 months ago

  • Subject changed from [ipmi][openqa][vnc] Massive test run failures due to "IO::Socket::INET: connect: Connection refused" to [ipmi][openqa][vnc] Massive test run failures with 'IO::Socket::INET: connect: Connection refused' due to auto_review:"Use of uninitialized value.*connect_timeout in addition.*consoles/VNC.pm line 13.*":retry
  • Description updated (diff)

#9 Updated by waynechen55 4 months ago

okurz wrote:

please follow the ticket template from https://progress.opensuse.org/projects/openqav3/wiki/#Defects . I saw that the problems only appear with the new build. Did you crosscheck with an older build? Did you crosscheck with an older state of tests?

I triggered investigation runs now with echo https://openqa.suse.de/tests/8140400 | host=openqa.suse.de openqa-investigate, comment available in https://openqa.suse.de/tests/8140400#comment-485767 . Awaiting test results.

Templates will be followed.

okurz wrote:

The fix was deployed with https://gitlab.suse.de/openqa/osd-deployment/-/pipelines/318539 . https://openqa.suse.de/tests/overview?build=okurz_investigation_poo106654_uefi-gi-guest_sles12sp5-on-host_developing-xen%3Ainvestigate%3Alast_good_tests_and_build%3A78a8b9064552b95543ab89a27a5678bb7987bbc8%2B91.2&distri=sle&version=15-SP4 shows 39/40 passed for tests that either just passed in before or with the fix applied. One job failed which was running before deployment. I retriggered this job as https://openqa.suse.de/tests/8152116 , currently running. Adding https://github.com/os-autoinst/scripts/blob/master/README.md#auto-review---automatically-detect-known-issues-in-openqa-jobs-label-openqa-jobs-with-ticket-references-and-optionally-retrigger control parameters.

Thanks. I will follow up on the results.

#10 Updated by waynechen55 4 months ago

mkittler wrote:

This PR should fix the problem, I'm currently testing it on openqaworker2: https://github.com/os-autoinst/os-autoinst/pull/1953

The test job I've created is still scheduled: https://openqa.suse.de/tests/8141551

Thanks for your info.

#11 Updated by waynechen55 4 months ago

Looks like the issue is resolved.

#12 Updated by waynechen55 4 months ago

  • Status changed from In Progress to Resolved

#13 Updated by openqa_review 4 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: jeos@RPi2B
https://openqa.opensuse.org/tests/2191462

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

#14 Updated by okurz 4 months ago

  • Due date deleted (2022-02-26)
  • Status changed from Resolved to Feedback

The above job is 21 days old but still the latest job in this scenario. I retriggered it in the hope to end up with a successful cloned job. Currently running https://openqa.opensuse.org/tests/2236473

#15 Updated by okurz 3 months ago

  • Subject changed from [ipmi][openqa][vnc] Massive test run failures with 'IO::Socket::INET: connect: Connection refused' due to auto_review:"Use of uninitialized value.*connect_timeout in addition.*consoles/VNC.pm line 13.*":retry to [ipmi][openqa][vnc] Massive test run failures with 'IO::Socket::INET: connect: Connection refused' due to "Use of uninitialized value.*connect_timeout in addition.*consoles/VNC.pm line 13.*":retry
  • Status changed from Feedback to Resolved

https://openqa.opensuse.org/tests/2282397 looks ok, at least not failing as in before.

Also available in: Atom PDF