Project

General

Profile

action #45047

[functional][u][s390x][zvm] test fails in reconnect_s390 - could not connect to host causing many incompletes

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Bugs in existing tests
Target version:
SUSE QA tests - Milestone 22
Start date:
2018-12-12
Due date:
% Done:

0%

Estimated time:
Difficulty:
Duration:

Description

Observation

openQA test in scenario sle-15-SP1-Installer-DVD-s390x-btrfs@s390x-zVM-ctc fails in
reconnect_s390
trying to reach the host over ssh, see the messages in https://openqa.suse.de/tests/2325162/file/autoinst-log.txt . We already try for multiple minutes multiple times. I will also report a product issue about this as I am pretty sure test code did not change and this happens now reproducibly with the product. However, our tests do not handle this well. incompletes should be avoided.

Reproducible

Fails since https://openqa.suse.de/tests/2312987 for 10 builds in a row now.

Expected result

Tests should not fail as incomplete in the case of not being able to reach the remote

Further details

Always latest result in this scenario: latest


Related issues

Blocks openQA Tests - action #41237: [functional][u][ipmi] test fails in first_boot after system shows text tty login prompt but fails to connect to machine over SSH -> need better post_fail_hook or retry, compare to s390x approachRejected2018-09-19

History

#1 Updated by okurz over 1 year ago

  • Target version set to Milestone 21

#2 Updated by mgriessmeier over 1 year ago

  • Status changed from New to In Progress
  • Assignee set to mgriessmeier

#3 Updated by mgriessmeier over 1 year ago

incompletes because the "Die" happens in the backend since the 'iucvconn' is a serial one which is not known in the tests

#4 Updated by okurz over 1 year ago

We would expect the test to fail and not complete. What is happening:

[2018-12-12T11:09:09.080 CET] [debug] Could not connect to root@s390ctc152.suse.de, Retry
[2018-12-12T11:09:19.080 CET] [debug] Backend process died, backend errors are reported below in the following lines Failed to login to root@s390ctc152.suse.de at /usr/lib/os-autoinst/backend/baseclass.pm line 1195.

So there is a call to "die" but it is in the backend and not the test context, see

https://github.com/os-autoinst/os-autoinst/blob/master/backend/baseclass.pm#L1195

Somehow we need to forward that information to test context

#5 Updated by mgriessmeier over 1 year ago

okurz wrote:

Somehow we need to forward that information to test context

I can not come up with a quick solution...
Will take a look on monday

#6 Updated by okurz over 1 year ago

  • Target version changed from Milestone 21 to Milestone 22

#7 Updated by mgriessmeier over 1 year ago

WIP PR which implements a common baseclass for remote backends to reuse the activate function as well as the error handling for both, VNC and SSH remote consoles:
https://github.com/os-autoinst/os-autoinst/pull/1081/

#8 Updated by mgriessmeier over 1 year ago

  • Status changed from In Progress to Feedback

#9 Updated by okurz over 1 year ago

  • Blocks action #41237: [functional][u][ipmi] test fails in first_boot after system shows text tty login prompt but fails to connect to machine over SSH -> need better post_fail_hook or retry, compare to s390x approach added

#10 Updated by mgriessmeier over 1 year ago

  • Status changed from Feedback to Resolved

merged! deployed...

#11 Updated by okurz over 1 year ago

well, it was merged already 15 days ago. You mean it was "deployed"? Does it work in production?

#13 Updated by okurz over 1 year ago

great! Thx

Also available in: Atom PDF