Project

General

Profile

action #105506

[sporadic][tools] openQA-in-openQA test sporadically fails in shutdown

Added by okurz 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
Start date:
2022-01-26
Due date:
2022-02-18
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario openqa-Tumbleweed-dev-x86_64-openqa_install+publish@64bit-2G fails in
shutdown. Stepping through the video frame by frame it looks like the test switches to the root console but never types poweroff. The line

wait_screen_change { send_key 'ctrl-alt-f3' }

might be the culprite because we don't explicitly wait for the root console to appear but just assume that after the screen content changed significantly enough that we have reached the root console. However the root console might not yet be responsive at this time.

Test suite description

Maintainer: okurz@suse.de Test for installation of openQA itself. To be used with "openqa" distri. Publishes an qcow2 image including the openQA installation ready to run as an appliance.

Reproducible

Fails since (at least) Build :TW.10346 (current job). https://openqa.opensuse.org/tests/2157847#next_previous shows 19/407 failures so a fail-ratio of roughly 5%.

Expected result

Last good: :TW.10345 (or more recent)

Suggestions

  • Reproduce the error locally and confirm the fail-ratio
  • Either properly synchronize the switch to the responsive root terminal with an explicit assert_screen or use backend power functions over the test API.

Further details

Always latest result in this scenario: latest


Related issues

Related to openQA Project - action #105429: openQA's fullstack test fails in `shutdown` moduleResolved2022-01-25

History

#1 Updated by okurz 4 months ago

  • Related to action #105429: openQA's fullstack test fails in `shutdown` module added

#2 Updated by osukup 4 months ago

why this test didnt't use power_action ?

#3 Updated by osukup 4 months ago

  • Assignee set to osukup

#4 Updated by osukup 4 months ago

okurz probadly is good idea to check openQA-in-openQA code to modermize/reuse code from os-autoins-distri-opensuse and reuse some things from there ( handling consoles/ serial console).

#5 Updated by osukup 4 months ago

  • Status changed from New to In Progress

#6 Updated by okurz 4 months ago

osukup wrote:

okurz probadly is good idea to check openQA-in-openQA code to modermize/reuse code from os-autoins-distri-opensuse and reuse some things from there ( handling consoles/ serial console).

yes. I would like that. But at best without copy-pasting big blocks of code. This can be yet another good opportunity to either move some code upstream to os-autoinst or to introduce an intermediate helper library or plugin for test distributions.

#7 Updated by openqa_review 4 months ago

  • Due date set to 2022-02-11

Setting due date based on mean cycle time of SUSE QE Tools

#8 Updated by osukup 4 months ago

okurz wrote:

osukup wrote:

okurz probadly is good idea to check openQA-in-openQA code to modermize/reuse code from os-autoins-distri-opensuse and reuse some things from there ( handling consoles/ serial console).

yes. I would like that. But at best without copy-pasting big blocks of code. This can be yet another good opportunity to either move some code upstream to os-autoinst or to introduce an intermediate helper library or plugin for test distributions.

I think best solution will be port openqa-in-openqa to standard 'os-autoinst-distri-opensuse' as new product like microos/sle/opensuse

btw ... about failing shutdown. Do we need for anything test which only select different console and then type poweroff ?

#9 Updated by cdywan 4 months ago

osukup wrote:

btw ... about failing shutdown. Do we need for anything test which only select different console and then type poweroff ?

I think it would be fine to avoid the switch if we can.

The only reason I'd be curious about keeping the switching is that #105429 could be hitting the same underlying issue.

#10 Updated by okurz 3 months ago

osukup wrote:

okurz wrote:

osukup wrote:

okurz probadly is good idea to check openQA-in-openQA code to modermize/reuse code from os-autoins-distri-opensuse and reuse some things from there ( handling consoles/ serial console).

yes. I would like that. But at best without copy-pasting big blocks of code. This can be yet another good opportunity to either move some code upstream to os-autoinst or to introduce an intermediate helper library or plugin for test distributions.

I think best solution will be port openqa-in-openqa to standard 'os-autoinst-distri-opensuse' as new product like microos/sle/opensuse

Same as I stated yesterday in https://suse.slack.com/archives/C02AJ1E568M/p1643710583739559?thread_ts=1643709665.704599&cid=C02AJ1E568M and before in meetings: What we definitely should not do is merge openqa-in-openqa into os-autoinst-distri-opensuse because os-autoinst-distri-opensuse is already too big to be managed efficiently and we are using also openqa-in-openqa as a playground to learn how to better handle separate test distributions

#12 Updated by osukup 3 months ago

  • Status changed from In Progress to Feedback

PR merged, now wait if changes solved issue on opensuse.org instance ( not able reproduce problem on private instance .. )

#13 Updated by cdywan 3 months ago

  • Due date changed from 2022-02-11 to 2022-02-18
  • Status changed from Feedback to In Progress

okurz wrote:

https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/77 merged

So it seems I missed that the PR got reverted. Good example of why we don't rely on personal instances for verification :-D

osukup Please prepare an updated PR with the typo fixed. I'm generously bumping the due date by a week, but feel free to surprise me and do it now so we can verify on Monday that it all works nicely

#15 Updated by osukup 3 months ago

  • Status changed from In Progress to Feedback

lets observe for day :D

#16 Updated by osukup 3 months ago

  • Status changed from Feedback to Resolved

no fail in shutdown, so we can set resolved

Also available in: Atom PDF