action #105506
closed[sporadic][tools] openQA-in-openQA test sporadically fails in shutdown
0%
Description
Observation¶
openQA test in scenario openqa-Tumbleweed-dev-x86_64-openqa_install+publish@64bit-2G fails in
shutdown. Stepping through the video frame by frame it looks like the test switches to the root console but never types poweroff. The line
wait_screen_change { send_key 'ctrl-alt-f3' }
might be the culprite because we don't explicitly wait for the root console to appear but just assume that after the screen content changed significantly enough that we have reached the root console. However the root console might not yet be responsive at this time.
Test suite description¶
Maintainer: okurz@suse.de Test for installation of openQA itself. To be used with "openqa" distri. Publishes an qcow2 image including the openQA installation ready to run as an appliance.
Reproducible¶
Fails since (at least) Build :TW.10346 (current job). https://openqa.opensuse.org/tests/2157847#next_previous shows 19/407 failures so a fail-ratio of roughly 5%.
Expected result¶
Last good: :TW.10345 (or more recent)
Suggestions¶
- Reproduce the error locally and confirm the fail-ratio
- Either properly synchronize the switch to the responsive root terminal with an explicit
assert_screenor use backend power functions over the test API.
Further details¶
Always latest result in this scenario: latest
Updated by okurz about 2 years ago
- Related to action #105429: openQA's fullstack test fails in `shutdown` module added
Updated by osukup about 2 years ago
@okurz probadly is good idea to check openQA-in-openQA code to modermize/reuse code from os-autoins-distri-opensuse and reuse some things from there ( handling consoles/ serial console).
Updated by okurz about 2 years ago
osukup wrote:
@okurz probadly is good idea to check openQA-in-openQA code to modermize/reuse code from os-autoins-distri-opensuse and reuse some things from there ( handling consoles/ serial console).
yes. I would like that. But at best without copy-pasting big blocks of code. This can be yet another good opportunity to either move some code upstream to os-autoinst or to introduce an intermediate helper library or plugin for test distributions.
Updated by openqa_review about 2 years ago
- Due date set to 2022-02-11
Setting due date based on mean cycle time of SUSE QE Tools
Updated by osukup about 2 years ago
okurz wrote:
osukup wrote:
@okurz probadly is good idea to check openQA-in-openQA code to modermize/reuse code from os-autoins-distri-opensuse and reuse some things from there ( handling consoles/ serial console).
yes. I would like that. But at best without copy-pasting big blocks of code. This can be yet another good opportunity to either move some code upstream to os-autoinst or to introduce an intermediate helper library or plugin for test distributions.
I think best solution will be port openqa-in-openqa to standard 'os-autoinst-distri-opensuse' as new product like microos/sle/opensuse
btw ... about failing shutdown. Do we need for anything test which only select different console and then type poweroff ?
Updated by livdywan about 2 years ago
osukup wrote:
btw ... about failing shutdown. Do we need for anything test which only select different console and then type
poweroff?
I think it would be fine to avoid the switch if we can.
The only reason I'd be curious about keeping the switching is that #105429 could be hitting the same underlying issue.
Updated by okurz about 2 years ago
osukup wrote:
okurz wrote:
osukup wrote:
@okurz probadly is good idea to check openQA-in-openQA code to modermize/reuse code from os-autoins-distri-opensuse and reuse some things from there ( handling consoles/ serial console).
yes. I would like that. But at best without copy-pasting big blocks of code. This can be yet another good opportunity to either move some code upstream to os-autoinst or to introduce an intermediate helper library or plugin for test distributions.
I think best solution will be port openqa-in-openqa to standard 'os-autoinst-distri-opensuse' as new product like microos/sle/opensuse
Same as I stated yesterday in https://suse.slack.com/archives/C02AJ1E568M/p1643710583739559?thread_ts=1643709665.704599&cid=C02AJ1E568M and before in meetings: What we definitely should not do is merge openqa-in-openqa into os-autoinst-distri-opensuse because os-autoinst-distri-opensuse is already too big to be managed efficiently and we are using also openqa-in-openqa as a playground to learn how to better handle separate test distributions
Updated by okurz about 2 years ago
Updated by osukup about 2 years ago
- Status changed from In Progress to Feedback
PR merged, now wait if changes solved issue on opensuse.org instance ( not able reproduce problem on private instance .. )
Updated by livdywan about 2 years ago
- Due date changed from 2022-02-11 to 2022-02-18
- Status changed from Feedback to In Progress
okurz wrote:
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/77 merged
So it seems I missed that the PR got reverted. Good example of why we don't rely on personal instances for verification :-D
@osukup Please prepare an updated PR with the typo fixed. I'm generously bumping the due date by a week, but feel free to surprise me and do it now so we can verify on Monday that it all works nicely
Updated by osukup about 2 years ago
Updated by osukup about 2 years ago
- Status changed from In Progress to Feedback
lets observe for day :D
Updated by osukup about 2 years ago
- Status changed from Feedback to Resolved
no fail in shutdown, so we can set resolved