action #175060
Updated by gpuliti 3 months ago
## Observation
fullstack failed on master
https://app.circleci.com/pipelines/github/os-autoinst/openQA/15433/workflows/dd25c1f6-7e00-4342-b808-f9f1398fcc47/jobs/147885/parallel-runs/0/steps/0-111
```
[14:08:58] t/full-stack.t ......... 31/? # stopped waiting for '(?^u:Result: passed)', result turned out to be 'failed,'
# full result panel contents:
# 2 1
# Result: failed, finished less than a minute ago (ran for 02:06 minutes)
# Scheduled product: job has not been created by posting an ISO
# Assigned worker: 31dcf6c6b299:1
# Actions:
#
# Restart job
# Failed test 'Expected result not found'
# at /home/squamata/project/t/lib/OpenQA/Test/FullstackUtils.pm line 76.
# Failed test 'job 5 passed'
# at t/full-stack.t line 349.
# 0
# Looks like you failed 2 tests of 19.
# Failed test 'results of test 5'
# at t/full-stack.t line 387.
```
* [2025-01-09](https://app.circleci.com/pipelines/github/os-autoinst/openQA/15446/workflows/107fa045-990a-49a2-aecb-08f4ead0742a/jobs/148009)
## Acceptance Criteria
* **AC1:** full-stack tests pass on CircleCI tests reliably with no retries
## Suggestions
* See full logs on https://output.circle-artifacts.com/output/job/bb84062f-82a9-49c3-815e-e875716cac82/artifacts/0/artifacts/full-stack.t
* Maybe CircleCI is being slow. We've seen that before
* Is this reproducible? Re-run seemed to pass. Seems to happen ~monthly https://app.circleci.com/pipelines/github/os-autoinst/openQA?branch=master&status=none&status=failing&status=failed
* As mentioned in #175060#note-3 this happens because `sudo poweroff` cannot be typed - the message `udevadm settle - timeout of 5 seconds reached …` appears in the middle.
* Maybe we can handle that condition in test code and retry typing? This is probably problematic as the issue is not easily reproducible and the exact behavior might differ from case to case (e.g. console output might be interleaved in some cases).
* Check the prompt e.g. look for a lot of blank space
* Maybe extend logging from within the shutdown function to make it easier to check?
* Override the load detection to more easily reproduce the issue?
## Rollback actions
* Revert https://github.com/os-autoinst/openQA/pull/6244 https://github.com/os-autoinst/openQA/pull/6115