action #110176
closedopenQA Project (public) - coordination #102915: [saga][epic] Automated classification of failures
coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests
[spike solution] [timeboxed:10h] Restart hook script in delayed minion job based on exit code size:M
0%
Description
Motivation¶
From hook scripts we need to know about results from other hooks scripts.
Acceptance criteria¶
- AC1: We know how a hook script can indicate it is not done yet, e.g. by returning a special exit code
- AC2: We know how openQA would be able to re-run the hook script again in a delayed Minion job in case of AC1
Suggestions¶
- See #109920 for further context and suggestions
- Note: The bash script openqa-investigate itself must not know anything about "openQA minion jobs" or schedule any
Updated by mkittler over 2 years ago
- Copied from action #109920: Identify reproducible product issues using openqa-investigate size:M added
Updated by okurz over 2 years ago
- Subject changed from [spike solution] [time-boxed:10h] Restart hook script in delayed minion job based on exit code size:M to [spike solution] [timeboxed:10h] Restart hook script in delayed minion job based on exit code size:M
Updated by mkittler over 2 years ago
- Related to action #95783: Provide support for multi-machine scenarios handled by openqa-investigate size:M added
Updated by openqa_review over 2 years ago
- Due date set to 2022-06-25
Setting due date based on mean cycle time of SUSE QE Tools
Updated by kraih over 2 years ago
Got to catch up a bit on the hook script code, since i've not worked on it before. But it's interesting at least.
Updated by kraih over 2 years ago
Ok, i'm up to speed now. Experimenting with moving hook scripts into a separate Minion task now.
Updated by kraih over 2 years ago
The first step in any solution will probably be moving the hook script execution into a separate Minion job. Which thanks to the good tests is not all that hard. https://github.com/os-autoinst/openQA/commit/8ab07cb29e179832cba1f7c33e53b11b40fdcd0e
Updated by kraih over 2 years ago
I've also prepared a proof of concept that uses the native Minion retry mechanism to run hook scripts multiple times (up to 3 here) that use the exit status 42. https://github.com/os-autoinst/openQA/compare/k/hook_scripts
Updated by kraih over 2 years ago
Open questions that remain:
- How to configure the number of retries?
- How to configure the delay?
- Which exit code to use in the real implementation?
- How to handle the output of more than one run? (proof of concept only keeps the latest result)
Updated by okurz over 2 years ago
kraih wrote:
Open questions that remain:
- How to configure the number of retries?
- How to configure the delay?
- Which exit code to use in the real implementation?
https://github.com/os-autoinst/openQA/pull/4703#discussion_r897747670
- How to handle the output of more than one run? (proof of concept only keeps the latest result)
log output for debugging into the gru log for all "skipped" runs, only keep latest result in minion job information.
Updated by kraih over 2 years ago
- Related to action #112523: Make hook scripts restartable with a special exit code added