Project

General

Profile

Actions

action #110176

closed

openQA Project - coordination #102915: [saga][epic] Automated classification of failures

coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests

[spike solution] [timeboxed:10h] Restart hook script in delayed minion job based on exit code size:M

Added by mkittler almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2022-06-15
Due date:
% Done:

0%

Estimated time:

Description

Motivation

From hook scripts we need to know about results from other hooks scripts.

Acceptance criteria

  • AC1: We know how a hook script can indicate it is not done yet, e.g. by returning a special exit code
  • AC2: We know how openQA would be able to re-run the hook script again in a delayed Minion job in case of AC1

Suggestions

  • See #109920 for further context and suggestions
  • Note: The bash script openqa-investigate itself must not know anything about "openQA minion jobs" or schedule any

Related issues 3 (0 open3 closed)

Related to openQA Project - action #95783: Provide support for multi-machine scenarios handled by openqa-investigate size:MResolvedmkittler

Actions
Related to openQA Project - action #112523: Make hook scripts restartable with a special exit codeResolvedkraih2022-06-15

Actions
Copied from openQA Project - action #109920: Identify reproducible product issues using openqa-investigate size:MResolvedtinita2023-03-23

Actions
Actions #1

Updated by mkittler almost 2 years ago

  • Copied from action #109920: Identify reproducible product issues using openqa-investigate size:M added
Actions #2

Updated by okurz almost 2 years ago

  • Subject changed from [spike solution] [time-boxed:10h] Restart hook script in delayed minion job based on exit code size:M to [spike solution] [timeboxed:10h] Restart hook script in delayed minion job based on exit code size:M
Actions #3

Updated by kraih almost 2 years ago

  • Assignee set to kraih
Actions #4

Updated by mkittler almost 2 years ago

  • Related to action #95783: Provide support for multi-machine scenarios handled by openqa-investigate size:M added
Actions #5

Updated by kraih almost 2 years ago

  • Status changed from Workable to In Progress
Actions #6

Updated by openqa_review almost 2 years ago

  • Due date set to 2022-06-25

Setting due date based on mean cycle time of SUSE QE Tools

Actions #7

Updated by kraih almost 2 years ago

Got to catch up a bit on the hook script code, since i've not worked on it before. But it's interesting at least.

Actions #8

Updated by kraih almost 2 years ago

Ok, i'm up to speed now. Experimenting with moving hook scripts into a separate Minion task now.

Actions #9

Updated by kraih almost 2 years ago

The first step in any solution will probably be moving the hook script execution into a separate Minion job. Which thanks to the good tests is not all that hard. https://github.com/os-autoinst/openQA/commit/8ab07cb29e179832cba1f7c33e53b11b40fdcd0e

Actions #10

Updated by kraih almost 2 years ago

I've also prepared a proof of concept that uses the native Minion retry mechanism to run hook scripts multiple times (up to 3 here) that use the exit status 42. https://github.com/os-autoinst/openQA/compare/k/hook_scripts

Actions #11

Updated by kraih almost 2 years ago

Open questions that remain:

  • How to configure the number of retries?
  • How to configure the delay?
  • Which exit code to use in the real implementation?
  • How to handle the output of more than one run? (proof of concept only keeps the latest result)
Actions #12

Updated by okurz almost 2 years ago

kraih wrote:

Open questions that remain:

  • How to configure the number of retries?
  • How to configure the delay?
  • Which exit code to use in the real implementation?

https://github.com/os-autoinst/openQA/pull/4703#discussion_r897747670

  • How to handle the output of more than one run? (proof of concept only keeps the latest result)

log output for debugging into the gru log for all "skipped" runs, only keep latest result in minion job information.

Actions #13

Updated by kraih almost 2 years ago

  • Status changed from In Progress to Feedback
Actions #14

Updated by kraih almost 2 years ago

  • Related to action #112523: Make hook scripts restartable with a special exit code added
Actions #15

Updated by kraih almost 2 years ago

  • Status changed from Feedback to Resolved

Followup: #112523

Actions

Also available in: Atom PDF