Project

General

Profile

action #110176

openQA Project - coordination #102915: [saga][epic] Automated classification of failures

coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests

[spike solution] [timeboxed:10h] Restart hook script in delayed minion job based on exit code size:M

Added by mkittler 2 months ago. Updated 11 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2022-06-15
Due date:
% Done:

0%

Estimated time:

Description

Motivation

From hook scripts we need to know about results from other hooks scripts.

Acceptance criteria

  • AC1: We know how a hook script can indicate it is not done yet, e.g. by returning a special exit code
  • AC2: We know how openQA would be able to re-run the hook script again in a delayed Minion job in case of AC1

Suggestions

  • See #109920 for further context and suggestions
  • Note: The bash script openqa-investigate itself must not know anything about "openQA minion jobs" or schedule any

Related issues

Related to openQA Project - action #95783: Provide support for multi-machine scenarios handled by openqa-investigate size:MWorkable

Related to openQA Project - action #112523: Make hook scripts restartable with a special exit codeResolved2022-06-15

Copied from QA - action #109920: Identify reproducible product issues using openqa-investigateNew2022-04-13

History

#1 Updated by mkittler 2 months ago

  • Copied from action #109920: Identify reproducible product issues using openqa-investigate added

#2 Updated by okurz 2 months ago

  • Subject changed from [spike solution] [time-boxed:10h] Restart hook script in delayed minion job based on exit code size:M to [spike solution] [timeboxed:10h] Restart hook script in delayed minion job based on exit code size:M

#3 Updated by kraih 27 days ago

  • Assignee set to kraih

#4 Updated by mkittler 17 days ago

  • Related to action #95783: Provide support for multi-machine scenarios handled by openqa-investigate size:M added

#5 Updated by kraih 16 days ago

  • Status changed from Workable to In Progress

#6 Updated by openqa_review 16 days ago

  • Due date set to 2022-06-25

Setting due date based on mean cycle time of SUSE QE Tools

#7 Updated by kraih 13 days ago

Got to catch up a bit on the hook script code, since i've not worked on it before. But it's interesting at least.

#8 Updated by kraih 12 days ago

Ok, i'm up to speed now. Experimenting with moving hook scripts into a separate Minion task now.

#9 Updated by kraih 12 days ago

The first step in any solution will probably be moving the hook script execution into a separate Minion job. Which thanks to the good tests is not all that hard. https://github.com/os-autoinst/openQA/commit/8ab07cb29e179832cba1f7c33e53b11b40fdcd0e

#10 Updated by kraih 12 days ago

I've also prepared a proof of concept that uses the native Minion retry mechanism to run hook scripts multiple times (up to 3 here) that use the exit status 42. https://github.com/os-autoinst/openQA/compare/k/hook_scripts

#11 Updated by kraih 12 days ago

Open questions that remain:

  • How to configure the number of retries?
  • How to configure the delay?
  • Which exit code to use in the real implementation?
  • How to handle the output of more than one run? (proof of concept only keeps the latest result)

#12 Updated by okurz 12 days ago

kraih wrote:

Open questions that remain:

  • How to configure the number of retries?
  • How to configure the delay?
  • Which exit code to use in the real implementation?

https://github.com/os-autoinst/openQA/pull/4703#discussion_r897747670

  • How to handle the output of more than one run? (proof of concept only keeps the latest result)

log output for debugging into the gru log for all "skipped" runs, only keep latest result in minion job information.

#13 Updated by kraih 11 days ago

  • Status changed from In Progress to Feedback

#14 Updated by kraih 11 days ago

  • Related to action #112523: Make hook scripts restartable with a special exit code added

#15 Updated by kraih 11 days ago

  • Status changed from Feedback to Resolved

Followup: #112523

Also available in: Atom PDF