Project

General

Profile

Actions

action #161879

closed

coordination #161168: [epic] Support hooks for obs-sync plugin

hook script support for obs-sync plugin - single hook script before "isos post" size:M

Added by okurz 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2024-06-04
Due date:
% Done:

0%

Estimated time:

Description

Motivation

See discussion in https://suse.slack.com/archives/C02CANHLANP/p1716882218853889 and #159828-9.

In general there is the need to be able to hook into different stages of how OBS Rsync plugin works, or an alternative solution that allows a release manager or maintenance coordinator (with help of QE) to dynamically select/alter the test variables and openqa cli parameters, using a script. Let's start with being able to pass a hook script called with the "isos post" command script and be able to adapt that.

Acceptance criteria

Suggestions

Out of scope

  • Implementing actual hook scripts, other than the hello world example (reference). #161750
  • Solution should allow for multiple hooks to coexist (think how git hooks behave)

Related issues 2 (2 open0 closed)

Related to QA - action #153421: [spike][timeboxed:10h] Replace scriptgen with executing rsync from pythonNew

Actions
Copied from openQA Project - action #161876: hook script support for obs-sync plugin - multiple hook scriptsNew2024-06-04

Actions
Actions #1

Updated by okurz 4 months ago

  • Copied from action #161876: hook script support for obs-sync plugin - multiple hook scripts added
Actions #2

Updated by okurz 4 months ago

  • Parent task changed from #159828 to #161168
Actions #3

Updated by okurz 4 months ago

  • Subject changed from hook script support for obs-sync plugin - single hook script before "isos post" to hook script support for obs-sync plugin - single hook script before "isos post" size:M
Actions #4

Updated by mkittler 4 months ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions #5

Updated by openqa_review 4 months ago

  • Due date set to 2024-06-26

Setting due date based on mean cycle time of SUSE QE Tools

Actions #6

Updated by mkittler 4 months ago

It looks like we have very little logic in openQA itself. The code of the openQA rsync plugin is mainly providing a UI for displaying generated scripts and acts as a generic runner for a shell script, see https://github.com/os-autoinst/openQA/pull/5693.

The shell script in question is https://github.com/os-autoinst/openqa-trigger-from-obs/blob/master/script/rsync.sh and that script defines all the logic. It actually defines the sequence in which to run some other (generated) scripts and lists of commands. So the place to add a hook script would probably be that script and thus the https://github.com/os-autoinst/openqa-trigger-from-obs repository (and not openQA).

I think that this makes things actually even simpler:

  • I just added a new file under my local openqa-trigger-from-obs/openSUSE:Factory:Staging:A/.run_240612_111353_551.1/ directory and it just showed up on http://localhost:9526/admin/obs_rsync/openSUSE:Factory:Staging:A/runs/.run_240612_111353_551.1 without code modifications. The web UI is really just displaying a directory listing here. So you can generate whatever scripts/artifacts you like and they will display on the web UI without extra effort.
  • To actually run an additional script and save its output you can just add a line like the existing line [ ! -e "$subfolder/print_rsync_iso.sh" ] || bash -e "$subfolder/print_rsync_iso.sh" > $logdir/rsync_iso.cmd 2> >(tee $logdir/generate_rsync_iso.err) somewhere in script/rsync.sh where needed. In this script you can call e.g. some OBS APIs to gather additional information about the project in question and store it in a way so it can be consumed in subsequent scripts as needed.
  • You probably need to modify some of the existing scripts as well as you probably want to modify the set of openQA jobs being created - and that is done by the script print_openqa.sh. You needed to modify it so it generates a different list of openqa-cli commands based on the information gathered before (as mentioned in the previous point). However, that script is actually just generated. It is generated by the function def gen_print_openqa(self, f): in script/scriptgen.py. So you probably needed to modify that function instead of the script itself.

Well, the last point of modifying the set of jobs to create is actually not that simple. In fact, modifying a Python script in order to modify a shell script is rather indirect. However, that is were it is decided what set of jobs will be scheduled - so that's what we need to modify and there's not much we can do about it from the openQA side. (In case this indirection gets more in the way than it is helpful it would be possible to not use it for new projects, though. You could simply not use script/scriptgen.py for your new projects and instead just handwrite one script to rule them all. The script could just get its parameters as needed (or consider its current working directory and files in it)¹. This would require only minor changes in script/rsync.sh about how the script generating jobs is invoked.)

That means: There is already a way to add a "hook" script. Just add another script invocation within script/rsync.sh where needed. The hard part is modifying the code that determines the set of jobs to create.


¹ And when I compare e.g. colordiff -Naur openSUSE:Factory:Staging:{A,B}/print_openqa.sh I see that those scripts really mainly differ in the name of the staging ("A" vs. "B"). So just considering that, using script/scriptgen.py is really an overkill in that case. One could just write one script once and it could determine the difference based on its project directory. Even the diff between e.g. colordiff -Naur openSUSE:{Factory:Staging:A,Slowroll:Staging}/print_openqa.sh is not that big.

Actions #7

Updated by szarate 4 months ago

You could simply not use script/scriptgen.py for your new projects and instead just handwrite one script to rule them all.

I was there Marius, 3000 days ago, when we had rsync.pl... and I would like to avoid the mess that rsync.pl was back then.

Well, the last point of modifying the set of jobs to create is actually not that simple. In fact, modifying a Python script in order to modify a shell script is rather indirect. However, that is were it is decided what set of jobs will be scheduled - so that's what we need to modify and there's not much we can do about it from the openQA side.

openQA only needs to sit there, be pretty and process the isos post. The sync script is just an rsync + glue code to generate an isos post.

Actions #8

Updated by livdywan 4 months ago

  • Related to action #153421: [spike][timeboxed:10h] Replace scriptgen with executing rsync from python added
Actions #9

Updated by mkittler 4 months ago

  • Status changed from In Progress to Feedback

This is how an additional hook script invocation would look like: https://github.com/os-autoinst/openqa-trigger-from-obs/pull/249

Actions #10

Updated by okurz 4 months ago

As discussed please break out a "[timeboxed:20h][spike solution] adaptable product scheduling in openqa-trigger-from-obs" ticket including pair-programming with a member of QE Core.

Actions #11

Updated by okurz 4 months ago

  • Status changed from Feedback to Workable
Actions #12

Updated by mkittler 4 months ago

  • Status changed from Workable to Resolved

I created #162515 - see the motivation of that ticked for why I consider this ticket resolved.

Actions #13

Updated by okurz 3 months ago

  • Due date deleted (2024-06-26)
Actions

Also available in: Atom PDF