action #161879
closedcoordination #161168: [epic] Support hooks for obs-sync plugin
hook script support for obs-sync plugin - single hook script before "isos post" size:M
Description
Motivation¶
See discussion in https://suse.slack.com/archives/C02CANHLANP/p1716882218853889 and #159828-9.
In general there is the need to be able to hook into different stages of how OBS Rsync plugin works, or an alternative solution that allows a release manager or maintenance coordinator (with help of QE) to dynamically select/alter the test variables and openqa cli parameters, using a script. Let's start with being able to pass a hook script called with the "isos post" command script and be able to adapt that.
Acceptance criteria¶
- AC1: A "hello world" hook script can be called before actual "isos post" calls in https://github.com/os-autoinst/openqa-trigger-from-obs/
- AC2: hook scripts execution can be skipped based on configuration/convention, e.g. from the .xml files in https://github.com/os-autoinst/openqa-trigger-from-obs/
- AC3: Solution should allow for the user to alter
isos post
behavior (cli parameters and variables, e.g.BETA=1
)
Suggestions¶
- Foresee optional hook scripts in https://github.com/os-autoinst/openqa-trigger-from-obs/ , e.g. in a "hooks/" folder, maybe replicating paths matching https://github.com/os-autoinst/openqa-trigger-from-obs/tree/master/xml/ paths, e.g. for a configuration https://github.com/os-autoinst/openqa-trigger-from-obs/tree/master/xml/openSUSE:Factory.xml read and execute any hook matching https://github.com/os-autoinst/openqa-trigger-from-obs/tree/master/hooks/openSUSE:Factory-* . For now it's enough if just one script is supported not multiple.
- Call the hook scripts as part of the openQA sync plugin, e.g. immediately before the actual "isos post" hook script invocation.
- Simply pass the product triggering script as string, e.g. https://openqa.opensuse.org/admin/obs_rsync/KDE:Medias%7Ckrypton/runs/.run_last/download/openqa.cmd, to the hook script and again return a script that will be called
- Emulate the hook scripts with unit tests within the openQA sync plugin code based on the above convention https://github.com/os-autoinst/openQA/blob/master/t/api/14-plugin_obs_rsync*.t or also the corresponding UI tests, e.g. t/ui/27-plugin_obs_rsync*.t
- Mention hook scripts in the README
Out of scope¶
- Implementing actual hook scripts, other than the hello world example (reference). #161750
- Solution should allow for multiple hooks to coexist (think how git hooks behave)
Updated by okurz 4 months ago
- Copied from action #161876: hook script support for obs-sync plugin - multiple hook scripts added
Updated by openqa_review 4 months ago
- Due date set to 2024-06-26
Setting due date based on mean cycle time of SUSE QE Tools
Updated by mkittler 4 months ago
It looks like we have very little logic in openQA itself. The code of the openQA rsync plugin is mainly providing a UI for displaying generated scripts and acts as a generic runner for a shell script, see https://github.com/os-autoinst/openQA/pull/5693.
The shell script in question is https://github.com/os-autoinst/openqa-trigger-from-obs/blob/master/script/rsync.sh and that script defines all the logic. It actually defines the sequence in which to run some other (generated) scripts and lists of commands. So the place to add a hook script would probably be that script and thus the https://github.com/os-autoinst/openqa-trigger-from-obs repository (and not openQA).
I think that this makes things actually even simpler:
- I just added a new file under my local
openqa-trigger-from-obs/openSUSE:Factory:Staging:A/.run_240612_111353_551.1/
directory and it just showed up on http://localhost:9526/admin/obs_rsync/openSUSE:Factory:Staging:A/runs/.run_240612_111353_551.1 without code modifications. The web UI is really just displaying a directory listing here. So you can generate whatever scripts/artifacts you like and they will display on the web UI without extra effort. - To actually run an additional script and save its output you can just add a line like the existing line
[ ! -e "$subfolder/print_rsync_iso.sh" ] || bash -e "$subfolder/print_rsync_iso.sh" > $logdir/rsync_iso.cmd 2> >(tee $logdir/generate_rsync_iso.err)
somewhere inscript/rsync.sh
where needed. In this script you can call e.g. some OBS APIs to gather additional information about the project in question and store it in a way so it can be consumed in subsequent scripts as needed. - You probably need to modify some of the existing scripts as well as you probably want to modify the set of openQA jobs being created - and that is done by the script
print_openqa.sh
. You needed to modify it so it generates a different list ofopenqa-cli
commands based on the information gathered before (as mentioned in the previous point). However, that script is actually just generated. It is generated by the functiondef gen_print_openqa(self, f):
inscript/scriptgen.py
. So you probably needed to modify that function instead of the script itself.
Well, the last point of modifying the set of jobs to create is actually not that simple. In fact, modifying a Python script in order to modify a shell script is rather indirect. However, that is were it is decided what set of jobs will be scheduled - so that's what we need to modify and there's not much we can do about it from the openQA side. (In case this indirection gets more in the way than it is helpful it would be possible to not use it for new projects, though. You could simply not use script/scriptgen.py
for your new projects and instead just handwrite one script to rule them all. The script could just get its parameters as needed (or consider its current working directory and files in it)¹. This would require only minor changes in script/rsync.sh
about how the script generating jobs is invoked.)
That means: There is already a way to add a "hook" script. Just add another script invocation within script/rsync.sh
where needed. The hard part is modifying the code that determines the set of jobs to create.
¹ And when I compare e.g. colordiff -Naur openSUSE:Factory:Staging:{A,B}/print_openqa.sh
I see that those scripts really mainly differ in the name of the staging ("A" vs. "B"). So just considering that, using script/scriptgen.py
is really an overkill in that case. One could just write one script once and it could determine the difference based on its project directory. Even the diff between e.g. colordiff -Naur openSUSE:{Factory:Staging:A,Slowroll:Staging}/print_openqa.sh
is not that big.
Updated by szarate 4 months ago
You could simply not use script/scriptgen.py for your new projects and instead just handwrite one script to rule them all.
I was there Marius, 3000 days ago, when we had rsync.pl... and I would like to avoid the mess that rsync.pl was back then.
Well, the last point of modifying the set of jobs to create is actually not that simple. In fact, modifying a Python script in order to modify a shell script is rather indirect. However, that is were it is decided what set of jobs will be scheduled - so that's what we need to modify and there's not much we can do about it from the openQA side.
openQA only needs to sit there, be pretty and process the isos post. The sync script is just an rsync + glue code to generate an isos post
.
Updated by livdywan 4 months ago
- Related to action #153421: [spike][timeboxed:10h] Replace scriptgen with executing rsync from python added
Updated by mkittler 4 months ago
- Status changed from In Progress to Feedback
This is how an additional hook script invocation would look like: https://github.com/os-autoinst/openqa-trigger-from-obs/pull/249