action #164577
closedcoordination #58184: [saga][epic][use case] full version control awareness within openQA
coordination #94750: [epic] Support needles from git-cloned or separate repo if casedir points to a git-repo
[timeboxed:6h][research] Understand the current behaviour of openQA test distributions including needles
Added by okurz 5 months ago. Updated 5 months ago.
Description
Motivation¶
See the unexpected current default behaviour in #94735. We plan to follow up in in #97112. As that is an older ticket we identified the need to learn what is the actual current behaviour.
Acceptance criteria¶
- G1: For tests with CASEDIR pointing to a git repo including tests+needles we understand what is necessary
Suggestions¶
- To understand how the current behaviour is it is suggested to trigger tests with CASEDIR=https://github.com/os-autoinst/os-autoinst-distri-example because the example distribution includes needles in the same repo
Updated by okurz 5 months ago
- Copied from action #97112: Support relative needle directories together with tests checked out from git added
Updated by mkittler 5 months ago · Edited
- Status changed from In Progress to Feedback
I've just ran the following command again from my notes on #162647 as it has all the bits required to understand the current behavior:
script/openqa-cli schedule --host http://localhost:9526 --monitor --param-file SCENARIO_DEFINITIONS_YAML=/hdd/openqa-devel/openqa/share/tests/example/scenario-definitions.yaml DISTRI=example VERSION=0 FLAVOR=DVD ARCH=x86_64 TEST=simple_boot BUILD=test-poo-162647 _GROUP_ID=0 CASEDIR=https://github.com/Martchus/os-autoinst-distri-example.git#test-poo-162647 NEEDLES_DIR=%%CASEDIR%%/needles PAUSE_AT=tests-boot
This test definitely ran tests from the Git branch specified via CASEDIR as it failed not finding need-yet-another-needle (a needle that is missing on purpose on that branch). It was however generally able to locate needles because the bootloader needle matched (which is the only needle these tests have so far). The symlink created by the worker was pointing to os-autoinst-distri-example/needles which is the needles directory checked out via CASEDIR. So everything works as expected.
Note that specifying NEEDLES_DIR=%%CASEDIR%%/needles would not have been necessary. However, without it, needles would not be loaded from the Git repo and branch specified via CASEDIR (and checked out by os-autoinst). They would instead be loaded from the default branch (checked out by the web UI and synced via the worker cache). I also verified that by leaving out NEEDLES_DIR=%%CASEDIR%%/needles. Then the symlink created the the worker was pointing to /hdd/openqa-devel/openqa/share/tests/example/needles (the location of the web UI checkout in my setup). This makes also sense because the CASEDIR often does not even contain needles. So setting CASEDIR should by default not influence NEEDLES_DIR.
All of this means that the documentation (as added in #94735) still matches the current behavior.
For #97112 we need to change the following behavior:
- Make the behavior we currently get via
NEEDLES_DIR=%%CASEDIR%%/needlesthe default ifCASEDIRactually contains aneedlesdirectory. The "if" is important because otherwise we'll break the most common use case (whenCASEDIRdoes not contain aneedlesdirectory). Unfortunately it isn't straight forward to check for this "if" because the openQA worker is evaluating these variables but only later os-autoinst is doing the actual checkouts.- Make the code for checking out Git repositories for the
CASEDIR/NEEDLES_DIRfrom os-autoinst available to the openQA worker. The easiest approach would be to ensure the code is in a separate module (not some bigutils.pm) and simply add that module to the openQA worker's Perl module search path. We can also make os-autoinst's install script install the module to the normal Perl search path. - Make the openQA worker (the worker code in openQA, distributed as openQA-worker package) handle
CASEDIR/NEEDLES_DIRreusing code from os-autoinst so it has the required checkouts available and we can test for the "if". This will not disturb os-autoinst as it'll just skip the checkouts if already present. This will also not break os-autoinst-only workflows as os-autoinst remains the capability to do the Git checkouts as needed.
- Make the code for checking out Git repositories for the
- Invent yet another syntax (e.g.
NEEDLES_DIR=%%DEFAULT_BRANCH%%) to load needles from the default branch as checked out by the web UI (even thoughCASEDIRcontains aneedlesdirectory). I'm not sure whether that is an actual use case but it is a behavior we currently support (as the default behavior) and by just implementing the first point we'd remove it.
Further remarks:
- We should probably keep the logic for
%%CASEDIR%%-syntax to not break backwards compatibility. Besides, depending on the next point we still need it to be able to point to a different sub directory withinCASEDIRthenneedles. - We should consider what we do if
NEEDLES_DIRcontains a relative path. So far it is treated relative to the default/web-UI checkout which makes also sense as this is so far also our default in the absence of the setting. Now we might want to change it so the%%CASEDIR%%/-prefix is the default if that directory exists (and e.g. the new prefix%%DEFAULT_BRANCH%%/would be used for the current default behavior). - I'm personally not sure whether all of this is worth it or even a good idea. Specifying
NEEDLES_DIR=%%CASEDIR%%/needlesdoesn't really hurt.
Updated by tinita 5 months ago · Edited
Make the code for checking out Git repositories for the CASEDIR/NEEDLES_DIR from os-autoinst available to the openQA worker. The easiest approach would be to ensure the code is in a separate module (not some big utils.pm) and simply add that module to the openQA worker's Perl module search path. We can also make os-autoinst's install script install the module to the normal Perl search path.
I don't understand what you have in mind here. Can you elaborate?
Especially add that module to the openQA worker's Perl module search path sounds hacky and I don't understand why we would manyally have to add something to the standard perl module paths.
Updated by tinita 5 months ago
Note that specifying NEEDLES_DIR=%%CASEDIR%%/needles would not have been necessary. However, ...
Necessary for what?
Maybe I also don't get the goal For tests with CASEDIR pointing to a git repo including tests+needles we understand what is necessary where it says we should understand what's necessary, but it doesn't say necessary for what.
Updated by mkittler 5 months ago · Edited
tinita wrote in #note-4:
Make the code for checking out Git repositories for the CASEDIR/NEEDLES_DIR from os-autoinst available to the openQA worker. The easiest approach would be to ensure the code is in a separate module (not some big utils.pm) and simply add that module to the openQA worker's Perl module search path. We can also make os-autoinst's install script install the module to the normal Perl search path.
I don't understand what you have in mind here. Can you elaborate?
Especially
add that module to the openQA worker's Perl module search pathsounds hacky and I don't understand why we would manyally have to add something to the standard perl module paths.
The Perl modules provided by os-autoinst are installed under /usr/lib/os-autoinst so we have to add this path to the search path in openQA worker to use them. Or we make sure the required subset of the Perl modules provided by os-autoinst is installed under the standard Perl module paths, e.g. /usr/lib/perl5/vendor_perl/5.40.0/os_autoinst.
tinita wrote in #note-5:
Note that specifying NEEDLES_DIR=%%CASEDIR%%/needles would not have been necessary. However, ...
Necessary for what?
It is not necessary to run the test. However, if you want to load needles also from the branch specified via CASEDIR=… you have to add it.
Maybe I also don't get the goal
For tests with CASEDIR pointing to a git repo including tests+needles we understand what is necessarywhere it says we should understand what's necessary, but it doesn't say necessary for what.
Yes, it doesn't say that. Hence I was trying to explain when you need NEEDLES_DIR=%%CASEDIR%%/needles and when you don't. It really depends on your use case. But the first paragraphs of my previous comment aren't that important. What's important is everything as of "For #97112 we need to change the following behavior:".
When I think more about it I'm really not sure whether I'm liking #97112. Why should specifying CASEDIR=…#some-branch change the branch we load needles from? After all, the variable is called CASEDIR and if I wanted needles from a custom branch as well then for this we have NEEDLES_DIR=…#some-branch or the also very explicit NEEDLES_DIR=%%CASEDIR%%/needles.
Updated by mkittler 5 months ago
PR to improve the documentation: https://github.com/os-autoinst/openQA/pull/5799
Updated by okurz 5 months ago
- Status changed from Feedback to Resolved
https://github.com/os-autoinst/openQA/pull/5799 merged. With this we can conclude.