action #164577: [timeboxed:6h][research] Understand the current behaviour of openQA test distributions including needles - openQA Project (public) - openSUSE Project Management Tool

Actions

Copy link

action #164577

closed

coordination #58184: [saga][epic][use case] full version control awareness within openQA

coordination #94750: [epic] Support needles from git-cloned or separate repo if casedir points to a git-repo

[timeboxed:6h][research] Understand the current behaviour of openQA test distributions including needles

Added by okurz 5 months ago. Updated 5 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

mkittler

Category:

Feature requests

Target version:

Ready

Start date:

2024-07-29

Due date:

% Done:

Estimated time:

Description

Motivation¶

See the unexpected current default behaviour in #94735. We plan to follow up in in #97112. As that is an older ticket we identified the need to learn what is the actual current behaviour.

Acceptance criteria¶

G1: For tests with CASEDIR pointing to a git repo including tests+needles we understand what is necessary

Suggestions¶

To understand how the current behaviour is it is suggested to trigger tests with CASEDIR=https://github.com/os-autoinst/os-autoinst-distri-example because the example distribution includes needles in the same repo

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by okurz 5 months ago

Copied from action #97112: Support relative needle directories together with tests checked out from git added

Actions

Copy link

Updated by mkittler 5 months ago

Status changed from New to In Progress
Assignee set to mkittler

Actions

Copy link

Updated by mkittler 5 months ago · Edited

Status changed from In Progress to Feedback

I've just ran the following command again from my notes on #162647 as it has all the bits required to understand the current behavior:

script/openqa-cli schedule --host http://localhost:9526 --monitor --param-file SCENARIO_DEFINITIONS_YAML=/hdd/openqa-devel/openqa/share/tests/example/scenario-definitions.yaml DISTRI=example VERSION=0 FLAVOR=DVD ARCH=x86_64 TEST=simple_boot BUILD=test-poo-162647 _GROUP_ID=0 CASEDIR=https://github.com/Martchus/os-autoinst-distri-example.git#test-poo-162647 NEEDLES_DIR=%%CASEDIR%%/needles PAUSE_AT=tests-boot

This test definitely ran tests from the Git branch specified via CASEDIR as it failed not finding need-yet-another-needle (a needle that is missing on purpose on that branch). It was however generally able to locate needles because the bootloader needle matched (which is the only needle these tests have so far). The symlink created by the worker was pointing to os-autoinst-distri-example/needles which is the needles directory checked out via CASEDIR. So everything works as expected.

Note that specifying NEEDLES_DIR=%%CASEDIR%%/needles would not have been necessary. However, without it, needles would not be loaded from the Git repo and branch specified via CASEDIR (and checked out by os-autoinst). They would instead be loaded from the default branch (checked out by the web UI and synced via the worker cache). I also verified that by leaving out NEEDLES_DIR=%%CASEDIR%%/needles. Then the symlink created the the worker was pointing to /hdd/openqa-devel/openqa/share/tests/example/needles (the location of the web UI checkout in my setup). This makes also sense because the CASEDIR often does not even contain needles. So setting CASEDIR should by default not influence NEEDLES_DIR.

All of this means that the documentation (as added in #94735) still matches the current behavior.

For #97112 we need to change the following behavior:

Make the behavior we currently get via NEEDLES_DIR=%%CASEDIR%%/needles the default if CASEDIR actually contains a needles directory. The "if" is important because otherwise we'll break the most common use case (when CASEDIR does not contain a needles directory). Unfortunately it isn't straight forward to check for this "if" because the openQA worker is evaluating these variables but only later os-autoinst is doing the actual checkouts.
- Make the code for checking out Git repositories for the CASEDIR/NEEDLES_DIR from os-autoinst available to the openQA worker. The easiest approach would be to ensure the code is in a separate module (not some big utils.pm) and simply add that module to the openQA worker's Perl module search path. We can also make os-autoinst's install script install the module to the normal Perl search path.
- Make the openQA worker (the worker code in openQA, distributed as openQA-worker package) handle CASEDIR/NEEDLES_DIR reusing code from os-autoinst so it has the required checkouts available and we can test for the "if". This will not disturb os-autoinst as it'll just skip the checkouts if already present. This will also not break os-autoinst-only workflows as os-autoinst remains the capability to do the Git checkouts as needed.
Invent yet another syntax (e.g. NEEDLES_DIR=%%DEFAULT_BRANCH%%) to load needles from the default branch as checked out by the web UI (even though CASEDIR contains a needles directory). I'm not sure whether that is an actual use case but it is a behavior we currently support (as the default behavior) and by just implementing the first point we'd remove it.

Further remarks:

We should probably keep the logic for %%CASEDIR%%-syntax to not break backwards compatibility. Besides, depending on the next point we still need it to be able to point to a different sub directory within CASEDIR then needles.
We should consider what we do if NEEDLES_DIR contains a relative path. So far it is treated relative to the default/web-UI checkout which makes also sense as this is so far also our default in the absence of the setting. Now we might want to change it so the %%CASEDIR%%/-prefix is the default if that directory exists (and e.g. the new prefix %%DEFAULT_BRANCH%%/ would be used for the current default behavior).
I'm personally not sure whether all of this is worth it or even a good idea. Specifying NEEDLES_DIR=%%CASEDIR%%/needles doesn't really hurt.

Actions

Copy link

Updated by tinita 5 months ago · Edited

Make the code for checking out Git repositories for the CASEDIR/NEEDLES_DIR from os-autoinst available to the openQA worker. The easiest approach would be to ensure the code is in a separate module (not some big utils.pm) and simply add that module to the openQA worker's Perl module search path. We can also make os-autoinst's install script install the module to the normal Perl search path.

I don't understand what you have in mind here. Can you elaborate?

Especially add that module to the openQA worker's Perl module search path sounds hacky and I don't understand why we would manyally have to add something to the standard perl module paths.

Actions

Copy link

Updated by tinita 5 months ago

Note that specifying NEEDLES_DIR=%%CASEDIR%%/needles would not have been necessary. However, ...

Necessary for what?

Maybe I also don't get the goal For tests with CASEDIR pointing to a git repo including tests+needles we understand what is necessary where it says we should understand what's necessary, but it doesn't say necessary for what.

Actions

Copy link

Updated by mkittler 5 months ago · Edited

tinita wrote in #note-4:

Make the code for checking out Git repositories for the CASEDIR/NEEDLES_DIR from os-autoinst available to the openQA worker. The easiest approach would be to ensure the code is in a separate module (not some big utils.pm) and simply add that module to the openQA worker's Perl module search path. We can also make os-autoinst's install script install the module to the normal Perl search path.

I don't understand what you have in mind here. Can you elaborate?

Especially add that module to the openQA worker's Perl module search path sounds hacky and I don't understand why we would manyally have to add something to the standard perl module paths.

The Perl modules provided by os-autoinst are installed under /usr/lib/os-autoinst so we have to add this path to the search path in openQA worker to use them. Or we make sure the required subset of the Perl modules provided by os-autoinst is installed under the standard Perl module paths, e.g. /usr/lib/perl5/vendor_perl/5.40.0/os_autoinst.

tinita wrote in #note-5:

Note that specifying NEEDLES_DIR=%%CASEDIR%%/needles would not have been necessary. However, ...

Necessary for what?

It is not necessary to run the test. However, if you want to load needles also from the branch specified via CASEDIR=… you have to add it.

Maybe I also don't get the goal For tests with CASEDIR pointing to a git repo including tests+needles we understand what is necessary where it says we should understand what's necessary, but it doesn't say necessary for what.

Yes, it doesn't say that. Hence I was trying to explain when you need NEEDLES_DIR=%%CASEDIR%%/needles and when you don't. It really depends on your use case. But the first paragraphs of my previous comment aren't that important. What's important is everything as of "For #97112 we need to change the following behavior:".

When I think more about it I'm really not sure whether I'm liking #97112. Why should specifying CASEDIR=…#some-branch change the branch we load needles from? After all, the variable is called CASEDIR and if I wanted needles from a custom branch as well then for this we have NEEDLES_DIR=…#some-branch or the also very explicit NEEDLES_DIR=%%CASEDIR%%/needles.

Actions

Copy link