action #36994
closedDynamic test flow definition + override
0%
Description
Motivation¶
Linked to UC4 as well as UC7. As a bug assignee or QA contact I want openQA to execute a custom scenario with only limited test modules and optional override test or investigation code to be able to reproduce bugs easily which are most easily or only reproduced in openQA context without needing to adapt the normal validation test code
Acceptance criteria¶
- AC1: A list of test modules to be scheduled can be specified on job trigger time overriding main.pm
- AC2: Override code for test modules can be specified
- AC3: New test modules can be defined
Suggestions¶
- openQA workers download additional test assets if specified by variables to allow injecting new test modules or override modules on-the-fly before isotovideo is called, e.g. using the existing openQA feature of downloading test assets using variables which specify the '_URL' suffix on test parameters.
- A test schedule can be specified completely by a test variable, e.g.
MODULES=boot/boot_to_desktop,qa_automation/patch_and_reboot,my_code
, skipping over all other loadtest calls loadtest
within main.pm looks in the pool directory if any override exists and loads that with preference over any that is specified in the test distribution. This should also work if the specified module does not even exist in the test distribution
Further details¶
Keep security considerations in mind, e.g. do not just allow everybody to execute random code from untrusted remote sources. openQA using the _URL
test variables already checks against a "whitelist" of trusted sources so maybe we are covered already.
Updated by okurz over 6 years ago
- Related to coordination #15132: [saga][epic] Better structure of test plans in main.pm added
Updated by okurz about 6 years ago
- Status changed from New to In Progress
- Assignee set to okurz
Updated by okurz about 6 years ago
Updated by okurz about 6 years ago
- Status changed from In Progress to Resolved
All working as expected. For example one can use a trigger line like
openqa-clone-job --skip-chained-deps --from http://openqa.suse.de 2163336 WORKER_CLASS=qemu_x86_64_sle SCHEDULE=tests/boot/boot_to_desktop,tests/console/consoletest_setup,sysrq ASSET_1_URL=https://w3.nue.suse.com/~okurz/sysrq.pm
where "sysrq.pm" is a file in test module format, e.g. the following content:
use base 'consoletest';
use strict;
use testapi;
use utils;
sub run {
select_console 'root-console';
script_run 'dmesg -n 7';
wait_screen_change { send_key 'alt-sysrq-w' };
assert_screen 'sysrq-output';
}
sub post_run_hook {}
1;
This could be used e.g. in bug investigations, trying out new test modules which are hard to test locally, etc. I assume documentation is hard to find for people but I guess this is always the case. https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc describes the "SCHEDULE" parameter, https://github.com/os-autoinst/openQA/blob/master/docs/UsersGuide.asciidoc#asset-handling describes how downloadable assets can be specified. The feature works as expected on osd, o3 can work as well but might have the small limitation that currently only the two domains "download.opensuse.org openqa.opensuse.org" are allowed to download from.
Updated by okurz about 6 years ago
- Status changed from Resolved to Feedback
feedback from sprint review: We should document this on another place describing the use case specifically, could be openQA users guide.
Updated by okurz almost 6 years ago
Updated by okurz almost 6 years ago
merged, need https://github.com/os-autoinst/openQA/pull/1853 now to have the asset download feature working with our o3 and osd setup using caching.
Updated by okurz over 5 years ago
- Blocked by action #43511: [functional][u] Asset cache does not support ASSET_ assets added
Updated by okurz over 5 years ago
- Status changed from Feedback to Blocked
- Target version set to Milestone 25
Updated by okurz over 5 years ago
- Related to action #44327: Trigger tests based on git refspec/branch added
Updated by okurz over 5 years ago
- Copied to action #48644: Dynamic scheduling of small script snippets to execute, e.g. scripts as downloadable assets added
Updated by okurz over 5 years ago
- Status changed from Blocked to Resolved
All the mentioned ACs are fulfilled with either the new SCHEDULE
, INCLUDE_MODULES
, EXCLUDE_MODULES
as well as #44327 to allow to specify any custom git refspec. I still see some value in providing small script snippets without needing a fork of os-autoinst-distri-opensuse so I carved that out into #48644