action #36994

Dynamic test flow definition + override

Added by okurz almost 2 years ago. Updated about 1 year ago.

Status:ResolvedStart date:08/06/2018
Priority:NormalDue date:
Assignee:okurz% Done:

0%

Category:Feature requests
Target version:SUSE QA tests - Milestone 25
Difficulty:
Duration:

Description

Motivation

Linked to UC4 as well as UC7. As a bug assignee or QA contact I want openQA to execute a custom scenario with only limited test modules and optional override test or investigation code to be able to reproduce bugs easily which are most easily or only reproduced in openQA context without needing to adapt the normal validation test code

Acceptance criteria

  • AC1: A list of test modules to be scheduled can be specified on job trigger time overriding main.pm
  • AC2: Override code for test modules can be specified
  • AC3: New test modules can be defined

Suggestions

  • openQA workers download additional test assets if specified by variables to allow injecting new test modules or override modules on-the-fly before isotovideo is called, e.g. using the existing openQA feature of downloading test assets using variables which specify the '_URL' suffix on test parameters.
  • A test schedule can be specified completely by a test variable, e.g. MODULES=boot/boot_to_desktop,qa_automation/patch_and_reboot,my_code, skipping over all other loadtest calls
  • loadtest within main.pm looks in the pool directory if any override exists and loads that with preference over any that is specified in the test distribution. This should also work if the specified module does not even exist in the test distribution

Further details

Keep security considerations in mind, e.g. do not just allow everybody to execute random code from untrusted remote sources. openQA using the _URL test variables already checks against a "whitelist" of trusted sources so maybe we are covered already.


Related issues

Related to openQA Tests - action #15132: [saga][epic] Better structure of test plans in main.pm Blocked 20/11/2018 27/11/2020
Related to openQA Project - action #44327: Trigger tests based on git refspec/branch Resolved 25/11/2018
Blocked by openQA Project - action #43511: [functional][u] Asset cache does not support ASSET_ assets Resolved 07/11/2018
Copied to openQA Project - action #48644: Dynamic scheduling of small script snippets to execute, e... New 08/06/2018

History

#1 Updated by okurz almost 2 years ago

  • Related to action #15132: [saga][epic] Better structure of test plans in main.pm added

#2 Updated by okurz over 1 year ago

  • Status changed from New to In Progress
  • Assignee set to okurz

#4 Updated by okurz over 1 year ago

  • Status changed from In Progress to Resolved

All working as expected. For example one can use a trigger line like

openqa-clone-job --skip-chained-deps --from http://openqa.suse.de 2163336 WORKER_CLASS=qemu_x86_64_sle SCHEDULE=tests/boot/boot_to_desktop,tests/console/consoletest_setup,sysrq ASSET_1_URL=https://w3.nue.suse.com/~okurz/sysrq.pm

where "sysrq.pm" is a file in test module format, e.g. the following content:

use base 'consoletest';
use strict;
use testapi;
use utils;

sub run {
    select_console 'root-console';
    script_run 'dmesg -n 7';
    wait_screen_change { send_key 'alt-sysrq-w' };
    assert_screen 'sysrq-output';
}

sub post_run_hook {}
1;

This could be used e.g. in bug investigations, trying out new test modules which are hard to test locally, etc. I assume documentation is hard to find for people but I guess this is always the case. https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc describes the "SCHEDULE" parameter, https://github.com/os-autoinst/openQA/blob/master/docs/UsersGuide.asciidoc#asset-handling describes how downloadable assets can be specified. The feature works as expected on osd, o3 can work as well but might have the small limitation that currently only the two domains "download.opensuse.org openqa.opensuse.org" are allowed to download from.

#5 Updated by okurz over 1 year ago

  • Status changed from Resolved to Feedback

feedback from sprint review: We should document this on another place describing the use case specifically, could be openQA users guide.

#7 Updated by okurz over 1 year ago

merged, need https://github.com/os-autoinst/openQA/pull/1853 now to have the asset download feature working with our o3 and osd setup using caching.

#8 Updated by okurz about 1 year ago

  • Blocked by action #43511: [functional][u] Asset cache does not support ASSET_ assets added

#9 Updated by okurz about 1 year ago

  • Status changed from Feedback to Blocked
  • Target version set to Milestone 25

#10 Updated by okurz about 1 year ago

  • Related to action #44327: Trigger tests based on git refspec/branch added

#11 Updated by okurz about 1 year ago

  • Copied to action #48644: Dynamic scheduling of small script snippets to execute, e.g. scripts as downloadable assets added

#12 Updated by okurz about 1 year ago

  • Status changed from Blocked to Resolved

All the mentioned ACs are fulfilled with either the new SCHEDULE, INCLUDE_MODULES, EXCLUDE_MODULES as well as #44327 to allow to specify any custom git refspec. I still see some value in providing small script snippets without needing a fork of os-autoinst-distri-opensuse so I carved that out into #48644

Also available in: Atom PDF