Project

General

Profile

Actions
Copy link

action #36994

closed

Dynamic test flow definition + override

Added by okurz over 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Category:
Feature requests
Target version:
SUSE QA (private) - Milestone 25
Start date:
2018-06-08
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Linked to UC4 as well as UC7. As a bug assignee or QA contact I want openQA to execute a custom scenario with only limited test modules and optional override test or investigation code to be able to reproduce bugs easily which are most easily or only reproduced in openQA context without needing to adapt the normal validation test code

Acceptance criteria

  • AC1: A list of test modules to be scheduled can be specified on job trigger time overriding main.pm
  • AC2: Override code for test modules can be specified
  • AC3: New test modules can be defined

Suggestions

  • openQA workers download additional test assets if specified by variables to allow injecting new test modules or override modules on-the-fly before isotovideo is called, e.g. using the existing openQA feature of downloading test assets using variables which specify the '_URL' suffix on test parameters.
  • A test schedule can be specified completely by a test variable, e.g. MODULES=boot/boot_to_desktop,qa_automation/patch_and_reboot,my_code, skipping over all other loadtest calls
  • loadtest within main.pm looks in the pool directory if any override exists and loads that with preference over any that is specified in the test distribution. This should also work if the specified module does not even exist in the test distribution

Further details

Keep security considerations in mind, e.g. do not just allow everybody to execute random code from untrusted remote sources. openQA using the _URL test variables already checks against a "whitelist" of trusted sources so maybe we are covered already.

Related issues 4 (2 open — 2 closed)

Related to openQA Tests (public) - coordination #15132: [saga][epic] Better structure of test plans in main.pmBlockedokurz2018-11-20

Actions
Related to openQA Project (public) - action #44327: Trigger tests based on git refspec/branchResolvedokurz2018-11-25

Actions
Blocked by openQA Project (public) - action #43511: [functional][u] Asset cache does not support ASSET_ assetsResolvedszarate2018-11-07

Actions
Copied to openQA Project (public) - action #48644: Dynamic scheduling of small script snippets to execute, e.g. scripts as downloadable assetsNew2018-06-08

Actions
Actions
Copy link
 #1

Updated by okurz over 6 years ago

Actions
Copy link
 #2

Updated by okurz over 6 years ago

  • Status changed from New to In Progress
  • Assignee set to okurz
Actions
Copy link
 #4

Updated by okurz about 6 years ago

  • Status changed from In Progress to Resolved

All working as expected. For example one can use a trigger line like

openqa-clone-job --skip-chained-deps --from http://openqa.suse.de 2163336 WORKER_CLASS=qemu_x86_64_sle SCHEDULE=tests/boot/boot_to_desktop,tests/console/consoletest_setup,sysrq ASSET_1_URL=https://w3.nue.suse.com/~okurz/sysrq.pm

where "sysrq.pm" is a file in test module format, e.g. the following content:

use base 'consoletest';
use strict;
use testapi;
use utils;

sub run {
    select_console 'root-console';
    script_run 'dmesg -n 7';
    wait_screen_change { send_key 'alt-sysrq-w' };
    assert_screen 'sysrq-output';
}

sub post_run_hook {}
1;

This could be used e.g. in bug investigations, trying out new test modules which are hard to test locally, etc. I assume documentation is hard to find for people but I guess this is always the case. https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc describes the "SCHEDULE" parameter, https://github.com/os-autoinst/openQA/blob/master/docs/UsersGuide.asciidoc#asset-handling describes how downloadable assets can be specified. The feature works as expected on osd, o3 can work as well but might have the small limitation that currently only the two domains "download.opensuse.org openqa.opensuse.org" are allowed to download from.

Actions
Copy link
 #5

Updated by okurz about 6 years ago

  • Status changed from Resolved to Feedback

feedback from sprint review: We should document this on another place describing the use case specifically, could be openQA users guide.

Actions
Copy link
 #7

Updated by okurz about 6 years ago

merged, need https://github.com/os-autoinst/openQA/pull/1853 now to have the asset download feature working with our o3 and osd setup using caching.

Actions
Copy link
 #8

Updated by okurz almost 6 years ago

  • Blocked by action #43511: [functional][u] Asset cache does not support ASSET_ assets added
Actions
Copy link
 #9

Updated by okurz almost 6 years ago

  • Status changed from Feedback to Blocked
  • Target version set to Milestone 25
Actions
Copy link
 #10

Updated by okurz almost 6 years ago

  • Related to action #44327: Trigger tests based on git refspec/branch added
Actions
Copy link
 #11

Updated by okurz almost 6 years ago

  • Copied to action #48644: Dynamic scheduling of small script snippets to execute, e.g. scripts as downloadable assets added
Actions
Copy link
 #12

Updated by okurz almost 6 years ago

  • Status changed from Blocked to Resolved

All the mentioned ACs are fulfilled with either the new SCHEDULE, INCLUDE_MODULES, EXCLUDE_MODULES as well as #44327 to allow to specify any custom git refspec. I still see some value in providing small script snippets without needing a fork of os-autoinst-distri-opensuse so I carved that out into #48644

Actions
Copy link

Also available in: Atom PDF