Project

General

Profile

Actions

action #155218

closed

coordination #58184: [saga][epic][use case] full version control awareness within openQA

coordination #154780: [epic] openQA scenario definitions fully in git

[spike][timeboxed:30h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse size:M

Added by okurz 8 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Motivation

With #132335 we showed that its feasible and not problematic to have the complete scenario definitions in the test distribution git repository. A much more complex test distribution is https://github.com/os-autoinst/os-autoinst-distri-opensuse with the separate job group templates in https://github.com/os-autoinst/opensuse-jobgroups/ as well as a corresponding gitlab.suse.de project for openqa.suse.de. We should elaborate what it would mean to use scenario definitions instead of the custom handling of job group templates.

Goals

  • G1: Proof of concept of scenario definitions for os-autoinst-distri-opensuse for multiple or all job group templates
  • G2: Plan for migration
  • G3: We know how to sustainably maintain all job group templates, e.g. including files, YAML anchors, aliases, etc.

Suggestions


Files

scenario-tumbleweed.yaml (4.73 KB) scenario-tumbleweed.yaml tinita, 2024-03-27 17:07

Related issues 3 (0 open3 closed)

Copied from openQA Project - action #132335: In openqa-in-openqa use scenario definitions instead of job group templates size:MResolvedosukup2023-06-01

Actions
Copied to openQA Project - action #160206: Minimum working example for job templates YAML import/include/inherit in kernelqa-openqa-yaml size:SResolvedtinita

Actions
Copied to openQA Project - action #160209: Reduce duplication in machine definitions in https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml#L8, e.g. with re-usable machine "defaults" size:SResolveddheidler

Actions
Actions #1

Updated by okurz 8 months ago

  • Copied from action #132335: In openqa-in-openqa use scenario definitions instead of job group templates size:M added
Actions #2

Updated by okurz 7 months ago

  • Description updated (diff)
  • Target version changed from future to Tools - Next
Actions #3

Updated by okurz 7 months ago

  • Target version changed from Tools - Next to Ready
Actions #4

Updated by okurz 7 months ago

  • Subject changed from [spike][timeboxed:10h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse to [spike][timeboxed:10h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #5

Updated by tinita 7 months ago

  • Description updated (diff)
Actions #6

Updated by tinita 7 months ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita
Actions #7

Updated by tinita 7 months ago · Edited

I think our scenario defintion data structure cannot map the current job group scenarios, because there are identical test names, like microos here:

https://github.com/os-autoinst/opensuse-jobgroups/blob/master/job_groups/opensuse_tumbleweed.yaml#L80-L85

scenarios:
  x86_64:
    microos-*-DVD-x86_64:
      # ...
      - microos:
          machine: uefi
      - microos-sdboot:
          machine: uefi
      - microos:
          machine: 64bit

We are using the test names as mapping keys, and they must be unique:
https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml#L33

job_templates:
  openqa_from_git:
    <<: *common
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: "1"
  openqa_from_containers:
    # ...

It's possible to write a custom constructor in YAML::PP that will collect the key/value pairs as a list of pairs and put it into an array.
But maybe we want to avoid this.
In any case, here's the code for that:

% cat yaml-scenarios.pl

#!/usr/bin/perl
use strict;
use warnings;
use v5.10;

package Templates {
    sub new { bless [], shift }
}

package main {
    use YAML::PP;
    my $yp = YAML::PP->new;
    $yp->schema->add_mapping_resolver(
        tag => '!templates',
        on_create => sub { Templates->new },
        on_data => sub {
            my ($constructor, $data, $list) = @_;
            for (my $i = 0; $i < $#$list; $i+= 2) {
                push @{ $$data }, { $list->[$i] => $list->[$i+1] };
            }
        },
    );
    my $yaml = do { local $/; <DATA> };
    my $data = $yp->load_string($yaml);
    say $yp->dump_string($data);
}

__DATA__
job_templates: !templates
  openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: "1"
  openqa_from_containers:
    settings:
      DESKTOP: minimalx
      LOAD_PYTHON_TEST_MODULES: "0"
      OPENQA_CONTAINERS: "1"
      OPENQA_FROM_GIT: "1"
  openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_BOOTSTRAP: "1"


% perl yaml-scenarios.pl
---
job_templates:
- openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: '1'
- openqa_from_containers:
    settings:
      DESKTOP: minimalx
      LOAD_PYTHON_TEST_MODULES: '0'
      OPENQA_CONTAINERS: '1'
      OPENQA_FROM_GIT: '1'
- openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_BOOTSTRAP: '1'

But I also see another problem, even if we fix the above problem in whatever way: Currently the job group templates are grouped by product.
In our structure, we group by test name, and the product has to be repeated everywhere. Even if it is done via aliases, that's annoying.
Looking at the Tumbleweed job group definition, grouping by product makes more sense.

We can easily redefine our scenario schema by using a new version, not breaking any existing files.

Actions #8

Updated by openqa_review 7 months ago

  • Due date set to 2024-04-09

Setting due date based on mean cycle time of SUSE QE Tools

Actions #9

Updated by tinita 7 months ago · Edited

I converted two products from the tumbleweed job group to a scenario file:
https://github.com/os-autoinst/opensuse-jobgroups/blob/master/job_groups/opensuse_tumbleweed.yaml#L76-L115
See the attached scenario-tumbleweed.yaml.
I scheduled openSUSE-MicroOS-DVD on my local instance with 9 jobs and compared the resulting job settings.

Additionally to the already mentioned issues we need to take into account the precedence handling.
Example: The machine 64bit has HDDSIZEGB: 20.
The testsuite microos_10G-disk has `HDDSIZEGB: 10.
Currently the test suite value 10 overrides the machine value.
In the job I scheduled the value is 20.
These are the relevant parts:

machines:
  64bit:
    backend: qemu
    settings:
      HDDSIZEGB: '20'  # <-- wins
# ...
.testsuites:
  microos_10G-disk: &microos_10G-disk
    HDDSIZEGB: '10'
    HDDSIZEGB_1: '10'
    SYSTEM_ROLE: microos
# ...
job_templates:
  microos_10G-disk:
    product: microos-*-DVD-x86_64
    machine: 64bit
    settings:
      <<: *microos_10G-disk

I'm looking at the code to see why the job template setting doesn't win over the machine setting.

edit: _merge_settings_and_worker_classes is called for product and machine settings, and it's just overriding any existing values.
I think it should check if a setting already exists to get the current behaviour:
OpenQA::JobSettings::generate_settings is processing settings in the following order: product machine test_suite job_template

Actions #10

Updated by tinita 7 months ago

I started a script to convert an existing job group template into a scenario file: https://gist.github.com/perlpunk/7510b201543ae9779bd25762fc3048c1

Actions #11

Updated by tinita 6 months ago

  • Status changed from In Progress to Feedback

Here is the script that automates the generation of a scenario file for a job group:
https://gist.github.com/perlpunk/f9022ce90f6dc1507a2b6afe61fcfbd7
Here are the job groups 1 and 50:
https://gist.github.com/perlpunk/86df06418b9a3d71ffa093fc2a4951c2

Comparing them to the existing job group YAML files, I have to say I find the current ones more readable.
We should also add a toplevel default group in order to not repeat the default priority and machines.
The resulting scenario files will be shorter, but maybe not better readable.

Actions #12

Updated by livdywan 6 months ago · Edited

tinita wrote in #note-11:

Here is the script that automates the generation of a scenario file for a job group:
https://gist.github.com/perlpunk/f9022ce90f6dc1507a2b6afe61fcfbd7
Here are the job groups 1 and 50:
https://gist.github.com/perlpunk/86df06418b9a3d71ffa093fc2a4951c2

Comparing them to the existing job group YAML files, I have to say I find the current ones more readable.

What do you find more or less readable?

To my mind the @machine syntax makes many scenarios less noisy.

It would be nice if for example QEMU_VIRTIO_RNG: "0" could be deduplicated, and if this is not currently possible we have a chance to consider that.

We should also add a toplevel default group in order to not repeat the default priority and machines.
The resulting scenario files will be shorter, but maybe not better readable.

The $ notation feels off to me since the motivation to allow scenarios without a testsuite was to define it within the YAML without relying on the database. We don't have that distinction here.

Actions #13

Updated by tinita 6 months ago

livdywan wrote in #note-12:

The $ notation feels off to me since the motivation to allow scenarios without a testsuite was to define it within the YAML without relying on the database. We don't have that distinction here.

What would you prefer instead?
Just use testsuite: null like we do currently? Or just don't add a $ and silently don't use a predefined testsuite if it isn't there?

$ in my example means: It's a test name that does not refer to a predefined testsuite defined in the toplevel testsuite key in YAML. (I haven't included that in my gist, you can see all files by running the script locally.)
So IMHO we still have that distinction, but instead of database it is a hash with testsuites coming from YAML.

Actions #14

Updated by livdywan 6 months ago

tinita wrote in #note-13:

$ in my example means: It's a test name that does not refer to a predefined testsuite defined in the toplevel testsuite key in YAML. (I haven't included that in my gist, you can see all files by running the script locally.)
So IMHO we still have that distinction, but instead of database it is a hash with testsuites coming from YAML.

I don't see what this is needed for. The test suite is defined in YAML in any case. In other words I would just define all test suites properly.

Actions #15

Updated by tinita 6 months ago

  • Status changed from Feedback to In Progress

I actually need to define the toplevel default, as I forgot to add the defaults to those tests that don't have any values (e.g. the ones that are just a string). Adding all defaults to them would make it really noisy.

I will also gist a complete example, so we can talk about it again in a meeting as there seem to be some misunderstandings.

Actions #17

Updated by tinita 6 months ago · Edited

NOTE: I had to update the script again and regenerated the files in the gists.

Here is an example of a predefined testsuite vs. an inline defined one:
https://gist.github.com/perlpunk/a08a131d10f2c8fd1dc55a2447805662#file-scenario-50-yaml-L28-L31

      RAID1_gpt: {}
      $RAID5_gpt:
        settings:
          YAML_SCHEDULE: schedule/yast/opensuse/raid/raid5_gpt.yaml

RAID1_gpt at the same time a test name as well as being predefined here: https://gist.github.com/perlpunk/a08a131d10f2c8fd1dc55a2447805662#file-testsuites-50-yaml-L28
The RAID5_gpt is only a testname.

And here we have an example where a test name microos-wizard-tpm uses a predefined testsuite with a different name:
https://gist.github.com/perlpunk/dae70fcfcb400b79b6e1cb0fe6b53e4a#file-scenario-1-yaml-L59

      $microos-wizard-tpm@uefi:
        testsuite: microos-wizard
        settings:
          ENCRYPT: '1'
          QEMUTPM: 'instance'
      microos-wizard@uefi:
        settings:
          ENCRYPT: '1'
Actions #18

Updated by tinita 6 months ago · Edited

I added two more examples:
https://gist.github.com/perlpunk/e4cbb5881030778a43045e2dcd7e7abe group 35
https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f group 102

I noticed a weirdness here: https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f#file-scenario-102-yaml-L18 and https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f#file-scenario-102-yaml-L25

      microos_installation_autoyast: &dvd_autoyast_test
        settings:
          <<:
            <<: *default_settings
            YAML_SCHEDULE: 'schedule/sle-micro/autoyast.yaml'
            # ...
Actions #19

Updated by tinita 6 months ago · Edited

my last comment shows what happens when there are nested merge keys (<<).
In my conversion script I am not enabling merge keys, so that we can actually preserve all of the used aliases, and in the code I didn't handle nested merge keys so far. But that should be possible.
Also we should probably take over any keys starting with ., which is where people define common aliases. Otherwise in the converted YAML they will be defined inline at their first usage, which might make it less readable.

We could talk about the format in our next thursday collab session.

Actions #20

Updated by livdywan 6 months ago

  • Due date changed from 2024-04-09 to 2024-04-12

Also we should probably take over any keys starting with ., which is where people define common aliases. Otherwise in the converted YAML they will be defined inline at their first usage, which might make it less readable.

Good catch. I would also vote for supporting them.

We could talk about the format in our next thursday collab session.

Sounds good. Let's plan follow-up tasks there (and hence bumping the due date)

Actions #21

Updated by tinita 6 months ago

  • Status changed from In Progress to Feedback
Actions #22

Updated by tinita 6 months ago

Just dumping ideas from the collab session:

---
products:
  openqa-*-dev-x86_64:
    distri: openqa
    …

machines:
  64bit-2G:
    backend: qemu
    settings:
      WORKER_CLASS: qemu_x86_64

.my_testsuites:
    openqa_from_git: &openqa_from_git
      settings: 
     DESKTOP: minimalx
     OPENQA_FROM_GIT: "1"

.my_shared_settings:
openqa_from_containers_settings:
    settings: &openqa_from_containers_settings
      DESKTOP: minimalx
      OPENQA_CONTAINERS: "1"

testsuites:
  openqa_from_bootstrap:
 settings:
  DESKTOP: minimalx
  OPENQA_FROM_BOOTSTRAP: "1"

job_templates:
  openqa-*-dev-x86_64:
 openqa_from_git: # reuse complete job templates, YAML-only
   *openqa_from_git

  openqa_from_containers: # re-use settings but be able to extend
    settings:
      <<: *openqa_from_containers_settings
      FOO: BAR

  =openqa_from_bootstrap: # our proposal for looking up existing testsuites

  =(openqa_from_bootstrap)_but_different: # proposal for testsuite look-up with custom name for job template with additional settings
    #testsuite: openqa_from_bootstrap  # alternative, explicit way to inherit from the specified test suite
    settings:
      BLA: blub

  openqa-*-dev-aarch64:
      openqa_from_git:
        settings:
          <<: *openqa_from_git

 openqa_from_containers:
   settings:
     <<: *openqa_from_containers_settings
     FOO: bar
Actions #23

Updated by okurz 6 months ago

  • Subject changed from [spike][timeboxed:10h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse size:S to [spike][timeboxed:30h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse size:S
Actions #24

Updated by okurz 6 months ago

  • Subject changed from [spike][timeboxed:30h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse size:S to [spike][timeboxed:30h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse size:M
Actions #25

Updated by livdywan 6 months ago

  • Due date changed from 2024-04-12 to 2024-04-19

I will assume with the re-estimation the due date should have been bumped.

Actions #26

Updated by livdywan 6 months ago

  • Due date changed from 2024-04-19 to 2024-04-26

Not good for metrics, but I guess with events and other tasks this should've been left for next week in the first place.

Actions #27

Updated by livdywan 6 months ago

  • Due date changed from 2024-04-26 to 2024-05-03

Correction. There was another conference this week.

Actions #28

Updated by livdywan 5 months ago

  • Due date changed from 2024-05-03 to 2024-05-10

No further comment

Actions #29

Updated by tinita 5 months ago

  • Status changed from Feedback to In Progress

I'm writing a little unittest for https://gitlab.suse.de/kernel-qa/kernelqa-openqa-yaml/-/blob/master/jobgroup_genconf.py?ref_type=heads in order to get a better unerstanding what it does and ideally have a short usage example of the otherwise easy to confuse import, include and inherit keywords.

Actions #30

Updated by tinita 5 months ago

I started to write a unittest, but still have to understand it all.
Pushed to my fork: https://gitlab.suse.de/tinita/kernelqa-openqa-yaml/-/tree/test-genconf?ref_type=heads
Apparently there is a mistake in https://gitlab.suse.de/kernel-qa/kernelqa-openqa-yaml/-/blob/master/jobgroup_genconf.md?ref_type=heads

scenarios:
  arch1:
    prod-ver-flavor-arch1:
      include:
      - joblist1
      import:
      - template1
      jobs:
      - testsuite1
    prod-ver-flavor-arch1:
      - testsuite1
      - testsuite2

The import there seems to be misplaced and is ignored, if I understand it correctly.

Actions #31

Updated by okurz 5 months ago

  • Copied to action #160206: Minimum working example for job templates YAML import/include/inherit in kernelqa-openqa-yaml size:S added
Actions #32

Updated by okurz 5 months ago

  • Copied to action #160209: Reduce duplication in machine definitions in https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml#L8, e.g. with re-usable machine "defaults" size:S added
Actions #33

Updated by tinita 5 months ago

Summarizing issues and ideas about the data structure

Group by architecture and product

https://github.com/os-autoinst/opensuse-jobgroups/blob/master/job_groups/opensuse_tumbleweed.yaml#L74

https://gitlab.suse.de/kernel-qa/kernelqa-openqa-yaml/-/blob/master/sle15/filesystems.yaml?ref_type=heads#L35

job_templates:
  arch1:
    prod-ver-flavor-arch1:

Remove redundant architecture in product string

job_templates:
  arch1:
    prod-ver-flavor:

Append @machine to test(suite) key

https://github.com/os-autoinst/opensuse-jobgroups/blob/master/job_groups/opensuse_tumbleweed.yaml#L76-L85

Job group:

scenarios:
  x86_64:
    microos-*-DVD-x86_64:
      - rcshell:
          machine: 64bit
      - microos_10G-disk
      - microos:
          machine: uefi
      - microos-sdboot:
          machine: uefi
      - microos:
          machine: 64bit
      - microos_textmode

Scenarios:

job_templates:
  x86_64:
    microos-*-DVD:
      - rcshell
      - microos_10G-disk
      - microos@uefi
      - microos-sdboot@uefi
      - microos
      - microos_textmode

No machine will still use the machine defined in the defaults:

defaults:
  x86_64:
    machine: 64bit
    priority: 50

Make list of tests a mapping instead of a sequence

With the above we can easily have unique test names as keys:

job_templates:
  x86_64:
    microos-*-DVD:
      rcshell:
      microos_10G-disk:
      microos@uefi:
      microos-sdboot@uefi:
      microos:
      microos_textmode:

Using "imported" testsuite or inline

The test(suite) name can refer to a testsuite defined elsewhere or just be a name

Currently:
Job group:

microos-wizard-tpm:
  testsuite: microos-wizard
  machine: uefi
  settings: ...
microos-wizard:
  machine: uefi
  settings: ...

Scenarios:

=(microos-wizard)-tpm@uefi:
  settings: ...
=microos-wizard@uefi:
  settings: ...
some_inline_testsuite:
  settings: ...

This way we get rid of testsuite: something and testsuite: null

Allow to specify settings in the architecture defaults

defaults:
  x86_64:
    machine: 64bit
    priority: 50
    settings: ...

I only found this feature in the jobgroup_genconf example though. It seems like an unused feature.

Use the same precedence for default settings as in job group templates

Current order is: product machine test_suite job_template

With architecture default settings (see above) this could be: product arch machine test_suite job_template

Loading testsuites, machines, products from other files

One way would be to simple allow to specify multiple files.

We would need the following toplevel keys:

defaults:
products:
machines:
testsuites:
scenarios:

They could all be in one file, or spread across multiple files.
For triggering a product one would specify one or more YAML file.
This way one can keep it all in one file if it's simple, and more if necessary, e.g. scenario-tumbleweed.yaml,machines.yaml,products.yaml,testsuites.yaml

This allows to also schedule a product if the scenarios are not (yet) in a git repo. For uploading you can just send all in one file, or concatenate several files with ---.

Actions #34

Updated by tinita 5 months ago

  • Status changed from In Progress to Resolved
Actions #35

Updated by okurz 5 months ago

  • Due date deleted (2024-05-10)
Actions

Also available in: Atom PDF