QA » openQA Project

I think our scenario defintion data structure cannot map the current job group scenarios, because there are identical test names, like microos here:

https://github.com/os-autoinst/opensuse-jobgroups/blob/master/job_groups/opensuse_tumbleweed.yaml#L80-L85

scenarios:
  x86_64:
    microos-*-DVD-x86_64:
      # ...
      - microos:
          machine: uefi
      - microos-sdboot:
          machine: uefi
      - microos:
          machine: 64bit

We are using the test names as mapping keys, and they must be unique:
https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml#L33

job_templates:
  openqa_from_git:
    <<: *common
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: "1"
  openqa_from_containers:
    # ...

It's possible to write a custom constructor in YAML::PP that will collect the key/value pairs as a list of pairs and put it into an array.
But maybe we want to avoid this.
In any case, here's the code for that:

% cat yaml-scenarios.pl

#!/usr/bin/perl
use strict;
use warnings;
use v5.10;

package Templates {
    sub new { bless [], shift }
}

package main {
    use YAML::PP;
    my $yp = YAML::PP->new;
    $yp->schema->add_mapping_resolver(
        tag => '!templates',
        on_create => sub { Templates->new },
        on_data => sub {
            my ($constructor, $data, $list) = @_;
            for (my $i = 0; $i < $#$list; $i+= 2) {
                push @{ $$data }, { $list->[$i] => $list->[$i+1] };
            }
        },
    );
    my $yaml = do { local $/; <DATA> };
    my $data = $yp->load_string($yaml);
    say $yp->dump_string($data);
}

__DATA__
job_templates: !templates
  openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: "1"
  openqa_from_containers:
    settings:
      DESKTOP: minimalx
      LOAD_PYTHON_TEST_MODULES: "0"
      OPENQA_CONTAINERS: "1"
      OPENQA_FROM_GIT: "1"
  openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_BOOTSTRAP: "1"


% perl yaml-scenarios.pl
---
job_templates:
- openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: '1'
- openqa_from_containers:
    settings:
      DESKTOP: minimalx
      LOAD_PYTHON_TEST_MODULES: '0'
      OPENQA_CONTAINERS: '1'
      OPENQA_FROM_GIT: '1'
- openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_BOOTSTRAP: '1'

But I also see another problem, even if we fix the above problem in whatever way: Currently the job group templates are grouped by product.
In our structure, we group by test name, and the product has to be repeated everywhere. Even if it is done via aliases, that's annoying.
Looking at the Tumbleweed job group definition, grouping by product makes more sense.

We can easily redefine our scenario schema by using a new version, not breaking any existing files.

I started a script to convert an existing job group template into a scenario file: https://gist.github.com/perlpunk/7510b201543ae9779bd25762fc3048c1

tinita wrote in #note-11:

Here is the script that automates the generation of a scenario file for a job group:
https://gist.github.com/perlpunk/f9022ce90f6dc1507a2b6afe61fcfbd7
Here are the job groups 1 and 50:
https://gist.github.com/perlpunk/86df06418b9a3d71ffa093fc2a4951c2

Comparing them to the existing job group YAML files, I have to say I find the current ones more readable.

What do you find more or less readable?

To my mind the @machine syntax makes many scenarios less noisy.

It would be nice if for example QEMU_VIRTIO_RNG: "0" could be deduplicated, and if this is not currently possible we have a chance to consider that.

We should also add a toplevel default group in order to not repeat the default priority and machines.
The resulting scenario files will be shorter, but maybe not better readable.

The $ notation feels off to me since the motivation to allow scenarios without a testsuite was to define it within the YAML without relying on the database. We don't have that distinction here.

livdywan wrote in #note-12:

The $ notation feels off to me since the motivation to allow scenarios without a testsuite was to define it within the YAML without relying on the database. We don't have that distinction here.

What would you prefer instead?
Just use testsuite: null like we do currently? Or just don't add a $ and silently don't use a predefined testsuite if it isn't there?

$ in my example means: It's a test name that does not refer to a predefined testsuite defined in the toplevel testsuite key in YAML. (I haven't included that in my gist, you can see all files by running the script locally.)
So IMHO we still have that distinction, but instead of database it is a hash with testsuites coming from YAML.

tinita wrote in #note-13:

$ in my example means: It's a test name that does not refer to a predefined testsuite defined in the toplevel testsuite key in YAML. (I haven't included that in my gist, you can see all files by running the script locally.)
So IMHO we still have that distinction, but instead of database it is a hash with testsuites coming from YAML.

I don't see what this is needed for. The test suite is defined in YAML in any case. In other words I would just define all test suites properly.

https://gist.github.com/perlpunk/dae70fcfcb400b79b6e1cb0fe6b53e4a group 1
https://gist.github.com/perlpunk/a08a131d10f2c8fd1dc55a2447805662 group 50

NOTE: I had to update the script again and regenerated the files in the gists.

Here is an example of a predefined testsuite vs. an inline defined one:
https://gist.github.com/perlpunk/a08a131d10f2c8fd1dc55a2447805662#file-scenario-50-yaml-L28-L31

      RAID1_gpt: {}
      $RAID5_gpt:
        settings:
          YAML_SCHEDULE: schedule/yast/opensuse/raid/raid5_gpt.yaml

RAID1_gpt at the same time a test name as well as being predefined here: https://gist.github.com/perlpunk/a08a131d10f2c8fd1dc55a2447805662#file-testsuites-50-yaml-L28
The RAID5_gpt is only a testname.

And here we have an example where a test name microos-wizard-tpm uses a predefined testsuite with a different name:
https://gist.github.com/perlpunk/dae70fcfcb400b79b6e1cb0fe6b53e4a#file-scenario-1-yaml-L59

      $microos-wizard-tpm@uefi:
        testsuite: microos-wizard
        settings:
          ENCRYPT: '1'
          QEMUTPM: 'instance'
      microos-wizard@uefi:
        settings:
          ENCRYPT: '1'

I added two more examples:
https://gist.github.com/perlpunk/e4cbb5881030778a43045e2dcd7e7abe group 35
https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f group 102

I noticed a weirdness here: https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f#file-scenario-102-yaml-L18 and https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f#file-scenario-102-yaml-L25

      microos_installation_autoyast: &dvd_autoyast_test
        settings:
          <<:
            <<: *default_settings
            YAML_SCHEDULE: 'schedule/sle-micro/autoyast.yaml'
            # ...

my last comment shows what happens when there are nested merge keys (<<).
In my conversion script I am not enabling merge keys, so that we can actually preserve all of the used aliases, and in the code I didn't handle nested merge keys so far. But that should be possible.
Also we should probably take over any keys starting with ., which is where people define common aliases. Otherwise in the converted YAML they will be defined inline at their first usage, which might make it less readable.

We could talk about the format in our next thursday collab session.

Just dumping ideas from the collab session:

---
products:
  openqa-*-dev-x86_64:
    distri: openqa
    …

machines:
  64bit-2G:
    backend: qemu
    settings:
      WORKER_CLASS: qemu_x86_64

.my_testsuites:
    openqa_from_git: &openqa_from_git
      settings: 
     DESKTOP: minimalx
     OPENQA_FROM_GIT: "1"

.my_shared_settings:
openqa_from_containers_settings:
    settings: &openqa_from_containers_settings
      DESKTOP: minimalx
      OPENQA_CONTAINERS: "1"

testsuites:
  openqa_from_bootstrap:
 settings:
  DESKTOP: minimalx
  OPENQA_FROM_BOOTSTRAP: "1"

job_templates:
  openqa-*-dev-x86_64:
 openqa_from_git: # reuse complete job templates, YAML-only
   *openqa_from_git

  openqa_from_containers: # re-use settings but be able to extend
    settings:
      <<: *openqa_from_containers_settings
      FOO: BAR

  =openqa_from_bootstrap: # our proposal for looking up existing testsuites

  =(openqa_from_bootstrap)_but_different: # proposal for testsuite look-up with custom name for job template with additional settings
    #testsuite: openqa_from_bootstrap  # alternative, explicit way to inherit from the specified test suite
    settings:
      BLA: blub

  openqa-*-dev-aarch64:
      openqa_from_git:
        settings:
          <<: *openqa_from_git

 openqa_from_containers:
   settings:
     <<: *openqa_from_containers_settings
     FOO: bar

I started to write a unittest, but still have to understand it all.
Pushed to my fork: https://gitlab.suse.de/tinita/kernelqa-openqa-yaml/-/tree/test-genconf?ref_type=heads
Apparently there is a mistake in https://gitlab.suse.de/kernel-qa/kernelqa-openqa-yaml/-/blob/master/jobgroup_genconf.md?ref_type=heads

scenarios:
  arch1:
    prod-ver-flavor-arch1:
      include:
      - joblist1
      import:
      - template1
      jobs:
      - testsuite1
    prod-ver-flavor-arch1:
      - testsuite1
      - testsuite2

The import there seems to be misplaced and is ignored, if I understand it correctly.

Summarizing issues and ideas about the data structure¶

Group by architecture and product¶

https://github.com/os-autoinst/opensuse-jobgroups/blob/master/job_groups/opensuse_tumbleweed.yaml#L74

https://gitlab.suse.de/kernel-qa/kernelqa-openqa-yaml/-/blob/master/sle15/filesystems.yaml?ref_type=heads#L35

job_templates:
  arch1:
    prod-ver-flavor-arch1:

Remove redundant architecture in product string¶

job_templates:
  arch1:
    prod-ver-flavor:

Append @machine to test(suite) key¶

https://github.com/os-autoinst/opensuse-jobgroups/blob/master/job_groups/opensuse_tumbleweed.yaml#L76-L85

Job group:

scenarios:
  x86_64:
    microos-*-DVD-x86_64:
      - rcshell:
          machine: 64bit
      - microos_10G-disk
      - microos:
          machine: uefi
      - microos-sdboot:
          machine: uefi
      - microos:
          machine: 64bit
      - microos_textmode

Scenarios:

job_templates:
  x86_64:
    microos-*-DVD:
      - rcshell
      - microos_10G-disk
      - microos@uefi
      - microos-sdboot@uefi
      - microos
      - microos_textmode

No machine will still use the machine defined in the defaults:

defaults:
  x86_64:
    machine: 64bit
    priority: 50

Make list of tests a mapping instead of a sequence¶

With the above we can easily have unique test names as keys:

job_templates:
  x86_64:
    microos-*-DVD:
      rcshell:
      microos_10G-disk:
      microos@uefi:
      microos-sdboot@uefi:
      microos:
      microos_textmode:

Using "imported" testsuite or inline¶

The test(suite) name can refer to a testsuite defined elsewhere or just be a name

Currently:
Job group:

microos-wizard-tpm:
  testsuite: microos-wizard
  machine: uefi
  settings: ...
microos-wizard:
  machine: uefi
  settings: ...

Scenarios:

=(microos-wizard)-tpm@uefi:
  settings: ...
=microos-wizard@uefi:
  settings: ...
some_inline_testsuite:
  settings: ...

This way we get rid of testsuite: something and testsuite: null

Allow to specify settings in the architecture defaults¶

defaults:
  x86_64:
    machine: 64bit
    priority: 50
    settings: ...

I only found this feature in the jobgroup_genconf example though. It seems like an unused feature.

Use the same precedence for default settings as in job group templates¶

Current order is: product machine test_suite job_template

With architecture default settings (see above) this could be: product arch machine test_suite job_template

Loading testsuites, machines, products from other files¶

One way would be to simple allow to specify multiple files.

We would need the following toplevel keys:

defaults:
products:
machines:
testsuites:
scenarios:

They could all be in one file, or spread across multiple files.
For triggering a product one would specify one or more YAML file.
This way one can keep it all in one file if it's simple, and more if necessary, e.g. scenario-tumbleweed.yaml,machines.yaml,products.yaml,testsuites.yaml

This allows to also schedule a product if the scenarios are not (yet) in a git repo. For uploading you can just send all in one file, or concatenate several files with ---.