QA » openQA Project

I think our scenario defintion data structure cannot map the current job group scenarios, because there are identical test names, like microos here:

https://github.com/os-autoinst/opensuse-jobgroups/blob/master/job_groups/opensuse_tumbleweed.yaml#L80-L85

scenarios:
  x86_64:
    microos-*-DVD-x86_64:
      # ...
      - microos:
          machine: uefi
      - microos-sdboot:
          machine: uefi
      - microos:
          machine: 64bit

We are using the test names as mapping keys, and they must be unique:
https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml#L33

job_templates:
  openqa_from_git:
    <<: *common
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: "1"
  openqa_from_containers:
    # ...

It's possible to write a custom constructor in YAML::PP that will collect the key/value pairs as a list of pairs and put it into an array.
But maybe we want to avoid this.
In any case, here's the code for that:

% cat yaml-scenarios.pl

#!/usr/bin/perl
use strict;
use warnings;
use v5.10;

package Templates {
    sub new { bless [], shift }
}

package main {
    use YAML::PP;
    my $yp = YAML::PP->new;
    $yp->schema->add_mapping_resolver(
        tag => '!templates',
        on_create => sub { Templates->new },
        on_data => sub {
            my ($constructor, $data, $list) = @_;
            for (my $i = 0; $i < $#$list; $i+= 2) {
                push @{ $$data }, { $list->[$i] => $list->[$i+1] };
            }
        },
    );
    my $yaml = do { local $/; <DATA> };
    my $data = $yp->load_string($yaml);
    say $yp->dump_string($data);
}

__DATA__
job_templates: !templates
  openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: "1"
  openqa_from_containers:
    settings:
      DESKTOP: minimalx
      LOAD_PYTHON_TEST_MODULES: "0"
      OPENQA_CONTAINERS: "1"
      OPENQA_FROM_GIT: "1"
  openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_BOOTSTRAP: "1"


% perl yaml-scenarios.pl
---
job_templates:
- openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_GIT: '1'
- openqa_from_containers:
    settings:
      DESKTOP: minimalx
      LOAD_PYTHON_TEST_MODULES: '0'
      OPENQA_CONTAINERS: '1'
      OPENQA_FROM_GIT: '1'
- openqa:
    settings:
      DESKTOP: minimalx
      OPENQA_FROM_BOOTSTRAP: '1'

But I also see another problem, even if we fix the above problem in whatever way: Currently the job group templates are grouped by product.
In our structure, we group by test name, and the product has to be repeated everywhere. Even if it is done via aliases, that's annoying.
Looking at the Tumbleweed job group definition, grouping by product makes more sense.

We can easily redefine our scenario schema by using a new version, not breaking any existing files.

I started a script to convert an existing job group template into a scenario file: https://gist.github.com/perlpunk/7510b201543ae9779bd25762fc3048c1

tinita wrote in #note-11:

Here is the script that automates the generation of a scenario file for a job group:
https://gist.github.com/perlpunk/f9022ce90f6dc1507a2b6afe61fcfbd7
Here are the job groups 1 and 50:
https://gist.github.com/perlpunk/86df06418b9a3d71ffa093fc2a4951c2

Comparing them to the existing job group YAML files, I have to say I find the current ones more readable.

What do you find more or less readable?

To my mind the @machine syntax makes many scenarios less noisy.

It would be nice if for example QEMU_VIRTIO_RNG: "0" could be deduplicated, and if this is not currently possible we have a chance to consider that.

We should also add a toplevel default group in order to not repeat the default priority and machines.
The resulting scenario files will be shorter, but maybe not better readable.

The $ notation feels off to me since the motivation to allow scenarios without a testsuite was to define it within the YAML without relying on the database. We don't have that distinction here.

livdywan wrote in #note-12:

The $ notation feels off to me since the motivation to allow scenarios without a testsuite was to define it within the YAML without relying on the database. We don't have that distinction here.

What would you prefer instead?
Just use testsuite: null like we do currently? Or just don't add a $ and silently don't use a predefined testsuite if it isn't there?

$ in my example means: It's a test name that does not refer to a predefined testsuite defined in the toplevel testsuite key in YAML. (I haven't included that in my gist, you can see all files by running the script locally.)
So IMHO we still have that distinction, but instead of database it is a hash with testsuites coming from YAML.

tinita wrote in #note-13:

$ in my example means: It's a test name that does not refer to a predefined testsuite defined in the toplevel testsuite key in YAML. (I haven't included that in my gist, you can see all files by running the script locally.)
So IMHO we still have that distinction, but instead of database it is a hash with testsuites coming from YAML.

I don't see what this is needed for. The test suite is defined in YAML in any case. In other words I would just define all test suites properly.

https://gist.github.com/perlpunk/dae70fcfcb400b79b6e1cb0fe6b53e4a group 1
https://gist.github.com/perlpunk/a08a131d10f2c8fd1dc55a2447805662 group 50

NOTE: I had to update the script again and regenerated the files in the gists.

Here is an example of a predefined testsuite vs. an inline defined one:
https://gist.github.com/perlpunk/a08a131d10f2c8fd1dc55a2447805662#file-scenario-50-yaml-L28-L31

      RAID1_gpt: {}
      $RAID5_gpt:
        settings:
          YAML_SCHEDULE: schedule/yast/opensuse/raid/raid5_gpt.yaml

RAID1_gpt at the same time a test name as well as being predefined here: https://gist.github.com/perlpunk/a08a131d10f2c8fd1dc55a2447805662#file-testsuites-50-yaml-L28
The RAID5_gpt is only a testname.

And here we have an example where a test name microos-wizard-tpm uses a predefined testsuite with a different name:
https://gist.github.com/perlpunk/dae70fcfcb400b79b6e1cb0fe6b53e4a#file-scenario-1-yaml-L59

      $microos-wizard-tpm@uefi:
        testsuite: microos-wizard
        settings:
          ENCRYPT: '1'
          QEMUTPM: 'instance'
      microos-wizard@uefi:
        settings:
          ENCRYPT: '1'

I added two more examples:
https://gist.github.com/perlpunk/e4cbb5881030778a43045e2dcd7e7abe group 35
https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f group 102

I noticed a weirdness here: https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f#file-scenario-102-yaml-L18 and https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f#file-scenario-102-yaml-L25

      microos_installation_autoyast: &dvd_autoyast_test
        settings:
          <<:
            <<: *default_settings
            YAML_SCHEDULE: 'schedule/sle-micro/autoyast.yaml'
            # ...

my last comment shows what happens when there are nested merge keys (<<).
In my conversion script I am not enabling merge keys, so that we can actually preserve all of the used aliases, and in the code I didn't handle nested merge keys so far. But that should be possible.
Also we should probably take over any keys starting with ., which is where people define common aliases. Otherwise in the converted YAML they will be defined inline at their first usage, which might make it less readable.

We could talk about the format in our next thursday collab session.

Just dumping ideas from the collab session:

---
products:
  openqa-*-dev-x86_64:
    distri: openqa
    …

machines:
  64bit-2G:
    backend: qemu
    settings:
      WORKER_CLASS: qemu_x86_64

.my_testsuites:
    openqa_from_git: &openqa_from_git
      settings: 
     DESKTOP: minimalx
     OPENQA_FROM_GIT: "1"

.my_shared_settings:
openqa_from_containers_settings:
    settings: &openqa_from_containers_settings
      DESKTOP: minimalx
      OPENQA_CONTAINERS: "1"

testsuites:
  openqa_from_bootstrap:
 settings:
  DESKTOP: minimalx
  OPENQA_FROM_BOOTSTRAP: "1"

job_templates:
  openqa-*-dev-x86_64:
 openqa_from_git: # reuse complete job templates, YAML-only
   *openqa_from_git

  openqa_from_containers: # re-use settings but be able to extend
    settings:
      <<: *openqa_from_containers_settings
      FOO: BAR

  =openqa_from_bootstrap: # our proposal for looking up existing testsuites

  =(openqa_from_bootstrap)_but_different: # proposal for testsuite look-up with custom name for job template with additional settings
    #testsuite: openqa_from_bootstrap  # alternative, explicit way to inherit from the specified test suite
    settings:
      BLA: blub

  openqa-*-dev-aarch64:
      openqa_from_git:
        settings:
          <<: *openqa_from_git

 openqa_from_containers:
   settings:
     <<: *openqa_from_containers_settings
     FOO: bar