action #60329

openQA Tests - action #15132: [epic] Better structure of test plans in main.pm

action #44360: [epic] Parameterize test suites within job groups

action #55730: [epic] Move parameters from test suites into job groups

Use more parameterized job templates for test suites only used once

Added by okurz 3 months ago. Updated 1 day ago.

Status:In ProgressStart date:27/11/2019
Priority:NormalDue date:27/11/2020
Assignee:tinita% Done:

0%

Category:Organisational
Target version:Current Sprint
Difficulty:
Duration: 263

Description

Motivation

See parent #55730 . We want to reduce the number of test suites. This ticket is for trying out the concept for test suites which are only defined in a single place.

Acceptance criteria

  • AC1: Test suites on o3 and osd used in only a single job group are replaced by parameterized job templates in the job groups

test_suite_usage_o3.csv Magnifier (9.41 KB) mkittler, 15/01/2020 11:42 am

History

#1 Updated by okurz 3 months ago

  • Status changed from In Progress to Feedback

Current situation

#55730#note-20 brought me to the idea for this ticket how to go forward. As we currently do not allow in-place definitions of test suites we need to have a test suite defined when we want to use it as a template so on https://openqa.opensuse.org/admin/test_suites I defined a test suite "empty" which no settings (but a description). I tried first with "null" which I later renamed as in our job group YAML document this is not allowed. Then in the openQA job group I defined a new job template with:

@@ -12,3 +12,12 @@
     openqa-*-dev-x86_64:
     - openqa_install+publish
     - openqa_from_git
+    - openqa_from_git_in_place_defined_testsuite_poo55730:
+        testsuite: empty
+        settings:
+          INSTALL: '1'
+          INSTALL_ONLY: '1'
+          OPENQA_FROM_GIT: '1'
+          QEMUCPU: host
+          UPDATE: '0'
+          DESCRIPTION: "Maintainer: okurz@suse.de Test for running openQA itself from git. To be used with the openqa distri. Derived from test suite 'openqa_from_git' but fully defined within the job templates based on the test suite 'null'"

Observations / caveats

  1. testsuite: null was not allowed so I renamed the test suite to "empty"
  2. integer values are not allowed so I needed to quote the values for the settings

Next steps

Let's see what happens for the next scheduled openQA products.

#2 Updated by okurz 3 months ago

  • Status changed from Feedback to Blocked

https://openqa.opensuse.org/tests/overview?build=%3ATW.3517&distri=openqa&version=Tumbleweed&groupid=24 was the build that was triggered with the new job https://openqa.opensuse.org/tests/1097122# which is named "empty" due to #59097

After #59097 is done we can move both test suites "openqa_from_git" and "openqa_install+publish" into the job group instead based on "empty".

#3 Updated by okurz 3 months ago

We finished the main part for #59097 so https://openqa.opensuse.org/tests/overview?groupid=24 looks nice, see the second test named openqa_from_git_in_place_defined_testsuite_poo55730" which merely inherits an "empty" test suite. The next requirement I see so that we can get rid of the actual test suite definitions is that we do not want the description to vanish. Originally when I proposed test suite descriptions I suggested just a setting value, not a column in the database. https://openqa.opensuse.org/admin/job_templates/24 shows how we could define this. Then we should be able to use the job template description instead. But we can also make "description" a key of the job template itself. -> #60782

#4 Updated by okurz about 1 month ago

  • Status changed from Blocked to In Progress
  • Assignee changed from okurz to mkittler

work in #60782 has sufficiently progressed so that we can go ahead and define templates based on "empty". Within https://openqa.opensuse.org/admin/job_templates/24 we now define both two tests in-place.

@mkittler over to you as you are currently working on this.

Next steps and ideas:

  • As visible in https://openqa.opensuse.org/admin/job_templates/24 would be good if we can reduce some duplication with inheriting common settings, e.g. with yaml sections referenced
  • Send an email explaining that it is possible to define settings on job template level and give examples how to do that in practice. Attach list with applicable job templates

#5 Updated by tinita about 1 month ago

I reduced the duplication by adding default settings.

#6 Updated by mkittler about 1 month ago

Yes, this ticket is a team effort.

I've created this PR so the syntax for the "description" is documented: https://github.com/os-autoinst/openQA/pull/2663

I also converted the openQA group on o3 and documented the procedure (first example below) and further findings (the other writings below).


Example: Move test suite settings completely into YAML

In this example I am migrating the test suites for openQA-in-openQA test on o3.
Previously the scenarios section of the YAML document looked like this:

scenarios:
  x86_64:
    openqa-*-dev-x86_64:
    - openqa_install+publish
    - openqa_from_git

The referenced test suites openqa_install+publish and openqa_from_git are not used
anywhere else. That makes them perfect candidates for being included directly within
the YAML so the scenarios section can be changed to:

scenarios:
  x86_64:
    openqa-*-dev-x86_64:
    - openqa_install+publish:
        testsuite: 'empty'
        settings:
          INSTALL: '1'
          INSTALL_ONLY: '1'
          PUBLISH_HDD_1: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%.qcow2'
          PUBLISH_PFLASH_VARS: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%-uefi-vars.qcow2'
          QEMUCPU: 'host'
          UPDATE: '0'
        description: >-
          Maintainer: okurz@suse.de Test for installation of openQA itself.
          To be used with "openqa" distri. Publishes an qcow2 image including the openQA installation
          ready to run as an appliance.
    - openqa_from_git:
        testsuite: 'empty'
        settings:
          INSTALL: '1'
          INSTALL_ONLY: '1'
          OPENQA_FROM_GIT: '1'
          QEMUCPU: 'host'
          UPDATE: '0'
        description: >-
          Maintainer: okurz@suse.de Test for running openQA itself from git. To be used with "openqa"
          distri.

The settings and description have just been copied over from the test suites which then
have been removed. Note that the names openqa_install+publish and openqa_from_git
are still present because the structure of the YAML document expects a name at this
level. As we do not want to use any settings from testsuites the assignment testsuite: 'empty'
is used. This refers to an actually existing but empty test suite.

Example: Merging similar test suites

On SUSE's internal openQA instance we have the test suites ext4_textmode and btrfs_textmode.
These could be combined into a test suite textmode_install_only with the common settings:

DESKTOP=textmode
INSTALLONLY=1

Then the YAML could be changed from:

scenarios:
  ppc64le:
    - ext4_textmode:
        machine: ppc64le-spvm

to:

scenarios:
  ppc64le:
    - textmode_install_only:
        machine: ppc64le-spvm
        settings:
          FILESYSTEM: 'ext4'

The same applies to the YAML document where btrfs_textmode is used.

Finding relevant test suites

An SQL query like this can be used to determine groups where a certain test
suite is used:

select
    group_id as job_group_id
from job_templates
where test_suite_id in (select id from test_suites where name =
'ext4_textmode');

The following SQL query shows test suites, their usage and relevant job groups:

select
    test_suite_id,
    (select name from test_suites where test_suites.id = job_templates.test_suite_id) as test_suite_name,
    count(job_templates.id) as test_suite_usage_count,
    array_to_string(array_agg(group_id), ' ') as job_group_id
from job_templates
group by test_suite_id
order by count(job_templates.id) asc, test_suite_name asc;

This reveals that on o3 there are 41 test suites which are only used by one job template and
could therefore be converted like in the examples. I have attached the result of this query.

#7 Updated by mkittler about 1 month ago

  • Assignee changed from mkittler to tinita

We also had a discussion about this in the chat with the outcome that a script for the migration mentioned in the previous comment is required. (On OSD there are over 900 relevant testsuites.) It appeared that @tinita likes to take over so I'm assigning the ticket to her. (If you want me to continue instead please assign the ticket back to me.)

#8 Updated by okurz about 1 month ago

https://github.com/os-autoinst/openQA/pull/2669 is a PR from @tinita to add a script that helps to validate the job templates YAML documents with a CLI.

#10 Updated by tinita about 1 month ago

I created a repo for converting jobtemplates: https://github.com/perlpunk/openqa-convert-jobtemplates

The usage can be found in the README.md.

For now it just creates a new file with the inlined testsuite, validates it and shows a diff.

My suggestion would be to first convert all templates where the testsuite is used as a plain string and does not do any overrides:

- foo
- bar

This is probably the majority, and in this case the conversion isn't that complicated.
I'll add a switch that tells the script to only do this conversion.
After that we can check how many testsuites are left.

How do we find out which templates are updated automatically via git? Then we can exclude them.

#11 Updated by tinita about 1 month ago

My script now has several options.
https://github.com/perlpunk/openqa-convert-jobtemplates

# for automatic conversion of testsuites only used once
jobtemplate-convert.pl --host o3 --update 34 1195 1196
jobtemplate-convert.pl --host o3 --update 34 name1 name2
# for converting a local jobtemplate file
jobtemplate-convert.pl --host o3 /path/to/local/jobtemplate.yaml name1 name2

Even if you specify --update, it will ask you for confirmation to proceed, and first do a preview on the API call.

#12 Updated by tinita 19 days ago

I wonder if it would be useful to be able to use settings from more than one testsuite:

- foo:
  settings:
    <<: [ !testsuite a, !testsuite b ]
    MORE: settings

or the same, more verbose:

- foo:
  settings:
    <<:
    - !testsuite a
    - !testsuite b
    MORE: settings

#13 Updated by okurz 17 days ago

yes, but not as part of this substory :) Feel welcome to provide something in the parent tickets.

#14 Updated by okurz 1 day ago

Is #60329#note-12 covered now with YAML merge keys? In any case, how are we faring regarding AC1?

#15 Updated by tinita 1 day ago

okurz wrote:

Is #60329#note-12 covered now with YAML merge keys?

No, that one is not covered and would need a custom constructor --> future

In any case, how are we faring regarding AC1?

I'm not sure how to proceed with that. The conversion is done for o3, but it seems people don't want that automatic conversion for osd. We can't really judge if a single-used testsuite is supposed to be used by more templates in the future.

Also available in: Atom PDF