Project

General

Profile

action #60329

openQA Tests - action #15132: [saga][epic] Better structure of test plans in main.pm

action #44360: [epic] Parameterize test suites within job groups

action #55730: [epic] Move parameters from test suites into job groups

Use more parameterized job templates for test suites only used once

Added by okurz 8 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Organisational
Target version:
Start date:
2019-11-27
Due date:
2020-11-27
% Done:

0%

Estimated time:
Difficulty:
Duration: 263

Description

Motivation

See parent #55730 . We want to reduce the number of test suites. This ticket is for trying out the concept for test suites which are only defined in a single place.

Acceptance criteria

  • AC1: Test suites on o3 and osd used in only a single job group are replaced by parameterized job templates in the job groups
test_suite_usage_o3.csv (9.41 KB) test_suite_usage_o3.csv mkittler, 2020-01-15 11:42

History

#1 Updated by okurz 8 months ago

  • Status changed from In Progress to Feedback

Current situation

#55730#note-20 brought me to the idea for this ticket how to go forward. As we currently do not allow in-place definitions of test suites we need to have a test suite defined when we want to use it as a template so on https://openqa.opensuse.org/admin/test_suites I defined a test suite "empty" which no settings (but a description). I tried first with "null" which I later renamed as in our job group YAML document this is not allowed. Then in the openQA job group I defined a new job template with:

@@ -12,3 +12,12 @@
     openqa-*-dev-x86_64:
     - openqa_install+publish
     - openqa_from_git
+    - openqa_from_git_in_place_defined_testsuite_poo55730:
+        testsuite: empty
+        settings:
+          INSTALL: '1'
+          INSTALL_ONLY: '1'
+          OPENQA_FROM_GIT: '1'
+          QEMUCPU: host
+          UPDATE: '0'
+          DESCRIPTION: "Maintainer: okurz@suse.de Test for running openQA itself from git. To be used with the openqa distri. Derived from test suite 'openqa_from_git' but fully defined within the job templates based on the test suite 'null'"

Observations / caveats

  1. testsuite: null was not allowed so I renamed the test suite to "empty"
  2. integer values are not allowed so I needed to quote the values for the settings

Next steps

Let's see what happens for the next scheduled openQA products.

#2 Updated by okurz 8 months ago

  • Status changed from Feedback to Blocked

https://openqa.opensuse.org/tests/overview?build=%3ATW.3517&distri=openqa&version=Tumbleweed&groupid=24 was the build that was triggered with the new job https://openqa.opensuse.org/tests/1097122# which is named "empty" due to #59097

After #59097 is done we can move both test suites "openqa_from_git" and "openqa_install+publish" into the job group instead based on "empty".

#3 Updated by okurz 7 months ago

We finished the main part for #59097 so https://openqa.opensuse.org/tests/overview?groupid=24 looks nice, see the second test named openqa_from_git_in_place_defined_testsuite_poo55730" which merely inherits an "empty" test suite. The next requirement I see so that we can get rid of the actual test suite definitions is that we do not want the description to vanish. Originally when I proposed test suite descriptions I suggested just a setting value, not a column in the database. https://openqa.opensuse.org/admin/job_templates/24 shows how we could define this. Then we should be able to use the job template description instead. But we can also make "description" a key of the job template itself. -> #60782

#4 Updated by okurz 6 months ago

  • Status changed from Blocked to In Progress
  • Assignee changed from okurz to mkittler

work in #60782 has sufficiently progressed so that we can go ahead and define templates based on "empty". Within https://openqa.opensuse.org/admin/job_templates/24 we now define both two tests in-place.

mkittler over to you as you are currently working on this.

Next steps and ideas:

  • As visible in https://openqa.opensuse.org/admin/job_templates/24 would be good if we can reduce some duplication with inheriting common settings, e.g. with yaml sections referenced
  • Send an email explaining that it is possible to define settings on job template level and give examples how to do that in practice. Attach list with applicable job templates

#5 Updated by tinita 6 months ago

I reduced the duplication by adding default settings.

#6 Updated by mkittler 6 months ago

Yes, this ticket is a team effort.

I've created this PR so the syntax for the "description" is documented: https://github.com/os-autoinst/openQA/pull/2663

I also converted the openQA group on o3 and documented the procedure (first example below) and further findings (the other writings below).


Example: Move test suite settings completely into YAML

In this example I am migrating the test suites for openQA-in-openQA test on o3.
Previously the scenarios section of the YAML document looked like this:

scenarios:
  x86_64:
    openqa-*-dev-x86_64:
    - openqa_install+publish
    - openqa_from_git

The referenced test suites openqa_install+publish and openqa_from_git are not used
anywhere else. That makes them perfect candidates for being included directly within
the YAML so the scenarios section can be changed to:

scenarios:
  x86_64:
    openqa-*-dev-x86_64:
    - openqa_install+publish:
        testsuite: 'empty'
        settings:
          INSTALL: '1'
          INSTALL_ONLY: '1'
          PUBLISH_HDD_1: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%.qcow2'
          PUBLISH_PFLASH_VARS: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%-uefi-vars.qcow2'
          QEMUCPU: 'host'
          UPDATE: '0'
        description: >-
          Maintainer: okurz@suse.de Test for installation of openQA itself.
          To be used with "openqa" distri. Publishes an qcow2 image including the openQA installation
          ready to run as an appliance.
    - openqa_from_git:
        testsuite: 'empty'
        settings:
          INSTALL: '1'
          INSTALL_ONLY: '1'
          OPENQA_FROM_GIT: '1'
          QEMUCPU: 'host'
          UPDATE: '0'
        description: >-
          Maintainer: okurz@suse.de Test for running openQA itself from git. To be used with "openqa"
          distri.

The settings and description have just been copied over from the test suites which then
have been removed. Note that the names openqa_install+publish and openqa_from_git
are still present because the structure of the YAML document expects a name at this
level. As we do not want to use any settings from testsuites the assignment testsuite: 'empty'
is used. This refers to an actually existing but empty test suite.

Example: Merging similar test suites

On SUSE's internal openQA instance we have the test suites ext4_textmode and btrfs_textmode.
These could be combined into a test suite textmode_install_only with the common settings:

DESKTOP=textmode
INSTALLONLY=1

Then the YAML could be changed from:

scenarios:
  ppc64le:
    - ext4_textmode:
        machine: ppc64le-spvm

to:

scenarios:
  ppc64le:
    - textmode_install_only:
        machine: ppc64le-spvm
        settings:
          FILESYSTEM: 'ext4'

The same applies to the YAML document where btrfs_textmode is used.

Finding relevant test suites

An SQL query like this can be used to determine groups where a certain test
suite is used:

select
    group_id as job_group_id
from job_templates
where test_suite_id in (select id from test_suites where name =
'ext4_textmode');

The following SQL query shows test suites, their usage and relevant job groups:

select
    test_suite_id,
    (select name from test_suites where test_suites.id = job_templates.test_suite_id) as test_suite_name,
    count(job_templates.id) as test_suite_usage_count,
    array_to_string(array_agg(group_id), ' ') as job_group_id
from job_templates
group by test_suite_id
order by count(job_templates.id) asc, test_suite_name asc;

This reveals that on o3 there are 41 test suites which are only used by one job template and
could therefore be converted like in the examples. I have attached the result of this query.

#7 Updated by mkittler 6 months ago

  • Assignee changed from mkittler to tinita

We also had a discussion about this in the chat with the outcome that a script for the migration mentioned in the previous comment is required. (On OSD there are over 900 relevant testsuites.) It appeared that tinita likes to take over so I'm assigning the ticket to her. (If you want me to continue instead please assign the ticket back to me.)

#8 Updated by okurz 6 months ago

https://github.com/os-autoinst/openQA/pull/2669 is a PR from tinita to add a script that helps to validate the job templates YAML documents with a CLI.

#10 Updated by tinita 6 months ago

I created a repo for converting jobtemplates: https://github.com/perlpunk/openqa-convert-jobtemplates

The usage can be found in the README.md.

For now it just creates a new file with the inlined testsuite, validates it and shows a diff.

My suggestion would be to first convert all templates where the testsuite is used as a plain string and does not do any overrides:

- foo
- bar

This is probably the majority, and in this case the conversion isn't that complicated.
I'll add a switch that tells the script to only do this conversion.
After that we can check how many testsuites are left.

How do we find out which templates are updated automatically via git? Then we can exclude them.

#11 Updated by tinita 6 months ago

My script now has several options.
https://github.com/perlpunk/openqa-convert-jobtemplates

# for automatic conversion of testsuites only used once
jobtemplate-convert.pl --host o3 --update 34 1195 1196
jobtemplate-convert.pl --host o3 --update 34 name1 name2
# for converting a local jobtemplate file
jobtemplate-convert.pl --host o3 /path/to/local/jobtemplate.yaml name1 name2

Even if you specify --update, it will ask you for confirmation to proceed, and first do a preview on the API call.

#12 Updated by tinita 5 months ago

I wonder if it would be useful to be able to use settings from more than one testsuite:

- foo:
  settings:
    <<: [ !testsuite a, !testsuite b ]
    MORE: settings

or the same, more verbose:

- foo:
  settings:
    <<:
    - !testsuite a
    - !testsuite b
    MORE: settings

#13 Updated by okurz 5 months ago

yes, but not as part of this substory :) Feel welcome to provide something in the parent tickets.

#14 Updated by okurz 5 months ago

Is #60329#note-12 covered now with YAML merge keys? In any case, how are we faring regarding AC1?

#15 Updated by tinita 5 months ago

okurz wrote:

Is #60329#note-12 covered now with YAML merge keys?

No, that one is not covered and would need a custom constructor --> future

In any case, how are we faring regarding AC1?

I'm not sure how to proceed with that. The conversion is done for o3, but it seems people don't want that automatic conversion for osd. We can't really judge if a single-used testsuite is supposed to be used by more templates in the future.

#16 Updated by tinita 4 months ago

  • Status changed from In Progress to Resolved

The script to inline testsuites https://github.com/perlpunk/openqa-convert-jobtemplates is ready to use and was already successfully used, as far as I know.

We decided that automatically inlining all single testsuites is not what we want.
Some testuites might be useful to use elsewhere in the future.
Also the jobtemplate files could grow quite large.

So if a testsuite should be inlined should rather be decided case by case.

Since our plan is to restructure how machines/testsuites/products are created in general, I consider this issue as resolved.

#17 Updated by riafarov 2 months ago

Hi all! So, after looking more closely to our job group, I would like provide feedback from our side.
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.
Second thing is that job group yaml files are already quite big, and for the test suites which are executed on different backends and archs, we can define it for one and reference it. However, it will be slightly confusing as now we will have to search where reference is defined, etc.
I had an idea of creating separate file for those definitions and import them. It means that we will build custom script which build yaml from multiple files and then does deployment. As we are not the only team here, I would like to discuss it in a bigger scope before we proceed with any implementation.
I also have to admit, that the main motivation for us, as openQA users is that we can distinguish between environment related settings, like NUMDISKS, or HDDMODEL and test related like FILESYSTEM or DESKTOP. And as things are currently, just moving all settings to the job group won't bring much benefits, as well as affect some of the currently working flows.

#18 Updated by cdywan 2 months ago

riafarov wrote:

One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.

Are you aware of the description key? It was introduced some time ago.

#19 Updated by okurz about 2 months ago

riafarov wrote:

Hi all! So, after looking more closely to our job group, I would like provide feedback from our side.
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.

Definitely you should not loose descriptions. There is a key "description" in job templates that you can use if you want to move test suites to parameterized job templates. The conversion script https://github.com/perlpunk/openqa-convert-jobtemplates also supports moving descriptions the same, see : https://github.com/perlpunk/openqa-convert-jobtemplates/blob/master/t/data/demo-template.yaml.expected1#L10

Second thing is that job group yaml files are already quite big, and for the test suites which are executed on different backends and archs, we can define it for one and reference it. However, it will be slightly confusing as now we will have to search where reference is defined, etc.
I had an idea of creating separate file for those definitions and import them. It means that we will build custom script which build yaml from multiple files and then does deployment. As we are not the only team here, I would like to discuss it in a bigger scope before we proceed with any implementation.
I also have to admit, that the main motivation for us, as openQA users is that we can distinguish between environment related settings, like NUMDISKS, or HDDMODEL and test related like FILESYSTEM or DESKTOP. And as things are currently, just moving all settings to the job group won't bring much benefits, as well as affect some of the currently working flows.

I am impressed that so far the job group YaST in particular looks like a rather clean example to me if you are talking about this job group. I do not see a need to move all settings to the job group. The goal of this ticket was to move only test suites that are used once and only once to the job group. I think for example the openQA job group is a good example to show a good use case for this: Only this job group defines test suites to test openQA-in-openQA like this using os-autoinst-distri-openQA so there is no benefit in defining something in a test suite with other test suites that share no relation. Please also see the tickets in the relation tree that we still have open, e.g. there is the saga #15132 still open. #64967 is to get rid of duplicate job template + testsuite definitions for the case of single-place definitions. #54179 is about "Re-use YAML betweens different groups" which I see strongly linked to #58184 for "full version control awareness within openQA". When we have more stored in git it can be easier to share information among the different job groups. https://github.com/os-autoinst/openQA/pull/2706 can give a very rough idea of what I plan: Basically have support to also have everything defined in a single git repo and not have information separated in the openQA database and test distri directory.

Also available in: Atom PDF