action #60329
closedopenQA Tests - coordination #15132: [saga][epic] Better structure of test plans in main.pm
coordination #44360: [epic] Parameterize test suites within job groups
coordination #55730: [epic] Move parameters from test suites into job groups
Use more parameterized job templates for test suites only used once
Added by okurz over 4 years ago. Updated over 3 years ago.
0%
Description
Motivation¶
See parent #55730 . We want to reduce the number of test suites. This ticket is for trying out the concept for test suites which are only defined in a single place.
Acceptance criteria¶
- AC1: Test suites on o3 and osd used in only a single job group are replaced by parameterized job templates in the job groups
Files
| test_suite_usage_o3.csv (9.41 KB) test_suite_usage_o3.csv | mkittler, 2020-01-15 11:42 |
Updated by okurz over 4 years ago
- Status changed from In Progress to Feedback
Current situation¶
#55730#note-20 brought me to the idea for this ticket how to go forward. As we currently do not allow in-place definitions of test suites we need to have a test suite defined when we want to use it as a template so on https://openqa.opensuse.org/admin/test_suites I defined a test suite "empty" which no settings (but a description). I tried first with "null" which I later renamed as in our job group YAML document this is not allowed. Then in the openQA job group I defined a new job template with:
@@ -12,3 +12,12 @@
openqa-*-dev-x86_64:
- openqa_install+publish
- openqa_from_git
+ - openqa_from_git_in_place_defined_testsuite_poo55730:
+ testsuite: empty
+ settings:
+ INSTALL: '1'
+ INSTALL_ONLY: '1'
+ OPENQA_FROM_GIT: '1'
+ QEMUCPU: host
+ UPDATE: '0'
+ DESCRIPTION: "Maintainer: okurz@suse.de Test for running openQA itself from git. To be used with the openqa distri. Derived from test suite 'openqa_from_git' but fully defined within the job templates based on the test suite 'null'"
Observations / caveats¶
testsuite: nullwas not allowed so I renamed the test suite to "empty"- integer values are not allowed so I needed to quote the values for the settings
Next steps¶
Let's see what happens for the next scheduled openQA products.
Updated by okurz over 4 years ago
- Status changed from Feedback to Blocked
https://openqa.opensuse.org/tests/overview?build=%3ATW.3517&distri=openqa&version=Tumbleweed&groupid=24 was the build that was triggered with the new job https://openqa.opensuse.org/tests/1097122# which is named "empty" due to #59097
After #59097 is done we can move both test suites "openqa_from_git" and "openqa_install+publish" into the job group instead based on "empty".
Updated by okurz over 4 years ago
We finished the main part for #59097 so https://openqa.opensuse.org/tests/overview?groupid=24 looks nice, see the second test named openqa_from_git_in_place_defined_testsuite_poo55730" which merely inherits an "empty" test suite. The next requirement I see so that we can get rid of the actual test suite definitions is that we do not want the description to vanish. Originally when I proposed test suite descriptions I suggested just a setting value, not a column in the database. https://openqa.opensuse.org/admin/job_templates/24 shows how we could define this. Then we should be able to use the job template description instead. But we can also make "description" a key of the job template itself. -> #60782
Updated by okurz over 4 years ago
- Status changed from Blocked to In Progress
- Assignee changed from okurz to mkittler
work in #60782 has sufficiently progressed so that we can go ahead and define templates based on "empty". Within https://openqa.opensuse.org/admin/job_templates/24 we now define both two tests in-place.
@mkittler over to you as you are currently working on this.
Next steps and ideas:
- As visible in https://openqa.opensuse.org/admin/job_templates/24 would be good if we can reduce some duplication with inheriting common settings, e.g. with yaml sections referenced
- Send an email explaining that it is possible to define settings on job template level and give examples how to do that in practice. Attach list with applicable job templates
Updated by tinita over 4 years ago
I reduced the duplication by adding default settings.
Updated by mkittler over 4 years ago
- File test_suite_usage_o3.csv test_suite_usage_o3.csv added
Yes, this ticket is a team effort.
I've created this PR so the syntax for the "description" is documented: https://github.com/os-autoinst/openQA/pull/2663
I also converted the openQA group on o3 and documented the procedure (first example below) and further findings (the other writings below).
Example: Move test suite settings completely into YAML¶
In this example I am migrating the test suites for openQA-in-openQA test on o3.
Previously the scenarios section of the YAML document looked like this:
scenarios:
x86_64:
openqa-*-dev-x86_64:
- openqa_install+publish
- openqa_from_git
The referenced test suites openqa_install+publish and openqa_from_git are not used
anywhere else. That makes them perfect candidates for being included directly within
the YAML so the scenarios section can be changed to:
scenarios:
x86_64:
openqa-*-dev-x86_64:
- openqa_install+publish:
testsuite: 'empty'
settings:
INSTALL: '1'
INSTALL_ONLY: '1'
PUBLISH_HDD_1: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%.qcow2'
PUBLISH_PFLASH_VARS: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%-uefi-vars.qcow2'
QEMUCPU: 'host'
UPDATE: '0'
description: >-
Maintainer: okurz@suse.de Test for installation of openQA itself.
To be used with "openqa" distri. Publishes an qcow2 image including the openQA installation
ready to run as an appliance.
- openqa_from_git:
testsuite: 'empty'
settings:
INSTALL: '1'
INSTALL_ONLY: '1'
OPENQA_FROM_GIT: '1'
QEMUCPU: 'host'
UPDATE: '0'
description: >-
Maintainer: okurz@suse.de Test for running openQA itself from git. To be used with "openqa"
distri.
The settings and description have just been copied over from the test suites which then
have been removed. Note that the names openqa_install+publish and openqa_from_git
are still present because the structure of the YAML document expects a name at this
level. As we do not want to use any settings from testsuites the assignment testsuite: 'empty'
is used. This refers to an actually existing but empty test suite.
Example: Merging similar test suites¶
On SUSE's internal openQA instance we have the test suites ext4_textmode and btrfs_textmode.
These could be combined into a test suite textmode_install_only with the common settings:
DESKTOP=textmode
INSTALLONLY=1
Then the YAML could be changed from:
scenarios:
ppc64le:
- ext4_textmode:
machine: ppc64le-spvm
to:
scenarios:
ppc64le:
- textmode_install_only:
machine: ppc64le-spvm
settings:
FILESYSTEM: 'ext4'
The same applies to the YAML document where btrfs_textmode is used.
Finding relevant test suites¶
An SQL query like this can be used to determine groups where a certain test
suite is used:
select
group_id as job_group_id
from job_templates
where test_suite_id in (select id from test_suites where name =
'ext4_textmode');
The following SQL query shows test suites, their usage and relevant job groups:
select
test_suite_id,
(select name from test_suites where test_suites.id = job_templates.test_suite_id) as test_suite_name,
count(job_templates.id) as test_suite_usage_count,
array_to_string(array_agg(group_id), ' ') as job_group_id
from job_templates
group by test_suite_id
order by count(job_templates.id) asc, test_suite_name asc;
This reveals that on o3 there are 41 test suites which are only used by one job template and
could therefore be converted like in the examples. I have attached the result of this query.
Updated by mkittler over 4 years ago
- Assignee changed from mkittler to tinita
We also had a discussion about this in the chat with the outcome that a script for the migration mentioned in the previous comment is required. (On OSD there are over 900 relevant testsuites.) It appeared that @tinita likes to take over so I'm assigning the ticket to her. (If you want me to continue instead please assign the ticket back to me.)
Updated by okurz over 4 years ago
https://github.com/os-autoinst/openQA/pull/2669 is a PR from @tinita to add a script that helps to validate the job templates YAML documents with a CLI.
Updated by tinita over 4 years ago
PR https://github.com/os-autoinst/openQA/pull/2669 was merged
Updated by tinita over 4 years ago
I created a repo for converting jobtemplates: https://github.com/perlpunk/openqa-convert-jobtemplates
The usage can be found in the README.md.
For now it just creates a new file with the inlined testsuite, validates it and shows a diff.
My suggestion would be to first convert all templates where the testsuite is used as a plain string and does not do any overrides:
- foo
- bar
This is probably the majority, and in this case the conversion isn't that complicated.
I'll add a switch that tells the script to only do this conversion.
After that we can check how many testsuites are left.
How do we find out which templates are updated automatically via git? Then we can exclude them.
Updated by tinita over 4 years ago
My script now has several options.
https://github.com/perlpunk/openqa-convert-jobtemplates
# for automatic conversion of testsuites only used once
jobtemplate-convert.pl --host o3 --update 34 1195 1196
jobtemplate-convert.pl --host o3 --update 34 name1 name2
# for converting a local jobtemplate file
jobtemplate-convert.pl --host o3 /path/to/local/jobtemplate.yaml name1 name2
Even if you specify --update, it will ask you for confirmation to proceed, and first do a preview on the API call.
Updated by tinita over 4 years ago
I wonder if it would be useful to be able to use settings from more than one testsuite:
- foo:
settings:
<<: [ !testsuite a, !testsuite b ]
MORE: settings
or the same, more verbose:
- foo:
settings:
<<:
- !testsuite a
- !testsuite b
MORE: settings
Updated by okurz about 4 years ago
yes, but not as part of this substory :) Feel welcome to provide something in the parent tickets.
Updated by okurz about 4 years ago
Is #60329#note-12 covered now with YAML merge keys? In any case, how are we faring regarding AC1?
Updated by tinita about 4 years ago
okurz wrote:
Is #60329#note-12 covered now with YAML merge keys?
No, that one is not covered and would need a custom constructor --> future
In any case, how are we faring regarding AC1?
I'm not sure how to proceed with that. The conversion is done for o3, but it seems people don't want that automatic conversion for osd. We can't really judge if a single-used testsuite is supposed to be used by more templates in the future.
Updated by tinita about 4 years ago
- Status changed from In Progress to Resolved
The script to inline testsuites https://github.com/perlpunk/openqa-convert-jobtemplates is ready to use and was already successfully used, as far as I know.
We decided that automatically inlining all single testsuites is not what we want.
Some testuites might be useful to use elsewhere in the future.
Also the jobtemplate files could grow quite large.
So if a testsuite should be inlined should rather be decided case by case.
Since our plan is to restructure how machines/testsuites/products are created in general, I consider this issue as resolved.
Updated by riafarov almost 4 years ago
Hi all! So, after looking more closely to our job group, I would like provide feedback from our side.
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.
Second thing is that job group yaml files are already quite big, and for the test suites which are executed on different backends and archs, we can define it for one and reference it. However, it will be slightly confusing as now we will have to search where reference is defined, etc.
I had an idea of creating separate file for those definitions and import them. It means that we will build custom script which build yaml from multiple files and then does deployment. As we are not the only team here, I would like to discuss it in a bigger scope before we proceed with any implementation.
I also have to admit, that the main motivation for us, as openQA users is that we can distinguish between environment related settings, like NUMDISKS, or HDDMODEL and test related like FILESYSTEM or DESKTOP. And as things are currently, just moving all settings to the job group won't bring much benefits, as well as affect some of the currently working flows.
Updated by livdywan almost 4 years ago
riafarov wrote:
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.
Are you aware of the description key? It was introduced some time ago.
Updated by okurz almost 4 years ago
riafarov wrote:
Hi all! So, after looking more closely to our job group, I would like provide feedback from our side.
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.
Definitely you should not loose descriptions. There is a key "description" in job templates that you can use if you want to move test suites to parameterized job templates. The conversion script https://github.com/perlpunk/openqa-convert-jobtemplates also supports moving descriptions the same, see : https://github.com/perlpunk/openqa-convert-jobtemplates/blob/master/t/data/demo-template.yaml.expected1#L10
Second thing is that job group yaml files are already quite big, and for the test suites which are executed on different backends and archs, we can define it for one and reference it. However, it will be slightly confusing as now we will have to search where reference is defined, etc.
I had an idea of creating separate file for those definitions and import them. It means that we will build custom script which build yaml from multiple files and then does deployment. As we are not the only team here, I would like to discuss it in a bigger scope before we proceed with any implementation.
I also have to admit, that the main motivation for us, as openQA users is that we can distinguish between environment related settings, likeNUMDISKS, orHDDMODELand test related likeFILESYSTEMorDESKTOP. And as things are currently, just moving all settings to the job group won't bring much benefits, as well as affect some of the currently working flows.
I am impressed that so far the job group YaST in particular looks like a rather clean example to me if you are talking about this job group. I do not see a need to move all settings to the job group. The goal of this ticket was to move only test suites that are used once and only once to the job group. I think for example the openQA job group is a good example to show a good use case for this: Only this job group defines test suites to test openQA-in-openQA like this using os-autoinst-distri-openQA so there is no benefit in defining something in a test suite with other test suites that share no relation. Please also see the tickets in the relation tree that we still have open, e.g. there is the saga #15132 still open. #64967 is to get rid of duplicate job template + testsuite definitions for the case of single-place definitions. #54179 is about "Re-use YAML betweens different groups" which I see strongly linked to #58184 for "full version control awareness within openQA". When we have more stored in git it can be easier to share information among the different job groups. https://github.com/os-autoinst/openQA/pull/2706 can give a very rough idea of what I plan: Basically have support to also have everything defined in a single git repo and not have information separated in the openQA database and test distri directory.