|Status:||In Progress||Start date:||27/11/2019|
|Target version:||Current Sprint|
See parent #55730 . We want to reduce the number of test suites. This ticket is for trying out the concept for test suites which are only defined in a single place.
- AC1: Test suites on o3 and osd used in only a single job group are replaced by parameterized job templates in the job groups
- Status changed from In Progress to Feedback
#55730#note-20 brought me to the idea for this ticket how to go forward. As we currently do not allow in-place definitions of test suites we need to have a test suite defined when we want to use it as a template so on https://openqa.opensuse.org/admin/test_suites I defined a test suite "empty" which no settings (but a description). I tried first with "null" which I later renamed as in our job group YAML document this is not allowed. Then in the openQA job group I defined a new job template with:
@@ -12,3 +12,12 @@ openqa-*-dev-x86_64: - openqa_install+publish - openqa_from_git + - openqa_from_git_in_place_defined_testsuite_poo55730: + testsuite: empty + settings: + INSTALL: '1' + INSTALL_ONLY: '1' + OPENQA_FROM_GIT: '1' + QEMUCPU: host + UPDATE: '0' + DESCRIPTION: "Maintainer: firstname.lastname@example.org Test for running openQA itself from git. To be used with the openqa distri. Derived from test suite 'openqa_from_git' but fully defined within the job templates based on the test suite 'null'"
Observations / caveats¶
testsuite: nullwas not allowed so I renamed the test suite to "empty"
- integer values are not allowed so I needed to quote the values for the settings
Let's see what happens for the next scheduled openQA products.
- Status changed from Feedback to Blocked
https://openqa.opensuse.org/tests/overview?build=%3ATW.3517&distri=openqa&version=Tumbleweed&groupid=24 was the build that was triggered with the new job https://openqa.opensuse.org/tests/1097122# which is named "empty" due to #59097
After #59097 is done we can move both test suites "openqa_from_git" and "openqa_install+publish" into the job group instead based on "empty".
We finished the main part for #59097 so https://openqa.opensuse.org/tests/overview?groupid=24 looks nice, see the second test named openqa_from_git_in_place_defined_testsuite_poo55730" which merely inherits an "empty" test suite. The next requirement I see so that we can get rid of the actual test suite definitions is that we do not want the description to vanish. Originally when I proposed test suite descriptions I suggested just a setting value, not a column in the database. https://openqa.opensuse.org/admin/job_templates/24 shows how we could define this. Then we should be able to use the job template description instead. But we can also make "description" a key of the job template itself. -> #60782
#4 Updated by okurz about 1 month ago
- Status changed from Blocked to In Progress
- Assignee changed from okurz to mkittler
work in #60782 has sufficiently progressed so that we can go ahead and define templates based on "empty". Within https://openqa.opensuse.org/admin/job_templates/24 we now define both two tests in-place.
@mkittler over to you as you are currently working on this.
Next steps and ideas:
- As visible in https://openqa.opensuse.org/admin/job_templates/24 would be good if we can reduce some duplication with inheriting common settings, e.g. with yaml sections referenced
- Send an email explaining that it is possible to define settings on job template level and give examples how to do that in practice. Attach list with applicable job templates
#6 Updated by mkittler about 1 month ago
- File test_suite_usage_o3.csv added
Yes, this ticket is a team effort.
I've created this PR so the syntax for the "description" is documented: https://github.com/os-autoinst/openQA/pull/2663
I also converted the openQA group on o3 and documented the procedure (first example below) and further findings (the other writings below).
Example: Move test suite settings completely into YAML¶
In this example I am migrating the test suites for openQA-in-openQA test on o3.
scenarios section of the YAML document looked like this:
scenarios: x86_64: openqa-*-dev-x86_64: - openqa_install+publish - openqa_from_git
The referenced test suites
openqa_from_git are not used
anywhere else. That makes them perfect candidates for being included directly within
the YAML so the
scenarios section can be changed to:
scenarios: x86_64: openqa-*-dev-x86_64: - openqa_install+publish: testsuite: 'empty' settings: INSTALL: '1' INSTALL_ONLY: '1' PUBLISH_HDD_1: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%.qcow2' PUBLISH_PFLASH_VARS: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%-uefi-vars.qcow2' QEMUCPU: 'host' UPDATE: '0' description: >- Maintainer: email@example.com Test for installation of openQA itself. To be used with "openqa" distri. Publishes an qcow2 image including the openQA installation ready to run as an appliance. - openqa_from_git: testsuite: 'empty' settings: INSTALL: '1' INSTALL_ONLY: '1' OPENQA_FROM_GIT: '1' QEMUCPU: 'host' UPDATE: '0' description: >- Maintainer: firstname.lastname@example.org Test for running openQA itself from git. To be used with "openqa" distri.
The settings and description have just been copied over from the test suites which then
have been removed. Note that the names
are still present because the structure of the YAML document expects a name at this
level. As we do not want to use any settings from testsuites the assignment
is used. This refers to an actually existing but empty test suite.
Example: Merging similar test suites¶
On SUSE's internal openQA instance we have the test suites
These could be combined into a test suite
textmode_install_only with the common settings:
Then the YAML could be changed from:
scenarios: ppc64le: - ext4_textmode: machine: ppc64le-spvm
scenarios: ppc64le: - textmode_install_only: machine: ppc64le-spvm settings: FILESYSTEM: 'ext4'
The same applies to the YAML document where
btrfs_textmode is used.
Finding relevant test suites¶
An SQL query like this can be used to determine groups where a certain test
suite is used:
select group_id as job_group_id from job_templates where test_suite_id in (select id from test_suites where name = 'ext4_textmode');
The following SQL query shows test suites, their usage and relevant job groups:
select test_suite_id, (select name from test_suites where test_suites.id = job_templates.test_suite_id) as test_suite_name, count(job_templates.id) as test_suite_usage_count, array_to_string(array_agg(group_id), ' ') as job_group_id from job_templates group by test_suite_id order by count(job_templates.id) asc, test_suite_name asc;
This reveals that on o3 there are 41 test suites which are only used by one job template and
could therefore be converted like in the examples. I have attached the result of this query.
#7 Updated by mkittler about 1 month ago
- Assignee changed from mkittler to tinita
We also had a discussion about this in the chat with the outcome that a script for the migration mentioned in the previous comment is required. (On OSD there are over 900 relevant testsuites.) It appeared that @tinita likes to take over so I'm assigning the ticket to her. (If you want me to continue instead please assign the ticket back to me.)
#10 Updated by tinita about 1 month ago
I created a repo for converting jobtemplates: https://github.com/perlpunk/openqa-convert-jobtemplates
The usage can be found in the README.md.
For now it just creates a new file with the inlined testsuite, validates it and shows a diff.
My suggestion would be to first convert all templates where the testsuite is used as a plain string and does not do any overrides:
- foo - bar
This is probably the majority, and in this case the conversion isn't that complicated.
I'll add a switch that tells the script to only do this conversion.
After that we can check how many testsuites are left.
How do we find out which templates are updated automatically via git? Then we can exclude them.
#11 Updated by tinita about 1 month ago
My script now has several options.
# for automatic conversion of testsuites only used once jobtemplate-convert.pl --host o3 --update 34 1195 1196 jobtemplate-convert.pl --host o3 --update 34 name1 name2 # for converting a local jobtemplate file jobtemplate-convert.pl --host o3 /path/to/local/jobtemplate.yaml name1 name2
Even if you specify
--update, it will ask you for confirmation to proceed, and first do a preview on the API call.
Is #60329#note-12 covered now with YAML merge keys?
No, that one is not covered and would need a custom constructor --> future
In any case, how are we faring regarding AC1?
I'm not sure how to proceed with that. The conversion is done for o3, but it seems people don't want that automatic conversion for osd. We can't really judge if a single-used testsuite is supposed to be used by more templates in the future.