QA » openQA Project

We finished the main part for #59097 so https://openqa.opensuse.org/tests/overview?groupid=24 looks nice, see the second test named openqa_from_git_in_place_defined_testsuite_poo55730" which merely inherits an "empty" test suite. The next requirement I see so that we can get rid of the actual test suite definitions is that we do not want the description to vanish. Originally when I proposed test suite descriptions I suggested just a setting value, not a column in the database. https://openqa.opensuse.org/admin/job_templates/24 shows how we could define this. Then we should be able to use the job template description instead. But we can also make "description" a key of the job template itself. -> #60782

I reduced the duplication by adding default settings.

https://github.com/os-autoinst/openQA/pull/2669 is a PR from @tinita to add a script that helps to validate the job templates YAML documents with a CLI.

PR https://github.com/os-autoinst/openQA/pull/2669 was merged

I created a repo for converting jobtemplates: https://github.com/perlpunk/openqa-convert-jobtemplates

The usage can be found in the README.md.

For now it just creates a new file with the inlined testsuite, validates it and shows a diff.

My suggestion would be to first convert all templates where the testsuite is used as a plain string and does not do any overrides:

- foo
- bar

This is probably the majority, and in this case the conversion isn't that complicated.
I'll add a switch that tells the script to only do this conversion.
After that we can check how many testsuites are left.

How do we find out which templates are updated automatically via git? Then we can exclude them.

My script now has several options.
https://github.com/perlpunk/openqa-convert-jobtemplates

# for automatic conversion of testsuites only used once
jobtemplate-convert.pl --host o3 --update 34 1195 1196
jobtemplate-convert.pl --host o3 --update 34 name1 name2
# for converting a local jobtemplate file
jobtemplate-convert.pl --host o3 /path/to/local/jobtemplate.yaml name1 name2

Even if you specify --update, it will ask you for confirmation to proceed, and first do a preview on the API call.

I wonder if it would be useful to be able to use settings from more than one testsuite:

- foo:
  settings:
    <<: [ !testsuite a, !testsuite b ]
    MORE: settings

or the same, more verbose:

- foo:
  settings:
    <<:
    - !testsuite a
    - !testsuite b
    MORE: settings

yes, but not as part of this substory :) Feel welcome to provide something in the parent tickets.

Is #60329#note-12 covered now with YAML merge keys? In any case, how are we faring regarding AC1?

okurz wrote:

Is #60329#note-12 covered now with YAML merge keys?

No, that one is not covered and would need a custom constructor --> future

In any case, how are we faring regarding AC1?

I'm not sure how to proceed with that. The conversion is done for o3, but it seems people don't want that automatic conversion for osd. We can't really judge if a single-used testsuite is supposed to be used by more templates in the future.

Hi all! So, after looking more closely to our job group, I would like provide feedback from our side.
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.
Second thing is that job group yaml files are already quite big, and for the test suites which are executed on different backends and archs, we can define it for one and reference it. However, it will be slightly confusing as now we will have to search where reference is defined, etc.
I had an idea of creating separate file for those definitions and import them. It means that we will build custom script which build yaml from multiple files and then does deployment. As we are not the only team here, I would like to discuss it in a bigger scope before we proceed with any implementation.
I also have to admit, that the main motivation for us, as openQA users is that we can distinguish between environment related settings, like NUMDISKS, or HDDMODEL and test related like FILESYSTEM or DESKTOP. And as things are currently, just moving all settings to the job group won't bring much benefits, as well as affect some of the currently working flows.

riafarov wrote:

One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.

Are you aware of the description key? It was introduced some time ago.

riafarov wrote:

Hi all! So, after looking more closely to our job group, I would like provide feedback from our side.
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.

Definitely you should not loose descriptions. There is a key "description" in job templates that you can use if you want to move test suites to parameterized job templates. The conversion script https://github.com/perlpunk/openqa-convert-jobtemplates also supports moving descriptions the same, see : https://github.com/perlpunk/openqa-convert-jobtemplates/blob/master/t/data/demo-template.yaml.expected1#L10

Second thing is that job group yaml files are already quite big, and for the test suites which are executed on different backends and archs, we can define it for one and reference it. However, it will be slightly confusing as now we will have to search where reference is defined, etc.
I had an idea of creating separate file for those definitions and import them. It means that we will build custom script which build yaml from multiple files and then does deployment. As we are not the only team here, I would like to discuss it in a bigger scope before we proceed with any implementation.
I also have to admit, that the main motivation for us, as openQA users is that we can distinguish between environment related settings, like NUMDISKS, or HDDMODEL and test related like FILESYSTEM or DESKTOP. And as things are currently, just moving all settings to the job group won't bring much benefits, as well as affect some of the currently working flows.

I am impressed that so far the job group YaST in particular looks like a rather clean example to me if you are talking about this job group. I do not see a need to move all settings to the job group. The goal of this ticket was to move only test suites that are used once and only once to the job group. I think for example the openQA job group is a good example to show a good use case for this: Only this job group defines test suites to test openQA-in-openQA like this using os-autoinst-distri-openQA so there is no benefit in defining something in a test suite with other test suites that share no relation. Please also see the tickets in the relation tree that we still have open, e.g. there is the saga #15132 still open. #64967 is to get rid of duplicate job template + testsuite definitions for the case of single-place definitions. #54179 is about "Re-use YAML betweens different groups" which I see strongly linked to #58184 for "full version control awareness within openQA". When we have more stored in git it can be easier to share information among the different job groups. https://github.com/os-autoinst/openQA/pull/2706 can give a very rough idea of what I plan: Basically have support to also have everything defined in a single git repo and not have information separated in the openQA database and test distri directory.