https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842019-11-27T10:22:07ZopenSUSE Project Management ToolopenQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2601592019-11-27T10:22:07Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><a name="Current-situation"></a>
<h2 >Current situation<a href="#Current-situation" class="wiki-anchor">¶</a></h2>
<p><a class="issue tracker-6 status-3 priority-4 priority-default closed child parent" title="coordination: [epic] Move parameters from test suites into job groups (Resolved)" href="https://progress.opensuse.org/issues/55730#note-20">#55730#note-20</a> brought me to the idea for this ticket how to go forward. As we currently do not allow in-place definitions of test suites we need to have a test suite defined when we want to use it as a template so on <a href="https://openqa.opensuse.org/admin/test_suites" class="external">https://openqa.opensuse.org/admin/test_suites</a> I defined a test suite "empty" which no settings (but a description). I tried first with "null" which I later renamed as in our job group YAML document this is not allowed. Then in the <a href="https://openqa.opensuse.org/admin/job_templates/24" class="external">openQA job group</a> I defined a new job template with:</p>
<pre><code class="patch syntaxhl" data-language="patch"><span class="p">@@ -12,3 +12,12 @@</span>
openqa-*-dev-x86_64:
- openqa_install+publish
- openqa_from_git
<span class="gi">+ - openqa_from_git_in_place_defined_testsuite_poo55730:
+ testsuite: empty
+ settings:
+ INSTALL: '1'
+ INSTALL_ONLY: '1'
+ OPENQA_FROM_GIT: '1'
+ QEMUCPU: host
+ UPDATE: '0'
+ DESCRIPTION: "Maintainer: okurz@suse.de Test for running openQA itself from git. To be used with the openqa distri. Derived from test suite 'openqa_from_git' but fully defined within the job templates based on the test suite 'null'"
</span></code></pre>
<a name="Observations-caveats"></a>
<h2 >Observations / caveats<a href="#Observations-caveats" class="wiki-anchor">¶</a></h2>
<ol>
<li><code>testsuite: null</code> was not allowed so I renamed the test suite to "empty"</li>
<li>integer values are not allowed so I needed to quote the values for the settings</li>
</ol>
<a name="Next-steps"></a>
<h2 >Next steps<a href="#Next-steps" class="wiki-anchor">¶</a></h2>
<p>Let's see what happens for the next scheduled openQA products.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2602012019-11-27T11:57:46Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Blocked</i></li></ul><p><a href="https://openqa.opensuse.org/tests/overview?build=%3ATW.3517&distri=openqa&version=Tumbleweed&groupid=24" class="external">https://openqa.opensuse.org/tests/overview?build=%3ATW.3517&distri=openqa&version=Tumbleweed&groupid=24</a> was the build that was triggered with the new job <a href="https://openqa.opensuse.org/tests/1097122#" class="external">https://openqa.opensuse.org/tests/1097122#</a> which is named "empty" due to <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: The test result overview page display the test suite name not job template name (Resolved)" href="https://progress.opensuse.org/issues/59097">#59097</a></p>
<p>After <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: The test result overview page display the test suite name not job template name (Resolved)" href="https://progress.opensuse.org/issues/59097">#59097</a> is done we can move both test suites "openqa_from_git" and "openqa_install+publish" into the job group instead based on "empty".</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2628382019-12-06T15:31:11Zokurzokurz@suse.com
<ul></ul><p>We finished the main part for <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: The test result overview page display the test suite name not job template name (Resolved)" href="https://progress.opensuse.org/issues/59097">#59097</a> so <a href="https://openqa.opensuse.org/tests/overview?groupid=24" class="external">https://openqa.opensuse.org/tests/overview?groupid=24</a> looks nice, see the second test named openqa_from_git_in_place_defined_testsuite_poo55730" which merely inherits an "empty" test suite. The next requirement I see so that we can get rid of the actual test suite definitions is that we do not want the description to vanish. Originally when I proposed test suite descriptions I suggested just a setting value, not a column in the database. <a href="https://openqa.opensuse.org/admin/job_templates/24" class="external">https://openqa.opensuse.org/admin/job_templates/24</a> shows how we could define this. Then we should be able to use the job template description instead. But we can also make "description" a key of the job template itself. -> <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: descriptions for parameterized job templates independant of test suite descriptions (Resolved)" href="https://progress.opensuse.org/issues/60782">#60782</a></p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2712112020-01-14T15:56:23Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Blocked</i> to <i>In Progress</i></li><li><strong>Assignee</strong> changed from <i>okurz</i> to <i>mkittler</i></li></ul><p>work in <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: descriptions for parameterized job templates independant of test suite descriptions (Resolved)" href="https://progress.opensuse.org/issues/60782">#60782</a> has sufficiently progressed so that we can go ahead and define templates based on "empty". Within <a href="https://openqa.opensuse.org/admin/job_templates/24" class="external">https://openqa.opensuse.org/admin/job_templates/24</a> we now define both two tests in-place.</p>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/22072">@mkittler</a> over to you as you are currently working on this.</p>
<p>Next steps and ideas:</p>
<ul>
<li>As visible in <a href="https://openqa.opensuse.org/admin/job_templates/24" class="external">https://openqa.opensuse.org/admin/job_templates/24</a> would be good if we can reduce some duplication with inheriting common settings, e.g. with yaml sections referenced</li>
<li>Send an email explaining that it is possible to define settings on job template level and give examples how to do that in practice. Attach list with applicable job templates</li>
</ul>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2712232020-01-14T16:09:51Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>I reduced the duplication by adding default settings.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2715052020-01-15T11:42:59Zmkittlermarius.kittler@suse.com
<ul><li><strong>File</strong> <a href="/attachments/9272">test_suite_usage_o3.csv</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/9272/test_suite_usage_o3.csv">test_suite_usage_o3.csv</a> added</li></ul><p>Yes, this ticket is a team effort.</p>
<p>I've created this PR so the syntax for the "description" is documented: <a href="https://github.com/os-autoinst/openQA/pull/2663">https://github.com/os-autoinst/openQA/pull/2663</a></p>
<p>I also converted the openQA group on o3 and documented the procedure (first example below) and further findings (the other writings below).</p>
<hr>
<a name="Example-Move-test-suite-settings-completely-into-YAML"></a>
<h3 >Example: Move test suite settings completely into YAML<a href="#Example-Move-test-suite-settings-completely-into-YAML" class="wiki-anchor">¶</a></h3>
<p>In this example I am migrating the test suites for openQA-in-openQA test on o3.<br>
Previously the <code>scenarios</code> section of the YAML document looked like this:</p>
<pre><code>scenarios:
x86_64:
openqa-*-dev-x86_64:
- openqa_install+publish
- openqa_from_git
</code></pre>
<p>The referenced test suites <code>openqa_install+publish</code> and <code>openqa_from_git</code> are not used<br>
anywhere else. That makes them perfect candidates for being included directly within<br>
the YAML so the <code>scenarios</code> section can be changed to:</p>
<pre><code>scenarios:
x86_64:
openqa-*-dev-x86_64:
- openqa_install+publish:
testsuite: 'empty'
settings:
INSTALL: '1'
INSTALL_ONLY: '1'
PUBLISH_HDD_1: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%.qcow2'
PUBLISH_PFLASH_VARS: 'opensuse-Tumbleweed-%ARCH%@%MACHINE%-%BUILD%-uefi-vars.qcow2'
QEMUCPU: 'host'
UPDATE: '0'
description: >-
Maintainer: okurz@suse.de Test for installation of openQA itself.
To be used with "openqa" distri. Publishes an qcow2 image including the openQA installation
ready to run as an appliance.
- openqa_from_git:
testsuite: 'empty'
settings:
INSTALL: '1'
INSTALL_ONLY: '1'
OPENQA_FROM_GIT: '1'
QEMUCPU: 'host'
UPDATE: '0'
description: >-
Maintainer: okurz@suse.de Test for running openQA itself from git. To be used with "openqa"
distri.
</code></pre>
<p>The settings and description have just been copied over from the test suites which then<br>
have been removed. Note that the names <code>openqa_install+publish</code> and <code>openqa_from_git</code><br>
are still present because the structure of the YAML document expects a name at this<br>
level. As we do not want to use any settings from testsuites the assignment <code>testsuite: 'empty'</code><br>
is used. This refers to an actually existing but empty test suite.</p>
<a name="Example-Merging-similar-test-suites"></a>
<h3 >Example: Merging similar test suites<a href="#Example-Merging-similar-test-suites" class="wiki-anchor">¶</a></h3>
<p>On SUSE's internal openQA instance we have the test suites <code>ext4_textmode</code> and <code>btrfs_textmode</code>.<br>
These could be combined into a test suite <code>textmode_install_only</code> with the common settings:</p>
<pre><code>DESKTOP=textmode
INSTALLONLY=1
</code></pre>
<p>Then the YAML could be changed from:</p>
<pre><code>scenarios:
ppc64le:
- ext4_textmode:
machine: ppc64le-spvm
</code></pre>
<p>to:</p>
<pre><code>scenarios:
ppc64le:
- textmode_install_only:
machine: ppc64le-spvm
settings:
FILESYSTEM: 'ext4'
</code></pre>
<p>The same applies to the YAML document where <code>btrfs_textmode</code> is used.</p>
<a name="Finding-relevant-test-suites"></a>
<h3 >Finding relevant test suites<a href="#Finding-relevant-test-suites" class="wiki-anchor">¶</a></h3>
<p>An SQL query like this can be used to determine groups where a certain test<br>
suite is used:</p>
<pre><code>select
group_id as job_group_id
from job_templates
where test_suite_id in (select id from test_suites where name =
'ext4_textmode');
</code></pre>
<p>The following SQL query shows test suites, their usage and relevant job groups:</p>
<pre><code>select
test_suite_id,
(select name from test_suites where test_suites.id = job_templates.test_suite_id) as test_suite_name,
count(job_templates.id) as test_suite_usage_count,
array_to_string(array_agg(group_id), ' ') as job_group_id
from job_templates
group by test_suite_id
order by count(job_templates.id) asc, test_suite_name asc;
</code></pre>
<p>This reveals that on o3 there are 41 test suites which are only used by one job template and<br>
could therefore be converted like in the examples. I have attached the result of this query.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2715082020-01-15T11:46:47Zmkittlermarius.kittler@suse.com
<ul><li><strong>Assignee</strong> changed from <i>mkittler</i> to <i>tinita</i></li></ul><p>We also had a discussion about this in the chat with the outcome that a script for the migration mentioned in the previous comment is required. (On OSD there are over 900 relevant testsuites.) It appeared that <a class="user active user-mention" href="https://progress.opensuse.org/users/33482">@tinita</a> likes to take over so I'm assigning the ticket to her. (If you want me to continue instead please assign the ticket back to me.)</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2724502020-01-18T08:13:20Zokurzokurz@suse.com
<ul></ul><p><a href="https://github.com/os-autoinst/openQA/pull/2669" class="external">https://github.com/os-autoinst/openQA/pull/2669</a> is a PR from <a class="user active user-mention" href="https://progress.opensuse.org/users/33482">@tinita</a> to add a script that helps to validate the job templates YAML documents with a CLI.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2732092020-01-21T15:37:35Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>PR <a href="https://github.com/os-autoinst/openQA/pull/2669" class="external">https://github.com/os-autoinst/openQA/pull/2669</a> was merged</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2732122020-01-21T15:40:44Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>I created a repo for converting jobtemplates: <a href="https://github.com/perlpunk/openqa-convert-jobtemplates" class="external">https://github.com/perlpunk/openqa-convert-jobtemplates</a></p>
<p>The usage can be found in the README.md.</p>
<p>For now it just creates a new file with the inlined testsuite, validates it and shows a diff.</p>
<p>My suggestion would be to first convert all templates where the testsuite is used as a plain string and does not do any overrides:</p>
<pre><code>- foo
- bar
</code></pre>
<p>This is probably the majority, and in this case the conversion isn't that complicated.<br>
I'll add a switch that tells the script to only do this conversion.<br>
After that we can check how many testsuites are left.</p>
<p>How do we find out which templates are updated automatically via git? Then we can exclude them.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2742412020-01-24T12:48:06Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>My script now has several options.<br>
<a href="https://github.com/perlpunk/openqa-convert-jobtemplates" class="external">https://github.com/perlpunk/openqa-convert-jobtemplates</a></p>
<pre><code># for automatic conversion of testsuites only used once
jobtemplate-convert.pl --host o3 --update 34 1195 1196
jobtemplate-convert.pl --host o3 --update 34 name1 name2
# for converting a local jobtemplate file
jobtemplate-convert.pl --host o3 /path/to/local/jobtemplate.yaml name1 name2
</code></pre>
<p>Even if you specify <code>--update</code>, it will ask you for confirmation to proceed, and first do a preview on the API call.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2774712020-02-07T18:14:27Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>I wonder if it would be useful to be able to use settings from more than one testsuite:</p>
<pre><code>- foo:
settings:
<<: [ !testsuite a, !testsuite b ]
MORE: settings
</code></pre>
<p>or the same, more verbose:</p>
<pre><code>- foo:
settings:
<<:
- !testsuite a
- !testsuite b
MORE: settings
</code></pre> openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2777832020-02-10T13:00:00Zokurzokurz@suse.com
<ul></ul><p>yes, but not as part of this substory :) Feel welcome to provide something in the parent tickets.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2805852020-02-25T11:45:15Zokurzokurz@suse.com
<ul></ul><p>Is <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Use more parameterized job templates for test suites only used once (Resolved)" href="https://progress.opensuse.org/issues/60329#note-12">#60329#note-12</a> covered now with YAML merge keys? In any case, how are we faring regarding AC1?</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2805942020-02-25T12:24:16Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p>Is <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Use more parameterized job templates for test suites only used once (Resolved)" href="https://progress.opensuse.org/issues/60329#note-12">#60329#note-12</a> covered now with YAML merge keys?</p>
</blockquote>
<p>No, that one is not covered and would need a custom constructor --> future</p>
<blockquote>
<p>In any case, how are we faring regarding AC1?</p>
</blockquote>
<p>I'm not sure how to proceed with that. The conversion is done for o3, but it seems people don't want that automatic conversion for osd. We can't really judge if a single-used testsuite is supposed to be used by more templates in the future.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2885042020-03-27T13:30:08Ztinitatina.mueller+trick-redmine@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>The script to inline testsuites <a href="https://github.com/perlpunk/openqa-convert-jobtemplates" class="external">https://github.com/perlpunk/openqa-convert-jobtemplates</a> is ready to use and was already successfully used, as far as I know.</p>
<p>We decided that automatically inlining all single testsuites is not what we want.<br>
Some testuites might be useful to use elsewhere in the future.<br>
Also the jobtemplate files could grow quite large.</p>
<p>So if a testsuite should be inlined should rather be decided case by case.</p>
<p>Since our plan is to restructure how machines/testsuites/products are created in general, I consider this issue as resolved.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2993202020-05-11T15:19:19Zriafarov
<ul></ul><p>Hi all! So, after looking more closely to our job group, I would like provide feedback from our side.<br>
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.<br>
Second thing is that job group yaml files are already quite big, and for the test suites which are executed on different backends and archs, we can define it for one and reference it. However, it will be slightly confusing as now we will have to search where reference is defined, etc.<br>
I had an idea of creating separate file for those definitions and import them. It means that we will build custom script which build yaml from multiple files and then does deployment. As we are not the only team here, I would like to discuss it in a bigger scope before we proceed with any implementation.<br>
I also have to admit, that the main motivation for us, as openQA users is that we can distinguish between environment related settings, like <code>NUMDISKS</code>, or <code>HDDMODEL</code> and test related like <code>FILESYSTEM</code> or <code>DESKTOP</code>. And as things are currently, just moving all settings to the job group won't bring much benefits, as well as affect some of the currently working flows.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2993232020-05-11T15:50:18Zlivdywanliv.dywan@suse.com
<ul></ul><p>riafarov wrote:</p>
<blockquote>
<p>One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.</p>
</blockquote>
<p>Are you aware of the <code>description</code> key? It was introduced some time ago.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=2993592020-05-12T05:22:22Zokurzokurz@suse.com
<ul></ul><p>riafarov wrote:</p>
<blockquote>
<p>Hi all! So, after looking more closely to our job group, I would like provide feedback from our side.<br>
One of the blocking issues we see is that with removal of the test suite in openQA, we won't see descriptions for those in the job group, and as there is not commonly accepted solution for storing it, we might want get some solution and e.g. allow setting it in the job group yaml with special key.</p>
</blockquote>
<p>Definitely you should not loose descriptions. There is a key "description" in job templates that you can use if you want to move test suites to parameterized job templates. The conversion script <a href="https://github.com/perlpunk/openqa-convert-jobtemplates">https://github.com/perlpunk/openqa-convert-jobtemplates</a> also supports moving descriptions the same, see : <a href="https://github.com/perlpunk/openqa-convert-jobtemplates/blob/master/t/data/demo-template.yaml.expected1#L10">https://github.com/perlpunk/openqa-convert-jobtemplates/blob/master/t/data/demo-template.yaml.expected1#L10</a></p>
<blockquote>
<p>Second thing is that job group yaml files are already quite big, and for the test suites which are executed on different backends and archs, we can define it for one and reference it. However, it will be slightly confusing as now we will have to search where reference is defined, etc.<br>
I had an idea of creating separate file for those definitions and import them. It means that we will build custom script which build yaml from multiple files and then does deployment. As we are not the only team here, I would like to discuss it in a bigger scope before we proceed with any implementation.<br>
I also have to admit, that the main motivation for us, as openQA users is that we can distinguish between environment related settings, like <code>NUMDISKS</code>, or <code>HDDMODEL</code> and test related like <code>FILESYSTEM</code> or <code>DESKTOP</code>. And as things are currently, just moving all settings to the job group won't bring much benefits, as well as affect some of the currently working flows.</p>
</blockquote>
<p>I am impressed that so far the job group <a href="https://openqa.suse.de/admin/job_templates/129" class="external">YaST</a> in particular looks like a rather clean example to me if you are talking about this job group. I do not see a need to move <em>all settings</em> to the job group. The goal of this ticket was to move only test suites that are used once and only once to the job group. I think for example the <a href="https://openqa.opensuse.org/admin/job_templates/24" class="external">openQA</a> job group is a good example to show a good use case for this: Only this job group defines test suites to test openQA-in-openQA like this using os-autoinst-distri-openQA so there is no benefit in defining something in a test suite with other test suites that share no relation. Please also see the tickets in the relation tree that we still have open, e.g. there is the saga <a class="issue tracker-6 status-15 priority-3 priority-lowest parent" title="coordination: [saga][epic] Better structure of test plans in main.pm (Blocked)" href="https://progress.opensuse.org/issues/15132">#15132</a> still open. <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: job templates are duplicated as job template in job groups as well as test suites (Resolved)" href="https://progress.opensuse.org/issues/64967">#64967</a> is to get rid of duplicate job template + testsuite definitions for the case of single-place definitions. <a class="issue tracker-4 status-1 priority-3 priority-lowest child" title="action: Re-use YAML betweens different groups (New)" href="https://progress.opensuse.org/issues/54179">#54179</a> is about "Re-use YAML betweens different groups" which I see strongly linked to <a class="issue tracker-6 status-15 priority-4 priority-default parent behind-schedule" title="coordination: [saga][epic][use case] full version control awareness within openQA (Blocked)" href="https://progress.opensuse.org/issues/58184">#58184</a> for "full version control awareness within openQA". When we have more stored in git it can be easier to share information among the different job groups. <a href="https://github.com/os-autoinst/openQA/pull/2706">https://github.com/os-autoinst/openQA/pull/2706</a> can give a very rough idea of what I plan: Basically have support to also have everything defined in a single git repo and not have information separated in the openQA database and test distri directory.</p>
openQA Project - action #60329: Use more parameterized job templates for test suites only used oncehttps://progress.opensuse.org/issues/60329?journal_id=3560442020-12-02T14:21:46Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> deleted (<del><i>2020-11-27</i></del>)</li></ul>