https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842019-08-13T14:34:34ZopenSUSE Project Management ToolopenQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2348152019-08-13T14:34:34Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>The job group description YAML should support multiple jobs with different variables, not just settings.</i> to <i>The job group YAML schedule should support multiple scenarios with different variables, not just settings</i></li><li><strong>Category</strong> set to <i>Feature requests</i></li></ul> openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2363392019-08-19T12:55:46Zokurzokurz@suse.com
<ul><li><strong>Parent task</strong> set to <i>#44360</i></li></ul> openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2363842019-08-19T13:42:18Zlivdywanliv.dywan@suse.com
<ul><li><strong>Assignee</strong> set to <i>livdywan</i></li></ul> openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2368462019-08-20T15:12:23Zokurzokurz@suse.com
<ul></ul><p>I can imagine the following different ways how the YAML format could look like as examples and what the consequences would be:</p>
<ul>
<li><strong>Option 1</strong>: Override the automatic scenario name with a variable "name"</li>
</ul>
<pre><code>scenarios:
ppc64le:
- gnome
- gnome:
- name: gnome-overrides
- settings:
MR_TEST: overrides
- gnome:
- name: gnome-sapconf
- settings:
MR_TEST: sapconf
</code></pre>
<p>which should end up in three scenarios, "gnome", "gnome-overrides" and "gnome-sapconf".</p>
<ul>
<li><strong>Option 2</strong>: Define testsuites in-place with the name</li>
</ul>
<pre><code>scenarios:
ppc64le:
- gnome
- gnome-overrides:
- settings:
DESKTOP: gnome
MR_TEST: overrides
- gnome-sapconf:
- settings:
DESKTOP: gnome
MR_TEST: sapconf
</code></pre>
<p>this should do the same as <em>Option 1</em>, three scenarios "gnome", "gnome-overrides" and "gnome-sapconf" where only the first maps to an existing test suite, the others are defined in-place. There would be no error-handling in this case about testsuites not being mapped. As an alternative, <strong>Option 2b</strong>, one can make that explicit, e.g. with <code>- testsuite: none</code> as an additional parameter. I like this option because it allows to migrate to a complete test plan definition managed in plain text files that we can eventually store next to the test modules and the test plan in YAML format we already have there.</p>
<ul>
<li><strong>Option 3</strong>: Auto-generated scenario names when ambiguous</li>
</ul>
<pre><code>scenarios:
ppc64le:
- gnome
- gnome:
- settings:
MR_TEST: overrides
- gnome:
- settings:
MR_TEST: sapconf
</code></pre>
<p>ending up in auto-generated names with the testsuite component replaced by "gnome", "gnome-overrides", "gnome-sapconf".</p>
<p>All three options are compatible and could be all allowed in parallel.</p>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2369872019-08-21T05:52:06Zcoolocoolo@suse.com
<ul></ul><p>OK, just for context: the user story of rbranco are 28 test suites with 13 settings, where only 1 is different between the various test suites. The perfect use case for what we did this :)</p>
<p>I kind of like the idea of potentially moving away from predefined test suites at all, <em>BUT</em> that would require in this use case either duplicating 13 settings in every section (and then there would be basically no distinction between test suite settings and parameters anymore) or some kind of inheritance.</p>
<p>Having the name of the job as key in yaml sounds to me the most natural way to express this, but then we need a testsuite setting. Aka option 2+</p>
<pre><code>scenarios:
ppc64le:
- gnome # follows test suite gnome blindly
- gnome-overrides:
- testsuite: gnome # overwrites gnome with a different name
- settings:
MR_TEST: overrides
- gnome: # follows test suite gnome too, but will be named gnome_2 (and the first gnome will be gnome_1)
- settings:
MR_TEST: sapconf
</code></pre> openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2375032019-08-22T15:57:32Zlivdywanliv.dywan@suse.com
<ul></ul><p>coolo wrote:</p>
<blockquote>
<p>I kind of like the idea of potentially moving away from predefined test suites at all, <em>BUT</em> that would require in this use case either duplicating 13 settings in every section (and then there would be basically no distinction between test suite settings and parameters anymore) or some kind of inheritance.</p>
</blockquote>
<p>There is <a class="issue tracker-4 status-1 priority-3 priority-lowest child" title="action: Re-use YAML betweens different groups (New)" href="https://progress.opensuse.org/issues/54179">#54179</a> which proposes sharing the YAML between groups.</p>
<p>Actually I find the idea of inheritance from existing test suites ie. <strong>option 2b</strong> a little more intuitive and straightforward without adding another layer. I'd probably stay away from generating names or even specifying names entirely and just re-use test suites where they are a good basis.<br>
Although <code>test-suite: none</code> sounds a bit hackish... it might be cleaner to just use unique names instead.</p>
<p>I've been pondering if we should have a visualization of the actual scenarios/ dynamic test suites that would make it very clear what's used and what's generated. That would address accidentally creating new test suites when it wasn't intended as well as aliases behaving different to what one was expecting.<br>
Although so far I'm not aware that anyone is asking for this so far. Seeing the build might be good enough.</p>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2375062019-08-22T16:02:49Zcoolocoolo@suse.com
<ul></ul><p>That sounds like a misunderstanding. We for sure don't want to generate test suites from here. Only name jobs</p>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2375152019-08-22T16:33:17Zokurzokurz@suse.com
<ul></ul><p>I think coolo means "scenario" when he says "job", as defined on <a href="http://open.qa/docs/#concepts" class="external">http://open.qa/docs/#concepts</a> . He will probably say that I am wrong because he can't admit that he's wrong ;)</p>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2375272019-08-22T17:44:13Zlivdywanliv.dywan@suse.com
<ul></ul><p>I didn't (consciously) suggest to generate test suites. But I'll try and clarify a bit.</p>
<p>We can generate names, or specify names, and I think both is easy enough in a technical sense. <code>test_suite_HASHED_SETTINGS</code> or some such.</p>
<p>But when you save the YAML, the job templates are updated. These are (currently) a unique combination of a test suite, a product and a machine - a scenario. So at the least we need a new column with the (generated) scenario name. This is where the job eventually comes from. We don't create jobs based on YAML.</p>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2395192019-08-29T13:58:21Zcoolocoolo@suse.com
<ul><li><strong>Target version</strong> set to <i>Current Sprint</i></li></ul><p>we are actually working on it, so move to CS</p>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2408092019-09-03T09:08:54Zokurzokurz@suse.com
<ul></ul><p>As discussed in the weekly jangouts call, we actually agreed on option <strong>2+</strong>. The likely necessary tasks to continue are:</p>
<ul>
<li>Add a column for the test suite name that is defined within the YAML document to the job templates table</li>
<li>Add parsing for that key into the job templates parser</li>
<li>Read out the test suite name from the job templates table when generating jobs</li>
</ul>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2416282019-09-05T23:02:01Zlivdywanliv.dywan@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li></ul> openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2417122019-09-06T10:52:59Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Feedback</i></li></ul><p>Feedback on the feature:</p>
<ul>
<li>The left column in the YAML editor does not seem to document the feature</li>
<li>The schema mentions the new parameter test suite but does not explain that the job template name is generated from the section header of scenarios when the new parameter is used (and does not need to match to an test suite)</li>
</ul>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2417152019-09-06T11:01:43Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>Just wanted to mention that the examples probably have mistakes.</p>
<p>The following YAML</p>
<pre><code> - settings:
MR_TEST: overrides
</code></pre>
<p>is aequivalent to</p>
<pre><code> - settings: null
MR_TEST: overrides
</code></pre>
<p>Probably it was meant to be this?</p>
<pre><code> - settings:
MR_TEST: overrides
</code></pre> openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2421562019-09-09T11:06:42Zlivdywanliv.dywan@suse.com
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p>Feedback on the feature:</p>
<ul>
<li>The left column in the YAML editor does not seem to document the feature</li>
<li>The schema mentions the new parameter test suite but does not explain that the job template name is generated from the section header of scenarios when the new parameter is used (and does not need to match to an test suite)</li>
</ul>
</blockquote>
<p><a href="https://github.com/os-autoinst/openQA/pull/2323" class="external">https://github.com/os-autoinst/openQA/pull/2323</a></p>
openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingshttps://progress.opensuse.org/issues/55454?journal_id=2426002019-09-11T07:00:04Zlivdywanliv.dywan@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul>