https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-04-21T12:40:55ZopenSUSE Project Management ToolopenQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=3995132021-04-21T12:40:55Zhurhajjhura@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/399513/diff?detail_id=379458">diff</a>)</li></ul> openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=3995162021-04-21T12:42:29ZVANASTASIADISvasilios.anastasiadis@suse.com
<ul><li><strong>Category</strong> set to <i>Feature requests</i></li><li><strong>Target version</strong> set to <i>future</i></li></ul> openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=3995252021-04-21T13:17:27Zokurzokurz@suse.com
<ul><li><strong>Parent task</strong> set to <i>#88229</i></li></ul> openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=3995282021-04-21T13:19:52Zokurzokurz@suse.com
<ul></ul><p>I have linked this ticket to already existing <a class="issue tracker-6 status-1 priority-5 priority-high3 child parent" title="coordination: [epic] Prevent unintended test coverage decrease (New)" href="https://progress.opensuse.org/issues/88229">#88229</a> . I have the feeling that we did not see this need as severly before most of SUSE openQA contributors have selected to use YAML schedule files per openQA scenario in comparison to previously using a shared schedule definition where differences would all be noted down explicitly. IMHO the whole problem of differing coverage was made worse with <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: [functional][y] Use new yaml structure for test suites to define different schedules for differen... (Resolved)" href="https://progress.opensuse.org/issues/54839">#54839</a> which of course helped to mitigate short-term pain because teams "felt" as if they would step less on each other's toes.<br>
Would we need a "tool" to compare coverage in case we would simply use the same schedule definitions by default?</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=3995402021-04-21T14:38:51Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>[tools] Create tool for checking and comparing coverage in openQAs</i> to <i>Easy way to check and compare coverage in multiple openQA instances</i></li></ul><p>As discussed in chat</p>
<p>To be able to proceed we need actual use cases. For example I wonder: What are the actual goals you want to achieve? The template <a href="https://progress.opensuse.org/projects/openqav3/wiki/#Feature-requests" class="external">https://progress.opensuse.org/projects/openqav3/wiki/#Feature-requests</a> should help to fill the necessary details.</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=3996662021-04-22T06:48:45Zhurhajjhura@suse.com
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p>To be able to proceed we need actual use cases. For example I wonder: What are the actual goals you want to achieve?</p>
</blockquote>
<p>From the description:</p>
<blockquote>
<p>The tool should be able to answer two main questions:</p>
<p>What is the coverage / What tests do we run for product X? (i.e. What tests are running on SLE 15 SP2?)<br>
What is the difference in coverage for products X and Y? (i.e. What tests are running on openSUSE Tumbleweed but not on SLE 15 SP2 and vice versa? )<br>
both of these questions were already asked, but it's hard to give quick and precise answer at this moment</p>
</blockquote>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=3996752021-04-22T07:12:01Zokurzokurz@suse.com
<ul></ul><p>I have read the description. But <em>why</em> do you need the coverage? What would you do with this information if you have it?</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=4002632021-04-23T14:57:21Zokurzokurz@suse.com
<ul><li><strong>Project</strong> changed from <i>openQA Project</i> to <i>openQA Tests</i></li><li><strong>Category</strong> changed from <i>Feature requests</i> to <i>Enhancement to existing tests</i></li></ul> openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=4026632021-04-30T07:48:22Ztjyrinki_susetjyrinki+redmine@suse.de
<ul><li><strong>Subject</strong> changed from <i>Easy way to check and compare coverage in multiple openQA instances</i> to <i>[tools] Easy way to check and compare coverage in multiple openQA instances</i></li></ul> openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=4088052021-05-19T10:50:25Zhurhajjhura@suse.com
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p>I have read the description. But <em>why</em> do you need the coverage? What would you do with this information if you have it?</p>
</blockquote>
<p>Mostly for filling the gaps, checking if all is OK during release of new service pack or even whole new SLES. Also there is a possibility that someone will want to use it for whatever statistics they need.</p>
<p>It seems to be very interesting for product owners, but any team in QE could find it useful.</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=4099852021-05-24T21:15:28Zokurzokurz@suse.com
<ul></ul><p>Just to get expectations aligned: SUSE QE Tools has not much experience with the test distribution os-autoinst-distri-opensuse itself and also not expectations regarding something like the expected test coverage data. AFAIK reading data from multiple instances and comparing against each other has never been done so far. What I could think of being possible here is an external script accessing the databases of each instance directly, reading test modules, sorting test modules by DISTRI, FLAVOR, VERSION, ARCH, MACHINE . I assume what we would end up with is a very big document that can be used for reference and searching for individual test modules. This can help to answer a question like "Is module X tested on Y at all". But I think it will not be usable to effectively compare test coverage to find gaps assuming that the actual difference will be very big. This is the reason why I proposed <a class="issue tracker-4 status-1 priority-5 priority-high3 child" title="action: [qe-core] os-autoinst-distri-opensuse YAML schedule file comparison (New)" href="https://progress.opensuse.org/issues/91656">#91656</a> assuming that it's easier to implement, easier to use and more helpful in the long run.</p>
<p>In the meantime maybe also <a href="https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module" class="external">https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module</a> can be helpful to find out in which scenarios specified modules are used.</p>
<p>Regarding a time expectation when we could come to implementing the current ticket my current estimate is in the range of months. See the complete current SUSE QE Tools team backlog under <a href="https://progress.opensuse.org/issues?query_id=230" class="external">https://progress.opensuse.org/issues?query_id=230</a></p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=4102912021-05-25T11:10:09Zhurhajjhura@suse.com
<ul></ul><p>I'm fully aware how difficult this issue is. And my personal expectations don't matter, really, as this is coming more from product owners and I'm here just a middleman who created the ticket. As I mentioned in chat, most interested people in this kind of functionality were Marita, Heiko and Timo. I suggest you talk to each other and align expectations, without me introducing needles noise to the discussion.</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=4103152021-05-25T11:43:38Zokurzokurz@suse.com
<ul></ul><p>I understand. Yes, that makes sense. I guess it's best to do that then in #72877</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=5372352022-07-15T09:54:11Zszarate
<ul></ul><p>These tickets are not on high prio</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=5373072022-07-15T10:07:03Zszarate
<ul><li><strong>Tags</strong> set to <i>bulkupdate</i></li></ul><p>These tickets are not on high pro</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=5374602022-07-15T10:59:39Zszarate
<ul><li><strong>Tags</strong> deleted (<del><i>bulkupdate</i></del>)</li><li><strong>Project</strong> changed from <i>openQA Tests</i> to <i>openQA Project</i></li><li><strong>Category</strong> changed from <i>Enhancement to existing tests</i> to <i>Feature requests</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>Maybe reject the ticket based on: <a class="issue tracker-4 status-1 priority-5 priority-high3 child parent" title="action: Easy way to check and compare coverage in multiple openQA instances (New)" href="https://progress.opensuse.org/issues/91509#note-13">#91509#note-13</a>?</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=5397432022-07-21T12:32:35Zokurzokurz@suse.com
<ul></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/23010">@szarate</a> I don't understand why you think that we should reject this topic. The original requirement is still not fulfilled, so why reject?</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=5398242022-07-21T13:47:41Zszarate
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/23010">@szarate</a> I don't understand why you think that we should reject this topic. The original requirement is still not fulfilled, so why reject?</p>
</blockquote>
<p>perhaps I misunderstood your comment, I guess you were referring to Juraj's discussion?</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=5398872022-07-21T17:55:09Zokurzokurz@suse.com
<ul><li><strong>Project</strong> changed from <i>openQA Project</i> to <i>openQA Tests</i></li><li><strong>Subject</strong> changed from <i>[tools] Easy way to check and compare coverage in multiple openQA instances</i> to <i>[qe-core] Easy way to check and compare coverage in multiple openQA instances</i></li><li><strong>Category</strong> changed from <i>Feature requests</i> to <i>Enhancement to existing tests</i></li><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li></ul><p>Well, in any case this isn't done. Moving back to "openQA Tests" as the problem seems to be really self-made due to the decision to have the yaml-based schedule separate per product/version/flavor in os-autoinst-distri-opensuse instead of (my preference) to try to keep the test schedule as much aligned as possible and only apply exceptions, e.g. skip "scc_registration" on openSUSE, skip "steam" on SLE but do the same otherwise for SLE and openSUSE. Also see <a class="issue tracker-4 status-1 priority-5 priority-high3 child parent" title="action: Easy way to check and compare coverage in multiple openQA instances (New)" href="https://progress.opensuse.org/issues/91509#note-4">#91509#note-4</a> where I have explained this.</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=5400072022-07-22T07:21:01Zszarate
<ul><li><strong>Project</strong> changed from <i>openQA Tests</i> to <i>openQA Project</i></li><li><strong>Subject</strong> changed from <i>[qe-core] Easy way to check and compare coverage in multiple openQA instances</i> to <i>Easy way to check and compare coverage in multiple openQA instances</i></li><li><strong>Category</strong> deleted (<del><i>Enhancement to existing tests</i></del>)</li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>Please reject the ticket if the tools team doesn't see a valid request, don't reassign to core, as it goes beyond our scope.</p>
openQA Project - action #91509: Easy way to check and compare coverage in multiple openQA instanceshttps://progress.opensuse.org/issues/91509?journal_id=5410632022-07-27T06:19:02Zokurzokurz@suse.com
<ul><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li></ul><p>Well, I still see this as a valid request though still specific to os-autoinst-distri-opensuse. Let's keep in future for now</p>