https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842018-09-14T12:32:32ZopenSUSE Project Management ToolopenQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1494712018-09-14T12:32:32Zcoolocoolo@suse.com
<ul></ul><p>I disagree - this is not a state. That's more a flag - just like 'important' is. </p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1584742018-10-18T08:58:19Zcoolocoolo@suse.com
<ul><li><strong>Category</strong> set to <i>124</i></li><li><strong>Target version</strong> set to <i>Ready</i></li></ul><p>I would like us to promote this to a property of the issue carried over. So we have open issue, closed issues and ignored issues. </p>
<p>If a job has ignored issues, it's result is mapped from failed to softfailed.</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1585402018-10-18T09:24:34Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>create new state "ignored"</i> to <i>Make "ignored" issues more prominent (was: create new state "ignored")</i></li></ul><p>What does it mean for single ignored jobs without a issue reference? E.g. the jobs which just have a comment '@ttm ignore' but no bug or ticket link to just ignore the single job? I guess it should still be "failed" as we should not consider the failure reason a "known issue" when we do not have any external reference pointing to that.</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1585432018-10-18T09:25:07Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed parent" href="/issues/39719">coordination #39719</a>: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues</i> added</li></ul> openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1585552018-10-18T09:30:22Zcoolocoolo@suse.com
<ul></ul><p>Right. that's a different use case IMO. For those I expect lnussel to have a fresh memory</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1585612018-10-18T09:35:10Zokurzokurz@suse.com
<ul></ul><p>My proposal for a fitting definition of <em>softfailed</em> covering "known, ignored issues" as well: <a href="https://github.com/os-autoinst/openQA/pull/1837" class="external">https://github.com/os-autoinst/openQA/pull/1837</a></p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1594432018-10-23T07:59:56Zlnussellnussel@suse.com
<ul></ul><p>I guess manually setting softfailed on some tests make sense, like random failures in chromium, vlc etc. But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1594462018-10-23T08:01:00Zlnussellnussel@suse.com
<ul></ul><p>wrt ignored without issue reference those are usually ones that are chained. like raid tests. Sometimes it's also just cumbersome to dig out reference numbers.</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=1601062018-10-25T07:37:15Zokurzokurz@suse.com
<ul></ul><p>lnussel wrote:</p>
<blockquote>
<p>But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.</p>
</blockquote>
<p>So far softfails would only be limited to a single test module and any subsequent test module failing would turn the whole job result into failed and overrule the softfail. However, thinking about the potentially new feature here, I guess it is still up to the reviewer putting the '@ttm ignore' flag if he is fine with all the lost testing coverage which is a consequence of the ignored bug.</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=2225062019-06-20T16:02:00Zokurzokurz@suse.com
<ul><li><strong>Category</strong> changed from <i>124</i> to <i>Feature requests</i></li></ul> openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=2635672019-12-10T14:30:26Zokurzokurz@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/263567/diff?detail_id=259748">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li></ul><p>With <a href="https://github.com/os-autoinst/openQA/pull/1837" class="external">https://github.com/os-autoinst/openQA/pull/1837</a> and <a href="https://github.com/os-autoinst/os-autoinst/pull/1052" class="external">https://github.com/os-autoinst/os-autoinst/pull/1052</a> it is possible to overwrite a fail into softfail on the module level. As described in <a class="issue tracker-4 status-12 priority-3 priority-lowest" title="action: Make "ignored" issues more prominent (was: create new state "ignored") (Workable)" href="https://progress.opensuse.org/issues/40382#note-2">#40382#note-2</a> and <a class="issue tracker-4 status-12 priority-3 priority-lowest" title="action: Make "ignored" issues more prominent (was: create new state "ignored") (Workable)" href="https://progress.opensuse.org/issues/40382#note-9">#40382#note-9</a> the next step would be to automatically transform a failed job labeled with an "ignored issue" as soft-fail, e.g. at the time of carry-over. I would not change a fail to soft-fail for an existing, done job however as it could be seen as confusing if a job any time later changes its result. However changing to soft-fail might come with implications, e.g. so far we do not show "failed modules" for soft-fails AFAIK.</p>
<p>I updated the description with clear acceptance criteria and suggestions. However this only relates to issues which are already marked as "ignored". In general I wonder how we can make it easier to actually mark test failures for ignoring. What I see as a challenge is that test reviewers have a too hard time to find known issues from issue trackers, e.g. bugzilla, and then they mostly only put a comment on individual jobs instead of introducing a "record_soft_failure" or "force_soft_failure" which would detect and mark the issue in a more long-term maintainable way but with the additional effort of needing to put something into the test code. In the end I recommend to focus this ticket here <em>only</em> on how known ignored issues are rendered. All the rest can go into <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="coordination: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tr... (Resolved)" href="https://progress.opensuse.org/issues/39719">#39719</a> for better handling of "known issues".</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=3123822020-07-09T11:12:02ZXiaojing_liuxliu1@suse.com
<ul></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/17668">@okurz</a> I am not sure if I understand correctly. Following those comments, I think maybe we could provide a button in UI, only used to change a job from <code>failed</code> to <code>ignored</code>. Or we could provide a function such as <code>record_soft_failure</code> in os-autoinst, but this needs users to modify the test code. Or I completely understand wrong.</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=3146802020-07-25T10:44:38Zokurzokurz@suse.com
<ul></ul><p>Xiaojing_liu wrote:</p>
<blockquote>
<p>Following those comments, I think maybe we could provide a button in UI, only used to change a job from <code>failed</code> to <code>ignored</code></p>
</blockquote>
<p>No button is needed. The Suggestions in #40382#Suggestions are up-to-date and should be followed here.</p>
openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")https://progress.opensuse.org/issues/40382?journal_id=3227232020-09-04T11:51:33Zokurzokurz@suse.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Low</i></li><li><strong>Target version</strong> changed from <i>Ready</i> to <i>future</i></li></ul><p>low team capacity</p>