action #40382

Make "ignored" issues more prominent (was: create new state "ignored")

Added by lnussel over 1 year ago. Updated 2 months ago.

Status:WorkableStart date:29/08/2018
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Feature requests
Target version:Ready
Difficulty:
Duration:

Description

Motivation

It is pretty hard to get an overview of which tests TTM ignores and which ones not, despite all attempts we made so far.

Acceptance criteria

  • AC1: Ignored issues are obvious for test results

Suggestions

  • Add "ignored" property for each issue object in openQA
  • Use the property on carry over to turn failed job into "soft-fail"

Further details

Original proposal by lnussel: So I am wondering whether a new state result "ignored" would help there. Would be pretty clear then that all red tests have to be handled. Only if there are none left (means all passed, softfailed or ignored) a snapshot would be considered good. That condition would then be also visible in openQA (instead of overall state result failed).
The UI would have to gain a button to set a test to ignored. That would replace the current method of adding "@ttm ignore" to a comment and then wait until TTM puts the job into its ignore list.


Related issues

Related to openQA Project - action #39719: [epic] Detect "known failures" and mark jobs as such Blocked 23/05/2018 31/12/2020

History

#1 Updated by coolo over 1 year ago

I disagree - this is not a state. That's more a flag - just like 'important' is.

#2 Updated by coolo over 1 year ago

  • Category set to 124
  • Target version set to Ready

I would like us to promote this to a property of the issue carried over. So we have open issue, closed issues and ignored issues.

If a job has ignored issues, it's result is mapped from failed to softfailed.

#3 Updated by okurz over 1 year ago

  • Subject changed from create new state "ignored" to Make "ignored" issues more prominent (was: create new state "ignored")

What does it mean for single ignored jobs without a issue reference? E.g. the jobs which just have a comment '@ttm ignore' but no bug or ticket link to just ignore the single job? I guess it should still be "failed" as we should not consider the failure reason a "known issue" when we do not have any external reference pointing to that.

#4 Updated by okurz over 1 year ago

  • Related to action #39719: [epic] Detect "known failures" and mark jobs as such added

#5 Updated by coolo over 1 year ago

Right. that's a different use case IMO. For those I expect lnussel to have a fresh memory

#6 Updated by okurz over 1 year ago

My proposal for a fitting definition of softfailed covering "known, ignored issues" as well: https://github.com/os-autoinst/openQA/pull/1837

#7 Updated by lnussel over 1 year ago

I guess manually setting softfailed on some tests make sense, like random failures in chromium, vlc etc. But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.

#8 Updated by lnussel over 1 year ago

wrt ignored without issue reference those are usually ones that are chained. like raid tests. Sometimes it's also just cumbersome to dig out reference numbers.

#9 Updated by okurz over 1 year ago

lnussel wrote:

But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.

So far softfails would only be limited to a single test module and any subsequent test module failing would turn the whole job result into failed and overrule the softfail. However, thinking about the potentially new feature here, I guess it is still up to the reviewer putting the '@ttm ignore' flag if he is fine with all the lost testing coverage which is a consequence of the ignored bug.

#10 Updated by okurz 8 months ago

  • Category changed from 124 to Feature requests

#11 Updated by okurz 2 months ago

  • Description updated (diff)
  • Status changed from New to Workable

With https://github.com/os-autoinst/openQA/pull/1837 and github.com/os-autoinst/os-autoinst/pull/1052 it is possible to overwrite a fail into softfail on the module level. As described in #40382#note-2 and #40382#note-9 the next step would be to automatically transform a failed job labeled with an "ignored issue" as soft-fail, e.g. at the time of carry-over. I would not change a fail to soft-fail for an existing, done job however as it could be seen as confusing if a job any time later changes its result. However changing to soft-fail might come with implications, e.g. so far we do not show "failed modules" for soft-fails AFAIK.

I updated the description with clear acceptance criteria and suggestions. However this only relates to issues which are already marked as "ignored". In general I wonder how we can make it easier to actually mark test failures for ignoring. What I see as a challenge is that test reviewers have a too hard time to find known issues from issue trackers, e.g. bugzilla, and then they mostly only put a comment on individual jobs instead of introducing a "record_soft_failure" or "force_soft_failure" which would detect and mark the issue in a more long-term maintainable way but with the additional effort of needing to put something into the test code. In the end I recommend to focus this ticket here only on how known ignored issues are rendered. All the rest can go into #39719 for better handling of "known issues".

Also available in: Atom PDF