Project

General

Profile

Actions

action #40382

open

Make "ignored" issues more prominent (was: create new state "ignored")

Added by lnussel over 6 years ago. Updated over 4 years ago.

Status:
Workable
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2018-08-29
Due date:
% Done:

0%

Estimated time:

Description

Motivation

It is pretty hard to get an overview of which tests TTM ignores and which ones not, despite all attempts we made so far.

Acceptance criteria

  • AC1: Ignored issues are obvious for test results

Suggestions

  • Add "ignored" property for each issue object in openQA
  • Use the property on carry over to turn failed job into "soft-fail"

Further details

Original proposal by lnussel: So I am wondering whether a new state result "ignored" would help there. Would be pretty clear then that all red tests have to be handled. Only if there are none left (means all passed, softfailed or ignored) a snapshot would be considered good. That condition would then be also visible in openQA (instead of overall state result failed).
The UI would have to gain a button to set a test to ignored. That would replace the current method of adding "@ttm ignore" to a comment and then wait until TTM puts the job into its ignore list.


Related issues 1 (0 open1 closed)

Related to openQA Project (public) - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issuesResolvedokurz2018-05-23

Actions
Actions #1

Updated by coolo over 6 years ago

I disagree - this is not a state. That's more a flag - just like 'important' is.

Actions #2

Updated by coolo about 6 years ago

  • Category set to 124
  • Target version set to Ready

I would like us to promote this to a property of the issue carried over. So we have open issue, closed issues and ignored issues.

If a job has ignored issues, it's result is mapped from failed to softfailed.

Actions #3

Updated by okurz about 6 years ago

  • Subject changed from create new state "ignored" to Make "ignored" issues more prominent (was: create new state "ignored")

What does it mean for single ignored jobs without a issue reference? E.g. the jobs which just have a comment '@ttm ignore' but no bug or ticket link to just ignore the single job? I guess it should still be "failed" as we should not consider the failure reason a "known issue" when we do not have any external reference pointing to that.

Actions #4

Updated by okurz about 6 years ago

  • Related to coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added
Actions #5

Updated by coolo about 6 years ago

Right. that's a different use case IMO. For those I expect lnussel to have a fresh memory

Actions #6

Updated by okurz about 6 years ago

My proposal for a fitting definition of softfailed covering "known, ignored issues" as well: https://github.com/os-autoinst/openQA/pull/1837

Actions #7

Updated by lnussel about 6 years ago

I guess manually setting softfailed on some tests make sense, like random failures in chromium, vlc etc. But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.

Actions #8

Updated by lnussel about 6 years ago

wrt ignored without issue reference those are usually ones that are chained. like raid tests. Sometimes it's also just cumbersome to dig out reference numbers.

Actions #9

Updated by okurz about 6 years ago

lnussel wrote:

But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.

So far softfails would only be limited to a single test module and any subsequent test module failing would turn the whole job result into failed and overrule the softfail. However, thinking about the potentially new feature here, I guess it is still up to the reviewer putting the '@ttm ignore' flag if he is fine with all the lost testing coverage which is a consequence of the ignored bug.

Actions #10

Updated by okurz over 5 years ago

  • Category changed from 124 to Feature requests
Actions #11

Updated by okurz about 5 years ago

  • Description updated (diff)
  • Status changed from New to Workable

With https://github.com/os-autoinst/openQA/pull/1837 and https://github.com/os-autoinst/os-autoinst/pull/1052 it is possible to overwrite a fail into softfail on the module level. As described in #40382#note-2 and #40382#note-9 the next step would be to automatically transform a failed job labeled with an "ignored issue" as soft-fail, e.g. at the time of carry-over. I would not change a fail to soft-fail for an existing, done job however as it could be seen as confusing if a job any time later changes its result. However changing to soft-fail might come with implications, e.g. so far we do not show "failed modules" for soft-fails AFAIK.

I updated the description with clear acceptance criteria and suggestions. However this only relates to issues which are already marked as "ignored". In general I wonder how we can make it easier to actually mark test failures for ignoring. What I see as a challenge is that test reviewers have a too hard time to find known issues from issue trackers, e.g. bugzilla, and then they mostly only put a comment on individual jobs instead of introducing a "record_soft_failure" or "force_soft_failure" which would detect and mark the issue in a more long-term maintainable way but with the additional effort of needing to put something into the test code. In the end I recommend to focus this ticket here only on how known ignored issues are rendered. All the rest can go into #39719 for better handling of "known issues".

Actions #12

Updated by Xiaojing_liu over 4 years ago

@okurz I am not sure if I understand correctly. Following those comments, I think maybe we could provide a button in UI, only used to change a job from failed to ignored. Or we could provide a function such as record_soft_failure in os-autoinst, but this needs users to modify the test code. Or I completely understand wrong.

Actions #13

Updated by okurz over 4 years ago

Xiaojing_liu wrote:

Following those comments, I think maybe we could provide a button in UI, only used to change a job from failed to ignored

No button is needed. The Suggestions in #40382#Suggestions are up-to-date and should be followed here.

Actions #14

Updated by okurz over 4 years ago

  • Priority changed from Normal to Low
  • Target version changed from Ready to future

low team capacity

Actions

Also available in: Atom PDF