action #40382
openMake "ignored" issues more prominent (was: create new state "ignored")
0%
Description
Motivation¶
It is pretty hard to get an overview of which tests TTM ignores and which ones not, despite all attempts we made so far.
Acceptance criteria¶
- AC1: Ignored issues are obvious for test results
Suggestions¶
- Add "ignored" property for each issue object in openQA
- Use the property on carry over to turn failed job into "soft-fail"
Further details¶
Original proposal by lnussel: So I am wondering whether a new state result "ignored" would help there. Would be pretty clear then that all red tests have to be handled. Only if there are none left (means all passed, softfailed or ignored) a snapshot would be considered good. That condition would then be also visible in openQA (instead of overall state result failed).
The UI would have to gain a button to set a test to ignored. That would replace the current method of adding "@ttm ignore" to a comment and then wait until TTM puts the job into its ignore list.
Updated by coolo over 6 years ago
I disagree - this is not a state. That's more a flag - just like 'important' is.
Updated by coolo about 6 years ago
- Category set to 124
- Target version set to Ready
I would like us to promote this to a property of the issue carried over. So we have open issue, closed issues and ignored issues.
If a job has ignored issues, it's result is mapped from failed to softfailed.
Updated by okurz about 6 years ago
- Subject changed from create new state "ignored" to Make "ignored" issues more prominent (was: create new state "ignored")
What does it mean for single ignored jobs without a issue reference? E.g. the jobs which just have a comment '@ttm ignore' but no bug or ticket link to just ignore the single job? I guess it should still be "failed" as we should not consider the failure reason a "known issue" when we do not have any external reference pointing to that.
Updated by okurz about 6 years ago
- Related to coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added
Updated by coolo about 6 years ago
Right. that's a different use case IMO. For those I expect lnussel to have a fresh memory
Updated by okurz about 6 years ago
My proposal for a fitting definition of softfailed covering "known, ignored issues" as well: https://github.com/os-autoinst/openQA/pull/1837
Updated by lnussel about 6 years ago
I guess manually setting softfailed on some tests make sense, like random failures in chromium, vlc etc. But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.
Updated by lnussel about 6 years ago
wrt ignored without issue reference those are usually ones that are chained. like raid tests. Sometimes it's also just cumbersome to dig out reference numbers.
Updated by okurz about 6 years ago
lnussel wrote:
But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.
So far softfails would only be limited to a single test module and any subsequent test module failing would turn the whole job result into failed and overrule the softfail. However, thinking about the potentially new feature here, I guess it is still up to the reviewer putting the '@ttm ignore' flag if he is fine with all the lost testing coverage which is a consequence of the ignored bug.
Updated by okurz over 5 years ago
- Category changed from 124 to Feature requests
Updated by okurz about 5 years ago
- Description updated (diff)
- Status changed from New to Workable
With https://github.com/os-autoinst/openQA/pull/1837 and https://github.com/os-autoinst/os-autoinst/pull/1052 it is possible to overwrite a fail into softfail on the module level. As described in #40382#note-2 and #40382#note-9 the next step would be to automatically transform a failed job labeled with an "ignored issue" as soft-fail, e.g. at the time of carry-over. I would not change a fail to soft-fail for an existing, done job however as it could be seen as confusing if a job any time later changes its result. However changing to soft-fail might come with implications, e.g. so far we do not show "failed modules" for soft-fails AFAIK.
I updated the description with clear acceptance criteria and suggestions. However this only relates to issues which are already marked as "ignored". In general I wonder how we can make it easier to actually mark test failures for ignoring. What I see as a challenge is that test reviewers have a too hard time to find known issues from issue trackers, e.g. bugzilla, and then they mostly only put a comment on individual jobs instead of introducing a "record_soft_failure" or "force_soft_failure" which would detect and mark the issue in a more long-term maintainable way but with the additional effort of needing to put something into the test code. In the end I recommend to focus this ticket here only on how known ignored issues are rendered. All the rest can go into #39719 for better handling of "known issues".
Updated by Xiaojing_liu over 4 years ago
@okurz I am not sure if I understand correctly. Following those comments, I think maybe we could provide a button in UI, only used to change a job from failed to ignored. Or we could provide a function such as record_soft_failure in os-autoinst, but this needs users to modify the test code. Or I completely understand wrong.
Updated by okurz over 4 years ago
Xiaojing_liu wrote:
Following those comments, I think maybe we could provide a button in UI, only used to change a job from
failedtoignored
No button is needed. The Suggestions in #40382#Suggestions are up-to-date and should be followed here.
Updated by okurz over 4 years ago
- Priority changed from Normal to Low
- Target version changed from Ready to future
low team capacity