Make "ignored" issues more prominent (was: create new state "ignored")
It is pretty hard to get an overview of which tests TTM ignores and which ones not, despite all attempts we made so far.
- AC1: Ignored issues are obvious for test results
- Add "ignored" property for each issue object in openQA
- Use the property on carry over to turn failed job into "soft-fail"
Original proposal by lnussel: So I am wondering whether a new state result "ignored" would help there. Would be pretty clear then that all red tests have to be handled. Only if there are none left (means all passed, softfailed or ignored) a snapshot would be considered good. That condition would then be also visible in openQA (instead of overall state result failed).
The UI would have to gain a button to set a test to ignored. That would replace the current method of adding "@ttm ignore" to a comment and then wait until TTM puts the job into its ignore list.
#3 Updated by okurz over 2 years ago
- Subject changed from create new state "ignored" to Make "ignored" issues more prominent (was: create new state "ignored")
What does it mean for single ignored jobs without a issue reference? E.g. the jobs which just have a comment '@ttm ignore' but no bug or ticket link to just ignore the single job? I guess it should still be "failed" as we should not consider the failure reason a "known issue" when we do not have any external reference pointing to that.
#9 Updated by okurz over 2 years ago
But then there are also tests that really are hard failed and didn't finish. Not sure if it's a good idea to mix that with the other softfails.
So far softfails would only be limited to a single test module and any subsequent test module failing would turn the whole job result into failed and overrule the softfail. However, thinking about the potentially new feature here, I guess it is still up to the reviewer putting the '@ttm ignore' flag if he is fine with all the lost testing coverage which is a consequence of the ignored bug.
#11 Updated by okurz over 1 year ago
- Description updated (diff)
- Status changed from New to Workable
With https://github.com/os-autoinst/openQA/pull/1837 and https://github.com/os-autoinst/os-autoinst/pull/1052 it is possible to overwrite a fail into softfail on the module level. As described in #40382#note-2 and #40382#note-9 the next step would be to automatically transform a failed job labeled with an "ignored issue" as soft-fail, e.g. at the time of carry-over. I would not change a fail to soft-fail for an existing, done job however as it could be seen as confusing if a job any time later changes its result. However changing to soft-fail might come with implications, e.g. so far we do not show "failed modules" for soft-fails AFAIK.
I updated the description with clear acceptance criteria and suggestions. However this only relates to issues which are already marked as "ignored". In general I wonder how we can make it easier to actually mark test failures for ignoring. What I see as a challenge is that test reviewers have a too hard time to find known issues from issue trackers, e.g. bugzilla, and then they mostly only put a comment on individual jobs instead of introducing a "record_soft_failure" or "force_soft_failure" which would detect and mark the issue in a more long-term maintainable way but with the additional effort of needing to put something into the test code. In the end I recommend to focus this ticket here only on how known ignored issues are rendered. All the rest can go into #39719 for better handling of "known issues".
#12 Updated by Xiaojing_liu 11 months ago
okurz I am not sure if I understand correctly. Following those comments, I think maybe we could provide a button in UI, only used to change a job from
ignored. Or we could provide a function such as
record_soft_failure in os-autoinst, but this needs users to modify the test code. Or I completely understand wrong.