action #113761
openjobs restarted with `RETRY` have a comment, so the build gets the grey "commented" 'badge' if all failed jobs were retried
0%
Description
The grey "commented" badge for builds is meant to indicate, I think, that at least someone has looked at all the failures: a build gets it if all failed jobs have a comment.
The RETRY system kind of breaks this, because any retried job gets an automated comment from the retry system. So if all failed jobs for a build were auto-retried, the build will get the grey 'commented' badge even though no human has reviewed the failures.
Not sure what would be the best way to deal with this. Indicate retried jobs in some other way than a comment? Filter retried-job comments out from the badge calculation?
Updated by okurz almost 2 years ago
- Category set to Feature requests
- Assignee set to okurz
- Target version set to Ready
AdamWill wrote:
The grey "commented" badge for builds is meant to indicate, I think, that at least someone has looked at all the failures: a build gets it if all failed jobs have a comment.
well, a "commented" badge basically only means that a comment is present. It would need to be checked in detail what the comment is.
The
RETRYsystem kind of breaks this, because any retried job gets an automated comment from the retry system. So if all failed jobs for a build were auto-retried, the build will get the grey 'commented' badge even though no human has reviewed the failures.
There are multiple other systems which can generate a comment, e.g. we have github.com/os-autoinst/scripts/ which are triggered by http://open.qa/docs/#_enable_custom_hook_scripts_on_job_done_based_on_result so we have comments from automatic systems which are not humans. Then a comment without a bugref would mean "there is additional information, maybe from a human, maybe from a bot, but no known referenced issue yet"
Not sure what would be the best way to deal with this. Indicate retried jobs in some other way than a comment? Filter retried-job comments out from the badge calculation?
I see the following options:
- Adapting the processes around the refined definition that the comment icon just means "presence of a comment with potentially more information"
- Your suggestion that retry was applied but without a job comment. This could also be configurable.
- Come up with a new type to distinguish from plain "comments" and also not including in badge calculation. I would not just filter out "special comments" from the badge calculation as that would be potentially hard to understand why certain comments are included and others not. Better make it explicit by having a separate type.
I think option 3 is becoming too confusing and we should avoid that. I could live with option 2 but configurable or rethink and actually not have a comment in openQA jobs. The comment was introduced in https://github.com/os-autoinst/openQA/pull/4582/commits/4935386b6b279244088f1d1c049a93a55f50ca0d and it was not part of the original design. @mkittler WDYT about this, do we actually need the retry comment in openQA jobs? If we would not have a comment would we have openqa-investigate runs on jobs that we retry anyway?
Of course, option 1 would be easiest for us :)
Updated by okurz almost 2 years ago
- Assignee deleted (
okurz) - Target version changed from Ready to future
Updated by AdamWill over 1 year ago
I mean, the 'process' here is just 'me looking at the list of builds'. But practically speaking, it's difficult to see how the grey icon has any value now with the retry comments. Before it told me 'ok, all the failures are either carried-over bug associations or ones we've actually looked at and commented on'. Now it tells me 'all the failures are either that, or automatic retries'. And since we auto-retry all our update tests, that makes it essentially useless, because essentially all builds with failed tests will get it. So now it's just useless decoration.
Updated by livdywan over 1 year ago
AdamWill wrote:
I mean, the 'process' here is just 'me looking at the list of builds'. But practically speaking, it's difficult to see how the grey icon has any value now with the retry comments. Before it told me 'ok, all the failures are either carried-over bug associations or ones we've actually looked at and commented on'. Now it tells me 'all the failures are either that, or automatic retries'. And since we auto-retry all our update tests, that makes it essentially useless, because essentially all builds with failed tests will get it. So now it's just useless decoration.
So when you add a comment it practically means "reviewed" to you, and there's no particular format that could be automatically recognized, is what I understand.
An icon specific to bot/ automatically written comments could be the easiest solution. And it would also be meaningful for other instances
Updated by AdamWill over 1 year ago
cdywan wrote:
So when you add a comment it practically means "reviewed" to you, and there's no particular format that could be automatically recognized, is what I understand.
Basically, yeah. Sometimes the 'review' can't be in the form of a link to a recognizable issue tracker (though this does remind me I really need to add a handler for pagure.io).