The information shown on the dashboard for incomplete jobs could be done better which can avoid things like "creating duplicate bugs" or "let's simply restart, it will work afterwards"
- AC1: Obviously wrong usage of test API shows test module details, e.g. an info box, rather than just log content
- show the actual module in which the incomplete happens to avoid different issues to be accepted as the same one
- Ensure label carryover for incompletes works (at least as a comment so the reviewers will not miss it)
#1 Updated by okurz almost 3 years ago
"failed modules" for "incompletes" actually already works, see https://openqa.suse.de/tests/2330525#next_previous
Funny thing is: A carryover "to" incompletes with failed modules does work as well, only "from" does not work. In https://openqa.suse.de/tests/2330525#next_previous any label on incompletes is disregarded for a carry-over but instead the label from the last "failing" job in the same module, https://openqa.suse.de/tests/2261045, is carried over.
So to me it looks like: How can we ensure every incomplete to happen within a module so that a failed module would show up?
#2 Updated by okurz almost 2 years ago
- Subject changed from [feature][tools] Better visualization of incompletes to [feature][tools] Better visualization of incompletes - show module in which incomplete happens
- Description updated (diff)
- Status changed from New to Workable
As a workaround with https://gitlab.suse.de/openqa/auto-review/ using https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues which labels incomplete jobs better. With https://github.com/os-autoinst/openQA/pull/2554 we also show the log content when there are no module results at all. What I see for this ticket is the task to provide proper test module results in more cases, especially when tests fail due to incorrect test API usage, not "random crashes".
#5 Updated by okurz about 1 year ago
- Subject changed from [feature][tools] Better visualization of incompletes - show module in which incomplete happens to Better visualization of incompletes - show module in which incomplete happens
- Priority changed from Normal to Low
- Target version set to future
Likely better to wait for #59273 first
- Status changed from Workable to Resolved
- Assignee set to okurz
- Target version changed from future to Ready
https://openqa.suse.de/tests/5148360#next_previous shows that we do have support for failed modules in incomplete jobs. So it could be that this only works seldomly or in limited cases, e.g. when first the test module fails and then an incomplete happens in the same module. However I think this is good enough.
For an approach to have automatic labels on incomplete jobs regardless of the failed module I recommend the approach of job hooks with "auto-review" as described on http://open.qa/docs/#_enable_custom_hook_scripts_on_job_done_based_on_result