action #45062
closedcoordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues
coordination #62420: [epic] Distinguish all types of incompletes
Better visualization of incompletes - show module in which incomplete happens
Description
User story¶
The information shown on the dashboard for incomplete jobs could be done better which can avoid things like "creating duplicate bugs" or "let's simply restart, it will work afterwards"
Acceptance criteria¶
- AC1: Obviously wrong usage of test API shows test module details, e.g. an info box, rather than just log content
Suggestions¶
- show the actual module in which the incomplete happens to avoid different issues to be accepted as the same one
- Ensure label carryover for incompletes works (at least as a comment so the reviewers will not miss it)
Updated by okurz about 6 years ago
"failed modules" for "incompletes" actually already works, see https://openqa.suse.de/tests/2330525#next_previous
Funny thing is: A carryover "to" incompletes with failed modules does work as well, only "from" does not work. In https://openqa.suse.de/tests/2330525#next_previous any label on incompletes is disregarded for a carry-over but instead the label from the last "failing" job in the same module, https://openqa.suse.de/tests/2261045, is carried over.
So to me it looks like: How can we ensure every incomplete to happen within a module so that a failed module would show up?
Updated by okurz about 5 years ago
- Subject changed from [feature][tools] Better visualization of incompletes to [feature][tools] Better visualization of incompletes - show module in which incomplete happens
- Description updated (diff)
- Status changed from New to Workable
As a workaround with https://gitlab.suse.de/openqa/auto-review/ using https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues which labels incomplete jobs better. With https://github.com/os-autoinst/openQA/pull/2554 we also show the log content when there are no module results at all. What I see for this ticket is the task to provide proper test module results in more cases, especially when tests fail due to incorrect test API usage, not "random crashes".
Updated by okurz over 4 years ago
- Related to action #59273: module result missing for incompleting job added
Updated by okurz over 4 years ago
- Subject changed from [feature][tools] Better visualization of incompletes - show module in which incomplete happens to Better visualization of incompletes - show module in which incomplete happens
- Priority changed from Normal to Low
- Target version set to future
Likely better to wait for #59273 first
Updated by okurz about 4 years ago
- Related to action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retry added
Updated by okurz about 4 years ago
- Related to deleted (action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retry)
Updated by okurz about 4 years ago
- Related to action #80746: test fails in await_install - openQA hasn't inherit the comments for incomplete job added
Updated by okurz about 4 years ago
- Status changed from Workable to Resolved
- Assignee set to okurz
- Target version changed from future to Ready
https://openqa.suse.de/tests/5148360#next_previous shows that we do have support for failed modules in incomplete jobs. So it could be that this only works seldomly or in limited cases, e.g. when first the test module fails and then an incomplete happens in the same module. However I think this is good enough.
For an approach to have automatic labels on incomplete jobs regardless of the failed module I recommend the approach of job hooks with "auto-review" as described on http://open.qa/docs/#_enable_custom_hook_scripts_on_job_done_based_on_result