Project

General

Profile

action #45062

coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

coordination #62420: [epic] Distinguish all types of incompletes

Better visualization of incompletes - show module in which incomplete happens

Added by mgriessmeier almost 3 years ago. Updated 10 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2018-12-12
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

User story

The information shown on the dashboard for incomplete jobs could be done better which can avoid things like "creating duplicate bugs" or "let's simply restart, it will work afterwards"

Acceptance criteria

  • AC1: Obviously wrong usage of test API shows test module details, e.g. an info box, rather than just log content

Suggestions

  • show the actual module in which the incomplete happens to avoid different issues to be accepted as the same one
  • Ensure label carryover for incompletes works (at least as a comment so the reviewers will not miss it)

Related issues

Related to openQA Project - action #59273: module result missing for incompleting jobResolved2019-11-10

Related to openQA Project - action #80746: test fails in await_install - openQA hasn't inherit the comments for incomplete jobResolved2020-12-07

History

#1 Updated by okurz almost 3 years ago

"failed modules" for "incompletes" actually already works, see https://openqa.suse.de/tests/2330525#next_previous

Funny thing is: A carryover "to" incompletes with failed modules does work as well, only "from" does not work. In https://openqa.suse.de/tests/2330525#next_previous any label on incompletes is disregarded for a carry-over but instead the label from the last "failing" job in the same module, https://openqa.suse.de/tests/2261045, is carried over.

So to me it looks like: How can we ensure every incomplete to happen within a module so that a failed module would show up?

#2 Updated by okurz almost 2 years ago

  • Subject changed from [feature][tools] Better visualization of incompletes to [feature][tools] Better visualization of incompletes - show module in which incomplete happens
  • Description updated (diff)
  • Status changed from New to Workable

As a workaround with https://gitlab.suse.de/openqa/auto-review/ using https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues which labels incomplete jobs better. With https://github.com/os-autoinst/openQA/pull/2554 we also show the log content when there are no module results at all. What I see for this ticket is the task to provide proper test module results in more cases, especially when tests fail due to incorrect test API usage, not "random crashes".

#3 Updated by okurz over 1 year ago

  • Parent task set to #62420

#4 Updated by okurz over 1 year ago

  • Related to action #59273: module result missing for incompleting job added

#5 Updated by okurz about 1 year ago

  • Subject changed from [feature][tools] Better visualization of incompletes - show module in which incomplete happens to Better visualization of incompletes - show module in which incomplete happens
  • Priority changed from Normal to Low
  • Target version set to future

Likely better to wait for #59273 first

#6 Updated by okurz 11 months ago

  • Related to action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retry added

#7 Updated by okurz 11 months ago

  • Related to deleted (action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retry)

#8 Updated by okurz 10 months ago

  • Related to action #80746: test fails in await_install - openQA hasn't inherit the comments for incomplete job added

#9 Updated by okurz 10 months ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz
  • Target version changed from future to Ready

https://openqa.suse.de/tests/5148360#next_previous shows that we do have support for failed modules in incomplete jobs. So it could be that this only works seldomly or in limited cases, e.g. when first the test module fails and then an incomplete happens in the same module. However I think this is good enough.

For an approach to have automatic labels on incomplete jobs regardless of the failed module I recommend the approach of job hooks with "auto-review" as described on http://open.qa/docs/#_enable_custom_hook_scripts_on_job_done_based_on_result

Also available in: Atom PDF