Project

General

Profile

Actions

action #45062

closed

coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

coordination #62420: [epic] Distinguish all types of incompletes

Better visualization of incompletes - show module in which incomplete happens

Added by mgriessmeier almost 6 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2018-12-12
Due date:
% Done:

0%

Estimated time:

Description

User story

The information shown on the dashboard for incomplete jobs could be done better which can avoid things like "creating duplicate bugs" or "let's simply restart, it will work afterwards"

Acceptance criteria

  • AC1: Obviously wrong usage of test API shows test module details, e.g. an info box, rather than just log content

Suggestions

  • show the actual module in which the incomplete happens to avoid different issues to be accepted as the same one
  • Ensure label carryover for incompletes works (at least as a comment so the reviewers will not miss it)

Related issues 2 (0 open2 closed)

Related to openQA Project (public) - action #59273: module result missing for incompleting jobResolvedokurz2019-11-10

Actions
Related to openQA Project (public) - action #80746: test fails in await_install - openQA hasn't inherit the comments for incomplete jobResolvedokurz2020-12-07

Actions
Actions #1

Updated by okurz almost 6 years ago

"failed modules" for "incompletes" actually already works, see https://openqa.suse.de/tests/2330525#next_previous

Funny thing is: A carryover "to" incompletes with failed modules does work as well, only "from" does not work. In https://openqa.suse.de/tests/2330525#next_previous any label on incompletes is disregarded for a carry-over but instead the label from the last "failing" job in the same module, https://openqa.suse.de/tests/2261045, is carried over.

So to me it looks like: How can we ensure every incomplete to happen within a module so that a failed module would show up?

Actions #2

Updated by okurz almost 5 years ago

  • Subject changed from [feature][tools] Better visualization of incompletes to [feature][tools] Better visualization of incompletes - show module in which incomplete happens
  • Description updated (diff)
  • Status changed from New to Workable

As a workaround with https://gitlab.suse.de/openqa/auto-review/ using https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues which labels incomplete jobs better. With https://github.com/os-autoinst/openQA/pull/2554 we also show the log content when there are no module results at all. What I see for this ticket is the task to provide proper test module results in more cases, especially when tests fail due to incorrect test API usage, not "random crashes".

Actions #3

Updated by okurz almost 5 years ago

  • Parent task set to #62420
Actions #4

Updated by okurz over 4 years ago

  • Related to action #59273: module result missing for incompleting job added
Actions #5

Updated by okurz over 4 years ago

  • Subject changed from [feature][tools] Better visualization of incompletes - show module in which incomplete happens to Better visualization of incompletes - show module in which incomplete happens
  • Priority changed from Normal to Low
  • Target version set to future

Likely better to wait for #59273 first

Actions #6

Updated by okurz about 4 years ago

  • Related to action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retry added
Actions #7

Updated by okurz about 4 years ago

  • Related to deleted (action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retry)
Actions #8

Updated by okurz about 4 years ago

  • Related to action #80746: test fails in await_install - openQA hasn't inherit the comments for incomplete job added
Actions #9

Updated by okurz about 4 years ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz
  • Target version changed from future to Ready

https://openqa.suse.de/tests/5148360#next_previous shows that we do have support for failed modules in incomplete jobs. So it could be that this only works seldomly or in limited cases, e.g. when first the test module fails and then an incomplete happens in the same module. However I think this is good enough.

For an approach to have automatic labels on incomplete jobs regardless of the failed module I recommend the approach of job hooks with "auto-review" as described on http://open.qa/docs/#_enable_custom_hook_scripts_on_job_done_based_on_result

Actions

Also available in: Atom PDF