Project

General

Profile

Actions
Copy link

coordination #102906

open

[saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA

Added by okurz over 2 years ago. Updated 4 months ago.

Status:
New
Priority:
High
Assignee:
-
Category:
Feature requests
Target version:
QA - future
Start date:
2018-04-16
Due date:
% Done:

40%

Estimated time:
(Total: 0.00 h)

Description

User Story

As a reviewer of failed openQA tests I want known failures of jobs be handled fully automatically to not waste time on investigating known failures

Acceptance criteria

  • AC1: If a job fails for any reason that is "known" already in the context of the current openQA instance no further "test review" effort is needed by human reviewers

Suggestions

Further details

Definitions:

  • "known" means that a certain symptom of a test failure has been described with e.g. a matching pattern in either a test distribution, os-autoinst or also openQA itself
  • "test review" means what we currently do in openSUSE or SLE by providing job labels with issue references in openQA comments which are carried over or automatically labeled

Subtasks 43 (34 open — 9 closed)

coordination #41057: [epic] Make reviewing results easierNew2018-04-16

Actions
action #35017: Show more test failure details with the "test failed" text thumbnail popupsWorkable2018-04-16

Actions
action #91601: Add "return to top" button on openQA pages, e.g. job details, index, group overviewResolvedkraih2021-04-22

Actions
action #91602: Inform users about keyboard shortcuts in openQAWorkable2021-04-22

Actions
coordination #88229: [epic] Prevent unintended test coverage decreaseNew2021-02-08

Actions
QA - action #88127: [tools][qem] Test coverage DB for maintenance updatesClosedjbaier_cz2021-02-08

Actions
QA - action #88485: [teregen] Fetch and store coverage info for each incidentResolvedjbaier_cz2021-02-08

Actions
QA - action #90401: [teregen] Integrate coverage information in a presentable way into test templateResolvedjbaier_cz2021-03-29

Actions
QA - action #90404: [teregen] Update TeReGen for deployment on qam2Resolvedjbaier_cz2021-04-09

Actions
QA - action #88536: Find out differences in openQA test coverage with metabaseResolvedhurhaj2021-02-12

Actions
action #91509: Easy way to check and compare coverage in multiple openQA instancesNew2021-04-23

Actions
action #91656: [qe-core] os-autoinst-distri-opensuse YAML schedule file comparisonNew2021-04-23

Actions
coordination #102909: [epic] Prevent more incompletes already within os-autoinst or openQANew2019-11-17

Actions
action #59926: test incompletes in middle of execution with auto_review:"Unexpected end of data 0":retry, system journal shows 'kernel: traps: /usr/bin/isotov[2300] general protection ip:7fd5ef11771e sp:7ffe066f2200 error:0 in libc-2.26.so[7fd5ef094000+1b1000]'New2019-11-17

Actions
action #69448: test incompletes with auto_review:"(?s)was downloaded by.*details.*unavailable here.*Failed to download":retry , not helpful detailsWorkable2020-07-30

Actions
coordination #69451: [epic] test incompletes with "(?s)Download.*successful.*Failed to download":retry, not helpful detailsNew2020-07-30

Actions
action #69691: Improve incomplete output for qemu related problems, e.g. auto_review:"Failed to allocate KVM HPT.*Cannot allocate memory":retry instead of "can't open qmp"Workable2020-08-07

Actions
action #71188: job incomplete with auto_review:"backend died: QEMU exited unexpectedly, see log for details" and no other obvious information in the logfile what went wrongWorkable2020-09-10

Actions
action #71227: [ux] job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."Workable2020-09-11

Actions
action #73273: job incompletes with auto_review:"setup failure: Cache service status error from API.* file is not a database .*":retryWorkable2020-10-13

Actions
action #73282: auto_review:"setup failure: Cache service status error from API: Minion job.*Worker went away":retryWorkable2020-10-13

Actions
action #73288: auto_review:"setup failure: Cache service status error from API: Minion job.*Job terminated unexpectedly":retryWorkable2020-10-13

Actions
action #73294: auto_review:"isotovideo died: needles_dir not found" should be 'tests died' or something similar obvious to test maintainers that they need to actWorkable2020-10-13

Actions
action #73369: Job incompletes with auto_review:"(?s)backend died: runcmd .*qemu-img create -f qcow2 .* failed with exit code 1: 'Formatting .*" on o3Workable2020-10-14

Actions
action #73375: Job incompletes with reason auto_review:"(?m)api failure$" (and no further details)Workable2020-10-14

Actions
action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retryNew2020-10-19

Actions
action #75388: Explicit error feedback to test reviewers on wrong test API usageWorkable

Actions
action #78055: job incomplete exiting prematurely before reaching needle check timeout auto_review:"(?s)called testapi::assert_screen.*no match: [^-0]+\.[0-9]s,[^\n]*\n[^\n]*backend process exited: 0.*\[autotest\] process exited: 1":retryNew2020-11-16

Actions
action #80226: job incomplete with autoinst-log.txt ending just in the middleNew2020-11-23

Actions
action #80356: incompletes with auto_review:"Cache service.*error: Connection refused":retryWorkable2020-11-18

Actions
action #80778: job incompletes with "Virtio terminal and svirt serial terminal do not support send_key", we should change the reason message to be ignored by auto-review, but be clear for the test maintainerWorkable2020-12-07

Actions
coordination #102912: [epic] Simplify investigation of job failures - 2ndNew2018-04-16

Actions
openQA Tests - action #46988: [qe-core][functional] Detect known bugs from system journalNew2018-04-16

Actions
action #62441: openqa-worker systemd service can timeout when stoppingNew2020-01-21

Actions
action #63065: [gsoc] dynamic detection of error conditions from test resultsResolvedokurz2020-02-04

Actions
action #64935: Provide test results of "related results" on job detailsWorkable2020-03-27

Actions
action #69319: Surface relevant versions used by job workerNew2020-07-24

Actions
action #99066: ci: Use a git commit message style checker in more reposNew

Actions
action #103035: Present changes between packages on openQA worker machines in "investigation" - enable in production on o3New

Actions
action #104007: Support retry of openQA jobs based on test variablesResolvedokurz2021-12-15

Actions
action #113758: Jobs restarted with `RETRY` are not shown as 'clones', so it is hard or impossible find the original jobNew2022-07-18

Actions
openQA Tests - action #115079: [qe-core][qem&functional] Many test failures due to low performance on arm workersNew2022-08-08

Actions
action #136154: multimachine tests restarted by RETRY test variable end up without the proper dependency size:MResolvedmkittler

Actions

Related issues 3 (2 open — 1 closed)

Related to openQA Project - action #45011: Allow detection of known failures at the autoinst-log.txtWorkable2018-12-11

Actions
Copied from openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issuesResolvedokurz2018-05-23

Actions
Copied to openQA Project - coordination #102915: [saga][epic] Automated classification of failuresNew2015-12-18

Actions
Actions
Copy link
 #1

Updated by okurz over 2 years ago

  • Copied from coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added
Actions
Copy link
 #2

Updated by okurz over 2 years ago

Actions
Copy link
 #3

Updated by okurz over 2 years ago

  • Related to action #45011: Allow detection of known failures at the autoinst-log.txt added
Actions
Copy link
 #4

Updated by szarate 7 months ago

  • Subtask #136154 added
Actions
Copy link
 #5

Updated by szarate 6 months ago

  • Subtask #115079 added
Actions
Copy link

Also available in: Atom PDF