Project

General

Profile

coordination #102906

[saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA

Added by okurz 2 months ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2018-04-16
Due date:
% Done:

42%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

User Story

As a reviewer of failed openQA tests I want known failures of jobs be handled fully automatically to not waste time on investigating known failures

Acceptance criteria

  • AC1: If a job fails for any reason that is "known" already in the context of the current openQA instance no further "test review" effort is needed by human reviewers

Suggestions

Further details

Definitions:

  • "known" means that a certain symptom of a test failure has been described with e.g. a matching pattern in either a test distribution, os-autoinst or also openQA itself
  • "test review" means what we currently do in openSUSE or SLE by providing job labels with issue references in openQA comments which are carried over or automatically labeled

Subtasks

coordination #41057: [epic] Make reviewing results easierNew

action #35017: Show more test failure details with the "test failed" text thumbnail popupsWorkable

action #91601: Add "return to top" button on openQA pages, e.g. job details, index, group overviewResolvedkraih

action #91602: Inform users about keyboard shortcuts in openQAWorkable

coordination #88229: [epic] Prevent unintended test coverage decreaseNew

QA - action #88127: [tools][qem] Test coverage DB for maintenance updatesClosedjbaier_cz

QA - action #88485: [teregen] Fetch and store coverage info for each incidentResolvedjbaier_cz

QA - action #90401: [teregen] Integrate coverage information in a presentable way into test templateResolvedjbaier_cz

QA - action #90404: [teregen] Update TeReGen for deployment on qam2Resolvedjbaier_cz

QA - action #88536: Find out differences in openQA test coverage with metabaseResolvedhurhaj

openQA Tests - action #91509: [tools] Easy way to check and compare coverage in multiple openQA instancesNew

openQA Tests - action #91656: [qe-core] os-autoinst-distri-opensuse YAML schedule file comparisonNew

coordination #102909: [epic] Prevent more incompletes already within os-autoinst or openQANew

action #59926: test incompletes in middle of execution with auto_review:"Unexpected end of data 0":retry, system journal shows 'kernel: traps: /usr/bin/isotov[2300] general protection ip:7fd5ef11771e sp:7ffe066f2200 error:0 in libc-2.26.so[7fd5ef094000+1b1000]'New

action #69448: test incompletes with auto_review:"(?s)was downloaded by.*details.*unavailable here.*Failed to download":retry , not helpful detailsWorkable

coordination #69451: [epic] test incompletes with "(?s)Download.*successful.*Failed to download":retry, not helpful detailsNew

action #69691: Improve incomplete output for qemu related problems, e.g. auto_review:"Failed to allocate KVM HPT.*Cannot allocate memory":retry instead of "can't open qmp"Workable

action #71188: job incomplete with auto_review:"backend died: QEMU exited unexpectedly, see log for details" and no other obvious information in the logfile what went wrongWorkable

action #71227: job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."Workable

action #73273: job incompletes with auto_review:"setup failure: Cache service status error from API.* file is not a database .*":retryWorkable

action #73282: auto_review:"setup failure: Cache service status error from API: Minion job.*Worker went away":retryWorkable

action #73288: auto_review:"setup failure: Cache service status error from API: Minion job.*Job terminated unexpectedly":retryWorkable

action #73294: auto_review:"isotovideo died: needles_dir not found" should be 'tests died' or something similar obvious to test maintainers that they need to actWorkable

action #73369: Job incompletes with auto_review:"(?s)backend died: runcmd .*qemu-img create -f qcow2 .* failed with exit code 1: 'Formatting .*" on o3Workable

action #73375: Job incompletes with reason auto_review:"(?m)api failure$" (and no further details)Workable

action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retryNew

action #75388: Explicit error feedback to test reviewers on wrong test API usageWorkable

action #78055: job incomplete exiting prematurely before reaching needle check timeout auto_review:"(?s)called testapi::assert_screen.*no match: [^-0]+\.[0-9]s,[^\n]*\n[^\n]*backend process exited: 0.*\[autotest\] process exited: 1":retryNew

action #80226: job incomplete with autoinst-log.txt ending just in the middleNew

action #80356: incompletes with auto_review:"Cache service.*error: Connection refused":retryWorkable

action #80778: job incompletes with "Virtio terminal and svirt serial terminal do not support send_key", we should change the reason message to be ignored by auto-review, but be clear for the test maintainerWorkable

coordination #102912: [epic] Simplify investigation of job failures - 2ndNew

openQA Tests - action #46988: [qe-core][functional] Detect known bugs from system journalNew

action #62441: openqa-worker systemd service can timeout when stoppingNew

action #63065: [gsoc] dynamic detection of error conditions from test resultsNew

action #64935: Provide test results of "related results" on job detailsWorkable

action #69319: Surface relevant versions used by job workerNew

action #99066: ci: Use a git commit message style checker in more reposNew

action #103035: Present changes between packages on openQA worker machines in "investigation" - enable in production on o3New

action #104007: Support retry of openQA jobs based on test variablesResolvedokurz


Related issues

Related to openQA Project - action #45011: Allow detection of known failures at the autoinst-log.txtWorkable2018-12-11

Copied from openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issuesResolved2018-05-23

Copied to openQA Project - coordination #102915: [saga][epic] Automated classification of failuresNew2021-07-20

History

#1 Updated by okurz 2 months ago

  • Copied from coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added

#2 Updated by okurz 2 months ago

#3 Updated by okurz 2 months ago

  • Related to action #45011: Allow detection of known failures at the autoinst-log.txt added

Also available in: Atom PDF