coordination #102906

[saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA

Added by okurz about 1 year ago. Updated 7 months ago.

Feature requests
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)


User Story

As a reviewer of failed openQA tests I want known failures of jobs be handled fully automatically to not waste time on investigating known failures

Acceptance criteria

  • AC1: If a job fails for any reason that is "known" already in the context of the current openQA instance no further "test review" effort is needed by human reviewers


Further details


  • "known" means that a certain symptom of a test failure has been described with e.g. a matching pattern in either a test distribution, os-autoinst or also openQA itself
  • "test review" means what we currently do in openSUSE or SLE by providing job labels with issue references in openQA comments which are carried over or automatically labeled


coordination #41057: [epic] Make reviewing results easierNew

action #35017: Show more test failure details with the "test failed" text thumbnail popupsWorkable

action #91601: Add "return to top" button on openQA pages, e.g. job details, index, group overviewResolvedkraih

action #91602: Inform users about keyboard shortcuts in openQAWorkable

coordination #88229: [epic] Prevent unintended test coverage decreaseNew

QA - action #88127: [tools][qem] Test coverage DB for maintenance updatesClosedjbaier_cz

QA - action #88485: [teregen] Fetch and store coverage info for each incidentResolvedjbaier_cz

QA - action #90401: [teregen] Integrate coverage information in a presentable way into test templateResolvedjbaier_cz

QA - action #90404: [teregen] Update TeReGen for deployment on qam2Resolvedjbaier_cz

QA - action #88536: Find out differences in openQA test coverage with metabaseResolvedhurhaj

action #91509: Easy way to check and compare coverage in multiple openQA instancesNew

action #91656: [qe-core] os-autoinst-distri-opensuse YAML schedule file comparisonNew

coordination #102909: [epic] Prevent more incompletes already within os-autoinst or openQANew

action #59926: test incompletes in middle of execution with auto_review:"Unexpected end of data 0":retry, system journal shows 'kernel: traps: /usr/bin/isotov[2300] general protection ip:7fd5ef11771e sp:7ffe066f2200 error:0 in[7fd5ef094000+1b1000]'New

action #69448: test incompletes with auto_review:"(?s)was downloaded by.*details.*unavailable here.*Failed to download":retry , not helpful detailsWorkable

coordination #69451: [epic] test incompletes with "(?s)Download.*successful.*Failed to download":retry, not helpful detailsNew

action #69691: Improve incomplete output for qemu related problems, e.g. auto_review:"Failed to allocate KVM HPT.*Cannot allocate memory":retry instead of "can't open qmp"Workable

action #71188: job incomplete with auto_review:"backend died: QEMU exited unexpectedly, see log for details" and no other obvious information in the logfile what went wrongWorkable

action #71227: [ux] job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/ line 932."Workable

action #73273: job incompletes with auto_review:"setup failure: Cache service status error from API.* file is not a database .*":retryWorkable

action #73282: auto_review:"setup failure: Cache service status error from API: Minion job.*Worker went away":retryWorkable

action #73288: auto_review:"setup failure: Cache service status error from API: Minion job.*Job terminated unexpectedly":retryWorkable

action #73294: auto_review:"isotovideo died: needles_dir not found" should be 'tests died' or something similar obvious to test maintainers that they need to actWorkable

action #73369: Job incompletes with auto_review:"(?s)backend died: runcmd .*qemu-img create -f qcow2 .* failed with exit code 1: 'Formatting .*" on o3Workable

action #73375: Job incompletes with reason auto_review:"(?m)api failure$" (and no further details)Workable

action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/*":retryNew

action #75388: Explicit error feedback to test reviewers on wrong test API usageWorkable

action #78055: job incomplete exiting prematurely before reaching needle check timeout auto_review:"(?s)called testapi::assert_screen.*no match: [^-0]+\.[0-9]s,[^\n]*\n[^\n]*backend process exited: 0.*\[autotest\] process exited: 1":retryNew

action #80226: job incomplete with autoinst-log.txt ending just in the middleNew

action #80356: incompletes with auto_review:"Cache service.*error: Connection refused":retryWorkable

action #80778: job incompletes with "Virtio terminal and svirt serial terminal do not support send_key", we should change the reason message to be ignored by auto-review, but be clear for the test maintainerWorkable

coordination #102912: [epic] Simplify investigation of job failures - 2ndNew

openQA Tests - action #46988: [qe-core][functional] Detect known bugs from system journalNew

action #62441: openqa-worker systemd service can timeout when stoppingNew

action #63065: [gsoc] dynamic detection of error conditions from test resultsNew

action #64935: Provide test results of "related results" on job detailsWorkable

action #69319: Surface relevant versions used by job workerNew

action #99066: ci: Use a git commit message style checker in more reposNew

action #103035: Present changes between packages on openQA worker machines in "investigation" - enable in production on o3New

action #104007: Support retry of openQA jobs based on test variablesResolvedokurz

action #113758: Jobs restarted with `RETRY` are not shown as 'clones', so it is hard or impossible find the original jobNew

Related issues

Related to openQA Project - action #45011: Allow detection of known failures at the autoinst-log.txtWorkable2018-12-11

Copied from openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issuesResolved2018-05-23

Copied to openQA Project - coordination #102915: [saga][epic] Automated classification of failuresNew2021-07-20


#1 Updated by okurz about 1 year ago

  • Copied from coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added

#2 Updated by okurz about 1 year ago

#3 Updated by okurz about 1 year ago

  • Related to action #45011: Allow detection of known failures at the autoinst-log.txt added

Also available in: Atom PDF