Project

General

Profile

coordination #39719

[saga][epic] Detect "known failures" and mark jobs as such to make tests more stable, reviewing test results and tracking known issues easier

Added by okurz over 2 years ago. Updated 7 days ago.

Status:
Blocked
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2018-04-16
Due date:
2021-01-26
% Done:

57%

Estimated time:
(Total: 8.00 h)
Difficulty:

Description

User Story

As a reviewer of failed openQA tests I want known failures of jobs regardless of the error source to be marked as such automatically to not waste time on investigating known failures

Acceptance criteria

  • AC1: If a job fails for any reason that is "known" already in the context of the current openQA instance no further "test review" effort is needed by human reviewers

Suggestions

  • Provide a mechanism to match on regex in serial0.txt (as provided by existing "serial exception catching"-feature) based on patterns defined in the test distribution
  • Same for autoinst-log.txt
  • Provide patterns defined in os-autoinst for backend specific stuff, e.g. the "key event queue full"-thingy -> look for that string in os-autoinst for existing code to handle that
  • Same as above but patterns defined in instance specific configuration, e.g. workers.ini (managed by salt for SLE)
  • Maybe the same based on needles? But maybe the current approach using the "workaround" property and soft-fail needles to be always preferred is already good enough :)
  • It might be necessary to re-define "soft-fail" as "known issue" and nothing more so that we can use the "known failure" detection to set a job to soft-failed referencing the known issue, immediately aborting the further execution of a job to prevent it failing at a sporadic later step which would pose the need to provide openQA comments to provide a label

Further details

Definitions:

  • "known" means that a certain symptom of a test failure has been described with e.g. a matching pattern in either a test distribution, os-autoinst or maybe openQA itself as for the later mentioned jenkins plugin
  • "test review" means what we currently do in openSUSE or SLE by providing job labels with issue references in openQA comments which are carried over – which so far only works within individual scenarios

See https://wiki.jenkins.io/display/JENKINS/Build+Failure+Analyzer for an example. This jenkins plugin uses a "knowledge base" with jenkins instance global "known failures" defined with description and pattern matching, e.g. on "build log parsing", to mark failures as known when any log content matches existing patterns


Subtasks

coordination #19720: [epic] Simplify investigation of job failuresBlockedokurz

action #61103: Use CodeMirror to render diffs in the Investigation tabRejectedokurz

action #69085: Make "last good" a link to a job instead of plain job IDResolvedokurz

action #69088: Present changes between packages on openQA worker machines in "investigation"Resolvedilausuch

action #69319: Surface relevant versions used by job workerNew

openQA Tests - action #38621: [functional][y] test fails in welcome - "Module is not signed with expected PKCS#7 message" (bsc#1093659) - Use serial exception catching feature from openQA to make sure the jobs reference the bug, e.g. as labelResolvedriafarov

openQA Tests - action #46988: [qe-core][functional] Detect known bugs from system journalNew

action #60560: Self-investigate potential reasons for failures in openQAResolvedokurz

coordination #62420: [epic] Distinguish all types of incompletesBlockedokurz

action #45062: Better visualization of incompletes - show module in which incomplete happensResolvedokurz

action #59926: test incompletes in middle of execution with auto_review:"Unexpected end of data 0", system journal shows "kernel: traps: /usr/bin/isotov[2300] general protection ip:7fd5ef11771e sp:7ffe066f2200 error:0 in libc-2.26.so[7fd5ef094000+1b1000]"New

coordination #61922: [epic] Incomplete jobs with no logs at allResolvedmkittler

action #62984: Fix problem with job-worker assignment resulting in API errorsResolvedmkittler

action #63718: incomplete reason with just "quit"/"died" could provide more informationResolvedmkittler

action #64854: qemu-img error message is incorrectly tried to be parsed as JSON auto_review:"malformed JSON string"Resolvedtinita

action #64857: Put single-line error messages into incomplete reason for "died"Resolvedcdywan

action #64884: Distinguish test contributor errors from unexpected backend crashesResolvedmkittler

action #64917: auto_review:"(?s)qemu-img.*runcmd.*failed with exit code 1" sometimes but no apparent error messageResolvedokurz

action #66066: incomplete with reason "died: terminated prematurely" but log shows error 404 failing to download asset into cache auto_review:"(?s)Download.*failed: 404.*No scripts"Rejectedokurz

action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retryResolvedmkittler

action #69448: test incompletes with auto_review:"(?s)was downloaded by.*details.*unavailable here.*Failed to download":retry , not helpful detailsWorkable

coordination #69451: [epic] test incompletes with "(?s)Download.*successful.*Failed to download":retry, not helpful detailsBlockedokurz

action #69553: job incompletes with "Failed to rsync tests: exit code 10":retry, improve user feedbackResolvedkraih

action #69691: Improve incomplete output for qemu related problems, e.g. auto_review:"Failed to allocate KVM HPT.*Cannot allocate memory":retry instead of "can't open qmp"Workable

action #71185: job incompletes with auto_review:"setup failure: Cache service status error: Premature connection close":retry and does not retry, should we just automatically retry the connection?Resolvedokurz

action #71188: job incomplete with auto_review:"backend died: QEMU exited unexpectedly, see log for details" and no other obvious information in the logfile what went wrongWorkable

action #71227: job incompletes with auto_review:"backend died: 'current_console' is not set at /usr/lib/os-autoinst/backend/baseclass.pm line 932."Workable

action #71827: test incompletes with auto_review:"(?s)Failed to download.*Asset was pruned immediately after download":retry because worker cache prunes the asset it just downloadedResolvedmkittler

action #73273: job incompletes with auto_review:"setup failure: Cache service status error from API.* file is not a database .*":retryWorkable

action #73282: auto_review:"setup failure: Cache service status error from API: Minion job.*Worker went away":retryWorkable

action #73285: test incompletes with auto_review:"(?s)Download of.*processed[^:].*Failed to download":retry , not helpful details about reason of errorBlockedokurz

action #73288: auto_review:"setup failure: Cache service status error from API: Minion job.*Job terminated unexpectedly":retryWorkable

action #73294: auto_review:"isotovideo died: needles_dir not found" should be 'tests died' or something similar obvious to test maintainers that they need to actWorkable

action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"Resolvedkraih

action #73369: Job incompletes with auto_review:"(?s)backend died: runcmd .*qemu-img create -f qcow2 .* failed with exit code 1: 'Formatting .*" on o3Workable

action #73375: Job incompletes with reason auto_review:"(?m)api failure$" (and no further details)Workable

action #73396: job incompletes with auto_review:"setup failure: Failed to rsync tests: exit code 23":retryResolvedXiaojing_liu

action #73525: Job incompletes with auto_review:"backend died: unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm.*":retryNew

action #75388: Explicit error feedback to test reviewers on wrong test API usageWorkable

action #78055: job incomplete exiting prematurely before reaching needle check timeout auto_review:"(?s)called testapi::assert_screen.*no match: [^-0]+\.[0-9]s,[^\n]*\n[^\n]*backend process exited: 0.*\[autotest\] process exited: 1":retryNew

action #78169: after osd-deploy 2020-11-18 incompletes with auto_review:"Cache service (status error from API|.*error 500: Internal Server Error)":retryResolvedmkittler

openQA Infrastructure - action #80106: corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarilyResolvednicksinger

action #80118: test incompletes with auto_review:"(?s)Failed to download.*Asset was pruned immediately after download":retry, not effective on osd, or second fix neededResolvedokurz

action #80226: job incomplete with autoinst-log.txt ending just in the middleWorkable

action #80334: job incompletes with auto_review:"(?s)terminated prematurely with corrupted state file.*No space left on device":retry , should automatically retriggerIn ProgressXiaojing_liu

action #80356: incompletes with auto_review:"Cache service.*error: Connection refused":retryWorkable

openQA Infrastructure - action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cacheResolvednicksinger

openQA Tests - action #80776: [jeos] job incomplete auto_review:"(?s)(podman|docker).*Virtio terminal and svirt serial terminal do not support send_key":retryNew

action #80778: job incompletes with "Virtio terminal and svirt serial terminal do not support send_key", we should change the reason message to be ignored by auto-review, but be clear for the test maintainerWorkable

action #63065: [gsoc] dynamic detection of error conditions from test resultsNew

action #64935: Provide test results of "related results" on job detailsWorkable

action #35017: Show more test failure details with the "test failed" text thumbnail popupsWorkable

coordination #41057: [epic] Make reviewing results easierBlockedokurz

action #62441: openqa-worker systemd service can timeout when stoppingWorkable

action #75232: error message when worker has no network (yet): Unable to serialize fatal error: Can't open file "base_state.json": Permission denied at /usr/lib/os-autoinst/bmwqemu.pm line 86."Feedbackcdywan

QA - coordination #77899: [epic] Extend "auto-review" for failed jobs as wellWorkableokurz

QA - action #80414: [proof-of-concept] Extend "auto-review" for failed jobs as well, start with o3Resolvedokurz

QA - action #80418: [learning] Fix parse errors in "openqa-investigate" "parse error: Invalid numeric literal at line 1, column 10"Resolvedmkittler

QA - action #80806: Extend "auto-review" for failed jobs as well - Generalize openqa-monitor-investigation-candidates to look at more than just one job groupResolvedokurz

QA - action #80808: Extend "auto-review" for failed jobs as well - enable same as on o3 but on osdResolvedokurz

QA - action #77944: Run "auto-review" more often but alarm lessResolvedokurz

action #80264: multimachine tests unable to get vars from its pair jobResolvedmkittler

action #80412: tests fail with auto_review:"(?s)version is 4\.6\.1606298538\.191b5988.*Can.*t locate object method.*code.*via package":retryResolvedokurz

action #80772: [jeos] auto_review:"(?s)GENERAL_HW_FLASH_CMD.*No space left on device":retry incomplete in flash scriptResolvedggardet_arm

action #80774: [jeos] auto_review:"(?s)GENERAL_HW_FLASH_CMD.*No route to host":retry incomplete in flash scriptResolvedggardet_arm

coordination #80828: [epic] Trigger 'auto-review' and 'openqa-investigate' from within openQA when jobs incomplete or fail on o3+osdBlockedokurz

action #80736: Trigger 'auto-review' from within openQA when jobs incomplete (or fail) , for testing: auto_review:"tests died: unable to load main.pm, check the log for the cause"Resolvedokurz

action #80826: Trigger 'auto-review' from within openQA when jobs incomplete on osd as wellResolvedokurz

action #80830: Trigger 'openqa-investigate' from within openQA when jobs fail on o3Resolvedokurz

action #81206: Trigger 'openqa-investigate' from within openQA when jobs fail on osdResolvedokurz

action #81859: openqa-investigate triggers incomplete sets for multi-machine scenariosWorkable


Related issues

Related to openQA Project - action #13242: WDYT: For every job that does not have a label or bugref, retrigger some times to see if it's sporadic. Like rescheduling on incomplete but on failedRejected2016-11-25

Related to openQA Project - coordination #13812: [epic][dashboard] openQA Dashboard ideasNew2017-01-10

Related to openQA Tests - action #42446: [qe-core][functional] many opensuse tests fail in desktop_runner or gimp or other modules in what I think is boo#1105691 – can we detect this bug from the journal and track as soft-fail?Blocked2018-10-13

Related to openQA Project - action #40382: Make "ignored" issues more prominent (was: create new state "ignored")Workable2018-08-29

Related to openQA Tests - action #43784: [functional][y][sporadic] test fails in yast2_snapper now reproducibly not exiting the "show differences" screenResolved2018-11-14

Related to openQA Project - action #57452: Automatic summary of failuresRejected2019-09-27

Blocked by openQA Project - action #45011: Allow detection of known failures at the autoinst-log.txtWorkable2018-12-11

History

#1 Updated by okurz over 2 years ago

  • Related to action #13242: WDYT: For every job that does not have a label or bugref, retrigger some times to see if it's sporadic. Like rescheduling on incomplete but on failed added

#2 Updated by okurz over 2 years ago

  • Related to action #38621: [functional][y] test fails in welcome - "Module is not signed with expected PKCS#7 message" (bsc#1093659) - Use serial exception catching feature from openQA to make sure the jobs reference the bug, e.g. as label added

#3 Updated by okurz over 2 years ago

#4 Updated by okurz over 2 years ago

  • Related to deleted (action #38621: [functional][y] test fails in welcome - "Module is not signed with expected PKCS#7 message" (bsc#1093659) - Use serial exception catching feature from openQA to make sure the jobs reference the bug, e.g. as label)

#5 Updated by nicksinger over 2 years ago

Another idea which could be checked/better reported to the user:

If a crucial component in the "os-autoinst-chain" fails (e.g. xterm for ipmi jobs), openQA could easily report this earlier. As it is right now, the job stalls (hangs as "running") but only shows a black screen. Example: https://openqa.suse.de/tests/1970948 (look for "PermissionError" in the osautoinst-log.txt)

#6 Updated by coolo over 2 years ago

  • Target version set to future

IMO this is best handled by an automated review from outside. The problem is not so much the detecting the issue, but how to handle it. For some projects/objects you would do a retrigger, for others you would prefer defining a label.

#7 Updated by okurz over 2 years ago

"outside", yes, I agree. Should be outside what is currently defined as "openQA" but it could be that we still call it "the openQA ecosystem" so I guess this issue tracker is still best suited. Some parts we have already covered with the proof-of-concept of detecting known failures in the serial port output.

#8 Updated by coolo over 2 years ago

I don't disagree with the issue tracker - I just don't want a High priority epic in my 'to be sorted' list

#9 Updated by okurz over 2 years ago

  • Related to action #42446: [qe-core][functional] many opensuse tests fail in desktop_runner or gimp or other modules in what I think is boo#1105691 – can we detect this bug from the journal and track as soft-fail? added

#10 Updated by okurz over 2 years ago

  • Subject changed from [epic] Detect "known failures" and mark jobs as such to [functional][y][u][epic] Detect "known failures" and mark jobs as such

Trying to bring it forward with help of QSF again…

#11 Updated by okurz over 2 years ago

  • Related to action #27004: [opensuse][sle][functional][yast][y][hard] yast2 gui modules fail to start in the defined time frame added

#12 Updated by okurz over 2 years ago

  • Related to deleted (action #27004: [opensuse][sle][functional][yast][y][hard] yast2 gui modules fail to start in the defined time frame)

#13 Updated by okurz over 2 years ago

  • Blocks action #27004: [opensuse][sle][functional][yast][y][hard] yast2 gui modules fail to start in the defined time frame added

#14 Updated by okurz over 2 years ago

  • Related to action #40382: Make "ignored" issues more prominent (was: create new state "ignored") added

#15 Updated by okurz about 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/1052 to "Add option to override status of test modules with soft-fail"

#16 Updated by okurz about 2 years ago

  • Status changed from New to Feedback
  • Assignee set to okurz

#17 Updated by okurz about 2 years ago

The feature is not working as intended as in https://github.com/os-autoinst/os-autoinst/blob/master/basetest.pm#L286 we overwrite the result again. I am trying to simply remove that method :)

-> https://github.com/os-autoinst/os-autoinst/pull/1062

Also presented my idea to riafarov and we identified one problematic scenario: What if we force the status of a parent job to "softfail"? For now openQA would still trigger the downstream jobs which then most likely should fail because a module in the parent job failed, in the worst case even making the downstream jobs incomplete because the HDD image was never published properly. We should avoid this though.

#18 Updated by okurz about 2 years ago

  • Related to action #43784: [functional][y][sporadic] test fails in yast2_snapper now reproducibly not exiting the "show differences" screen added

#19 Updated by szarate about 2 years ago

  • Related to action #45011: Allow detection of known failures at the autoinst-log.txt added

#20 Updated by szarate about 2 years ago

I see that one of the suggestions on this ticket was exactly what poo#45011 is about :)

#21 Updated by agraul about 2 years ago

  • Related to deleted (action #45011: Allow detection of known failures at the autoinst-log.txt)

#22 Updated by agraul about 2 years ago

  • Blocked by action #45011: Allow detection of known failures at the autoinst-log.txt added

#23 Updated by agraul about 2 years ago

  • Status changed from Feedback to Blocked

#24 Updated by okurz almost 2 years ago

  • Due date changed from 2018-08-28 to 2019-03-12

due to changes in a related task

#25 Updated by okurz almost 2 years ago

  • Due date changed from 2019-03-12 to 2019-06-30

due to changes in a related task

#26 Updated by okurz over 1 year ago

  • Assignee changed from okurz to riafarov

Move to new QSF-y PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-y. Feel free to reassign to me or someone else in this case. Thanks.

#27 Updated by riafarov over 1 year ago

  • Blocks deleted (action #27004: [opensuse][sle][functional][yast][y][hard] yast2 gui modules fail to start in the defined time frame)

#28 Updated by riafarov over 1 year ago

  • Due date changed from 2019-06-30 to 2019-08-06

due to changes in a related task

#29 Updated by riafarov over 1 year ago

  • Due date changed from 2019-08-06 to 2019-12-31

due to changes in a related task

#30 Updated by okurz over 1 year ago

#31 Updated by okurz about 1 year ago

Using https://github.com/os-autoinst/scripts/blob/master/monitor-openqa_job and https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues I setup a gitlab CI pipeline in https://gitlab.suse.de/openqa/auto-review/ that automatically labels (and restarts) incompletes for which we know the reasons. The approach could also be extended to cover not only incompletes.

#32 Updated by okurz about 1 year ago

#33 Updated by riafarov about 1 year ago

  • Assignee changed from riafarov to okurz

As it's mainly tools team working on this epic, okurz I will set you as an assignee to track the progress. Feel free to change it, I rely on your expertise to set more suitable person if it's not you. Thanks!

#34 Updated by okurz about 1 year ago

  • Subject changed from [functional][y][u][epic] Detect "known failures" and mark jobs as such to [epic] Detect "known failures" and mark jobs as such

that's ok, it's me :)

There is currently only one subtask open #46988 on QSF-u though.

#35 Updated by okurz about 1 year ago

  • Due date changed from 2019-12-31 to 2020-12-31

due to changes in a related task

#36 Updated by okurz 11 months ago

  • Subject changed from [epic] Detect "known failures" and mark jobs as such to [saga] Detect "known failures" and mark jobs as such

#37 Updated by okurz 11 months ago

  • Subject changed from [saga] Detect "known failures" and mark jobs as such to [saga][epic] Detect "known failures" and mark jobs as such

#38 Updated by SLindoMansilla 9 months ago

  • Due date changed from 2020-12-31 to 2020-03-27

due to changes in a related task: #46988

#39 Updated by okurz 7 months ago

  • Due date changed from 2020-06-09 to 2020-03-27

due to changes in a related task: #62420

#40 Updated by okurz 7 months ago

  • Due date changed from 2018-08-28 to 2020-03-27

due to changes in a related task: #38621

#41 Updated by szarate 3 months ago

  • Tracker changed from action to coordination
  • Status changed from Blocked to New

#43 Updated by okurz 3 months ago

  • Status changed from New to Blocked
  • Target version changed from future to Ready

Discussed the topic of "auto-review" with SUSE QA Tools team and the general opinion was that this epic is interesting to follow up with so putting it to the backlog now.

#44 Updated by okurz about 2 months ago

  • Subject changed from [saga][epic] Detect "known failures" and mark jobs as such to [saga][epic] Detect "known failures" and mark jobs as such to make tests more stable, reviewing test results and tracking known issues easier

#45 Updated by cdywan about 2 months ago

Once again wondering: where's the due date coming from? It's not visible. Do we need to go through every single ticket again to check?

#46 Updated by okurz about 2 months ago

Maybe the API helps to find that easily but in this case it's #80264

Also available in: Atom PDF