coordination #102912

coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA

[epic] Simplify investigation of job failures - 2nd

Added by okurz about 1 year ago. Updated about 1 year ago.

Feature requests
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)



Make job failure investigation easier to save time and ensure we do not miss failures


  • all package changes, e.g. save rpm -qa in file and provide diff and/or changelog for the worker and the SUT. For openSUSE e.g. read out changelog diff from , for SLE from

  • diff of test schedule

  • if best needle candidate matches 0% it is most likely not a trivial needle issue

  • In settings table mark origin of settings and changed settings, e.g. for setting "foo" instead of the table row "foo | 1" one could have

    • "foo | 1 (testsuites table)" when the settings comes from the test suites database table, e.g. compared to job templates, machines, etc. . This would also help when we allow even more sources for settings, e.g. load job templates from test distributions in parallel to database tables
    • update the settings table from vars.json after job run to included changes but then show which settings changed since the job was initially created
    • "foo | 1 (+)" when the setting is new in the scenario, with the table row and/or "(+)" in green (as in common colored diffs) and on hover it shows the explanation that this was added, linked to the commit, showing which job it compares against
    • "foo | 1 (<->)" or similar when the setting changed against "last good" where it was e.g. 0, with "(<->)" being a link to the "last good" job, with the table row in different color


openQA Tests - action #46988: [qe-core][functional] Detect known bugs from system journalNew

action #62441: openqa-worker systemd service can timeout when stoppingNew

action #63065: [gsoc] dynamic detection of error conditions from test resultsNew

action #64935: Provide test results of "related results" on job detailsWorkable

action #69319: Surface relevant versions used by job workerNew

action #99066: ci: Use a git commit message style checker in more reposNew

action #103035: Present changes between packages on openQA worker machines in "investigation" - enable in production on o3New

Related issues

Copied from openQA Project - coordination #19720: [epic] Simplify investigation of job failuresResolved2019-12-17


#1 Updated by okurz about 1 year ago

Also available in: Atom PDF