coordination #88229

coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA

[epic] Prevent unintended test coverage decrease

Added by okurz about 2 years ago. Updated 8 months ago.

Feature requests
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)



In general the idea or demand for "prevent test coverage decrease" is nothing new and also not limited to QAM. Similar example would be in Tumbleweed as well as in other products: At any point in time it could happen that an openQA test scenario or test module or parts of test code or a combination of those is not executed anymore for a certain product and that is unlikely to be realized because neither TTM nor the openQA maintenance bot would count a reduction in test coverage as an alert condition. reports about this as well as which would not publish snapshots if less scenarios are found than in before. Both are not actively used for decision making about product releases.

User story

As a QE engineer, I would like to have a way of comparing openQA's tests (*.pm) utilization between our products, mostly before their release and after the release. With ~1500 tests available at this moment, it's very difficult to find out whether particular test is being used to it's full potential, if it is running on all products, code streams and architectures that it could and should run on. That brings a risk of needlessly lowering the testing coverage that is already available to us.

Acceptance criteria

  • AC1: An alert is raised if the test coverage within openQA tests is reduced without intention
  • AC2: Intended and acceptable openQA test coverage decreases are explicitly referenced, e.g. in open issues
  • AC3: Possibility to filter the data, to easily compare coverage between whichever products, code streams.
  • AC4: Possibility to access data by script/bot to process them further (dashboards, alerts, etc.)

Further details

As per previous discussions, Metabase can help for some parts of this story.


QA - action #88127: [tools][qem] Test coverage DB for maintenance updatesClosedjbaier_cz

QA - action #88485: [teregen] Fetch and store coverage info for each incidentResolvedjbaier_cz

QA - action #90401: [teregen] Integrate coverage information in a presentable way into test templateResolvedjbaier_cz

QA - action #90404: [teregen] Update TeReGen for deployment on qam2Resolvedjbaier_cz

QA - action #88536: Find out differences in openQA test coverage with metabaseResolvedhurhaj

action #91509: Easy way to check and compare coverage in multiple openQA instancesNew

action #91656: [qe-core] os-autoinst-distri-opensuse YAML schedule file comparisonNew

Related issues

Related to openQA Project - action #113614: [timeboxed:20h] [spike] Add openQA API for computing diff of executed test modulesNew2022-07-14


#1 Updated by okurz almost 2 years ago

  • Related to action #88536: Find out differences in openQA test coverage with metabase added

#2 Updated by okurz almost 2 years ago

  • Description updated (diff)
  • Status changed from Workable to Blocked
  • Assignee set to okurz

incorporated content from #88536

#3 Updated by okurz over 1 year ago

#88127 was resolved assuming that the initial request was fulfilled

#4 Updated by okurz over 1 year ago

  • Target version changed from Ready to future

#5 Updated by okurz about 1 year ago

  • Status changed from Blocked to New
  • Assignee deleted (okurz)

#6 Updated by okurz about 1 year ago

  • Parent task changed from #39719 to #102906

#8 Updated by jbaier_cz 7 months ago

  • Related to action #113614: [timeboxed:20h] [spike] Add openQA API for computing diff of executed test modules added

Also available in: Atom PDF