Project

General

Profile

coordination #88229

coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

[epic] Prevent unintended test coverage decrease

Added by okurz 9 months ago. Updated 4 months ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2021-02-08
Due date:
% Done:

75%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

Motivation

In general the idea or demand for "prevent test coverage decrease" is nothing new and also not limited to QAM. Similar example would be in Tumbleweed as well as in other products: At any point in time it could happen that an openQA test scenario or test module or parts of test code or a combination of those is not executed anymore for a certain product and that is unlikely to be realized because neither TTM nor the openQA maintenance bot would count a reduction in test coverage as an alert condition. github.com/os-autoinst/openqa_review reports about this as well as https://github.com/os-autoinst/openqa_review/blob/master/openqa_review/tumblesle_release.py which would not publish snapshots if less scenarios are found than in before. Both are not actively used for decision making about product releases.

User story

As a QE engineer, I would like to have a way of comparing openQA's tests (*.pm) utilization between our products, mostly before their release and after the release. With ~1500 tests available at this moment, it's very difficult to find out whether particular test is being used to it's full potential, if it is running on all products, code streams and architectures that it could and should run on. That brings a risk of needlessly lowering the testing coverage that is already available to us.

Acceptance criteria

  • AC1: An alert is raised if the test coverage within openQA tests is reduced without intention
  • AC2: Intended and acceptable openQA test coverage decreases are explicitly referenced, e.g. in open issues
  • AC3: Possibility to filter the data, to easily compare coverage between whichever products, code streams.
  • AC4: Possibility to access data by script/bot to process them further (dashboards, alerts, etc.)

Further details

As per previous discussions, Metabase can help for some parts of this story.


Subtasks

QA - action #88127: [tools][qem] Test coverage DB for maintenance updatesClosedjbaier_cz

QA - action #88485: [teregen] Fetch and store coverage info for each incidentResolvedjbaier_cz

QA - action #90401: [teregen] Integrate coverage information in a presentable way into test templateResolvedjbaier_cz

QA - action #90404: [teregen] Update TeReGen for deployment on qam2Resolvedjbaier_cz

QA - action #88536: Find out differences in openQA test coverage with metabaseResolvedhurhaj

openQA Tests - action #91509: [tools] Easy way to check and compare coverage in multiple openQA instancesNew

openQA Tests - action #91656: [qe-core] os-autoinst-distri-opensuse YAML schedule file comparisonNew

History

#1 Updated by okurz 9 months ago

  • Related to action #88536: Find out differences in openQA test coverage with metabase added

#2 Updated by okurz 8 months ago

  • Description updated (diff)
  • Status changed from Workable to Blocked
  • Assignee set to okurz

incorporated content from #88536

#3 Updated by okurz 5 months ago

#88127 was resolved assuming that the initial request was fulfilled

#4 Updated by okurz 4 months ago

  • Target version changed from Ready to future

Also available in: Atom PDF