action #88127
closedopenQA Project - coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA
openQA Project - coordination #88229: [epic] Prevent unintended test coverage decrease
[tools][qem] Test coverage DB for maintenance updates
100%
Description
User story¶
As a member of QEM update squad, I want to know the extent of automation available and running for the package in the maintenance update I'm testing, to decide whether additional manual testing is needed.
Acceptance criteria¶
- AC1: output about the maximum coverage needs to take the tested product/code-stream into account
- AC2: URL to the test suite where tests are running needs to be provided in test report
- AC3: list of tests that are not running needs to be provided in test report
- AC4: entries in DB needs to be updatable for both adding new available coverage but also removing tests from the list
Further details¶
To illustrate rough idea of expected outcome in test report, lets assume we have MU containing packages ABC and XYZ for SLE 15 SP0,1,2 and that that both packages have two dedicated tests that have always been running only on SP1,2 and there's also one indirect test. The possible output in test report might look something like:
regression tests:
--------------------------
Regression tests are covered in openQA:
SLE 15
- indirect_test in <URL_to_latest>
SLE 15 SP1
- abc_first, abc_second, xyz_first in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>
SLE 15 SP2
- abc_first, abc_second, xyz_first, xyz_second in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>
WARNING: Decreased coverage, replace with manual testing!
SLE 15 SP1 - xyz_second