Project

General

Profile

action #88127

openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

openQA Project - coordination #88229: [epic] Prevent unintended test coverage decrease

[tools][qem] Test coverage DB for maintenance updates

Added by hurhaj 5 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2021-02-08
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

User story

As a member of QEM update squad, I want to know the extent of automation available and running for the package in the maintenance update I'm testing, to decide whether additional manual testing is needed.

Acceptance criteria

  • AC1: output about the maximum coverage needs to take the tested product/code-stream into account
  • AC2: URL to the test suite where tests are running needs to be provided in test report
  • AC3: list of tests that are not running needs to be provided in test report
  • AC4: entries in DB needs to be updatable for both adding new available coverage but also removing tests from the list

Further details

To illustrate rough idea of expected outcome in test report, lets assume we have MU containing packages ABC and XYZ for SLE 15 SP0,1,2 and that that both packages have two dedicated tests that have always been running only on SP1,2 and there's also one indirect test. The possible output in test report might look something like:

regression tests:
--------------------------
Regression tests are covered in openQA:
SLE 15
- indirect_test in <URL_to_latest>
SLE 15 SP1
- abc_first, abc_second, xyz_first in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>
SLE 15 SP2
- abc_first, abc_second, xyz_first, xyz_second in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>

WARNING: Decreased coverage, replace with manual testing!
SLE 15 SP1 - xyz_second

Subtasks

action #88485: [teregen] Fetch and store coverage info for each incidentResolvedjbaier_cz

action #90401: [teregen] Integrate coverage information in a presentable way into test templateResolvedjbaier_cz

action #90404: [teregen] Update TeReGen for deployment on qam2Resolvedjbaier_cz


Related issues

Related to QA - action #88536: Find out differences in openQA test coverage with metabaseResolved2021-02-12

Related to QA - action #93799: Improvement of usability of disabled testcases notificationNew2021-06-102021-07-01

History

#1 Updated by jbaier_cz 5 months ago

  • Target version set to Ready

#2 Updated by okurz 5 months ago

  • Description updated (diff)

#3 Updated by okurz 5 months ago

  • Parent task set to #88229

Discussed with vpelcak. We agreed that the topic is important and should be followed up with within the team SUSE QE Tools. We agreed that the general topic is broader than how the ticket currently reflects. To cover that I created #88229 as a parent to catch the more generic concept.

Further topics we discussed:

hurhaj jbaier_cz I would like to support this story further but before we can start the implementation we should flesh out some details:

#4 Updated by hurhaj 5 months ago

  • Description updated (diff)

#6 Updated by szarate 4 months ago

  • Subject changed from Test coverage DB for maintenance updates to [tools][qem] Test coverage DB for maintenance updates

#7 Updated by okurz 4 months ago

  • Related to action #88536: Find out differences in openQA test coverage with metabase added

#9 Updated by okurz 3 months ago

Setting due date based on mean cycle time of SUSE QE Tools

#10 Updated by cdywan about 2 months ago

  • Description updated (diff)
  • Status changed from New to Workable

#11 Updated by cdywan about 1 month ago

jbaier_cz Can this be considered done? We talked about it a few days ago and afair the subtasks are all done now. But the ticket has no acceptance tests so not sure how to verify that everything does work 🤔️

#12 Updated by jbaier_cz about 1 month ago

  • Status changed from Workable to Resolved

As the required changes in the template generator are done, I will considered it done. The template was enhanced to indicate problems as suggested in the further details. Additional steps will be needed to visualize internal data for checking and managing, however that is out of scope of this ticket.

#13 Updated by jbaier_cz 7 days ago

  • Related to action #93799: Improvement of usability of disabled testcases notification added

Also available in: Atom PDF