Project

General

Profile

Actions

action #88127

closed

openQA Project - coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA

openQA Project - coordination #88229: [epic] Prevent unintended test coverage decrease

[tools][qem] Test coverage DB for maintenance updates

Added by hurhaj over 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Start date:
2021-02-08
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

User story

As a member of QEM update squad, I want to know the extent of automation available and running for the package in the maintenance update I'm testing, to decide whether additional manual testing is needed.

Acceptance criteria

  • AC1: output about the maximum coverage needs to take the tested product/code-stream into account
  • AC2: URL to the test suite where tests are running needs to be provided in test report
  • AC3: list of tests that are not running needs to be provided in test report
  • AC4: entries in DB needs to be updatable for both adding new available coverage but also removing tests from the list

Further details

To illustrate rough idea of expected outcome in test report, lets assume we have MU containing packages ABC and XYZ for SLE 15 SP0,1,2 and that that both packages have two dedicated tests that have always been running only on SP1,2 and there's also one indirect test. The possible output in test report might look something like:

regression tests:
--------------------------
Regression tests are covered in openQA:
SLE 15
- indirect_test in <URL_to_latest>
SLE 15 SP1
- abc_first, abc_second, xyz_first in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>
SLE 15 SP2
- abc_first, abc_second, xyz_first, xyz_second in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>

WARNING: Decreased coverage, replace with manual testing!
SLE 15 SP1 - xyz_second

Subtasks 3 (0 open3 closed)

action #88485: [teregen] Fetch and store coverage info for each incidentResolvedjbaier_cz2021-02-08

Actions
action #90401: [teregen] Integrate coverage information in a presentable way into test templateResolvedjbaier_cz2021-03-29

Actions
action #90404: [teregen] Update TeReGen for deployment on qam2Resolvedjbaier_cz2021-04-09

Actions

Related issues 2 (0 open2 closed)

Related to QA - action #88536: Find out differences in openQA test coverage with metabaseResolvedhurhaj2021-02-12

Actions
Related to QA - action #93799: teregen: Improvement of usability of disabled testcases notification size:MResolvedjbaier_cz2021-06-10

Actions
Actions #1

Updated by jbaier_cz over 3 years ago

  • Target version set to Ready
Actions #2

Updated by okurz over 3 years ago

  • Description updated (diff)
Actions #3

Updated by okurz over 3 years ago

  • Parent task set to #88229

Discussed with vpelcak. We agreed that the topic is important and should be followed up with within the team SUSE QE Tools. We agreed that the general topic is broader than how the ticket currently reflects. To cover that I created #88229 as a parent to catch the more generic concept.

Further topics we discussed:

@hurhaj @jbaier_cz I would like to support this story further but before we can start the implementation we should flesh out some details:

Actions #4

Updated by hurhaj over 3 years ago

  • Description updated (diff)
Actions #6

Updated by szarate over 3 years ago

  • Subject changed from Test coverage DB for maintenance updates to [tools][qem] Test coverage DB for maintenance updates
Actions #7

Updated by okurz over 3 years ago

  • Related to action #88536: Find out differences in openQA test coverage with metabase added
Actions #9

Updated by okurz about 3 years ago

Setting due date based on mean cycle time of SUSE QE Tools

Actions #10

Updated by livdywan about 3 years ago

  • Description updated (diff)
  • Status changed from New to Workable
Actions #11

Updated by livdywan about 3 years ago

@jbaier_cz Can this be considered done? We talked about it a few days ago and afair the subtasks are all done now. But the ticket has no acceptance tests so not sure how to verify that everything does work 🤔️

Actions #12

Updated by jbaier_cz about 3 years ago

  • Status changed from Workable to Resolved

As the required changes in the template generator are done, I will considered it done. The template was enhanced to indicate problems as suggested in the further details. Additional steps will be needed to visualize internal data for checking and managing, however that is out of scope of this ticket.

Actions #13

Updated by jbaier_cz almost 3 years ago

  • Related to action #93799: teregen: Improvement of usability of disabled testcases notification size:M added
Actions #14

Updated by hurhaj almost 3 years ago

  • Status changed from Resolved to Feedback

I've checked several test reports currently in the queue and it looks like only AC3 was met, list of tests that are not running.

Is there a plan to meet also AC1,2,4? If so, can I have a ticket number?

Actions #15

Updated by jbaier_cz almost 3 years ago

From my point of view, looking at http://qam.suse.de/testreports/SUSE:Maintenance:19738:241935/log

AC1 is covered by the list of missing test suites (added by template generator) plus the list of results from openQA incidents jobs (added by mtui) plus the list of installation tests (added by template generator). Not sure what else can we add here to help the tester and at the same time keep the report still readable. Listing all individual tests would, in this case, create almost 260 records in the template.
AC2 was partly omitted as it is impossible to link to a missing test, we can however add at least a link to a test overview page (in this case it would be https://openqa.suse.de/tests/overview?distri=sle&build=%3A19738%3Alibxml2) if it helps. More (and up to date info, as template generator can't handle reruns and results generated after the template is commited) can be always found inside smelt (info from openqabot: https://maintenance.suse.de/request/241935/#comments)
AC3 is the new addition inside the report, surely we can improve here as well
AC4 is only partly implemented, addition of new tests is done automatically, all new discovered tests are immediately added into database; removing/editing of current entries is poo#90914

If you have any idea how to improve, please share you suggestions. Also, please note, there is a relevant poo#93799 about output format improvements.

Actions #16

Updated by hurhaj almost 3 years ago

Let's try something easier first, look at https://qam.suse.de/testreports/SUSE:Maintenance:20432:245401/log for supportutils.

In https://openqa.suse.de/tests/6629518# there is a dedicated test https://openqa.suse.de/tests/6629518/modules/supportutils/steps/1/src with filled in metadata. But there is no mention of this in test report. It seems like for now there are only incidents being mentioned, but Test Repo is left out, correct? Yes, I understand there's new run twice a day, which makes it a nightmare. Maybe a simple path like 'Maintenance: Test Repo / Maintenance: SLE 15 SP2 Updates / mau-extratests2 / supportutils' would be helpfull.

As for the 'Results from openQA incidents jobs', I have doubts about how many people read it. It's all from one incident and personally I see no reason to list it whole instead of just providing a link to https://openqa.suse.de/tests/overview?distri=sle&version=15-SP2&build=%3A20432%3Asupportutils&groupid=306 .

Would that be a possible way out or did I just provide more mess to this?

Actions #17

Updated by jbaier_cz almost 3 years ago

hurhaj wrote:

Let's try something easier first, look at https://qam.suse.de/testreports/SUSE:Maintenance:20432:245401/log for supportutils.

In https://openqa.suse.de/tests/6629518# there is a dedicated test https://openqa.suse.de/tests/6629518/modules/supportutils/steps/1/src with filled in metadata. But there is no mention of this in test report. It seems like for now there are only incidents being mentioned, but Test Repo is left out, correct? Yes, I understand there's new run twice a day, which makes it a nightmare. Maybe a simple path like 'Maintenance: Test Repo / Maintenance: SLE 15 SP2 Updates / mau-extratests2 / supportutils' would be helpfull.

Well, yes. This whole think is only for incident jobs and was not (yet) indented to include aggregates. The metadata from tests are not parsed in any way, if I am not entirely mistaken, that should be covered by poo#62114. Will that help you more with the AC1?

As for the 'Results from openQA incidents jobs', I have doubts about how many people read it. It's all from one incident and personally I see no reason to list it whole instead of just providing a link to https://openqa.suse.de/tests/overview?distri=sle&version=15-SP2&build=%3A20432%3Asupportutils&groupid=306 .

That's partly the same solution as proposed in poo#94600

Would that be a possible way out or did I just provide more mess to this?

If you agree, it might be worth to resurrect some of the mentioned poo's back to live.

Actions #18

Updated by hurhaj almost 3 years ago

jbaier_cz wrote:

If you agree, it might be worth to resurrect some of the mentioned poo's back to live.

Definitely. And there's also https://progress.opensuse.org/issues/94480 , so again several poos that are stepping on each others' toes.

Actions #19

Updated by jbaier_cz almost 3 years ago

  • Status changed from Feedback to Closed
Actions

Also available in: Atom PDF