Project

General

Profile

action #88127

openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

openQA Project - coordination #88229: [epic] Prevent unintended test coverage decrease

[tools][qem] Test coverage DB for maintenance updates

Added by hurhaj 8 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Start date:
2021-02-08
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

User story

As a member of QEM update squad, I want to know the extent of automation available and running for the package in the maintenance update I'm testing, to decide whether additional manual testing is needed.

Acceptance criteria

  • AC1: output about the maximum coverage needs to take the tested product/code-stream into account
  • AC2: URL to the test suite where tests are running needs to be provided in test report
  • AC3: list of tests that are not running needs to be provided in test report
  • AC4: entries in DB needs to be updatable for both adding new available coverage but also removing tests from the list

Further details

To illustrate rough idea of expected outcome in test report, lets assume we have MU containing packages ABC and XYZ for SLE 15 SP0,1,2 and that that both packages have two dedicated tests that have always been running only on SP1,2 and there's also one indirect test. The possible output in test report might look something like:

regression tests:
--------------------------
Regression tests are covered in openQA:
SLE 15
- indirect_test in <URL_to_latest>
SLE 15 SP1
- abc_first, abc_second, xyz_first in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>
SLE 15 SP2
- abc_first, abc_second, xyz_first, xyz_second in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>

WARNING: Decreased coverage, replace with manual testing!
SLE 15 SP1 - xyz_second

Subtasks

action #88485: [teregen] Fetch and store coverage info for each incidentResolvedjbaier_cz

action #90401: [teregen] Integrate coverage information in a presentable way into test templateResolvedjbaier_cz

action #90404: [teregen] Update TeReGen for deployment on qam2Resolvedjbaier_cz


Related issues

Related to QA - action #88536: Find out differences in openQA test coverage with metabaseResolved2021-02-12

Related to QA - action #93799: teregen: Improvement of usability of disabled testcases notificationNew2021-06-10

History

#1 Updated by jbaier_cz 8 months ago

  • Target version set to Ready

#2 Updated by okurz 8 months ago

  • Description updated (diff)

#3 Updated by okurz 8 months ago

  • Parent task set to #88229

Discussed with vpelcak. We agreed that the topic is important and should be followed up with within the team SUSE QE Tools. We agreed that the general topic is broader than how the ticket currently reflects. To cover that I created #88229 as a parent to catch the more generic concept.

Further topics we discussed:

hurhaj jbaier_cz I would like to support this story further but before we can start the implementation we should flesh out some details:

#4 Updated by hurhaj 8 months ago

  • Description updated (diff)

#6 Updated by szarate 8 months ago

  • Subject changed from Test coverage DB for maintenance updates to [tools][qem] Test coverage DB for maintenance updates

#7 Updated by okurz 8 months ago

  • Related to action #88536: Find out differences in openQA test coverage with metabase added

#9 Updated by okurz 6 months ago

Setting due date based on mean cycle time of SUSE QE Tools

#10 Updated by cdywan 5 months ago

  • Description updated (diff)
  • Status changed from New to Workable

#11 Updated by cdywan 5 months ago

jbaier_cz Can this be considered done? We talked about it a few days ago and afair the subtasks are all done now. But the ticket has no acceptance tests so not sure how to verify that everything does work 🤔️

#12 Updated by jbaier_cz 5 months ago

  • Status changed from Workable to Resolved

As the required changes in the template generator are done, I will considered it done. The template was enhanced to indicate problems as suggested in the further details. Additional steps will be needed to visualize internal data for checking and managing, however that is out of scope of this ticket.

#13 Updated by jbaier_cz 4 months ago

  • Related to action #93799: teregen: Improvement of usability of disabled testcases notification added

#14 Updated by hurhaj about 2 months ago

  • Status changed from Resolved to Feedback

I've checked several test reports currently in the queue and it looks like only AC3 was met, list of tests that are not running.

Is there a plan to meet also AC1,2,4? If so, can I have a ticket number?

#15 Updated by jbaier_cz about 2 months ago

From my point of view, looking at http://qam.suse.de/testreports/SUSE:Maintenance:19738:241935/log

AC1 is covered by the list of missing test suites (added by template generator) plus the list of results from openQA incidents jobs (added by mtui) plus the list of installation tests (added by template generator). Not sure what else can we add here to help the tester and at the same time keep the report still readable. Listing all individual tests would, in this case, create almost 260 records in the template.
AC2 was partly omitted as it is impossible to link to a missing test, we can however add at least a link to a test overview page (in this case it would be https://openqa.suse.de/tests/overview?distri=sle&build=%3A19738%3Alibxml2) if it helps. More (and up to date info, as template generator can't handle reruns and results generated after the template is commited) can be always found inside smelt (info from openqabot: https://maintenance.suse.de/request/241935/#comments)
AC3 is the new addition inside the report, surely we can improve here as well
AC4 is only partly implemented, addition of new tests is done automatically, all new discovered tests are immediately added into database; removing/editing of current entries is poo#90914

If you have any idea how to improve, please share you suggestions. Also, please note, there is a relevant poo#93799 about output format improvements.

#16 Updated by hurhaj about 2 months ago

Let's try something easier first, look at https://qam.suse.de/testreports/SUSE:Maintenance:20432:245401/log for supportutils.

In https://openqa.suse.de/tests/6629518# there is a dedicated test https://openqa.suse.de/tests/6629518/modules/supportutils/steps/1/src with filled in metadata. But there is no mention of this in test report. It seems like for now there are only incidents being mentioned, but Test Repo is left out, correct? Yes, I understand there's new run twice a day, which makes it a nightmare. Maybe a simple path like 'Maintenance: Test Repo / Maintenance: SLE 15 SP2 Updates / mau-extratests2 / supportutils' would be helpfull.

As for the 'Results from openQA incidents jobs', I have doubts about how many people read it. It's all from one incident and personally I see no reason to list it whole instead of just providing a link to https://openqa.suse.de/tests/overview?distri=sle&version=15-SP2&build=%3A20432%3Asupportutils&groupid=306 .

Would that be a possible way out or did I just provide more mess to this?

#17 Updated by jbaier_cz about 2 months ago

hurhaj wrote:

Let's try something easier first, look at https://qam.suse.de/testreports/SUSE:Maintenance:20432:245401/log for supportutils.

In https://openqa.suse.de/tests/6629518# there is a dedicated test https://openqa.suse.de/tests/6629518/modules/supportutils/steps/1/src with filled in metadata. But there is no mention of this in test report. It seems like for now there are only incidents being mentioned, but Test Repo is left out, correct? Yes, I understand there's new run twice a day, which makes it a nightmare. Maybe a simple path like 'Maintenance: Test Repo / Maintenance: SLE 15 SP2 Updates / mau-extratests2 / supportutils' would be helpfull.

Well, yes. This whole think is only for incident jobs and was not (yet) indented to include aggregates. The metadata from tests are not parsed in any way, if I am not entirely mistaken, that should be covered by poo#62114. Will that help you more with the AC1?

As for the 'Results from openQA incidents jobs', I have doubts about how many people read it. It's all from one incident and personally I see no reason to list it whole instead of just providing a link to https://openqa.suse.de/tests/overview?distri=sle&version=15-SP2&build=%3A20432%3Asupportutils&groupid=306 .

That's partly the same solution as proposed in poo#94600

Would that be a possible way out or did I just provide more mess to this?

If you agree, it might be worth to resurrect some of the mentioned poo's back to live.

#18 Updated by hurhaj about 2 months ago

jbaier_cz wrote:

If you agree, it might be worth to resurrect some of the mentioned poo's back to live.

Definitely. And there's also https://progress.opensuse.org/issues/94480 , so again several poos that are stepping on each others' toes.

#19 Updated by jbaier_cz about 1 month ago

  • Status changed from Feedback to Closed

Also available in: Atom PDF