action #88127
closedopenQA Project - coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA
openQA Project - coordination #88229: [epic] Prevent unintended test coverage decrease
[tools][qem] Test coverage DB for maintenance updates
100%
Description
User story¶
As a member of QEM update squad, I want to know the extent of automation available and running for the package in the maintenance update I'm testing, to decide whether additional manual testing is needed.
Acceptance criteria¶
- AC1: output about the maximum coverage needs to take the tested product/code-stream into account
- AC2: URL to the test suite where tests are running needs to be provided in test report
- AC3: list of tests that are not running needs to be provided in test report
- AC4: entries in DB needs to be updatable for both adding new available coverage but also removing tests from the list
Further details¶
To illustrate rough idea of expected outcome in test report, lets assume we have MU containing packages ABC and XYZ for SLE 15 SP0,1,2 and that that both packages have two dedicated tests that have always been running only on SP1,2 and there's also one indirect test. The possible output in test report might look something like:
regression tests:
--------------------------
Regression tests are covered in openQA:
SLE 15
- indirect_test in <URL_to_latest>
SLE 15 SP1
- abc_first, abc_second, xyz_first in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>
SLE 15 SP2
- abc_first, abc_second, xyz_first, xyz_second in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,...
- indirect_test in <URL_to_latest>
WARNING: Decreased coverage, replace with manual testing!
SLE 15 SP1 - xyz_second
Updated by okurz over 3 years ago
- Parent task set to #88229
Discussed with vpelcak. We agreed that the topic is important and should be followed up with within the team SUSE QE Tools. We agreed that the general topic is broader than how the ticket currently reflects. To cover that I created #88229 as a parent to catch the more generic concept.
Further topics we discussed:
- http://10.67.183.130:8081/test/unscheduled_test_tracker_v1.html shows "recently unscheduled tests"
- https://qam.suse.de/openqa-summary-tests/ shows how "Packages" and "Summary" are parsed from openQA test modules. This should at best be provided for all products, including openSUSE products in a publically available form. Could the software that automatically parses the reports about "Packages" and "Summary" be published as free software and the reports be provided so that the openSUSE community has access for the regarding openSUSE products, e.g. Tumbleweed and Leap? This would even help with the new E&I goals by Sheng to have more free software projects published :)
- https://confluence.suse.com/display/~vpelcak/Draft+-+Closing+the+Gap+Between+Manual+Testing+and+Automation is a related document
- We have a "Definition of DONE" for openQA Tests, see https://progress.opensuse.org/projects/openqatests/wiki#Definition-of-DONEREADY . It already mentions 'Test modules that have been touched have updated metadata, e.g. "Maintainer" and "Summary" (#13034)'. All contributors should be reminded about this
@hurhaj @jbaier_cz I would like to support this story further but before we can start the implementation we should flesh out some details:
- Could you try to update the description of the ticket according to https://progress.opensuse.org/projects/openqav3/wiki/#Feature-requests ?
- Would you plan for this to reuse an existing database setup or develop something new?
- How would test maintainers be notified about unintended test coverage reductions?
Updated by szarate about 3 years ago
- Subject changed from Test coverage DB for maintenance updates to [tools][qem] Test coverage DB for maintenance updates
Updated by okurz about 3 years ago
- Related to action #88536: Find out differences in openQA test coverage with metabase added
Updated by okurz about 3 years ago
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan almost 3 years ago
- Description updated (diff)
- Status changed from New to Workable
Updated by livdywan almost 3 years ago
@jbaier_cz Can this be considered done? We talked about it a few days ago and afair the subtasks are all done now. But the ticket has no acceptance tests so not sure how to verify that everything does work 🤔️
Updated by jbaier_cz almost 3 years ago
- Status changed from Workable to Resolved
As the required changes in the template generator are done, I will considered it done. The template was enhanced to indicate problems as suggested in the further details. Additional steps will be needed to visualize internal data for checking and managing, however that is out of scope of this ticket.
Updated by jbaier_cz almost 3 years ago
- Related to action #93799: teregen: Improvement of usability of disabled testcases notification size:M added
Updated by hurhaj over 2 years ago
- Status changed from Resolved to Feedback
I've checked several test reports currently in the queue and it looks like only AC3 was met, list of tests that are not running.
Is there a plan to meet also AC1,2,4? If so, can I have a ticket number?
Updated by jbaier_cz over 2 years ago
From my point of view, looking at http://qam.suse.de/testreports/SUSE:Maintenance:19738:241935/log
AC1 is covered by the list of missing test suites (added by template generator) plus the list of results from openQA incidents jobs (added by mtui) plus the list of installation tests (added by template generator). Not sure what else can we add here to help the tester and at the same time keep the report still readable. Listing all individual tests would, in this case, create almost 260 records in the template.
AC2 was partly omitted as it is impossible to link to a missing test, we can however add at least a link to a test overview page (in this case it would be https://openqa.suse.de/tests/overview?distri=sle&build=%3A19738%3Alibxml2) if it helps. More (and up to date info, as template generator can't handle reruns and results generated after the template is commited) can be always found inside smelt (info from openqabot: https://maintenance.suse.de/request/241935/#comments)
AC3 is the new addition inside the report, surely we can improve here as well
AC4 is only partly implemented, addition of new tests is done automatically, all new discovered tests are immediately added into database; removing/editing of current entries is poo#90914
If you have any idea how to improve, please share you suggestions. Also, please note, there is a relevant poo#93799 about output format improvements.
Updated by hurhaj over 2 years ago
Let's try something easier first, look at https://qam.suse.de/testreports/SUSE:Maintenance:20432:245401/log for supportutils.
In https://openqa.suse.de/tests/6629518# there is a dedicated test https://openqa.suse.de/tests/6629518/modules/supportutils/steps/1/src with filled in metadata. But there is no mention of this in test report. It seems like for now there are only incidents being mentioned, but Test Repo is left out, correct? Yes, I understand there's new run twice a day, which makes it a nightmare. Maybe a simple path like 'Maintenance: Test Repo / Maintenance: SLE 15 SP2 Updates / mau-extratests2 / supportutils' would be helpfull.
As for the 'Results from openQA incidents jobs', I have doubts about how many people read it. It's all from one incident and personally I see no reason to list it whole instead of just providing a link to https://openqa.suse.de/tests/overview?distri=sle&version=15-SP2&build=%3A20432%3Asupportutils&groupid=306 .
Would that be a possible way out or did I just provide more mess to this?
Updated by jbaier_cz over 2 years ago
hurhaj wrote:
Let's try something easier first, look at https://qam.suse.de/testreports/SUSE:Maintenance:20432:245401/log for supportutils.
In https://openqa.suse.de/tests/6629518# there is a dedicated test https://openqa.suse.de/tests/6629518/modules/supportutils/steps/1/src with filled in metadata. But there is no mention of this in test report. It seems like for now there are only incidents being mentioned, but Test Repo is left out, correct? Yes, I understand there's new run twice a day, which makes it a nightmare. Maybe a simple path like 'Maintenance: Test Repo / Maintenance: SLE 15 SP2 Updates / mau-extratests2 / supportutils' would be helpfull.
Well, yes. This whole think is only for incident jobs and was not (yet) indented to include aggregates. The metadata from tests are not parsed in any way, if I am not entirely mistaken, that should be covered by poo#62114. Will that help you more with the AC1?
As for the 'Results from openQA incidents jobs', I have doubts about how many people read it. It's all from one incident and personally I see no reason to list it whole instead of just providing a link to https://openqa.suse.de/tests/overview?distri=sle&version=15-SP2&build=%3A20432%3Asupportutils&groupid=306 .
That's partly the same solution as proposed in poo#94600
Would that be a possible way out or did I just provide more mess to this?
If you agree, it might be worth to resurrect some of the mentioned poo's back to live.
Updated by hurhaj over 2 years ago
jbaier_cz wrote:
If you agree, it might be worth to resurrect some of the mentioned poo's back to live.
Definitely. And there's also https://progress.opensuse.org/issues/94480 , so again several poos that are stepping on each others' toes.