As a member of QEM update squad, I want to know the extent of automation available and running for the package in the maintenance update I'm testing, to decide whether additional manual testing is needed.
- AC1: output about the maximum coverage needs to take the tested product/code-stream into account
- AC2: URL to the test suite where tests are running needs to be provided in test report
- AC3: list of tests that are not running needs to be provided in test report
- AC4: entries in DB needs to be updatable for both adding new available coverage but also removing tests from the list
To illustrate rough idea of expected outcome in test report, lets assume we have MU containing packages ABC and XYZ for SLE 15 SP0,1,2 and that that both packages have two dedicated tests that have always been running only on SP1,2 and there's also one indirect test. The possible output in test report might look something like:
regression tests: -------------------------- Regression tests are covered in openQA: SLE 15 - indirect_test in <URL_to_latest> SLE 15 SP1 - abc_first, abc_second, xyz_first in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,... - indirect_test in <URL_to_latest> SLE 15 SP2 - abc_first, abc_second, xyz_first, xyz_second in <URL_to_latest_SERVER>, <URL_to_latest_DESKTOP>,... - indirect_test in <URL_to_latest> WARNING: Decreased coverage, replace with manual testing! SLE 15 SP1 - xyz_second
- Parent task set to #88229
Discussed with vpelcak. We agreed that the topic is important and should be followed up with within the team SUSE QE Tools. We agreed that the general topic is broader than how the ticket currently reflects. To cover that I created #88229 as a parent to catch the more generic concept.
Further topics we discussed:
- http://10.67.183.130:8081/test/unscheduled_test_tracker_v1.html shows "recently unscheduled tests"
- https://qam.suse.de/openqa-summary-tests/ shows how "Packages" and "Summary" are parsed from openQA test modules. This should at best be provided for all products, including openSUSE products in a publically available form. Could the software that automatically parses the reports about "Packages" and "Summary" be published as free software and the reports be provided so that the openSUSE community has access for the regarding openSUSE products, e.g. Tumbleweed and Leap? This would even help with the new E&I goals by Sheng to have more free software projects published :)
- https://confluence.suse.com/display/~vpelcak/Draft+-+Closing+the+Gap+Between+Manual+Testing+and+Automation is a related document
- We have a "Definition of DONE" for openQA Tests, see https://progress.opensuse.org/projects/openqatests/wiki#Definition-of-DONEREADY . It already mentions 'Test modules that have been touched have updated metadata, e.g. "Maintainer" and "Summary" (#13034)'. All contributors should be reminded about this
- Could you try to update the description of the ticket according to https://progress.opensuse.org/projects/openqav3/wiki/#Feature-requests ?
- Would you plan for this to reuse an existing database setup or develop something new?
- How would test maintainers be notified about unintended test coverage reductions?
#12 Updated by jbaier_cz about 1 month ago
- Status changed from Workable to Resolved
As the required changes in the template generator are done, I will considered it done. The template was enhanced to indicate problems as suggested in the further details. Additional steps will be needed to visualize internal data for checking and managing, however that is out of scope of this ticket.