action #124473
opencoordination #99303: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, approval and release
[tools] Automatic regression tests export from openQA
0%
Description
Motivation¶
With our progression in automation of regression tests we reached the state that many updates just need the results to be looked up in openQA and linked in our testreports.
That is pretty repetitive manual work which begs for the automation.
Acceptance criteria¶
- AC1: Automation fills the links to the appropriate regression tests in openQA and their result into the testreport of the update.
Suggestions¶
- This will perhaps require knowledge of our test coverage
- Maybe extension of openQA API will be needed
Updated by okurz almost 2 years ago
- Target version set to future
Discussed with vpelcak. The test report needs to include information about the test coverage which allows a 2nd level reviewer to crosscheck the test coverage reviewed by test reviewers. The generated test reports already contain openQA references but here we need additional job informations. This is about aggregate tests only. vpelcak showed me an example looking for "multipath" tests. For that we needed to look around on the webUI and look primarily for test modules, maybe other scenarios as well. We can try to use the openQA built-in search functionality.
One idea: An openQA search link like https://openqa.suse.de/search?q=multipath but then need to filter the thousands of search results to match only something like "Maintenance". We could include such search term automatically in reports but often the exact search term is not obvious.
Also there is the "metadata" field in test modules.
There should be more related tickets that we should link here.
Updated by okurz almost 2 years ago
- Related to action #94600: [tools][mtui] Communicate reduced visibility of openQA incident related coverage in log added
Updated by okurz almost 2 years ago
https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Definition-of-DONE mentions already Test modules that have been touched have updated metadata, e.g. "Maintainer" and "Summary" which is also ensured by CI so we could extend that for additional metadata as described in e.g. #77002
Updated by okurz almost 2 years ago
- Related to coordination #110016: [epic][teregen] teregen (maintenance test report template generator) improvements added