coordination #89062
closedQA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release
[epic] Simplify review for SUSE QAM
Description
Motivation¶
https://confluence.suse.com/pages/viewpage.action?spaceKey=openqa&title=QAM+openQA+review+guide explains the current process for openQA test review within SUSE QAM. We have already learned that they rely on some special rules which have not been foreseen like this within openQA, e.g. they see the black certificate as "tests reviewed and failures can be ignored" rather than "every job has a comment but not necessarily more than that" . We can try to improve openQA so that it features can be used in the correct way and review workflows become even easier.
Ideas¶
/tests can already be configured like https://openqa.opensuse.org/tests/?resultfilter=Failed&resultfilter=Incomplete&resultfilter=timeout_exceeded , even over UI elements. Also /tests already shows a comments icon but makes no distinction like on /tests/overview or "next & previous" between "plain comment", "label" or "bugref".
- DONE: Distinguish between "plain comment", "label" and "bugref" same as we do in /tests/overview and "next & previous" -> #94937
- DONE: Add query parameter to filter only jobs that need review, e.g. "todo=1", same as on /tests/overview . At best use the same logic to prevent duplication
Updated by okurz about 3 years ago
Talked with geor. Mainly the information "openQA API what jobs were/are testing X incident/package", see #91542 , is missing from the maintenance test template. But the "comments" tab on smelt includes the most recent test results and hence. Any non-passed test on the most recent comment must be seen as a stopper preventing the approval. In the worst case the corresponding test scenario is pulled from the test schedule according to process, in the better case the test is fixed within a reasonable time frame.
Other ideas and suggestions:
- geor: The template generator output should just output non-passed tests to not overwhelm the reader
- okurz: the template generator output mixes both "optional information" with "necessary status". Both should be separated because every power user otherwise ignores relevant text very soon
Updated by okurz almost 3 years ago
Conducted a SUSE QE tools workshop
open points and ideas from API playground:
- Q: how to filter by test modules and their results from API
- A: Use the
modulesandmodules_resultparameter, e.g.openqa-cli api --osd jobs version=15-SP3 modules=shutdown modules_result=failed
- A: Use the
- Q: API equivalent of /tests/overview
- A: You can use the JSON representation of the same page. Just append ".json" to the URL, e.g.
https://openqa.suse.de/tests/overview.json?groupid=110&distri=sle&build=187.1&version=15-SP3
- A: You can use the JSON representation of the same page. Just append ".json" to the URL, e.g.
- Q: select jobs with "settings" filter, e.g. "ISSUE" for SUSE QAM aggregrated tests
- A: Part of it is answered in #91542#note-23 already but this could be made simpler in the openQA API
- Idea: have more subcommands in openqa-cli, e.g. to look up module results or jobs where certain modules are executed or where these modules fail
- Q:
openqa-cli api --osd jobs version=15-SP3 modules=shutdown modules_result=failed result=failed limit=1000 latest=1 | jq '.[][] | "\(.settings.FLAVOR), \(.group), \(.name): https://openqa.suse.de/t\(.id) (\(.result))"'shows the following jobs: "Online, Functional, sle-15-SP3-Online-x86_64-Build183.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5951445 (failed)" "Online, Functional, sle-15-SP3-Online-x86_64-Build186.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5975627 (failed)" "Online, Functional, sle-15-SP3-Online-x86_64-Build187.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5990471 (failed)" which are all from the same scenario. Shouldn'tlatest=1mean to show only one of these?
Updated by okurz almost 3 years ago
- Description updated (diff)
- Status changed from New to Blocked
- Assignee set to okurz