Project

General

Profile

Actions

coordination #89062

closed

QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

[epic] Simplify review for SUSE QAM

Added by okurz about 3 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2021-04-21
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Motivation

https://confluence.suse.com/pages/viewpage.action?spaceKey=openqa&title=QAM+openQA+review+guide explains the current process for openQA test review within SUSE QAM. We have already learned that they rely on some special rules which have not been foreseen like this within openQA, e.g. they see the black certificate as "tests reviewed and failures can be ignored" rather than "every job has a comment but not necessarily more than that" . We can try to improve openQA so that it features can be used in the correct way and review workflows become even easier.

Ideas

/tests can already be configured like https://openqa.opensuse.org/tests/?resultfilter=Failed&resultfilter=Incomplete&resultfilter=timeout_exceeded , even over UI elements. Also /tests already shows a comments icon but makes no distinction like on /tests/overview or "next & previous" between "plain comment", "label" or "bugref".

  • DONE: Distinguish between "plain comment", "label" and "bugref" same as we do in /tests/overview and "next & previous" -> #94937
  • DONE: Add query parameter to filter only jobs that need review, e.g. "todo=1", same as on /tests/overview . At best use the same logic to prevent duplication

Subtasks 13 (0 open13 closed)

action #91542: openQA API what jobs were/are testing X incident/packageResolvedilausuch2021-04-21

Actions
action #91658: Make "black certificate" stricter to only show when /tests/overview?todo=1 is empty, i.e. no unlabeled failuresResolvedmkittler2021-04-23

Actions
action #91773: Automatic replacement of openQA job URLs preview of openQA size:MResolvedtinita2021-04-26

Actions
openQA Infrastructure - action #92034: Re-enable openqa-investigate options after the black certificate now only shows properly "reviewed" jobsResolvedmkittler

Actions
QA - action #92851: Workshop series proposal "How SUSE QE teams review openQA test results"Resolvedlivdywan2021-05-19

Actions
coordination #93246: [epic] List all unreviewed failed (or incomplete) jobs on /tests on request size:MResolvedokurz2021-12-09

Actions
action #103765: Support for "todo" query parameter on /tests, same as /tests/overview size:MResolvedtinita2021-12-09

Actions
action #104995: Add UI element and help text for "todo" query parameter on /tests, similar as /tests/overviewResolvedmkittler

Actions
action #105001: Add doc for "todo" query parameter on /tests, similar to /tests/overview size:SResolvedkodymo2022-01-18

Actions
action #94937: Distinguish comment types on jobs on /tests (maybe optional) size:SResolvedtinita2021-05-28

Actions
action #102374: Support use of force_result via ticket title in auto-review size:MResolvedtinita2021-11-14

Actions
openqa-force-result #109857: Secure auto-review+force_result size:M auto_review:"Failed to download gobbledeegoop":force_result:softfailedResolvedlivdywanActions
QA - action #107014: trigger openqa-trigger-bisect-jobs from our automatic investigations whenever the cause is not already known size:MResolvedtinita2022-02-17

Actions
Actions #1

Updated by okurz about 3 years ago

Talked with geor. Mainly the information "openQA API what jobs were/are testing X incident/package", see #91542 , is missing from the maintenance test template. But the "comments" tab on smelt includes the most recent test results and hence. Any non-passed test on the most recent comment must be seen as a stopper preventing the approval. In the worst case the corresponding test scenario is pulled from the test schedule according to process, in the better case the test is fixed within a reasonable time frame.

Other ideas and suggestions:

  • geor: The template generator output should just output non-passed tests to not overwhelm the reader
  • okurz: the template generator output mixes both "optional information" with "necessary status". Both should be separated because every power user otherwise ignores relevant text very soon
Actions #2

Updated by okurz about 3 years ago

  • Parent task changed from #39719 to #91646
Actions #3

Updated by okurz almost 3 years ago

Conducted a SUSE QE tools workshop

open points and ideas from API playground:

  • Q: how to filter by test modules and their results from API
    • A: Use the modules and modules_result parameter, e.g. openqa-cli api --osd jobs version=15-SP3 modules=shutdown modules_result=failed
  • Q: API equivalent of /tests/overview
    • A: You can use the JSON representation of the same page. Just append ".json" to the URL, e.g. https://openqa.suse.de/tests/overview.json?groupid=110&distri=sle&build=187.1&version=15-SP3
  • Q: select jobs with "settings" filter, e.g. "ISSUE" for SUSE QAM aggregrated tests
    • A: Part of it is answered in #91542#note-23 already but this could be made simpler in the openQA API
  • Idea: have more subcommands in openqa-cli, e.g. to look up module results or jobs where certain modules are executed or where these modules fail
  • Q: openqa-cli api --osd jobs version=15-SP3 modules=shutdown modules_result=failed result=failed limit=1000 latest=1 | jq '.[][] | "\(.settings.FLAVOR), \(.group), \(.name): https://openqa.suse.de/t\(.id) (\(.result))"' shows the following jobs: "Online, Functional, sle-15-SP3-Online-x86_64-Build183.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5951445 (failed)" "Online, Functional, sle-15-SP3-Online-x86_64-Build186.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5975627 (failed)" "Online, Functional, sle-15-SP3-Online-x86_64-Build187.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5990471 (failed)" which are all from the same scenario. Shouldn't latest=1 mean to show only one of these?
Actions #4

Updated by okurz almost 3 years ago

  • Description updated (diff)
  • Status changed from New to Blocked
  • Assignee set to okurz
Actions #5

Updated by okurz almost 3 years ago

  • Target version changed from future to Ready
Actions #6

Updated by okurz almost 3 years ago

  • Description updated (diff)
Actions #7

Updated by okurz almost 3 years ago

  • Description updated (diff)
Actions #8

Updated by okurz about 2 years ago

  • Description updated (diff)
  • Status changed from Blocked to Resolved

Moved some optional ideas to a new openQA filter specific ticket #111072, rest is resolved.

Actions

Also available in: Atom PDF