Project

General

Profile

coordination #89062

QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

[epic] Simplify review for SUSE QAM

Added by okurz 10 months ago. Updated 13 days ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2021-04-21
Due date:
2021-12-09
% Done:

60%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

Motivation

https://confluence.suse.com/pages/viewpage.action?spaceKey=openqa&title=QAM+openQA+review+guide explains the current process for openQA test review within SUSE QAM. We have already learned that they rely on some special rules which have not been foreseen like this within openQA, e.g. they see the black certificate as "tests reviewed and failures can be ignored" rather than "every job has a comment but not necessarily more than that" . We can try to improve openQA so that it features can be used in the correct way and review workflows become even easier.

Ideas

/tests can already be configured like https://openqa.opensuse.org/tests/?resultfilter=Failed&resultfilter=Incomplete&resultfilter=timeout_exceeded , even over UI elements. Also /tests already shows a comments icon but makes no distinction like on /tests/overview or "next & previous" between "plain comment", "label" or "bugref".

  • DONE: Distinguish between "plain comment", "label" and "bugref" same as we do in /tests/overview and "next & previous" -> #94937
  • Add query parameter to filter only jobs that need review, e.g. "todo=1", same as on /tests/overview . At best use the same logic to prevent duplication
  • Allow to show more than the default 500 finished jobs on /tests on request
  • On /tests also limit "running" and "scheduled" jobs, same as finished, but give the user the opportunity to show more or the additional ones, e.g. on an extra page for each running, scheduled, finished
  • Optional: Add UI element to configure this, e.g. again like on /tests/overview the "todo" checkbox
  • Optional: Add further filter query parameters (+doc, +UI control elements) for
    • job group include/exclude regex (similar to what we have on the index page)
    • filter for "no group/any group"
    • test module include/exclude regex
    • further filter possibilities, e.g. medium, flavor, arch, machine

Subtasks

action #91542: openQA API what jobs were/are testing X incident/packageResolvedilausuch

action #91658: Make "black certificate" stricter to only show when /tests/overview?todo=1 is empty, i.e. no unlabeled failuresResolvedmkittler

action #91770: Optional job investigation information in "investigation" tab rather than commentsNew

action #91773: Automatic replacement of openQA job URLs preview of openQA size:MResolvedtinita

openQA Infrastructure - action #92034: Re-enable openqa-investigate options after the black certificate now only shows properly "reviewed" jobsResolvedmkittler

QA - action #92851: Workshop series proposal "How SUSE QE teams review openQA test results"Resolvedcdywan

action #93246: List all unreviewed failed (or incomplete) jobs on /tests on request size:MWorkable

action #94937: Distinguish comment types on jobs on /tests (maybe optional) size:SResolvedtinita

action #102374: Support use of "force_result" via ticket title in auto-review size:MIn Progresstinita

History

#1 Updated by okurz 8 months ago

Talked with geor. Mainly the information "openQA API what jobs were/are testing X incident/package", see #91542 , is missing from the maintenance test template. But the "comments" tab on smelt includes the most recent test results and hence. Any non-passed test on the most recent comment must be seen as a stopper preventing the approval. In the worst case the corresponding test scenario is pulled from the test schedule according to process, in the better case the test is fixed within a reasonable time frame.

Other ideas and suggestions:

  • geor: The template generator output should just output non-passed tests to not overwhelm the reader
  • okurz: the template generator output mixes both "optional information" with "necessary status". Both should be separated because every power user otherwise ignores relevant text very soon

#2 Updated by okurz 8 months ago

  • Parent task changed from #39719 to #91646

#3 Updated by okurz 7 months ago

Conducted a SUSE QE tools workshop

open points and ideas from API playground:

  • Q: how to filter by test modules and their results from API
    • A: Use the modules and modules_result parameter, e.g. openqa-cli api --osd jobs version=15-SP3 modules=shutdown modules_result=failed
  • Q: API equivalent of /tests/overview
    • A: You can use the JSON representation of the same page. Just append ".json" to the URL, e.g. https://openqa.suse.de/tests/overview.json?groupid=110&distri=sle&build=187.1&version=15-SP3
  • Q: select jobs with "settings" filter, e.g. "ISSUE" for SUSE QAM aggregrated tests
    • A: Part of it is answered in #91542#note-23 already but this could be made simpler in the openQA API
  • Idea: have more subcommands in openqa-cli, e.g. to look up module results or jobs where certain modules are executed or where these modules fail
  • Q: openqa-cli api --osd jobs version=15-SP3 modules=shutdown modules_result=failed result=failed limit=1000 latest=1 | jq '.[][] | "\(.settings.FLAVOR), \(.group), \(.name): https://openqa.suse.de/t\(.id) (\(.result))"' shows the following jobs: "Online, Functional, sle-15-SP3-Online-x86_64-Build183.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5951445 (failed)" "Online, Functional, sle-15-SP3-Online-x86_64-Build186.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5975627 (failed)" "Online, Functional, sle-15-SP3-Online-x86_64-Build187.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5990471 (failed)" which are all from the same scenario. Shouldn't latest=1 mean to show only one of these?

#4 Updated by okurz 6 months ago

  • Description updated (diff)
  • Status changed from New to Blocked
  • Assignee set to okurz

#5 Updated by okurz 5 months ago

  • Target version changed from future to Ready

#6 Updated by okurz 5 months ago

  • Description updated (diff)

#7 Updated by okurz 4 months ago

  • Description updated (diff)

Also available in: Atom PDF