⚲

Project

General

Profile

Home
Projects
Help

Search:

openQA Project

All Projects

QA » openQA Project

Overview
Activity
Roadmap
Issues
Wiki

Custom queries

All 'new' issues w/o assignee, sorted by version/priority
All auto_review tickets
All auto_review+force_result tickets
openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (ready issues)
QE tools team - backlog (w/o infra)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE Tools team - due soon
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (w/o infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

coordination #89062

closed

QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

[epic] Simplify review for SUSE QAM

Added by okurz about 3 years ago. Updated almost 2 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

okurz

Category:

Feature requests

Target version:

Ready

Start date:

2021-04-21

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Description

Motivation¶

https://confluence.suse.com/pages/viewpage.action?spaceKey=openqa&title=QAM+openQA+review+guide explains the current process for openQA test review within SUSE QAM. We have already learned that they rely on some special rules which have not been foreseen like this within openQA, e.g. they see the black certificate as "tests reviewed and failures can be ignored" rather than "every job has a comment but not necessarily more than that" . We can try to improve openQA so that it features can be used in the correct way and review workflows become even easier.

Ideas¶

/tests can already be configured like https://openqa.opensuse.org/tests/?resultfilter=Failed&resultfilter=Incomplete&resultfilter=timeout_exceeded , even over UI elements. Also /tests already shows a comments icon but makes no distinction like on /tests/overview or "next & previous" between "plain comment", "label" or "bugref".

DONE: Distinguish between "plain comment", "label" and "bugref" same as we do in /tests/overview and "next & previous" -> #94937
DONE: Add query parameter to filter only jobs that need review, e.g. "todo=1", same as on /tests/overview . At best use the same logic to prevent duplication

Subtasks 13 (0 open — 13 closed)

action #91542: openQA API what jobs were/are testing X incident/package

Resolved

ilausuch

2021-04-21

Actions

action #91658: Make "black certificate" stricter to only show when /tests/overview?todo=1 is empty, i.e. no unlabeled failures

Resolved

mkittler

2021-04-23

Actions

action #91773: Automatic replacement of openQA job URLs preview of openQA size:M

Resolved

tinita

2021-04-26

Actions

openQA Infrastructure - action #92034: Re-enable openqa-investigate options after the black certificate now only shows properly "reviewed" jobs

Resolved

mkittler

Actions

QA - action #92851: Workshop series proposal "How SUSE QE teams review openQA test results"

Resolved

livdywan

2021-05-19

Actions

coordination #93246: [epic] List all unreviewed failed (or incomplete) jobs on /tests on request size:M

Resolved

okurz

2021-12-09

Actions

action #103765: Support for "todo" query parameter on /tests, same as /tests/overview size:M

Resolved

tinita

2021-12-09

Actions

action #104995: Add UI element and help text for "todo" query parameter on /tests, similar as /tests/overview

Resolved

mkittler

Actions

action #105001: Add doc for "todo" query parameter on /tests, similar to /tests/overview size:S

Resolved

kodymo

2022-01-18

Actions

action #94937: Distinguish comment types on jobs on /tests (maybe optional) size:S

Resolved

tinita

2021-05-28

Actions

action #102374: Support use of force_result via ticket title in auto-review size:M

Resolved

tinita

2021-11-14

Actions

openqa-force-result #109857: Secure auto-review+force_result size:M auto_review:"Failed to download gobbledeegoop":force_result:softfailed

Resolved

livdywan

Actions

QA - action #107014: trigger openqa-trigger-bisect-jobs from our automatic investigations whenever the cause is not already known size:M

Resolved

tinita

2022-02-17

Actions

History
Notes
Property changes

Actions

Copy link

Updated by okurz about 3 years ago

Talked with geor. Mainly the information "openQA API what jobs were/are testing X incident/package", see #91542 , is missing from the maintenance test template. But the "comments" tab on smelt includes the most recent test results and hence. Any non-passed test on the most recent comment must be seen as a stopper preventing the approval. In the worst case the corresponding test scenario is pulled from the test schedule according to process, in the better case the test is fixed within a reasonable time frame.

Other ideas and suggestions:

geor: The template generator output should just output non-passed tests to not overwhelm the reader
okurz: the template generator output mixes both "optional information" with "necessary status". Both should be separated because every power user otherwise ignores relevant text very soon

Actions

Copy link

Updated by okurz about 3 years ago

Parent task changed from #39719 to #91646

Actions

Copy link

Updated by okurz almost 3 years ago

Conducted a SUSE QE tools workshop

open points and ideas from API playground:

Q: how to filter by test modules and their results from API
- A: Use the modules and modules_result parameter, e.g. openqa-cli api --osd jobs version=15-SP3 modules=shutdown modules_result=failed
Q: API equivalent of /tests/overview
- A: You can use the JSON representation of the same page. Just append ".json" to the URL, e.g. https://openqa.suse.de/tests/overview.json?groupid=110&distri=sle&build=187.1&version=15-SP3
Q: select jobs with "settings" filter, e.g. "ISSUE" for SUSE QAM aggregrated tests
- A: Part of it is answered in #91542#note-23 already but this could be made simpler in the openQA API
Idea: have more subcommands in openqa-cli, e.g. to look up module results or jobs where certain modules are executed or where these modules fail
Q: openqa-cli api --osd jobs version=15-SP3 modules=shutdown modules_result=failed result=failed limit=1000 latest=1 | jq '.[][] | "\(.settings.FLAVOR), \(.group), \(.name): https://openqa.suse.de/t\(.id) (\(.result))"' shows the following jobs: "Online, Functional, sle-15-SP3-Online-x86_64-Build183.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5951445 (failed)" "Online, Functional, sle-15-SP3-Online-x86_64-Build186.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5975627 (failed)" "Online, Functional, sle-15-SP3-Online-x86_64-Build187.1-create_hdd_gnome@svirt-xen-hvm: https://openqa.suse.de/t5990471 (failed)" which are all from the same scenario. Shouldn't latest=1 mean to show only one of these?