Project

General

Profile

Actions
Copy link

coordination #91646

closed

[saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

Added by okurz over 3 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Target version:
openQA Project - Ready
Start date:
2021-04-21
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Motivation

Scale up the business while ensuring highest quality with automated testing, approval and release of package updates for SLE and openSUSE

Subtasks 63 (0 open — 63 closed)

openQA Project - coordination #89062: [epic] Simplify review for SUSE QAMResolvedokurz2021-04-21

Actions
openQA Project - action #91542: openQA API what jobs were/are testing X incident/packageResolvedilausuch2021-04-21

Actions
openQA Project - action #91658: Make "black certificate" stricter to only show when /tests/overview?todo=1 is empty, i.e. no unlabeled failuresResolvedmkittler2021-04-23

Actions
openQA Project - action #91773: Automatic replacement of openQA job URLs preview of openQA size:MResolvedtinita2021-04-26

Actions
openQA Infrastructure - action #92034: Re-enable openqa-investigate options after the black certificate now only shows properly "reviewed" jobsResolvedmkittler

Actions
action #92851: Workshop series proposal "How SUSE QE teams review openQA test results"Resolvedlivdywan2021-05-19

Actions
openQA Project - coordination #93246: [epic] List all unreviewed failed (or incomplete) jobs on /tests on request size:MResolvedokurz2021-12-09

Actions
openQA Project - action #103765: Support for "todo" query parameter on /tests, same as /tests/overview size:MResolvedtinita2021-12-09

Actions
openQA Project - action #104995: Add UI element and help text for "todo" query parameter on /tests, similar as /tests/overviewResolvedmkittler

Actions
openQA Project - action #105001: Add doc for "todo" query parameter on /tests, similar to /tests/overview size:SResolvedkodymo2022-01-18

Actions
openQA Project - action #94937: Distinguish comment types on jobs on /tests (maybe optional) size:SResolvedtinita2021-05-28

Actions
openQA Project - action #102374: Support use of force_result via ticket title in auto-review size:MResolvedtinita2021-11-14

Actions
openQA Project - openqa-force-result #109857: Secure auto-review+force_result size:M auto_review:"Failed to download gobbledeegoop":force_result:softfailedResolvedlivdywanActions
action #107014: trigger openqa-trigger-bisect-jobs from our automatic investigations whenever the cause is not already known size:MResolvedtinita2022-02-17

Actions
openQA Project - coordination #91467: [epic] Surface openQA failures per squad in a single placeResolvedokurz2021-04-23

Actions
openQA Project - action #91647: Making option to filter by flavor, test name on /tests/overview more prominentResolvedkraih2021-04-23

Actions
openQA Project - action #91650: Resolve the most recent builds per job group on /tests/overview when showing multiple job groupsResolvedilausuch2021-04-23

Actions
openQA Project - action #91652: Remind about the use of openqa-review in squadsResolvedokurz2021-04-23

Actions
openQA Project - action #92957: Add option to openqa-review to skip displaying all passed resultsResolvedtinita

Actions
openQA Project - action #93727: Publish openqa-review reports with "--skip-passed"Resolvedokurz

Actions
openQA Project - action #94732: Provide link to /tests/overview of latest builds of all job groups within a parent job group size:MResolvedilausuch

Actions
openQA Project - action #94762: openqa-review: Add mode of single-line todo lists size:MResolvedtinita2021-05-21

Actions
openQA Project - action #96058: [spike] Filter test results on /tests or /tests/overview by regex match in modules size:MResolvedosukup2021-07-22

Actions
action #97403: openqa-review: Polish job group section titles in todo-only mode size:SResolvedkodymo

Actions
openQA Project - action #98258: No results on /tests/overview w/o buildResolvedosukup2021-07-22

Actions
openQA Project - action #98445: improve description for "Test module" UI element as followup to #96058Resolvedosukup

Actions
openQA Project - action #98460: Filter actual test results on /tests or /tests/overview by regex match in modulesResolvedokurz

Actions
openQA Project - coordination #91914: [epic] Make reviewing openQA results per squad easierResolvedokurz2021-05-25

Actions
openQA Project - action #91605: notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) size:MResolvedtinita

Actions
openQA Project - action #93065: "parallel_failed" jobs show up on /tests/overview?todo=1 but these do not need a labelResolvedokurz2021-05-25

Actions
openQA Project - action #110629: openqa-label-known-issues: Fallback notification address in openqa-label-known-issues if no email address could be parsed from group_overviewResolvedtinita2022-05-04

Actions
openQA Project - action #105382: Reconsider coloring of "failed modules" to make it obvious that they actually failResolvedokurz2022-01-24

Actions
action #107923: qem-bot: Ignore not-ok openQA jobs for specific incident based on openQA job comment size:MResolvedjbaier_cz

Actions
openQA Project - action #111066: Document suggested workflows for multiple teams reviewing openQA test results size:MResolvedokurz2022-05-13

Actions
coordination #98457: [epic] Handle openqa-review reminder comments on very old jobs betterResolvedokurz2021-10-14

Actions
action #100982: openqa-review: Do not post reminder comment if comment would be exactly the same as the last size:MResolvedjbaier_cz2021-10-14

Actions
action #101722: openqa-review: Do not post reminder comment in progress.opensuse.org if comment would be exactly the same as the last size:MResolvedtinita2021-10-14

Actions
action #106368: openqa-review: Configurable no-reminder message pattern size:MResolvedjbaier_cz2022-02-09

Actions
action #106907: Exponential backoff for reminders based on previous reminders size:MResolvedjbaier_cz2022-02-16

Actions
action #106909: [openqa-review] reminder comments point to specific openQA test details steps or openQA comments size:MResolvedjbaier_cz2022-02-16

Actions
action #103701: Resubmited incident (ID) with new release request (RR) inherits incident test results from previous RRResolvedosukup2021-12-08

Actions
openQA Project - action #104199: Prevent confusion when openQA comments look like both a bugref as well as label at the same time size:MResolvedkraih2021-12-20

Actions
openQA Project - action #104616: Make openQA labels clearly visibleResolvedkraih2021-12-20

Actions
action #108869: Missing (re-)schedules of SLE maintenance tests size:MResolvedosukup2022-03-24

Actions
action #108944: 5 whys follow-up to Missing (re-)schedules of SLE maintenance tests size:MResolvedosukup2022-03-24

Actions
coordination #109641: [epic] qem-bot improvementsResolvedokurz2022-02-08

Actions
action #106179: No aggregate maintenance runs scheduled today on osd - dashboard.qem.suse.de down size:SResolvedosukup2022-02-08

Actions
openQA Project - action #109310: qem-bot/dashboard - mixed old and new incidents size:MResolvedkraih2022-03-31

Actions
action #109488: qem-bot - better loggingResolvedosukup2022-04-05

Actions
action #109512: qem-bot - add vars with GitlabCI job link and qem-dashboard linkResolvedmkittler

Actions
action #109623: Allow adding scheduling settings for informal purposes that are not added to openQA jobsResolvedmkittler2022-03-24

Actions
action #109977: qem-bot - approve pipeline failed with 403 forbidden size:MResolvedokurz2022-04-14

Actions
action #110167: Tests for qem-botResolvedokurz2022-04-21

Actions
action #110409: qem-dashboard - remove old openQA jobs when rr_number changes size:MResolvedkraih2022-04-28

Actions
action #111075: Collect code coverage for qem-botResolvedokurz2022-05-13

Actions
action #111078: Simple automatic test exercising one of the existing happy path workflows of qem-bot size:MResolvedosukup2022-05-13

Actions
action #109779: Cannot approve incident due to test report parsing errorResolvedjbaier_cz2022-04-11

Actions
coordination #109818: [epic] qa-maintenance/openQAbot improvementsResolvedokurz2022-04-11

Actions
openQA Project - action #109815: Add retry for HTTP requests in openQABot where missingResolvedjbaier_cz2022-04-11

Actions
coordination #117694: [epic] Stable and reliable qem-botResolvedokurz2022-07-26

Actions
action #114694: Incident seems to have missing aggregate test results in qem-dashboard but openQA jobs exists size:MResolvedkraih2022-07-26

Actions
action #117619: Bot approved update request with failing tests size:MResolvedtinita

Actions
openQA Project - action #117655: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does size:MResolvedkraih2022-10-06

Actions

Related issues 2 (2 open — 0 closed)

Related to openQA Project - coordination #102915: [saga][epic] Automated classification of failuresNew2015-12-182024-11-27

Actions
Copied to QA - coordination #99303: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, approval and releaseNew2016-02-092025-01-31

Actions
Actions
Copy link
 #1

Updated by okurz over 3 years ago

  • Project changed from 46 to QA
Actions
Copy link
 #3

Updated by okurz over 3 years ago

  • Tracker changed from action to coordination
Actions
Copy link
 #4

Updated by okurz over 3 years ago

  • Subject changed from [saga][epic] Improve SUSE Maintenance QA workflows with fully automated testing, approval and release to [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release
Actions
Copy link
 #5

Updated by okurz over 3 years ago

  • Target version changed from Ready to future

with #91605 put to future again, see #91605#note-4 , with the support of more internal stakeholders, with the need of focussing on existing sagas to get something done before we pick up more new stories, with the quick solutions applied as part of #91542 and #91467 we decide that currently for the foreseeable future of next weeks we will not continue work here and focus on other sagas instead, see https://progress.opensuse.org/issues?query_id=524

Actions
Copy link
 #7

Updated by okurz over 3 years ago

  • Target version changed from future to Ready
Actions
Copy link
 #9

Updated by okurz about 3 years ago

  • Copied to coordination #99303: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, approval and release added
Actions
Copy link
 #10

Updated by okurz almost 3 years ago

Had a discussion with vpelcak about some topics:

  • mawerner has stated again that "gut-feeling" is the base for product validation -> direct consequence is that maintenance tests will fail and nobody cares to fix them
  • proposal to move more tests from aggregate to incident tests
  • I think that the cost-benefit ratio investing effort in automatic classification #102915 is better than for #104209
Actions
Copy link
 #11

Updated by okurz over 2 years ago

A comment like https://build.opensuse.org/project/show/openSUSE:Maintenance:17578#comment-1652027 with a nice content

Group openSUSE Leap 15.4 Backports@DVD-Backports-Incidents (1 tests passed, 6 tests failed)

make me wonder. What was the reason that qem-bot was implemented and does not have such feature anymore?

Actions
Copy link
 #12

Updated by okurz over 1 year ago

  • Status changed from Blocked to Resolved

All subtasks resolved. With this we call the whole saga resolved \o/

Actions
Copy link
 #13

Updated by okurz 2 months ago

Actions
Copy link

Also available in: Atom PDF