Project

General

Profile

action #124473

coordination #99303: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, approval and release

[tools] Automatic regression tests export from openQA

Added by vpelcak 4 months ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
future
Start date:
2023-02-14
Due date:
% Done:

0%

Estimated time:

Description

Motivation

With our progression in automation of regression tests we reached the state that many updates just need the results to be looked up in openQA and linked in our testreports.
That is pretty repetitive manual work which begs for the automation.

Acceptance criteria

  • AC1: Automation fills the links to the appropriate regression tests in openQA and their result into the testreport of the update.

Suggestions

  • This will perhaps require knowledge of our test coverage
  • Maybe extension of openQA API will be needed

Related issues

Related to QA - action #94600: [tools][mtui] Communicate reduced visibility of openQA incident related coverage in logNew2021-06-23

Related to QA - coordination #110016: [epic][teregen] teregen (maintenance test report template generator) improvementsNew2021-01-14

History

#1 Updated by okurz 4 months ago

  • Target version set to future

Discussed with vpelcak. The test report needs to include information about the test coverage which allows a 2nd level reviewer to crosscheck the test coverage reviewed by test reviewers. The generated test reports already contain openQA references but here we need additional job informations. This is about aggregate tests only. vpelcak showed me an example looking for "multipath" tests. For that we needed to look around on the webUI and look primarily for test modules, maybe other scenarios as well. We can try to use the openQA built-in search functionality.

One idea: An openQA search link like https://openqa.suse.de/search?q=multipath but then need to filter the thousands of search results to match only something like "Maintenance". We could include such search term automatically in reports but often the exact search term is not obvious.

Also there is the "metadata" field in test modules.

There should be more related tickets that we should link here.

#2 Updated by okurz 4 months ago

  • Related to action #94600: [tools][mtui] Communicate reduced visibility of openQA incident related coverage in log added

#4 Updated by okurz 4 months ago

  • Parent task set to #99303

#7 Updated by okurz 4 months ago

https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Definition-of-DONE mentions already Test modules that have been touched have updated metadata, e.g. "Maintainer" and "Summary" which is also ensured by CI so we could extend that for additional metadata as described in e.g. #77002

#10 Updated by okurz 4 months ago

  • Related to coordination #110016: [epic][teregen] teregen (maintenance test report template generator) improvements added

Also available in: Atom PDF