Project

General

Profile

Actions

action #97118

closed

coordination #99303: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, approval and release

enhance bot automatic approval: check multiple days

Added by mgrifalconi about 3 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2021-08-18
Due date:
% Done:

0%

Estimated time:

Description

Context:
Right now the bot will approve an incident only if all tests that include such incident are green at a certain point in time.

Problem:
Day 1 after incident creation: Test 1 to 10 are green, test 11 fails.
Day 2 after incident creation: Test 1 fails, test 2 to 11 are green.
Day 3 after incident creation: Test 1 is green, test 2 fails, test 3 to 11 are green.

You get the point.
Update is likely fine since all tests were green at least once with the same update code.

While we always work to improve tests stability, it is tricky to find a time where "everything is green" due to test development, new updates that break aggregate runs, infra hiccups etc.

Right now it is slow manual work to crosscheck such results and very error prone.

AC:
The bot can check between multiple days to make sure every test was green at least one and approve the update.

Implementation proposal:
Same logic that checks latest run, but if a failure is found, go back in history and check if there was a green result up until incident creation.


Related issues 2 (1 open1 closed)

Related to QA - action #104209: [qem] dashboard.qam.suse.de checkpoints for aggregatesRejectedokurz2021-12-21

Actions
Related to openQA Project - action #122296: Fix openqa-trigger-bisect-jobs to actually remove incidents from bisection tests againNewszarate2024-02-13

Actions
Actions

Also available in: Atom PDF