Project

General

Profile

action #97118

coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

enhance bot automatic approval: check multiple days

Added by mgrifalconi 5 months ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
Start date:
2021-08-18
Due date:
% Done:

0%

Estimated time:

Description

Context:
Right now the bot will approve an incident only if all tests that include such incident are green at a certain point in time.

Problem:
Day 1 after incident creation: Test 1 to 10 are green, test 11 fails.
Day 2 after incident creation: Test 1 fails, test 2 to 11 are green.
Day 3 after incident creation: Test 1 is green, test 2 fails, test 3 to 11 are green.

You get the point.
Update is likely fine since all tests were green at least once with the same update code.

While we always work to improve tests stability, it is tricky to find a time where "everything is green" due to test development, new updates that break aggregate runs, infra hiccups etc.

Right now it is slow manual work to crosscheck such results and very error prone.

AC:
The bot can check between multiple days to make sure every test was green at least one and approve the update.

Implementation proposal:
Same logic that checks latest run, but if a failure is found, go back in history and check if there was a green result up until incident creation.


Related issues

Related to QA - action #104209: [qem] dashboard.qam.suse.de checkpoints for aggregatesNew2021-12-21

History

#1 Updated by okurz 5 months ago

  • Project changed from openQA Project to QA
  • Target version set to future

#2 Updated by dzedro 3 months ago

I would lock the result on dashboard on passed job group for the update, add check box or color to make it visible and here we are done.
Now we have constant overlapping and restarting of failures and waiting for results.

#3 Updated by kraih 2 months ago

The required data to implement this should be present in the dashboard already. It could provide a new REST API endpoint for the bot.

#4 Updated by okurz about 1 month ago

  • Related to action #104209: [qem] dashboard.qam.suse.de checkpoints for aggregates added

#5 Updated by okurz about 1 month ago

  • Parent task set to #80194

Also available in: Atom PDF