Project

General

Profile

Actions

action #97274

open

coordination #99303: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, approval and release

qam dashboard improvement ideas

Added by mgrifalconi over 3 years ago. Updated almost 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
Start date:
2021-06-29
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Hello, doing openQA review I always used smelt comments to find out which test run needs to be checked to approve an update.

Ideally approval is automated, but when a single test fails (out of dozens/hundreds) it still needs some manual work to decide if such failures can be ignored for that particular test.

I won't mention crosscheck aggregate runs with precedent days (see https://progress.opensuse.org/issues/97118).

These are the current issues I found while using the dashboard for my week of review:

  • Sorting order: I like to sort on smelt the priority or due date to have an idea on the situation. Neither of which is available. Incidents are sorted by incident ID, which I do not care
  • Missing Release Request ID: If I am given only a RR ID, I must go to smelt to find the incident and back to the dashboard.
  • Result History: I can only see latest results, so I find more painful to crosscheck different days, but I would be happier to see such think automated (see other poo linked earlier). In the meantime though, it is just more painful than before. I also have a good overview of the situation near the end of the day, because in the morning all runs are still ongoing and cannot do review based on yesterday's results.
  • Development Job Groups: such job groups are not ignored, also some test groups will fit in. This creates some confusion and time wasted.

Extra thought:
The dashboard and smelt might be duplicating some work. Why not having a link in smelt to the list of related tests on the dashboard? I would be using the indexing/priority/informations on smelt and then go on the dashboard to check tests, possibly with result history.
What I am basically asking for is the same features as smelt comments, whichever implementation is used.


Subtasks 1 (0 open1 closed)

openQA Project (public) - action #102206: Make bot-ng a proper public open source project size:MResolvedokurz2021-06-29

Actions

Related issues 2 (0 open2 closed)

Related to openQA Project (public) - action #94838: Make qem-dashboard a proper public open source project size:MResolvedkraih2021-06-29

Actions
Related to QA (public) - action #104209: [qem] dashboard.qam.suse.de checkpoints for aggregatesRejectedokurz2021-12-21

Actions
Actions #1

Updated by okurz over 3 years ago

  • Related to action #94838: Make qem-dashboard a proper public open source project size:M added
Actions #2

Updated by okurz over 3 years ago

  • Priority changed from Normal to Low
  • Target version set to future

thanks for all your nice ideas. We might be able to consider them depending on if work on the dashboard will be followed up. At the very least I consider #94838 a prerequisite to prevent the situation that we end up with a hard to maintain one-man-show

Actions #3

Updated by okurz over 3 years ago

  • Parent task set to #80194

Discussed with others in meeting

mgrifalconi wrote:

  • Development Job Groups: such job groups are not ignored, also some test groups will fit in. This creates some confusion and time wasted.

Might be solved already or maybe same issue as described in https://progress.opensuse.org/issues/80194#Further-tasks

Actions #4

Updated by mgrifalconi about 3 years ago

One more limitation of the dashboard that might be considered:
When a job gets unscheduled, the dashboard will still get the latest run of that job and present it. If the job was failing, auto approval will be blocked forever and will need manual action.

Dashboard might get smarter by looking at the dates of the last run, compared with others or simply allow a "ignore this" comment on a job to allow auto approval.

Actions #5

Updated by okurz about 3 years ago

  • Related to action #104209: [qem] dashboard.qam.suse.de checkpoints for aggregates added
Actions

Also available in: Atom PDF