Project

General

Profile

action #97274

openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

qam dashboard improvement ideas

Added by mgrifalconi 2 months ago. Updated 25 days ago.

Status:
New
Priority:
Low
Assignee:
-
Target version:
Start date:
2021-08-20
Due date:
% Done:

0%

Estimated time:

Description

Hello, doing openQA review I always used smelt comments to find out which test run needs to be checked to approve an update.

Ideally approval is automated, but when a single test fails (out of dozens/hundreds) it still needs some manual work to decide if such failures can be ignored for that particular test.

I won't mention crosscheck aggregate runs with precedent days (see https://progress.opensuse.org/issues/97118).

These are the current issues I found while using the dashboard for my week of review:

  • Sorting order: I like to sort on smelt the priority or due date to have an idea on the situation. Neither of which is available. Incidents are sorted by incident ID, which I do not care
  • Missing Release Request ID: If I am given only a RR ID, I must go to smelt to find the incident and back to the dashboard.
  • Result History: I can only see latest results, so I find more painful to crosscheck different days, but I would be happier to see such think automated (see other poo linked earlier). In the meantime though, it is just more painful than before. I also have a good overview of the situation near the end of the day, because in the morning all runs are still ongoing and cannot do review based on yesterday's results.
  • Development Job Groups: such job groups are not ignored, also some test groups will fit in. This creates some confusion and time wasted.

Extra thought:
The dashboard and smelt might be duplicating some work. Why not having a link in smelt to the list of related tests on the dashboard? I would be using the indexing/priority/informations on smelt and then go on the dashboard to check tests, possibly with result history.
What I am basically asking for is the same features as smelt comments, whichever implementation is used.


Related issues

Related to openQA Project - action #94838: Make qem-dashboard a proper public open source projectNew2021-06-29

History

#1 Updated by okurz 2 months ago

  • Related to action #94838: Make qem-dashboard a proper public open source project added

#2 Updated by okurz 2 months ago

  • Priority changed from Normal to Low
  • Target version set to future

thanks for all your nice ideas. We might be able to consider them depending on if work on the dashboard will be followed up. At the very least I consider #94838 a prerequisite to prevent the situation that we end up with a hard to maintain one-man-show

#3 Updated by okurz about 1 month ago

  • Parent task set to #80194

Discussed with others in meeting

mgrifalconi wrote:

  • Development Job Groups: such job groups are not ignored, also some test groups will fit in. This creates some confusion and time wasted.

Might be solved already or maybe same issue as described in https://progress.opensuse.org/issues/80194#Further-tasks

#4 Updated by mgrifalconi 25 days ago

One more limitation of the dashboard that might be considered:
When a job gets unscheduled, the dashboard will still get the latest run of that job and present it. If the job was failing, auto approval will be blocked forever and will need manual action.

Dashboard might get smarter by looking at the dates of the last run, compared with others or simply allow a "ignore this" comment on a job to allow auto approval.

Also available in: Atom PDF