Project

General

Profile

action #98820

openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

Various requirements for qem-dashboard (was: Design document for openQA CI dashboard)

Added by vpelcak about 1 month ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
Start date:
2021-09-16
Due date:
% Done:

0%

Estimated time:

Description

The dashboard itself consists of 2 components. The dashboard itself openQA bot performing the operations underneath.

Dashboard

  • Test state overview - The openQA reviewer can easily see what tests passed/failed... for the particular update to see commonalities, e.g. the same test modules failing in all codestreams and versions
    • Alternative: A test module centric openQA view
  • Have a section for each squad (need to be able to assign tests/job groups to squads before)
    • It should be sufficient or better suitable to just have that in openQA itself unless there is benefit in knowing which updates are blocked by tests maintained by particular squads
  • History - be able to see which tests passed/failed for the updates in recent history to not need to wait for all jobs for incidents to complete
    • alternative: seen as less helpful than automatic actions by any approving automation that looks into the history
    • further details: openQA knows the history for each scenario
      • Is it possible some example how verify this easily, as an old history from smelt?
        • yes, openQA can provide test overviews for specific parameters, e.g. a certain incident, for a specific time
        • remark: Right now maintenance openQA test results on openqa.suse.de are mostly stored for only about 2 weeks, would need storage investment to store results going back longer into the past

Bot

  • Ensure that all to-be-scheduled tests are considered for any approval decision
    • rejection, especially manual rejection, can be done on individual already finished test results
    • alternative: Only approve if there are at least as many passed results as in previous incident releases
      • Would need possibility to mark the decrease of test numbers or test coverage as acceptable

History

#2 Updated by okurz about 1 month ago

  • Subject changed from Design document for openQA CI dashboard to Various requirements for qem-dashboard (was: Design document for openQA CI dashboard)
  • Target version set to future
  • Parent task set to #80194

Thanks. this can be a subtask of #80194 which in itself is currently also not on our backlog but I expect the work to continue there as soon as we resolved #91467 and neighboring stories.

An overall observation in the above findings is that there are no really big visionary ideas challenging the design which can be a good thing because it means the proposed features can be done as evolutionary, individual tasks.

vpelcak if you agree then we can say that this ticket turned out to just specify additional specific feature requests for the qem-dashboard and not a full design document for it. Also, I assume with "openQA CI dashboard" you just mean the qem-dashboard. A term "openQA CI dashboard" sounds confusing any likely even wrong because openQA itself can be considered a "CI dashboard". So I am updating the ticket subject accordingly. Please revert my changes to the subject or update again if you disagree.

For all three we found good alternative proposals to implement just within openQA which should be preferred if we find both benefit for non-SUSE users and also if the effort could be lower, mostly depending on existing feature sets and availability of specific data that we would need. The "Bot" feature request is good on it's own and IMHO applicable to https://gitlab.suse.de/qa-maintenance/bot-ng

#3 Updated by cdywan about 1 month ago

okurz wrote:

An overall observation in the above findings is that there are no really big visionary ideas challenging the design which can be a good thing because it means the proposed features can be done as evolutionary, individual tasks.

vpelcak if you agree then we can say that this ticket turned out to just specify additional specific feature requests for the qem-dashboard and not a full design document for it. Also, I assume with "openQA CI dashboard" you just mean the qem-dashboard. A term "openQA CI dashboard" sounds confusing any likely even wrong because openQA itself can be considered a "CI dashboard". So I am updating the ticket subject accordingly. Please revert my changes to the subject or update again if you disagree.

I filed #99252 more specifically about the design document (or section in the README.md), which I think is good to split out in any case. And maybe assessing what we have is a great step towards finding out if we have bigger goals.

Also available in: Atom PDF