Project

General

Profile

Actions

coordination #98457

closed

coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

[epic] Handle openqa-review reminder comments on very old jobs better

Added by okurz over 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2021-10-14
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Motivation

openqa-review puts reminder comments jobs even they are multiple months old. This can be confusing to some users and the existing options in https://github.com/os-autoinst/openqa_review/blob/master/openqa_review/openqa_review.py#L196 might not be good enough for some users.
Also see https://suse.slack.com/archives/C02CANHLANP/p1631255003169600

Ideas

  • DONE Don't comment if the content of the comment would be the same with same job -> #100982 #101722
  • DONE Maybe just extend the reminder comment with the option to specify a label instead of a bugref, e.g. label:wontfix:bsc1234 -> https://github.com/os-autoinst/openqa_review/pull/177
  • DONE Change the reminder comments if there are already existing reminder comments:
    1. Compare the existing reminder comments with the template. If it's the same except for $name and $url then do not repeat the steps what can be done to prevent more reminders, just the starting line and $name and $url and maybe a pointer to the previous comment
    2. If there are already two (or more?) similar reminder comments as above vary the message even more
    3. If there are more than N reminder comments write a final notice and that openqa-review gives up to remind
    4. For now we will try exponential backoff, see #106907
  • DONE Extend the reminder comments to state where exactly the ticket reference comes from, at best point the step detail in openQA or to a code location or the workaround needle that references a ticket
    • openQA test detail step covered in #106909
    • for reference, the autoinst-log.txt has a line [2022-02-08T11:34:07.525532+01:00] [debug] needle 'generic-desktop-test-20220206' is a workaround. The reason is bsc#123456
  • DONE Additional suggestions by mdoucha: Don't remind on soft-fails: "Most soft-fails in kernel groups are WONTFIX issues and stuff that isn't important enough to fix immediately. We know about it, we'll fix what needs to be fixed eventually, we don't need a reminder". This seems to be a squad specific workflow which is not generally true. So here I would stick with the current behaviour -> It's suggested to use "record_info" instead of "record_soft_failure" for such issues. However we decided to support a specific "no reminder" pattern that is configurable and can be used to avoid reminders -> #106368

Discarded ideas

  • Suggestion by foursixnine: Don't comment at all on a job result that is multiple months old. okurz sees a problem with that approach because if these results are still the "most recent" ones then users might be able to follow why such jobs would not be referenced in reminder comments -> It's suggested to move according job groups to "Released" or "EOL" or simply ensure that testing starts earlier, e.g. for Alpha0 of new product iterations

Subtasks 5 (0 open5 closed)

action #100982: openqa-review: Do not post reminder comment if comment would be exactly the same as the last size:MResolvedjbaier_cz2021-10-14

Actions
action #101722: openqa-review: Do not post reminder comment in progress.opensuse.org if comment would be exactly the same as the last size:MResolvedtinita2021-10-14

Actions
action #106368: openqa-review: Configurable no-reminder message pattern size:MResolvedjbaier_cz2022-02-09

Actions
action #106907: Exponential backoff for reminders based on previous reminders size:MResolvedjbaier_cz2022-02-16

Actions
action #106909: [openqa-review] reminder comments point to specific openQA test details steps or openQA comments size:MResolvedjbaier_cz2022-02-16

Actions
Actions #1

Updated by okurz over 2 years ago

Actions #2

Updated by okurz over 2 years ago

  • Description updated (diff)
Actions #3

Updated by okurz over 2 years ago

  • Description updated (diff)
Actions #4

Updated by okurz over 2 years ago

  • Tracker changed from action to coordination
  • Subject changed from Handle openqa-review reminder comments on very old jobs better to [epic] Handle openqa-review reminder comments on very old jobs better

from estimation meeting

Actions #5

Updated by mkittler over 2 years ago

  • Status changed from New to Workable
Actions #6

Updated by okurz over 2 years ago

  • Description updated (diff)
  • Status changed from Workable to Blocked
  • Assignee set to okurz

added one idea how to handle multiple reminder comments with similar content and no response in tickets. Have not created a dedicated subticket for that. First we can go with the existing subtask and see how we proceed.

Actions #7

Updated by okurz over 2 years ago

  • Description updated (diff)
  • Status changed from Blocked to Workable

both subtasks finished. We can continue with the other ideas.

Actions #8

Updated by okurz over 2 years ago

  • Assignee deleted (okurz)
Actions #10

Updated by okurz about 2 years ago

  • Parent task set to #91646
Actions #11

Updated by okurz about 2 years ago

  • Description updated (diff)
  • Status changed from Workable to Blocked
  • Assignee set to okurz

In the weekly SUSE QE tools unblock 2022-02-16 we refined the ticket, came up with new tasks and new ideas. Noted down in subtasks and description so I can track the ticket as "Blocked" by subtasks.

Actions #12

Updated by okurz almost 2 years ago

  • Description updated (diff)
  • Status changed from Blocked to Resolved

We have completed all the mentioned ideas. I am out of further ideas right now and have received no complaints since then, only a question which I could answer with the now existing features. This concludes this epic successfully.

Actions

Also available in: Atom PDF