Project

General

Profile

action #77944

openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

Run "auto-review" more often but alarm less

Added by okurz 7 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Target version:
openQA Project - Ready
Start date:
2020-11-14
Due date:
% Done:

0%

Estimated time:

Description

Motivation

auto-review does a good job but is at the border of annoying with too many alerts. But on the other hand we could benefit from running auto-review more often. We should consider counting the gitlab CI pipeline as passed as long as only less than N unknown, new issues show up, potentially even not fail gitlab CI pipeline at all as an alert but alarm differently, e.g. just in grafana

Acceptance criteria

  • AC1: gitlab CI pipeline only fails if more than N unknown new issues
  • AC2: auto-review pipeline runs more often

Suggestions

  • Don't fail gitlab CI pipelines in case jobs are not known or introduce threshold N
  • Adjust gitlab CI pipeline schedule

Related issues

Related to openQA Project - action #76912: OpenQA::Script::Client throws perl warning "Wide character in print", should not be thereResolved2020-11-032020-12-09

Copied from QA - coordination #77899: [epic] Extend "auto-review" for failed jobs as wellResolved2020-11-26

Copied to openQA Project - action #80736: Trigger 'auto-review' from within openQA when jobs incomplete (or fail) , for testing: auto_review:"tests died: unable to load main.pm, check the log for the cause"Resolved

History

#1 Updated by okurz 7 months ago

#2 Updated by okurz 7 months ago

  • Estimated time set to 39719.00 h

#3 Updated by okurz 7 months ago

  • Parent task set to #39719

#4 Updated by okurz 7 months ago

  • Estimated time deleted (39719.00 h)

#5 Updated by okurz 7 months ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz

Unfortunately our users of (at least) o3 and osd are suffering from many unhandled incompletes related to #67000 so as nobody has a better clue and our users are suffering from incompletes I have enabled the "hourly" schedule in https://gitlab.suse.de/openqa/auto-review/-/pipeline_schedules . This will send more email alerts for any new unhandled incompletes which need a ticket

#6 Updated by okurz 7 months ago

  • Related to action #76912: OpenQA::Script::Client throws perl warning "Wide character in print", should not be there added

#7 Updated by okurz 7 months ago

  • Description updated (diff)
  • Status changed from In Progress to Feedback

With the patch for #76912 applied already while the PR is pending the pipeline did run just fine overnight.

We can still consider preventing repeated alerts. How about we add a mode to openqa-label-known-issues to write "unknown issue" (similar to what TTM already does as well) and/or optionally just retrigger automatically. Then, in a separate step we can still check for the number of incompletes with the text "unknown issue" or incompletes which still have no clone.

  • TODO: @waitfor #76912
  • TODO: consider the mentioned extensions to openqa-label-known-issues

#8 Updated by okurz 7 months ago

Had an idea: https://github.com/os-autoinst/openQA/pull/3603 This could also be for another ticket about "integrate auto-review more closely into openQA"

#9 Updated by okurz 7 months ago

  • Status changed from Feedback to In Progress

#10 Updated by okurz 6 months ago

  • Copied to action #80736: Trigger 'auto-review' from within openQA when jobs incomplete (or fail) , for testing: auto_review:"tests died: unable to load main.pm, check the log for the cause" added

#11 Updated by okurz 6 months ago

  • Status changed from In Progress to Resolved

https://github.com/os-autoinst/openQA/pull/3603 is merged. I plan to conduct the rollout to o3 and potentially osd in #80736

The original idea for this ticket was to run auto-review more often which works fine so far

Also available in: Atom PDF