Project

General

Profile

action #77887

[tools][openqa] Enable automatic openQA investigation jobs for osd as well

Added by okurz 9 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2020-11-14
Due date:
% Done:

0%

Estimated time:

Description

Motivation

SUSE QEM is challenged by a relative high false-positive rate of openQA tests. This is due to objectively higher product quality of released products in comparison to products in development, i.e. pre-GM SLE including Tumbleweed snapshots before release. We already use "openqa-investigate" for o3 which has been running there for multiple months and I received positive feedback. We can now extend the solution to osd.

Acceptance criteria

  • AC1: Automatic investigation jobs are triggered and commented on new, unlabeled failures within osd production groups, e.g. not development groups
  • AC2: Automatic investigation jobs run within a reasonable time to provide useful feedback to reviewers in their regular review routines
  • AC3: No harmful performance impact on infrastructure due to too many automatic investigation jobs

Suggestions

  • There is a script based solution "openqa-investigate" which we already use automatically within o3, see https://gitlab.suse.de/openqa/auto-review/-/blob/master/.gitlab-ci.yml#L110
  • Try out dry-runs of "openqa-investigate" against OSD and check for obvious things missing or going wrong, e.g. immediate errors or crashes or unreasonable results
  • Extending to osd can be just as simple as applying a similar block in .gitlab-ci.yml for osd
  • When activated monitor over couple of days for usefulness and performance impact
  • Consider changing the schedule of the scheduled pipeline, e.g. trigger more often over the day, or even "continuous" :)
  • Optional: Ensure that auto-review walks first over all issues, potentially even failed ones to detect known issues, and if unknown to auto-review, only then trigger investigation jobs

History

#1 Updated by okurz 9 months ago

  • Description updated (diff)

#2 Updated by okurz 9 months ago

  • Project changed from SUSE QA to openQA Infrastructure

#3 Updated by okurz 6 months ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz

All this was covered with openqa-investigate triggered as part of job-done-hooks on osd: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/420

Also available in: Atom PDF