Project

General

Profile

action #102374

QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

coordination #89062: [epic] Simplify review for SUSE QAM

Support use of force_result via ticket title in auto-review size:M

Added by okurz 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2021-11-14
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

We have the new "force_result" openQA feature (see #95479) and there is auto-review which allows to automatically comment on jobs. As an alternative to retriggering on sporadic issues we can use auto-review to change the result of jobs showing known acceptable failures. This allows to mark failures that should be ignored for an auto-approval process, e.g. as used by qa-maintenance/bot-ng, while ensuring that there is an according open ticket to properly fix the failures. Additionally this allows to search easily for jobs related to tickets.

Acceptance criteria

  • AC1: tickets with a special command in the subject line can change the job result of openQA jobs
  • AC2: The "retry" command of auto-review still works
  • AC3: jobs with changed results still have a reference to the according tickets

Suggestions

History

#1 Updated by okurz 2 months ago

  • Tracker changed from coordination to action

#2 Updated by okurz 2 months ago

  • Description updated (diff)

#3 Updated by cdywan 2 months ago

  • Subject changed from Combine "force_result" with auto-review to Support use of "force_result" via ticket title in auto-review size:M
  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by tinita 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita

#5 Updated by openqa_review 2 months ago

  • Due date set to 2021-12-09

Setting due date based on mean cycle time of SUSE QE Tools

#6 Updated by tinita about 2 months ago

  • Status changed from In Progress to Feedback

PR: https://github.com/os-autoinst/scripts/pull/121

In the example job I used for testing you will find several comments: https://openqa.opensuse.org/tests/1990037#comments

I also checked that retry is still called.

#7 Updated by tinita about 2 months ago

  • Status changed from Feedback to In Progress

Still need to work on the PR

#8 Updated by tinita about 2 months ago

  • Due date changed from 2021-12-09 to 2021-12-10
  • Status changed from In Progress to Feedback

PR https://github.com/os-autoinst/scripts/pull/121 ready for review.
As a plus, added some automatic testing, that's also why it took longer.

#9 Updated by tinita about 2 months ago

  • Due date changed from 2021-12-10 to 2021-12-13

Still waiting for feedback

#10 Updated by tinita about 2 months ago

PR was merged, I will check logs on o3 / osd.

#11 Updated by tinita about 1 month ago

  • Subject changed from Support use of "force_result" via ticket title in auto-review size:M to Support use of "force_result" via ticket title in auto-review size:M auto_review:"systemctl start docker' failed at .*/openqa_webui.pm":force_result:softfailed

Using this ticket as a live test

#12 Updated by tinita about 1 month ago

  • Due date changed from 2021-12-13 to 2021-12-15

As there are unfortunately luckily no failures in openqa-in-openqa tests today, I couldn't check if the feature works live.
I will try to create a failing test myself.

Meanwhile, I don't see any newly introduced problems in the gru logs of o3 and openqa.

#13 Updated by okurz about 1 month ago

This should not have anything to do with openqa-in-openqa tests. But as we already discussed you can create a new test from your tests distri fork which fails early, e.g. a die "my message"

#14 Updated by tinita about 1 month ago

  • Subject changed from Support use of "force_result" via ticket title in auto-review size:M auto_review:"systemctl start docker' failed at .*/openqa_webui.pm":force_result:softfailed to Support use of "force_result" via ticket title in auto-review size:M auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed

#15 Updated by tinita about 1 month ago

  • Subject changed from Support use of "force_result" via ticket title in auto-review size:M auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed to Support use of force_result via ticket title in auto-review size:M auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed

#16 Updated by tinita about 1 month ago

  • Subject changed from Support use of force_result via ticket title in auto-review size:M auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed to Support use of force_result via ticket title in auto-review size:M
  • Status changed from Feedback to Resolved

I had added the following to the ticket title:
auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed

I cloned an os-autoinst-distri-openQA test https://openqa.opensuse.org/tests/2086384 on my fork:

https://openqa.opensuse.org/tests/2086434#comments

As you can see, the comment was posted and the result set to softfailed.

#17 Updated by okurz about 1 month ago

  • Due date deleted (2021-12-15)

Also available in: Atom PDF