Project

General

Profile

Actions
Copy link

action #102374

closed

QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

coordination #89062: [epic] Simplify review for SUSE QAM

Support use of force_result via ticket title in auto-review size:M

Added by okurz over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
tinita
Category:
Feature requests
Target version:
Ready
Start date:
2021-11-14
Due date:
% Done:

0%

Estimated time:

Description

Motivation

We have the new "force_result" openQA feature (see #95479) and there is auto-review which allows to automatically comment on jobs. As an alternative to retriggering on sporadic issues we can use auto-review to change the result of jobs showing known acceptable failures. This allows to mark failures that should be ignored for an auto-approval process, e.g. as used by qa-maintenance/bot-ng, while ensuring that there is an according open ticket to properly fix the failures. Additionally this allows to search easily for jobs related to tickets.

Acceptance criteria

  • AC1: tickets with a special command in the subject line can change the job result of openQA jobs
  • AC2: The "retry" command of auto-review still works
  • AC3: jobs with changed results still have a reference to the according tickets

Suggestions

Actions
Copy link
 #1

Updated by okurz over 2 years ago

  • Tracker changed from coordination to action
Actions
Copy link
 #2

Updated by okurz over 2 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by livdywan over 2 years ago

  • Subject changed from Combine "force_result" with auto-review to Support use of "force_result" via ticket title in auto-review size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #4

Updated by tinita over 2 years ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita
Actions
Copy link
 #5

Updated by openqa_review over 2 years ago

  • Due date set to 2021-12-09

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #6

Updated by tinita over 2 years ago

  • Status changed from In Progress to Feedback

PR: https://github.com/os-autoinst/scripts/pull/121

In the example job I used for testing you will find several comments: https://openqa.opensuse.org/tests/1990037#comments

I also checked that retry is still called.

Actions
Copy link
 #7

Updated by tinita over 2 years ago

  • Status changed from Feedback to In Progress

Still need to work on the PR

Actions
Copy link
 #8

Updated by tinita over 2 years ago

  • Due date changed from 2021-12-09 to 2021-12-10
  • Status changed from In Progress to Feedback

PR https://github.com/os-autoinst/scripts/pull/121 ready for review.
As a plus, added some automatic testing, that's also why it took longer.

Actions
Copy link
 #9

Updated by tinita over 2 years ago

  • Due date changed from 2021-12-10 to 2021-12-13

Still waiting for feedback

Actions
Copy link
 #10

Updated by tinita over 2 years ago

PR was merged, I will check logs on o3 / osd.

Actions
Copy link
 #11

Updated by tinita over 2 years ago

  • Subject changed from Support use of "force_result" via ticket title in auto-review size:M to Support use of "force_result" via ticket title in auto-review size:M auto_review:"systemctl start docker' failed at .*/openqa_webui.pm":force_result:softfailed

Using this ticket as a live test

Actions
Copy link
 #12

Updated by tinita over 2 years ago

  • Due date changed from 2021-12-13 to 2021-12-15

As there are unfortunately luckily no failures in openqa-in-openqa tests today, I couldn't check if the feature works live.
I will try to create a failing test myself.

Meanwhile, I don't see any newly introduced problems in the gru logs of o3 and openqa.

Actions
Copy link
 #13

Updated by okurz over 2 years ago

This should not have anything to do with openqa-in-openqa tests. But as we already discussed you can create a new test from your tests distri fork which fails early, e.g. a die "my message"

Actions
Copy link
 #14

Updated by tinita over 2 years ago

  • Subject changed from Support use of "force_result" via ticket title in auto-review size:M auto_review:"systemctl start docker' failed at .*/openqa_webui.pm":force_result:softfailed to Support use of "force_result" via ticket title in auto-review size:M auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed
Actions
Copy link
 #15

Updated by tinita over 2 years ago

  • Subject changed from Support use of "force_result" via ticket title in auto-review size:M auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed to Support use of force_result via ticket title in auto-review size:M auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed
Actions
Copy link
 #16

Updated by tinita over 2 years ago

  • Subject changed from Support use of force_result via ticket title in auto-review size:M auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed to Support use of force_result via ticket title in auto-review size:M
  • Status changed from Feedback to Resolved

I had added the following to the ticket title:
auto_review:"Test died: fail on purpose at .*/openqa_webui.pm":force_result:softfailed

I cloned an os-autoinst-distri-openQA test https://openqa.opensuse.org/tests/2086384 on my fork:

https://openqa.opensuse.org/tests/2086434#comments

As you can see, the comment was posted and the result set to softfailed.

Actions
Copy link
 #17

Updated by okurz over 2 years ago

  • Due date deleted (2021-12-15)
Actions
Copy link

Also available in: Atom PDF