Project

General

Profile

Actions

action #132272

closed

coordination #102915: [saga][epic] Automated classification of failures

QA - coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests

Identify reproducible *TEST* issues (not product issues anymore) using openqa-investigate size:M

Added by okurz 11 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Motivation

With #109920 openqa-investigate can identify clear PRODUCT REGRESSIONS. We can extend that same concept to now clearly identify test regressions.

Acceptance criteria

  • AC1: On failed openQA jobs with openqa-investigate info "retry X, last_good_test V, last_good_build X, last_good_test+build V" (X: failed, V: passed) a comment is written pointing to a likely test regression
  • AC2: No such comment is written on other jobs

Suggestions

Steps


Related issues 3 (0 open3 closed)

Copied from openQA Project - action #109920: Identify reproducible product issues using openqa-investigate size:MResolvedtinita2023-03-23

Actions
Copied to openQA Project - action #138299: Make the final aggregation messages from openqa-investigate more prominent size:SResolveddheidler

Actions
Copied to openQA Project - action #151399: Identify reproducible *infrastructure* issues using openqa-investigate size:MResolvedtinita2023-11-24

Actions
Actions #1

Updated by okurz 11 months ago

  • Copied from action #109920: Identify reproducible product issues using openqa-investigate size:M added
Actions #2

Updated by okurz 11 months ago

  • Subject changed from Identify reproducible test issues using openqa-investigate to Identify reproducible *TEST* issues (not product issues anymore) using openqa-investigate size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #3

Updated by okurz 11 months ago

  • Priority changed from High to Normal
Actions #4

Updated by tinita 9 months ago

  • Target version changed from Ready to future

Our backlog is full with more urgent tasks

Actions #5

Updated by okurz 8 months ago

  • Target version changed from future to Ready
Actions #6

Updated by tinita 7 months ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita
Actions #8

Updated by openqa_review 7 months ago

  • Due date set to 2023-10-31

Setting due date based on mean cycle time of SUSE QE Tools

Actions #9

Updated by tinita 7 months ago

  • Description updated (diff)

Ready: https://github.com/os-autoinst/scripts/pull/265 Identify reproducible test issues

Actions #10

Updated by tinita 7 months ago

  • Status changed from In Progress to Feedback
Actions #11

Updated by okurz 7 months ago

https://github.com/os-autoinst/scripts/pull/265 merged. We can wait for next investigation jobs to show up on https://openqa.opensuse.org/tests/?match=:investigate: as well as https://openqa.suse.de/tests/?match=:investigate: and see if we have test issues identified today. Likely the best approach to actually find test issues identified is to query the database for the text "Jobs including the last good test are ok, likely a test issue"

Tried that on o3

openqa=> select job_id,text from comments where job_id is not null and text ~ 'Jobs including the last good test are ok, likely a test issue' limit 10; 
 job_id | text 
--------+------
(0 rows)

so no comment yet.

Actions #12

Updated by tinita 7 months ago

  • Status changed from Feedback to Resolved
Actions #13

Updated by okurz 7 months ago

  • Due date deleted (2023-10-31)
Actions #14

Updated by okurz 7 months ago

  • Copied to action #138299: Make the final aggregation messages from openqa-investigate more prominent size:S added
Actions #15

Updated by tinita 7 months ago

  • Status changed from Resolved to In Progress

There is an error in the code when there is no last good build or test:

Oct 20 12:36:13 new-ariel openqa-gru[12895]: /opt/os-autoinst-scripts/openqa-investigate: line 239: : command not found
Oct 20 12:36:13 new-ariel openqa-gru[12897]: /opt/os-autoinst-scripts/openqa-investigate: line 241: : command not found
if ([[ -z "$pass_lgtb" ]] || "$pass_lgtb") && ([[ -z "$pass_lgt" ]] || ! "$pass_lgt") && "$pass_lgb"; then
elif ([[ -z "$pass_lgtb" ]] || "$pass_lgtb") && "$pass_lgt" && ([[ -z "$pass_lgb" ]] || ! "$pass_lgb"); then

Not sure why it doesn't complain in the unit test...

Actions #16

Updated by tinita 7 months ago

The unit test didn't have a case yet where certain investigate jobs were not triggered.
https://github.com/os-autoinst/scripts/pull/268 Fix warnings about empty variables

Actions #17

Updated by openqa_review 7 months ago

  • Due date set to 2023-11-04

Setting due date based on mean cycle time of SUSE QE Tools

Actions #18

Updated by tinita 7 months ago

  • Status changed from In Progress to Resolved

The warning doesn't appear in the logs anymore

Actions #19

Updated by okurz 6 months ago

  • Due date deleted (2023-11-04)
Actions #20

Updated by okurz 6 months ago

  • Copied to action #151399: Identify reproducible *infrastructure* issues using openqa-investigate size:M added
Actions

Also available in: Atom PDF