action #132272
closedcoordination #102915: [saga][epic] Automated classification of failures
QA - coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests
Identify reproducible *TEST* issues (not product issues anymore) using openqa-investigate size:M
Description
Motivation¶
With #109920 openqa-investigate can identify clear PRODUCT REGRESSIONS. We can extend that same concept to now clearly identify test regressions.
Acceptance criteria¶
- AC1: On failed openQA jobs with openqa-investigate info "retry X, last_good_test V, last_good_build X, last_good_test+build V" (X: failed, V: passed) a comment is written pointing to a likely test regression
- AC2: No such comment is written on other jobs
Suggestions¶
- Learn what had been done in #109920 for product regressions
- Do what had been done in https://github.com/os-autoinst/scripts/pull/223 but for test regression identification
Steps¶
- https://github.com/os-autoinst/scripts/pull/265 Identify reproducible test issues
Updated by okurz 10 months ago
- Copied from action #109920: Identify reproducible product issues using openqa-investigate size:M added
Updated by openqa_review 7 months ago
- Due date set to 2023-10-31
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz 7 months ago
https://github.com/os-autoinst/scripts/pull/265 merged. We can wait for next investigation jobs to show up on https://openqa.opensuse.org/tests/?match=:investigate: as well as https://openqa.suse.de/tests/?match=:investigate: and see if we have test issues identified today. Likely the best approach to actually find test issues identified is to query the database for the text "Jobs including the last good test are ok, likely a test issue"
Tried that on o3
openqa=> select job_id,text from comments where job_id is not null and text ~ 'Jobs including the last good test are ok, likely a test issue' limit 10;
job_id | text
--------+------
(0 rows)
so no comment yet.
Updated by tinita 7 months ago
- Status changed from Feedback to Resolved
We have one job on osd: https://openqa.suse.de/tests/12563482#comments
Updated by okurz 7 months ago
- Copied to action #138299: Make the final aggregation messages from openqa-investigate more prominent size:S added
Updated by tinita 7 months ago
- Status changed from Resolved to In Progress
There is an error in the code when there is no last good build or test:
Oct 20 12:36:13 new-ariel openqa-gru[12895]: /opt/os-autoinst-scripts/openqa-investigate: line 239: : command not found
Oct 20 12:36:13 new-ariel openqa-gru[12897]: /opt/os-autoinst-scripts/openqa-investigate: line 241: : command not found
if ([[ -z "$pass_lgtb" ]] || "$pass_lgtb") && ([[ -z "$pass_lgt" ]] || ! "$pass_lgt") && "$pass_lgb"; then
elif ([[ -z "$pass_lgtb" ]] || "$pass_lgtb") && "$pass_lgt" && ([[ -z "$pass_lgb" ]] || ! "$pass_lgb"); then
Not sure why it doesn't complain in the unit test...
Updated by tinita 7 months ago
The unit test didn't have a case yet where certain investigate jobs were not triggered.
https://github.com/os-autoinst/scripts/pull/268 Fix warnings about empty variables
Updated by openqa_review 7 months ago
- Due date set to 2023-11-04
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz 5 months ago
- Copied to action #151399: Identify reproducible *infrastructure* issues using openqa-investigate size:M added