action #132272
closedcoordination #102915: [saga][epic] Automated classification of failures
QA (public) - coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests
Identify reproducible *TEST* issues (not product issues anymore) using openqa-investigate size:M
Description
Motivation¶
With #109920 openqa-investigate can identify clear PRODUCT REGRESSIONS. We can extend that same concept to now clearly identify test regressions.
Acceptance criteria¶
- AC1: On failed openQA jobs with openqa-investigate info "retry X, last_good_test V, last_good_build X, last_good_test+build V" (X: failed, V: passed) a comment is written pointing to a likely test regression
- AC2: No such comment is written on other jobs
Suggestions¶
- Learn what had been done in #109920 for product regressions
- Do what had been done in https://github.com/os-autoinst/scripts/pull/223 but for test regression identification
Steps¶
- https://github.com/os-autoinst/scripts/pull/265 Identify reproducible test issues
Updated by okurz over 1 year ago
- Copied from action #109920: Identify reproducible product issues using openqa-investigate size:M added
Updated by okurz over 1 year ago
- Subject changed from Identify reproducible test issues using openqa-investigate to Identify reproducible *TEST* issues (not product issues anymore) using openqa-investigate size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by tinita over 1 year ago
- Target version changed from Ready to future
Our backlog is full with more urgent tasks
Updated by tinita about 1 year ago
- Status changed from Workable to In Progress
- Assignee set to tinita
Updated by tinita about 1 year ago
Updated by openqa_review about 1 year ago
- Due date set to 2023-10-31
Setting due date based on mean cycle time of SUSE QE Tools
Updated by tinita about 1 year ago
- Description updated (diff)
Ready: https://github.com/os-autoinst/scripts/pull/265 Identify reproducible test issues
Updated by okurz about 1 year ago
https://github.com/os-autoinst/scripts/pull/265 merged. We can wait for next investigation jobs to show up on https://openqa.opensuse.org/tests/?match=:investigate: as well as https://openqa.suse.de/tests/?match=:investigate: and see if we have test issues identified today. Likely the best approach to actually find test issues identified is to query the database for the text "Jobs including the last good test are ok, likely a test issue"
Tried that on o3
openqa=> select job_id,text from comments where job_id is not null and text ~ 'Jobs including the last good test are ok, likely a test issue' limit 10;
job_id | text
--------+------
(0 rows)
so no comment yet.
Updated by tinita about 1 year ago
- Status changed from Feedback to Resolved
We have one job on osd: https://openqa.suse.de/tests/12563482#comments
Updated by okurz about 1 year ago
- Copied to action #138299: Make the final aggregation messages from openqa-investigate more prominent size:S added
Updated by tinita about 1 year ago
- Status changed from Resolved to In Progress
There is an error in the code when there is no last good build or test:
Oct 20 12:36:13 new-ariel openqa-gru[12895]: /opt/os-autoinst-scripts/openqa-investigate: line 239: : command not found
Oct 20 12:36:13 new-ariel openqa-gru[12897]: /opt/os-autoinst-scripts/openqa-investigate: line 241: : command not found
if ([[ -z "$pass_lgtb" ]] || "$pass_lgtb") && ([[ -z "$pass_lgt" ]] || ! "$pass_lgt") && "$pass_lgb"; then
elif ([[ -z "$pass_lgtb" ]] || "$pass_lgtb") && "$pass_lgt" && ([[ -z "$pass_lgb" ]] || ! "$pass_lgb"); then
Not sure why it doesn't complain in the unit test...
Updated by tinita about 1 year ago
The unit test didn't have a case yet where certain investigate jobs were not triggered.
https://github.com/os-autoinst/scripts/pull/268 Fix warnings about empty variables
Updated by openqa_review about 1 year ago
- Due date set to 2023-11-04
Setting due date based on mean cycle time of SUSE QE Tools
Updated by tinita about 1 year ago
- Status changed from In Progress to Resolved
The warning doesn't appear in the logs anymore
Updated by okurz about 1 year ago
- Copied to action #151399: Identify reproducible *infrastructure* issues using openqa-investigate size:M added