action #132332
closed
coordination #102915: [saga][epic] Automated classification of failures
QA - coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests
Multiple investigation comments for multimachine tests size:M
Added by tinita 11 months ago.
Updated 10 months ago.
Category:
Regressions/Crashes
Description
Observation¶
We see in https://openqa.suse.de/tests/11507412#comments that 8 several investigate:retry jobs are commenting back to the original job.
Acceptance criteria¶
- AC1: There is only one comment based on the result of the "retry" job reporting back the investigation results even if multiple jobs have been cloned (like for parallel clusters)
Suggestions¶
- We don't need to display any other job from an investigation cluster than one corresponding to the original scenario. For example for a salt-master+salt-minion job where the salt-minion job fails we are only interested in results of investigation jobs for the salt-minion testsuite
- Find out in the retry job if we are the actual job we're interested in. E.g. we have
salt-master:investigate:retry and salt-minion:investigate:retry. Look into the OPENQA_INVESTIGATE_ORIGIN job data and look at the TEST name and compare it to the current TEST, e.g. if "salt-minion" matches the beginning of the string "salt-minion:investigate:retry" then it's the same test, "salt-master" is not the same as "salt-minion:investigate:retry". If it doesn't match, just return from the post-investigate function.
Out of scope¶
- Don't care if job links for related jobs of a cluster show up or not, e.g. if the "salt-minion" job failed, it does not matter if a parallel "salt-master" job URL shows up in the original investigate comment or not
Further details¶
- Copied from action #109920: Identify reproducible product issues using openqa-investigate size:M added
- Description updated (diff)
- Status changed from New to In Progress
- Assignee set to tinita
The actual fix is not that hard I think:
post-investigate() {
- local id=$1 old_name=$2
+ local id=$1 retry_name=$2
local rc=0 status
- [[ ! "$old_name" =~ investigate:retry$ ]] && echo "Job is ':investigate:' already, skipping investigation" && return 0
+ [[ ! "$retry_name" =~ investigate:retry$ ]] && echo "Job is ':investigate:' already, skipping investigation" && return 0
# We are in the investigate:retry job now. From here we will check the
# results of the other investigation jobs, if necessary
retry_result="$(echo "$job_data" | runjq -r '.job.result')" || return $?
investigate_origin="$(echo "$job_data" | runjq -r '.job.settings.OPENQA_INVESTIGATE_ORIGIN')" || return 1
origin_job_id=${investigate_origin#"$host_url/t"}
+ origin_job_data=$(openqa-cli "${client_args[@]}" --json jobs/"$origin_job_id") || return $?
+ origin_name="$(echo "$origin_job_data" | runjq -r '.job.test')" || return $?
+ # cluster jobs might have the same OPENQA_INVESTIGATE_ORIGIN as the root retry job
+ [[ $retry_name != "$origin_name:investigate:retry" ]] && echo "Job $retry_name ($id) is not a retry of $origin_name ($origin_job_id), skipping investigation" && return 0
But now it's time to cleanup the tests a bit, because the mocking data is not well structured anymore.
- Subject changed from Multiple investigation comments for multimachine tests to Multiple investigation comments for multimachine tests size:M
- Description updated (diff)
- Status changed from In Progress to Feedback
Merged and deployed. Will check later if there are still multiple comments.
A bit hard to find out because we also get multiple comments for jobs which have a RETRY set:
https://openqa.suse.de/tests/11530247#comments
I wonder if investigate:retry jobs should actually have the RETRY setting removed?
tinita wrote:
Merged and deployed. Will check later if there are still multiple comments.
A bit hard to find out because we also get multiple comments for jobs which have a RETRY set:
https://openqa.suse.de/tests/11530247#comments
I wonder if investigate:retry jobs should actually have the RETRY setting removed?
Don't care about the RETRY flag or whatever else retriggered the same test over and over again. We should only prevent the comments from sibling in a cluster.
- Due date changed from 2023-07-08 to 2023-07-21
Updating the due-date which was originally from the clonee ticket.
Looks good. Can we resolve or is there something else you wait for, like for validation?
Yes, I would like to validate and haven't come up with a good SQL statement yet
- Status changed from Feedback to Resolved
Ok, let's assume this works. I will be away for one week, so no time to check right now. Also I actually trust the unit tests.
- Due date deleted (
2023-07-21)
Also available in: Atom
PDF