action #98862
closed
coordination #102915: [saga][epic] Automated classification of failures
QA (public) - coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests
Comment about intermittent/sporadic test issues on original job if openqa-investigate retry job passes size:M
Added by Xiaojing_liu over 3 years ago.
Updated about 2 years ago.
Category:
Feature requests
Description
Comment about intermittent/sporadic test issues on original job if openqa-investigate retry job passes
User story¶
When a job fails, we will investigate the job by triggering a new job with different conditions, such as re-run, re-run with the last good build, or last commit in the test distribution.
If a retry job passes, the original job may be a intermittent/sporadic case.
We should provide that assessment as a comment back on the original job.
Acceptance criteria¶
- AC1: If an openqa-investigate retry job passes, a comment is created on the original job with the assessment that the issue is intermittent or a sporadic test issue
Suggestions¶
- Call the hook script with a setting to run the hook on passed jobs, e.g.
_TRIGGER_JOB_DONE_HOOK=0 or =1
- Consider exiting early from openqa-label-known-issues to ensure it does not handle passed jobs
- Take a look at how we identify likely sporadic issues as a result of failed "retry" jobs in https://github.com/os-autoinst/scripts/blob/master/openqa-investigate#L136=
- Extend to react according on passed retry
- Ensure that hook scripts in github.com/os-autoinst/scripts/ act as one should expect when called on passed jobs
- Alternative: Extend o3 config to also run job_done_hooks on passed jobs and monitor performance impact
- Same on osd as on o3
- Related to action #95746: Identify likely "sporadic" openQA tests with "openqa-investigate" size:M added
- Subject changed from Comment back to the origin job according to the investigate job' result to Comment back to the origin job according to the investigate job's result
- Category set to Feature requests
- Target version set to future
- Parent task set to #94105
thanks. Let's collect a bit of experience first with reacting on failed jobs as done in #95746 before coming back here reacting on passed jobs.
- Subject changed from Comment back to the origin job according to the investigate job's result to Comment about intermittent/sporadic test issues on original job if openqa-investigate retry job passes
- Description updated (diff)
- Target version changed from future to Ready
- Copied to action #110518: Call job_done_hooks if requested by test setting (not only openQA config as done so far) size:M added
- Description updated (diff)
- Status changed from New to Blocked
- Assignee set to okurz
- Status changed from Blocked to New
- Assignee deleted (
okurz)
With #110518 resolved we can now trigger investigation jobs within openqa-investigate with the new test setting to trigger hooks on itself so that we can evaluate the result and comment accordingly on the original job.
- Target version changed from Ready to future
- Target version changed from future to Ready
- Subject changed from Comment about intermittent/sporadic test issues on original job if openqa-investigate retry job passes to Comment about intermittent/sporadic test issues on original job if openqa-investigate retry job passes size:M
- Description updated (diff)
- Status changed from New to Workable
- Priority changed from Low to High
- Status changed from Workable to In Progress
- Assignee set to tinita
- Due date set to 2023-01-05
Setting due date based on mean cycle time of SUSE QE Tools
- Due date changed from 2023-01-05 to 2023-01-20
christmas grace due date bump :)
I have been busy with other things, e.g. setting up new laptop
- Due date changed from 2023-01-20 to 2023-01-27
I still need a little bit of time for the actual feature, hoping that it is now a bit easier to test.
- Due date deleted (
2023-01-27)
- Status changed from In Progress to Resolved
- Due date set to 2023-01-27
- Status changed from Resolved to In Progress
We still have to enable the triggering of jobs
- Status changed from In Progress to Feedback
I checked passed retry jobs and it seems like the hook is not working:
https://openqa.opensuse.org/tests/3068720#settings
This has the correct setting _TRIGGER_JOB_DONE_HOOK but apparently it didn't trigger the hook. Will check minion logs/db if there is more information.
I created a job myself with the setting, and it did't appear in the minion dashboard at all:
https://openqa.opensuse.org/tests/3069790
So the logic that runs the hook in lib/OpenQA/Task/Job/FinalizeResults.pm might be broken. Will verify locally what's happening.
Ok, I needed to configure job_done_hook in openqa.ini also. I did that now on o3 and restarted gru.
Let's see what happens.
So I enabled it on osd too.
You must do that with salt though to make permanent
- Status changed from Feedback to Resolved
- Due date deleted (
2023-01-27)
- Related to action #124274: openQA reports non-sporadic issue when retry job just softfailed size:M added
Also available in: Atom
PDF