Project

General

Profile

action #98862

coordination #102915: [saga][epic] Automated classification of failures

QA - coordination #94105: [epic] Use feedback from openqa-investigate to automatically inform on github pull requests, open tickets, weed out automatically failed tests

Comment about intermittent/sporadic test issues on original job if openqa-investigate retry job passes

Added by Xiaojing_liu 9 months ago. Updated 23 days ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
QA - future
Start date:
2021-09-18
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Comment about intermittent/sporadic test issues on original job if openqa-investigate retry job passes

User story

When a job fails, we will investigate the job by triggering new job with different conditions, such as rerun, re-run with last good build, or last commit in the test distribution.
If a retry job passes, the original job may be a intermittent/sporadic case.
We should provide that assessment as a comment back on the original job.

Acceptance criteria

  • AC1: If an openqa-investigate retry job passes a comment is created on the original job with the assessment that the issue is intermittent or a sporadic test issue

Suggestions

  • Take a look how we identify likely sporadic issues as a result of failed "retry" jobs in https://github.com/os-autoinst/scripts/blob/master/openqa-investigate#L136=
  • Extend to react according on passed retry
  • Ensure that hook scripts in github.com/os-autoinst/scripts/ act as one should expect when called on passed jobs
  • Use
  • Alternative: Extend o3 config to also run job_done_hooks on passed jobs and monitor performance impact
  • Same on osd as on o3

Related issues

Related to QA - action #95746: Identify likely "sporadic" openQA tests with "openqa-investigate" size:MResolved2021-07-20

Copied to openQA Project - action #110518: Call job_done_hooks if requested by test setting (not only openQA config as done so far) size:MResolved2021-09-18

History

#1 Updated by Xiaojing_liu 9 months ago

  • Related to action #95746: Identify likely "sporadic" openQA tests with "openqa-investigate" size:M added

#2 Updated by okurz 9 months ago

  • Subject changed from Comment back to the origin job according to the investigate job' result to Comment back to the origin job according to the investigate job's result
  • Category set to Feature requests
  • Target version set to future
  • Parent task set to #94105

thanks. Let's collect a bit of experience first with reacting on failed jobs as done in #95746 before coming back here reacting on passed jobs.

#3 Updated by okurz 2 months ago

  • Subject changed from Comment back to the origin job according to the investigate job's result to Comment about intermittent/sporadic test issues on original job if openqa-investigate retry job passes
  • Description updated (diff)
  • Target version changed from future to Ready

#4 Updated by okurz about 2 months ago

  • Copied to action #110518: Call job_done_hooks if requested by test setting (not only openQA config as done so far) size:M added

#5 Updated by okurz about 2 months ago

  • Description updated (diff)
  • Status changed from New to Blocked
  • Assignee set to okurz

blocked by #110518

#6 Updated by mkittler about 1 month ago

  • Status changed from Blocked to New
  • Assignee deleted (okurz)

#7 Updated by okurz about 1 month ago

With #110518 resolved we can now trigger investigation jobs within openqa-investigate with the new test setting to trigger hooks on itself so that we can evaluate the result and comment accordingly on the original job.

#8 Updated by okurz 23 days ago

  • Target version changed from Ready to future

Also available in: Atom PDF