Project

General

Profile

Actions

action #114451

closed

Incidents from all test issues variables are collected during bisect size:M

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2022-07-21
Due date:
2022-09-07
% Done:

0%

Estimated time:

Description

Observation

https://openqa.suse.de/tests/9184859#comments shows a comment from openqa-investigate. As this job is an aggregate test I would have expected also a result from https://github.com/os-autoinst/scripts/blob/master/openqa-trigger-bisect-jobs but we only have one entry in the job setting OS_TEST_ISSUES. Likely we should also trigger based on entries in BASE_TEST_ISSUES, DESKTOP_TEST_ISSUES, etc.

Acceptance criteria

  • AC1: Results from openqa-trigger-bisect-jobs are shown in job comments whenever an aggregate test includes entries in multiple TEST_ISSUES variables
  • AC2: Incidents are pulled from all *_TEST_ISSUES variables

Suggestions

Out of scope

  • Do not care about performance impact of "too many openQA jobs" triggered, we should have enough capacity

Related issues 1 (0 open1 closed)

Copied to openQA Project - action #115178: openqa-investigate: Ensure proper error handling size:MResolvedtinita2022-07-212022-08-26

Actions
Actions #1

Updated by livdywan over 1 year ago

  • Subject changed from gimme more bisect jobs! to Incidents from all test issues variables are collected during bisect size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #2

Updated by tinita over 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita
Actions #3

Updated by okurz over 1 year ago

  • Description updated (diff)
Actions #4

Updated by okurz over 1 year ago

  • Description updated (diff)
Actions #5

Updated by openqa_review over 1 year ago

  • Due date set to 2022-08-16

Setting due date based on mean cycle time of SUSE QE Tools

Actions #6

Updated by tinita over 1 year ago

  • Status changed from In Progress to Workable
  • Assignee deleted (tinita)

Currently I'm not able to work on it

Actions #7

Updated by tinita over 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita
Actions #8

Updated by tinita over 1 year ago

I think I have the basic implementation and will now add unit tests
https://github.com/perlpunk/scripts/tree/bisect-all-issue-kinds

Actions #9

Updated by tinita over 1 year ago

PR: https://github.com/os-autoinst/scripts/pull/177 - Bisect all *_TEST_ISSUES variables

Actions #10

Updated by tinita over 1 year ago

  • Status changed from In Progress to Feedback
Actions #11

Updated by tinita over 1 year ago

PR was merged, I will check if it works in production

Actions #12

Updated by osukup over 1 year ago

think there will be small problem with #111710 on OSD

Actions #13

Updated by tinita over 1 year ago

I was able to search for jobs with this query:

select j.id, jg.name, j.clone_id, js.key, js.value,j.t_created
 from job_settings js join jobs j on js.job_id=j.id left join job_groups jg on j.group_id=jg.id
 where js.key like '%_TEST_ISSUES' and js.value like '%,%' and j.result='failed'
 and j.t_created >= '2022-08-10' and j.clone_id is null
 and jg.name !~ '(Development|Public Cloud|Released|Others|Kernel|Virtualization).*';

I looked at this job https://openqa.suse.de/tests/9298857 but there were only the normal last_good_* investigate jobs.

I ran the hook manually but commented out line 18 echo "$test" | "$script_dir/openqa-investigate":

env script_dir=/opt/os-autoinst-scripts enable_force_result=true host=openqa.suse.de exclude_group_regex='.*(Development|Public Cloud|Released|Others|Kernel|Virtualization).*' grep_timeout=60 nice ionice -c idle /tmp/openqa-label-known-issues-and-investigate-hook 9298857

and now the investigation jobs have been created.

I enabled line 18 again and got this:

Skipping investigation of job 9298857: job cluster is already being investigated, see comment on job 9298857
403 Forbidden
404 Not Found
{"error_status":404}

So a new task is to improve error handling, so we can actually see where this comes from.
But no bisect jobs were created this time. I think the openqa-investigate exits with non-zero and aborts the script.

To search for actual bisect jobs I did:

select id, clone_id, test, t_created, state, result from jobs where t_created >= '2022-08-10' and test like '%:bisect_without_%';

and got 19 jobs for today, so in general it works. Example: https://openqa.suse.de/tests/9296997#comments

Actions #14

Updated by tinita over 1 year ago

  • Copied to action #115178: openqa-investigate: Ensure proper error handling size:M added
Actions #15

Updated by tinita over 1 year ago

  • Status changed from Feedback to Resolved

It works, so I resolve it.

Created #115178 as a followup.

Actions #16

Updated by tinita over 1 year ago

  • Status changed from Resolved to Workable

The logic needs to be a bit different from what I implemented:

Actions #17

Updated by okurz over 1 year ago

  • Due date deleted (2022-08-16)
Actions #18

Updated by tinita over 1 year ago

  • Assignee deleted (tinita)
Actions #19

Updated by livdywan over 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to osukup
Actions #20

Updated by openqa_review over 1 year ago

  • Due date set to 2022-09-07

Setting due date based on mean cycle time of SUSE QE Tools

Actions #21

Updated by osukup over 1 year ago

  • Status changed from In Progress to Feedback

PR merged...

Actions #22

Updated by livdywan over 1 year ago

  • Status changed from Feedback to Resolved

@osukup confirmed this can be resolved.

Actions

Also available in: Atom PDF