Project

General

Profile

action #114451

Incidents from all test issues variables are collected during bisect size:M

Added by okurz 2 months ago. Updated 20 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2022-07-21
Due date:
2022-09-07
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

https://openqa.suse.de/tests/9184859#comments shows a comment from openqa-investigate. As this job is an aggregate test I would have expected also a result from https://github.com/os-autoinst/scripts/blob/master/openqa-trigger-bisect-jobs but we only have one entry in the job setting OS_TEST_ISSUES. Likely we should also trigger based on entries in BASE_TEST_ISSUES, DESKTOP_TEST_ISSUES, etc.

Acceptance criteria

  • AC1: Results from openqa-trigger-bisect-jobs are shown in job comments whenever an aggregate test includes entries in multiple TEST_ISSUES variables
  • AC2: Incidents are pulled from all *_TEST_ISSUES variables

Suggestions

Out of scope

  • Do not care about performance impact of "too many openQA jobs" triggered, we should have enough capacity

Related issues

Copied to openQA Project - action #115178: openqa-investigate: Ensure proper error handling size:MResolved2022-07-212022-08-26

History

#1 Updated by cdywan 2 months ago

  • Subject changed from gimme more bisect jobs! to Incidents from all test issues variables are collected during bisect size:M
  • Description updated (diff)
  • Status changed from New to Workable

#2 Updated by tinita about 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita

#3 Updated by okurz about 2 months ago

  • Description updated (diff)

#4 Updated by okurz about 2 months ago

  • Description updated (diff)

#5 Updated by openqa_review about 2 months ago

  • Due date set to 2022-08-16

Setting due date based on mean cycle time of SUSE QE Tools

#6 Updated by tinita about 2 months ago

  • Status changed from In Progress to Workable
  • Assignee deleted (tinita)

Currently I'm not able to work on it

#7 Updated by tinita about 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita

#8 Updated by tinita about 2 months ago

I think I have the basic implementation and will now add unit tests
https://github.com/perlpunk/scripts/tree/bisect-all-issue-kinds

#9 Updated by tinita about 2 months ago

PR: https://github.com/os-autoinst/scripts/pull/177 - Bisect all *_TEST_ISSUES variables

#10 Updated by tinita about 2 months ago

  • Status changed from In Progress to Feedback

#11 Updated by tinita about 2 months ago

PR was merged, I will check if it works in production

#12 Updated by osukup about 2 months ago

think there will be small problem with #111710 on OSD

#13 Updated by tinita about 2 months ago

I was able to search for jobs with this query:

select j.id, jg.name, j.clone_id, js.key, js.value,j.t_created
 from job_settings js join jobs j on js.job_id=j.id left join job_groups jg on j.group_id=jg.id
 where js.key like '%_TEST_ISSUES' and js.value like '%,%' and j.result='failed'
 and j.t_created >= '2022-08-10' and j.clone_id is null
 and jg.name !~ '(Development|Public Cloud|Released|Others|Kernel|Virtualization).*';

I looked at this job https://openqa.suse.de/tests/9298857 but there were only the normal last_good_* investigate jobs.

I ran the hook manually but commented out line 18 echo "$test" | "$script_dir/openqa-investigate":

env script_dir=/opt/os-autoinst-scripts enable_force_result=true host=openqa.suse.de exclude_group_regex='.*(Development|Public Cloud|Released|Others|Kernel|Virtualization).*' grep_timeout=60 nice ionice -c idle /tmp/openqa-label-known-issues-and-investigate-hook 9298857

and now the investigation jobs have been created.

I enabled line 18 again and got this:

Skipping investigation of job 9298857: job cluster is already being investigated, see comment on job 9298857
403 Forbidden
404 Not Found
{"error_status":404}

So a new task is to improve error handling, so we can actually see where this comes from.
But no bisect jobs were created this time. I think the openqa-investigate exits with non-zero and aborts the script.

To search for actual bisect jobs I did:

select id, clone_id, test, t_created, state, result from jobs where t_created >= '2022-08-10' and test like '%:bisect_without_%';

and got 19 jobs for today, so in general it works. Example: https://openqa.suse.de/tests/9296997#comments

#14 Updated by tinita about 2 months ago

  • Copied to action #115178: openqa-investigate: Ensure proper error handling size:M added

#15 Updated by tinita about 2 months ago

  • Status changed from Feedback to Resolved

It works, so I resolve it.

Created #115178 as a followup.

#16 Updated by tinita about 1 month ago

  • Status changed from Resolved to Workable

The logic needs to be a bit different from what I implemented:

#17 Updated by okurz about 1 month ago

  • Due date deleted (2022-08-16)

#18 Updated by tinita about 1 month ago

  • Assignee deleted (tinita)

#19 Updated by cdywan about 1 month ago

  • Status changed from Workable to In Progress
  • Assignee set to osukup

#20 Updated by openqa_review about 1 month ago

  • Due date set to 2022-09-07

Setting due date based on mean cycle time of SUSE QE Tools

#21 Updated by osukup 28 days ago

  • Status changed from In Progress to Feedback

PR merged...

#22 Updated by cdywan 20 days ago

  • Status changed from Feedback to Resolved

osukup confirmed this can be resolved.

Also available in: Atom PDF