coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release
openQA Project - coordination #89062: [epic] Simplify review for SUSE QAM
trigger openqa-trigger-bisect-jobs from our automatic investigations whenever the cause is not already known size:M
SUSE SLE maintenance aggregate tests can fail if any of the included incident update causes a problem. The challenge is to find out which of the incident updates caused it. For this okurz created https://github.com/os-autoinst/scripts/blob/master/openqa-trigger-bisect-jobs . We already have automatic investigation jobs when a failure is not known and labeled accordingly in openQA. So we should combine both and trigger "openqa-trigger-bisect-jobs" as part of the automatic investigation as well.
- AC1: For SUSE SLE maintenance aggregate test failures,
openqa-trigger-bisect-jobsis run additionally to
openqa-investigate, creating automatic investigation jobs
- AC2: We see all those investigation jobs listed in comments on the failed job
- AC3: Other jobs do not trigger
openqa-trigger-bisect-jobs(or the script aborts early without failure)
- AC4: OSD is not overwhelmed with
- Take a look into https://github.com/os-autoinst/salt-states-openqa/blob/master/openqa/server.sls#L81 how we trigger investigation jobs from job done hooks. We basically call https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues-and-investigate-hook
- Either extend that hook or create another one
- Either create a new comment besides the one from openqa-investigate, or list all investigate-jobs in one comment
See https://suse.slack.com/archives/C02D16TCP99/p1645022828339319 for more context if needed
- Subject changed from trigger https://github.com/os-autoinst/scripts/blob/master/openqa-trigger-bisect-jobs from our automatic investigations whenever the cause is not already known to trigger https://github.com/os-autoinst/scripts/blob/master/openqa-trigger-bisect-jobs from our automatic investigations whenever the cause is not already known size:M
- Description updated (diff)
- Status changed from New to Workable
- Due date changed from 2022-03-19 to 2022-03-25
I'm bumping the due date to account for availability
PR: https://github.com/os-autoinst/scripts/pull/136 to make the script more robust
Maybe next week we can see if it makes sense to e.g. take tests from this PR, assuming the minimal change is otherwise enough
- Subject changed from trigger https://github.com/os-autoinst/scripts/blob/master/openqa-trigger-bisect-jobs from our automatic investigations whenever the cause is not already known size:M to trigger openqa-trigger-bisect-jobs from our automatic investigations whenever the cause is not already known size:M
- Due date changed from 2022-03-25 to 2022-04-01
Bumping the due date based on availability
I spent the whole afternoon trying to find out how to:
- load a module from a filename like 'openqa-trigger-bisect-jobs' without a suffix and with dashes in the filename
openqa = __import__("openqa-trigger-bisect-jobs")only works if the script has a
- I was pointed to
imp.load_sourceand that works
openqa = imp.load_source('openqa', rootpath + '/openqa-trigger-bisect-jobs'). It took me a while because it wasn't clear what parameters it would get
- then I was pointed to
impis going to be deprecated
- I fought with
importliband found no way to load a script as a module with it
- so I'm going to go back to
impfor now and let python lovers actually replace it when the time has come
btw, in perl it would be simply
require 'path/to/script'; my $mock = Test::MockModule->new('main'); $mock->redefine(script_function_name => $new_coderef);
(btw, thanks to Alberto Planas Dominguez and Martin Doucha for the help)
- Due date changed from 2022-04-05 to 2022-04-09
This is an endless task. There are things which are better in python than in perl, sure, but python makes up for this by doing other things worse :(
tests/test_trigger_bisect_jobs.py:8 36 /home/runner/work/scripts/scripts/tests/test_trigger_bisect_jobs.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses 37 import imp
Thanks to Martin Doucha and Nick Singer and others for your help.
The replacement for
importlib.machinery.SourceFileLoader, and as okurz requested, I also replaced the
_replace call with something else which is IMHO much less readable, but so be it.
- added openqa-trigger-bisect-jobs to the hook script
- added a test for the hook script
Still would like to run it on a real osd job tomorrow.
https://github.com/os-autoinst/scripts/pull/143 - Write comment after creating bisect investigate jobs
See example comments here: https://openqa.opensuse.org/tests/2282708#comments
- Due date changed from 2022-04-15 to 2022-04-22
- Assignee changed from tinita to okurz
Quite some jobs were already scheduled on OSD, see https://openqa.suse.de/tests/8538952 . coolo found the jobs and mentioned in https://suse.slack.com/archives/C02AJ1E568M/p1649840218112839 . Maybe the load is actually too high but I would give it more time to decide on that. I will monitor for the next days.
One thing I found is that there are also jobs triggered for single-incident tests which does not make sense as we already have the "retry" jobs.
- Assignee changed from okurz to tinita
I created https://github.com/os-autoinst/scripts/pull/146, merged, to prevent investigation jobs triggered when the list of incidents is only 1 (or less entries). https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1 shows that we still have significantly high number of scheduled jobs. It looks to be slowly decreasing and certainly public Easter holiday will help as much. But afterwards we should identify which jobs might still be redundant or not providing enough value compared to the additional load.