action #13560
closedPrevent incomplete builds by waiting for all architectures before testing
Added by okurz over 7 years ago. Updated over 7 years ago.
100%
Description
E.g. https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2094&groupid=25 is only showing ppc64le. What would we loose if we just wait until all architectures are there?
Updated by maritawerner over 7 years ago
- Status changed from New to Rejected
Actually A LOT OF time. We have seen builds were it took hours to get all 4 arches. While waiting the HW would be idle and in addition to that some arches take a lot of time to get through. So it could happen that we first wait for the last arch and then we wait for tests getting finished though they could be done already?
Close that as invalid.
Updated by okurz over 7 years ago
- Status changed from Rejected to In Progress
- Assignee set to okurz
As discussed with coolo: I reopen the ticket because what the ticket describes is what coolo is actually doing right now. He disabled the automatic trigger script to ensure less "noise", i.e. not obsoleting old builds and syncing only complete builds. So what we should do is:
- trigger osc api, grep for dirty
- if no dirty, check unfinished on openqa parent group overview
- then sync
So something like:
osc -A https://api.suse.de api /build/SUSE:SLE-12-SP3:GA/_result?package=_product | grep dirty || (curl -s https://openqa.suse.de/parent_group_overview/5.json?limit_builds=1 | jq -e '.result | .[] | .unfinished < 10' && /opt/openqa-scripts/openqa-iso-sync-sles)
but geekotest has no .oscrc so maybe we should do that on a jenkins instance, e.g. lord.arch.suse.de:8080 and then login as other user with ssh permissions for only executing the specific sync command.
Updated by okurz over 7 years ago
We can also get the data from ibs not needing authentication:
curl -s https://api.suse.de/public/build/SUSE:SLE-12-SP3:GA/_result?package=_product
Updated by okurz over 7 years ago
- Category changed from Enhancement to existing tests to Infrastructure
Additional option (discussed in today's call): After an incomplete/obsoleted build shows up all the jobs could be moved to another job group. Also proposed on https://wiki.microfocus.net/index.php/RD-OPS_QA/openQA_review#Incomplete_or_obsoleted_builds . If possible we should do this move using API calls if that is supported, if not we have to do it in the database itself and probably come up with a way to do it over the API.
Updated by okurz over 7 years ago
So the current updated command would be
curl -s https://api.suse.de/public/build/SUSE:SLE-12-SP3:GA/_result?package=_product | grep dirty || (curl -s https://openqa.suse.de/parent_group_overview/5.json?limit_builds=1 | jq -e '.result | .[] | .unfinished < 10' && /opt/openqa-scripts/openqa-iso-sync-sles)
As osd (the host itself) can call the OBS API call with curl without authentication fine but does not have "jq" available I see the following options:
- package jq for SLES in an additional repo, add this repo, call the above command as part of the cron job which is calling the sync script for now every 6h but then every 10 minutes
- run the above command from an external jenkins instance and call remotely the openqa-iso-sync-sles with ssh authentication
- run a workaround command with python, e.g.
python -c "import sys, json; sys.exit(not sum([i['unfinished'] for i in json.load(sys.stdin)['result'].values()]) < 10)"instead of the jq command but then same as 1, call that as part of the cron job
For now I prefer 3. Do you agree?
Updated by coolo over 7 years ago
3 sounds good - I would have python only output the unfinished jobs and have the comparision in the level above. But that's it
Updated by okurz over 7 years ago
Please see https://gitlab.suse.de/openqa/scripts/merge_requests/59 for my recommendation. The change looks bigger then I intended originally and I am not sure the changed script is the right location.
Updated by okurz over 7 years ago
last mentioned MR would allow to go ahead for some time, additional ideas are blocked by #15494
Updated by okurz over 7 years ago
- Status changed from In Progress to Resolved
subtask complete, all jobs from incomplete builds can now be moved into another group (or group deleted?) with openqa-client. https://wiki.microfocus.net/index.php/RD-OPS_QA/openQA_review was updated with a workflow description.