action #68260
closed[packages] prevent submission of failed packages from devel:openQA:tested to Factory
Description
Observation¶
[19/06/2020 15:19:45] <DimStar> okurz: wasn't the idea of openQA:tested that this was actually tested? And that sort of would require it to at least build, no? :)
[19/06/2020 15:20:54] <okurz> DimStar: unfortunately not. I remember clearly that I opted for a different name "testing" or similar and IRC logs might prove that maybe you and others opted for "tested" :D
[19/06/2020 15:22:19] <DimStar> shame...
[19/06/2020 15:23:03] <okurz> however the name is of course no excuse for creating a SR when the build does not work. … There was ticket for this somewhere around here … *diggingintobacklog*
[19/06/2020 15:23:58] <okurz> DimStar: https://build.opensuse.org/request/show/815729 would fix that though :)
[19/06/2020 15:24:00] <|Anna|> sr#815729 [submit devel:languages:perl/perl-Mojolicious -> openSUSE:Factory/perl-Mojolicious] No description text (State: review) -- https://build.opensuse.org/request/show/815729
[19/06/2020 15:24:29] <DimStar> okurz: ok, then I merge those adis together
[19/06/2020 15:25:01] <DimStar> can you reopen openQA in this case (as I declined it as FTBFS)
[19/06/2020 15:25:07] <okurz> that would be great. Anyway, I will still plan to not create submit requests when the build fails
Acceptance criteria¶
- AC1: No submit requests are created when build in https://build.opensuse.org/project/show/devel:openQA:tested fails
- AC2: openQA packages are still submitted automatically to Factory on no errors
Suggestions¶
- Take a look into what http://jenkins.qa.suse.de/view/openQA-in-openQA/ does
- Prevent that http://jenkins.qa.suse.de/view/openQA-in-openQA/job/submit-openQA-TW-to-oS_Fctry/ which uses https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit creates a submit request if the package (or any package in that repo?) is not succeeded, probably just add a waiting check for "all packages succeeded" after the line https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit#L44
- Consider as alternative doing the same in nother CI system, e.g. circleCI or github actions
Updated by okurz over 4 years ago
- Subject changed from prevent submission of failed packages from devel:openQA:tested to Factory to [packages] prevent submission of failed packages from devel:openQA:tested to Factory
Updated by okurz about 4 years ago
- Description updated (diff)
- Priority changed from Normal to High
Due to the broken perl-Mojolicious update we did multiple submit requests which all failed and were subsequently rightfully rejected. This "annoyed" DimStar at at least enough to point this out to me :) So I think we can regard this with higher prio now cosidering that there are not many other high prio issues right now.
Updated by okurz about 4 years ago
what happened, any particular reason which got in your way? :)
Updated by okurz about 4 years ago
so we had the chance to address this topic in the weekly meeting and I tried to explain a little bit more. More clarifications: You mentioned in the chat trigger-openqa_in_openqa
. The https://progress.opensuse.org/issues/68260#Suggestions should already cover this: I think the relevant script is https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit .I even suggested a specific line. I might be wrong and you are very welcome to explain to me where this is wrong :)
Updated by mkittler about 4 years ago
Right, the suggestion actually specify the exact line number. Then I assume my PR https://github.com/os-autoinst/scripts/pull/41 is wrong because it is done before the osc check-in.
probably just add a waiting check for "all packages succeeded"
So far I've tried osc results --repo openSUSE_Factory --failed "$src_project" "$package"
to check whether the packages haven't failed. Should I put something like that in a loop or do you have anything specific in mind with "waiting check"?
Updated by okurz about 4 years ago
yeah either that osc call or just curl like in https://gitlab.suse.de/openqa/osd-deployment/-/blob/master/.gitlab-ci.yml#L58 to query for package results in this project. Probably I would look at all packages in a repo, not a specific package.
Updated by okurz about 4 years ago
PR merged. You can either wait for jenkins to pick up the new version and the pipeline is triggered or trigger it in jenkins explicitly.
Updated by okurz about 4 years ago
all jobs fail now with:
Can't locate object method "allow_dupkeys" via package "Cpanel::JSON::XS" at /usr/share/openqa/script/../lib/OpenQA/Command.pm line 26.
Compilation failed in require at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Base.pm line 134.
BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/CLI/api.pm line 17.
Compilation failed in require at (eval 19) line 1.
+ qcow=
e.g. see http://jenkins.qa.suse.de/view/openQA-in-openQA/job/trigger-openQA_in_openQA-TW/6264/console
Updated by mkittler about 4 years ago
- Status changed from In Progress to Resolved
I fixed the installation on the Jenkins host, it seems to work now:
+ find_latest_published_tumbleweed_image
++ curl https://openqa.opensuse.org/group_overview/1.json
++ jq -r '[.build_results[] | select(.tag.description=="published") | select(.version=="Tumbleweed") | .build] | sort | reverse | .[0]'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 8503 100 8503 0 0 5357 0 0:00:01 0:00:01 --:--:-- 5354
100 8503 100 8503 0 0 5354 0 0:00:01 0:00:01 --:--:-- 5354
+ latest_published_tw_build=20201110
+ [[ -n 20201110 ]]
++ /usr/share/openqa/script/openqa-cli api --host https://openqa.opensuse.org assets get
++ jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-20201110-Tumbleweed\\@64bit.qcow"))] | .[0] | .name'
+ qcow=opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2
+ [[ -n opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2 ]]
+ '[' https://openqa.opensuse.org '!=' https://openqa.opensuse.org ']'
++ echo jenkins-trigger-openQA_in_openQA-TW-6333
++ sed -e s/jenkins-trigger-openQA_in_openQA-/:/ -e s/-/./g
+ build=:TW.6333
+ trigger
+ '[' openqa.opensuse.org = openqa.opensuse.org ']'
+ ARGS=OPENQA_HOST=http://openqa.opensuse.org
+ /usr/share/openqa/script/client --host https://openqa.opensuse.org isos post VERSION=Tumbleweed DISTRI=openQA FLAVOR=dev BUILD=:TW.6333 ARCH=x86_64 BACKEND=qemu WORKER_CLASS=qemu_x86_64 HDD_1=opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2
Updated by mkittler about 4 years ago
- Status changed from Resolved to Feedback
My previous comment was actually for #77008; but my change for that issue lead to the error you've mentioned here.
Not sure whether the submission of failed packages is already successfully prevented. At least it didn't break everything, e.g. http://jenkins.qa.suse.de/view/openQA-in-openQA/job/submit-openQA-TW-to-oS_Fctry/506/ was successful. However, it still doesn't work:
+ local 'osc_query=https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ local 'wip_states=\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ local 'bad_states=\(failed\|unresolvable\|broken\)'
+ local attempts=5
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 5 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 5)'
Waiting for re-build of os-autoinst to be considered (attempts left: 5)
+ sleep 2
+ attempts=4
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 4 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 4)'
Waiting for re-build of os-autoinst to be considered (attempts left: 4)
+ sleep 2
+ attempts=3
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 3 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 3)'
Waiting for re-build of os-autoinst to be considered (attempts left: 3)
+ sleep 2
+ attempts=2
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ [[ 2 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 2)'
Waiting for re-build of os-autoinst to be considered (attempts left: 2)
+ sleep 2
+ attempts=1
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 1 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 1)'
Waiting for re-build of os-autoinst to be considered (attempts left: 1)
+ sleep 2
+ attempts=0
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 0 -le 0 ]]
+ echo 'warning: Re-build of os-autoinst has not been considered in time (or package has been re-built so fast that we'\''ve missed it)'
warning: Re-build of os-autoinst has not been considered in time (or package has been re-built so fast that we've missed it)
+ break
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(failed\|unresolvable\|broken\)'
+ cmd=' osc sr'
The problem is that it passes openSUSE:Factory
instead of openSUSE_Factory
. The 10 seconds it gives OBS to change the package status are likely not enough as well.
Updated by mkittler about 4 years ago
It seems like the positive case works:
Updated by okurz about 4 years ago
Given that commonly our packages do not fail to build when we submit them you can just assume it works. I already told Dimstar that we have that safety measure in place and he should yell at us if it does not work :D
You can set to "Resolved"