Project

General

Profile

action #68260

[packages] prevent submission of failed packages from devel:openQA:tested to Factory

Added by okurz about 1 year ago. Updated 8 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-06-19
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

[19/06/2020 15:19:45] <DimStar> okurz: wasn't the idea of openQA:tested that this was actually tested? And that sort of would require it to at least build, no? :)
[19/06/2020 15:20:54] <okurz> DimStar: unfortunately not. I remember clearly that I opted for a different name "testing" or similar and IRC logs might prove that maybe you and others opted for "tested" :D
[19/06/2020 15:22:19] <DimStar> shame...
[19/06/2020 15:23:03] <okurz> however the name is of course no excuse for creating a SR when the build does not work. … There was ticket for this somewhere around here … *diggingintobacklog*
[19/06/2020 15:23:58] <okurz> DimStar: https://build.opensuse.org/request/show/815729 would fix that though :)
[19/06/2020 15:24:00] <|Anna|> sr#815729 [submit devel:languages:perl/perl-Mojolicious -> openSUSE:Factory/perl-Mojolicious] No description text (State: review) -- https://build.opensuse.org/request/show/815729
[19/06/2020 15:24:29] <DimStar> okurz: ok, then I merge those adis together
[19/06/2020 15:25:01] <DimStar> can you reopen openQA in this case (as I declined it as FTBFS)
[19/06/2020 15:25:07] <okurz> that would be great. Anyway, I will still plan to not create submit requests when the build fails

Acceptance criteria

Suggestions

History

#1 Updated by okurz about 1 year ago

  • Target version set to Ready

#2 Updated by okurz 12 months ago

  • Description updated (diff)

#3 Updated by okurz 12 months ago

  • Subject changed from prevent submission of failed packages from devel:openQA:tested to Factory to [packages] prevent submission of failed packages from devel:openQA:tested to Factory

#4 Updated by okurz 10 months ago

  • Description updated (diff)
  • Priority changed from Normal to High

Due to the broken perl-Mojolicious update we did multiple submit requests which all failed and were subsequently rightfully rejected. This "annoyed" DimStar at at least enough to point this out to me :) So I think we can regard this with higher prio now cosidering that there are not many other high prio issues right now.

#5 Updated by mkittler 9 months ago

  • Assignee set to mkittler

#6 Updated by mkittler 9 months ago

  • Assignee deleted (mkittler)

#7 Updated by okurz 9 months ago

what happened, any particular reason which got in your way? :)

#8 Updated by mkittler 9 months ago

  • Assignee set to mkittler

#9 Updated by okurz 9 months ago

so we had the chance to address this topic in the weekly meeting and I tried to explain a little bit more. More clarifications: You mentioned in the chat trigger-openqa_in_openqa. The https://progress.opensuse.org/issues/68260#Suggestions should already cover this: I think the relevant script is https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit .I even suggested a specific line. I might be wrong and you are very welcome to explain to me where this is wrong :)

#10 Updated by mkittler 9 months ago

Right, the suggestion actually specify the exact line number. Then I assume my PR https://github.com/os-autoinst/scripts/pull/41 is wrong because it is done before the osc check-in.

probably just add a waiting check for "all packages succeeded"

So far I've tried osc results --repo openSUSE_Factory --failed "$src_project" "$package" to check whether the packages haven't failed. Should I put something like that in a loop or do you have anything specific in mind with "waiting check"?

#11 Updated by okurz 9 months ago

yeah either that osc call or just curl like in https://gitlab.suse.de/openqa/osd-deployment/-/blob/master/.gitlab-ci.yml#L58 to query for package results in this project. Probably I would look at all packages in a repo, not a specific package.

#12 Updated by okurz 9 months ago

PR merged. You can either wait for jenkins to pick up the new version and the pipeline is triggered or trigger it in jenkins explicitly.

#13 Updated by okurz 9 months ago

  • Status changed from Workable to In Progress

#14 Updated by okurz 9 months ago

all jobs fail now with:

Can't locate object method "allow_dupkeys" via package "Cpanel::JSON::XS" at /usr/share/openqa/script/../lib/OpenQA/Command.pm line 26.
Compilation failed in require at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Base.pm line 134.
BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/CLI/api.pm line 17.
Compilation failed in require at (eval 19) line 1.
+ qcow=

e.g. see http://jenkins.qa.suse.de/view/openQA-in-openQA/job/trigger-openQA_in_openQA-TW/6264/console

#15 Updated by mkittler 9 months ago

  • Status changed from In Progress to Resolved

I fixed the installation on the Jenkins host, it seems to work now:

+ find_latest_published_tumbleweed_image
++ curl https://openqa.opensuse.org/group_overview/1.json
++ jq -r '[.build_results[] | select(.tag.description=="published") | select(.version=="Tumbleweed") | .build] | sort | reverse | .[0]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100  8503  100  8503    0     0   5357      0  0:00:01  0:00:01 --:--:--  5354
100  8503  100  8503    0     0   5354      0  0:00:01  0:00:01 --:--:--  5354
+ latest_published_tw_build=20201110
+ [[ -n 20201110 ]]
++ /usr/share/openqa/script/openqa-cli api --host https://openqa.opensuse.org assets get
++ jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-20201110-Tumbleweed\\@64bit.qcow"))] | .[0] | .name'
+ qcow=opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2
+ [[ -n opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2 ]]
+ '[' https://openqa.opensuse.org '!=' https://openqa.opensuse.org ']'
++ echo jenkins-trigger-openQA_in_openQA-TW-6333
++ sed -e s/jenkins-trigger-openQA_in_openQA-/:/ -e s/-/./g
+ build=:TW.6333
+ trigger
+ '[' openqa.opensuse.org = openqa.opensuse.org ']'
+ ARGS=OPENQA_HOST=http://openqa.opensuse.org
+ /usr/share/openqa/script/client --host https://openqa.opensuse.org isos post VERSION=Tumbleweed DISTRI=openQA FLAVOR=dev BUILD=:TW.6333 ARCH=x86_64 BACKEND=qemu WORKER_CLASS=qemu_x86_64 HDD_1=opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2

#16 Updated by mkittler 9 months ago

  • Status changed from Resolved to Feedback

My previous comment was actually for #77008; but my change for that issue lead to the error you've mentioned here.

Not sure whether the submission of failed packages is already successfully prevented. At least it didn't break everything, e.g. http://jenkins.qa.suse.de/view/openQA-in-openQA/job/submit-openQA-TW-to-oS_Fctry/506/ was successful. However, it still doesn't work:

+ local 'osc_query=https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ local 'wip_states=\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ local 'bad_states=\(failed\|unresolvable\|broken\)'
+ local attempts=5
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 5 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 5)'
Waiting for re-build of os-autoinst to be considered (attempts left: 5)
+ sleep 2
+ attempts=4
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 4 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 4)'
Waiting for re-build of os-autoinst to be considered (attempts left: 4)
+ sleep 2
+ attempts=3
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 3 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 3)'
Waiting for re-build of os-autoinst to be considered (attempts left: 3)
+ sleep 2
+ attempts=2
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ [[ 2 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 2)'
Waiting for re-build of os-autoinst to be considered (attempts left: 2)
+ sleep 2
+ attempts=1
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 1 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 1)'
Waiting for re-build of os-autoinst to be considered (attempts left: 1)
+ sleep 2
+ attempts=0
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 0 -le 0 ]]
+ echo 'warning: Re-build of os-autoinst has not been considered in time (or package has been re-built so fast that we'\''ve missed it)'
warning: Re-build of os-autoinst has not been considered in time (or package has been re-built so fast that we've missed it)
+ break
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(failed\|unresolvable\|broken\)'
+ cmd=' osc sr'

The problem is that it passes openSUSE:Factory instead of openSUSE_Factory. The 10 seconds it gives OBS to change the package status are likely not enough as well.

#18 Updated by okurz 8 months ago

Given that commonly our packages do not fail to build when we submit them you can just assume it works. I already told Dimstar that we have that safety measure in place and he should yell at us if it does not work :D
You can set to "Resolved"

#19 Updated by okurz 8 months ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF