Project

General

Profile

Actions

action #68260

closed

[packages] prevent submission of failed packages from devel:openQA:tested to Factory

Added by okurz almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-06-19
Due date:
% Done:

0%

Estimated time:

Description

Observation

[19/06/2020 15:19:45] <DimStar> okurz: wasn't the idea of openQA:tested that this was actually tested? And that sort of would require it to at least build, no? :)
[19/06/2020 15:20:54] <okurz> DimStar: unfortunately not. I remember clearly that I opted for a different name "testing" or similar and IRC logs might prove that maybe you and others opted for "tested" :D
[19/06/2020 15:22:19] <DimStar> shame...
[19/06/2020 15:23:03] <okurz> however the name is of course no excuse for creating a SR when the build does not work. … There was ticket for this somewhere around here … *diggingintobacklog*
[19/06/2020 15:23:58] <okurz> DimStar: https://build.opensuse.org/request/show/815729 would fix that though :)
[19/06/2020 15:24:00] <|Anna|> sr#815729 [submit devel:languages:perl/perl-Mojolicious -> openSUSE:Factory/perl-Mojolicious] No description text (State: review) -- https://build.opensuse.org/request/show/815729
[19/06/2020 15:24:29] <DimStar> okurz: ok, then I merge those adis together
[19/06/2020 15:25:01] <DimStar> can you reopen openQA in this case (as I declined it as FTBFS)
[19/06/2020 15:25:07] <okurz> that would be great. Anyway, I will still plan to not create submit requests when the build fails

Acceptance criteria

Suggestions

Actions #1

Updated by okurz over 3 years ago

  • Target version set to Ready
Actions #2

Updated by okurz over 3 years ago

  • Description updated (diff)
Actions #3

Updated by okurz over 3 years ago

  • Subject changed from prevent submission of failed packages from devel:openQA:tested to Factory to [packages] prevent submission of failed packages from devel:openQA:tested to Factory
Actions #4

Updated by okurz over 3 years ago

  • Description updated (diff)
  • Priority changed from Normal to High

Due to the broken perl-Mojolicious update we did multiple submit requests which all failed and were subsequently rightfully rejected. This "annoyed" DimStar at at least enough to point this out to me :) So I think we can regard this with higher prio now cosidering that there are not many other high prio issues right now.

Actions #5

Updated by mkittler over 3 years ago

  • Assignee set to mkittler
Actions #6

Updated by mkittler over 3 years ago

  • Assignee deleted (mkittler)
Actions #7

Updated by okurz over 3 years ago

what happened, any particular reason which got in your way? :)

Actions #8

Updated by mkittler over 3 years ago

  • Assignee set to mkittler
Actions #9

Updated by okurz over 3 years ago

so we had the chance to address this topic in the weekly meeting and I tried to explain a little bit more. More clarifications: You mentioned in the chat trigger-openqa_in_openqa. The https://progress.opensuse.org/issues/68260#Suggestions should already cover this: I think the relevant script is https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit .I even suggested a specific line. I might be wrong and you are very welcome to explain to me where this is wrong :)

Actions #10

Updated by mkittler over 3 years ago

Right, the suggestion actually specify the exact line number. Then I assume my PR https://github.com/os-autoinst/scripts/pull/41 is wrong because it is done before the osc check-in.

probably just add a waiting check for "all packages succeeded"

So far I've tried osc results --repo openSUSE_Factory --failed "$src_project" "$package" to check whether the packages haven't failed. Should I put something like that in a loop or do you have anything specific in mind with "waiting check"?

Actions #11

Updated by okurz over 3 years ago

yeah either that osc call or just curl like in https://gitlab.suse.de/openqa/osd-deployment/-/blob/master/.gitlab-ci.yml#L58 to query for package results in this project. Probably I would look at all packages in a repo, not a specific package.

Actions #12

Updated by okurz over 3 years ago

PR merged. You can either wait for jenkins to pick up the new version and the pipeline is triggered or trigger it in jenkins explicitly.

Actions #13

Updated by okurz over 3 years ago

  • Status changed from Workable to In Progress
Actions #14

Updated by okurz over 3 years ago

all jobs fail now with:

Can't locate object method "allow_dupkeys" via package "Cpanel::JSON::XS" at /usr/share/openqa/script/../lib/OpenQA/Command.pm line 26.
Compilation failed in require at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Base.pm line 134.
BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/CLI/api.pm line 17.
Compilation failed in require at (eval 19) line 1.
+ qcow=

e.g. see http://jenkins.qa.suse.de/view/openQA-in-openQA/job/trigger-openQA_in_openQA-TW/6264/console

Actions #15

Updated by mkittler over 3 years ago

  • Status changed from In Progress to Resolved

I fixed the installation on the Jenkins host, it seems to work now:

+ find_latest_published_tumbleweed_image
++ curl https://openqa.opensuse.org/group_overview/1.json
++ jq -r '[.build_results[] | select(.tag.description=="published") | select(.version=="Tumbleweed") | .build] | sort | reverse | .[0]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100  8503  100  8503    0     0   5357      0  0:00:01  0:00:01 --:--:--  5354
100  8503  100  8503    0     0   5354      0  0:00:01  0:00:01 --:--:--  5354
+ latest_published_tw_build=20201110
+ [[ -n 20201110 ]]
++ /usr/share/openqa/script/openqa-cli api --host https://openqa.opensuse.org assets get
++ jq -r '[.assets[] | select(.name | test("Tumbleweed-x86_64-20201110-Tumbleweed\\@64bit.qcow"))] | .[0] | .name'
+ qcow=opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2
+ [[ -n opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2 ]]
+ '[' https://openqa.opensuse.org '!=' https://openqa.opensuse.org ']'
++ echo jenkins-trigger-openQA_in_openQA-TW-6333
++ sed -e s/jenkins-trigger-openQA_in_openQA-/:/ -e s/-/./g
+ build=:TW.6333
+ trigger
+ '[' openqa.opensuse.org = openqa.opensuse.org ']'
+ ARGS=OPENQA_HOST=http://openqa.opensuse.org
+ /usr/share/openqa/script/client --host https://openqa.opensuse.org isos post VERSION=Tumbleweed DISTRI=openQA FLAVOR=dev BUILD=:TW.6333 ARCH=x86_64 BACKEND=qemu WORKER_CLASS=qemu_x86_64 HDD_1=opensuse-Tumbleweed-x86_64-20201110-Tumbleweed@64bit.qcow2
Actions #16

Updated by mkittler over 3 years ago

  • Status changed from Resolved to Feedback

My previous comment was actually for #77008; but my change for that issue lead to the error you've mentioned here.

Not sure whether the submission of failed packages is already successfully prevented. At least it didn't break everything, e.g. http://jenkins.qa.suse.de/view/openQA-in-openQA/job/submit-openQA-TW-to-oS_Fctry/506/ was successful. However, it still doesn't work:

+ local 'osc_query=https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ local 'wip_states=\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ local 'bad_states=\(failed\|unresolvable\|broken\)'
+ local attempts=5
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 5 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 5)'
Waiting for re-build of os-autoinst to be considered (attempts left: 5)
+ sleep 2
+ attempts=4
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 4 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 4)'
Waiting for re-build of os-autoinst to be considered (attempts left: 4)
+ sleep 2
+ attempts=3
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 3 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 3)'
Waiting for re-build of os-autoinst to be considered (attempts left: 3)
+ sleep 2
+ attempts=2
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ [[ 2 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 2)'
Waiting for re-build of os-autoinst to be considered (attempts left: 2)
+ sleep 2
+ attempts=1
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 1 -le 0 ]]
+ echo 'Waiting for re-build of os-autoinst to be considered (attempts left: 1)'
Waiting for re-build of os-autoinst to be considered (attempts left: 1)
+ sleep 2
+ attempts=0
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ [[ 0 -le 0 ]]
+ echo 'warning: Re-build of os-autoinst has not been considered in time (or package has been re-built so fast that we'\''ve missed it)'
warning: Re-build of os-autoinst has not been considered in time (or package has been re-built so fast that we've missed it)
+ break
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(unknown\|blocked\|scheduled\|dispatching\|building\|signing\|finished\)'
+ curl -s 'https://api.opensuse.org/public/build/devel:openQA:tested/_result?repository=openSUSE:Factory&package=os-autoinst'
+ grep -e '\(failed\|unresolvable\|broken\)'
+ cmd=' osc sr'

The problem is that it passes openSUSE:Factory instead of openSUSE_Factory. The 10 seconds it gives OBS to change the package status are likely not enough as well.

Actions #18

Updated by okurz over 3 years ago

Given that commonly our packages do not fail to build when we submit them you can just assume it works. I already told Dimstar that we have that safety measure in place and he should yell at us if it does not work :D
You can set to "Resolved"

Actions #19

Updated by okurz over 3 years ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF