action #178102
closedQA (public) - coordination #162890: [saga][epic] feature discoverability
coordination #162896: [epic] Job triggering on jobless openQA instances
No automatic submission of os-autoinst+openQA as of 2025-02-28 due to failing builds in openSUSE:Factory size:S
0%
Description
Observation¶
It seems that there have not been recent submissions of os-autoinst+openQA to openSUSE:Factory, likely related to #177901. Last openQA submission was https://build.opensuse.org/requests/1247815 from 2025-02-22.
http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/2827/console and similar jobs say
Skipping submission, reason:
Only triggering tests from devel:openQA (not overriding devel:openQA:testing and doing a submission) because openQA-in-openQA tests or a submission is still pending (as devel:openQA:testing still contains packages)
I think the assumption that a submission is pending only because devel:openQA:testing contains packages is wrong.
The problem was originally triggered by #177901
Acceptance criteria¶
- AC1: There are regular submissions of os-autoinst+openQA to openSUSE:Factory even if builds (of the version currently in Factory) fail in openSUSE:Factory
Suggestions¶
- DONE Mitigate with manual intervention -> done by okurz
- Check the logic within https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit, in particular https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit#L183
- As necessary watch https://youtu.be/VtQ68AQhfAQ to learn more about "continuous releases" . Since the time of the recording we introduced another intermediate step "devel:openQA:testing"
- If a package got skipped because of the version, possibly for the wrong reason, it should still show in the logs
Updated by okurz about 2 months ago
- Copied from action #174451: openQA-in-openQA tests can get stuck with an inconsistent repository size:S added
Updated by okurz about 2 months ago
- Copied to action #178105: Ensure automatic submissions os-autoinst+openQA if previous submissions have been too long in the past added
Updated by okurz about 2 months ago · Edited
On jenkins.qe.nue2.suse.org I did rm /var/lib/jenkins/workspace/submit-openQA-TW-to-oS_Fctry/job_post_skip_submission and triggered http://jenkins.qa.suse.de/job/trigger-openQA_in_openQA-TW/ again. I hope that's enough as mitigation.
EDIT: Seems to have no effect as per http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/lastSuccessfulBuild/console . I now deleted os-autoinst+openQA from https://build.opensuse.org/project/show/devel:openQA:testing and copypac'd them from devel:openQA and called the submit script manually. Finally a new submission https://build.opensuse.org/requests/1249335 . Was that ok?
Updated by mkittler about 2 months ago
I think the assumption that a submission is pending only because devel:openQA:testing contains packages is wrong.
I now deleted os-autoinst+openQA from https://build.opensuse.org/project/show/devel:openQA:testing and …
Ok, so the packages in that project were old?
… and copypac'd them from devel:openQA and called the submit script manually.
Was that ok?
I guess we normally don't use copypac in this case but I guess it was ok.
The packages currently on https://build.opensuse.org/project/show/devel:openQA:testing are quite new (only 17 to 35 minutes old right now). So if old packages got stuck there then the problem is not repeating.
Updated by okurz about 2 months ago
mkittler wrote in #note-4:
I think the assumption that a submission is pending only because devel:openQA:testing contains packages is wrong.
I now deleted os-autoinst+openQA from https://build.opensuse.org/project/show/devel:openQA:testing and …
Ok, so the packages in that project were old?
Yes
… and copypac'd them from devel:openQA and called the submit script manually.
Was that ok?
I guess we normally don't use copypac in this case but I guess it was ok.
The packages currently on https://build.opensuse.org/project/show/devel:openQA:testing are quite new (only 17 to 35 minutes old right now). So if old packages got stuck there then the problem is not repeating.
Yes, I assume the problem was that builds were failing hence packages got never updated to a fixed version
Updated by okurz about 2 months ago
- Related to action #177901: OBS build in openSUSE:Factory/openQA/standard/x86_64 fails with "make: *** No rule to make target 'package-lock.json', needed by 'node_modules'. Stop." size:S added
Updated by okurz about 2 months ago
- Subject changed from No automatic submission of os-autoinst+openQA as of 2025-02-28 to No automatic submission of os-autoinst+openQA as of 2025-02-28 due to failing builds in openSUSE:Factory size:S
- Description updated (diff)
- Status changed from New to Workable
Updated by mkittler about 2 months ago
- Status changed from Workable to In Progress
- Assignee set to mkittler
Updated by mkittler about 2 months ago · Edited
There are many Git related errors in the cleanup code:
+ [[ passed != \p\a\s\s\e\d ]]
+ [[ -n 4875278 ]]
+ delete_packages_from_obs_project devel:openQA:testing
+ local obs_project=devel:openQA:testing
+ local packages
++ list_packages devel:openQA:testing
++ local obs_project=devel:openQA:testing
++ osc ls devel:openQA:testing
++ grep -v '\-test$'
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
+ packages='openQA
openQA-devel-container
os-autoinst
os-autoinst-distri-opensuse-deps
xterm-console'
+ for package in $packages
+ osc rdelete -m 'Cleaning up openQA to allow triggering new jobs' devel:openQA:testing openQA
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
+ for package in $packages
+ osc rdelete -m 'Cleaning up openQA-devel-container to allow triggering new jobs' devel:openQA:testing openQA-devel-container
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
+ for package in $packages
+ osc rdelete -m 'Cleaning up os-autoinst to allow triggering new jobs' devel:openQA:testing os-autoinst
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
+ for package in $packages
+ osc rdelete -m 'Cleaning up os-autoinst-distri-opensuse-deps to allow triggering new jobs' devel:openQA:testing os-autoinst-distri-opensuse-deps
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
+ for package in $packages
+ osc rdelete -m 'Cleaning up xterm-console to allow triggering new jobs' devel:openQA:testing xterm-console
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
+ [[ -n '' ]]
+ exit 1
(from http://jenkins.qa.suse.de/job/monitor-openQA_in_openQA-TW/29451/console)
However, those are not relevant, e.g. when I run this command locally it still works (despite the Git error):
$ osc ls "devel:openQA:testing"
> pruning cache
> pruned 0 files comprised of 0 bytes
Schwerwiegend: Kein Git-Repository (oder irgendeines der Elternverzeichnisse): .git
openQA
openQA:openQA-devel-test
openQA:openQA-test
openQA:openQA-worker-test
openQA:openQA-client-test
openQA-devel-container
os-autoinst
os-autoinst:os-autoinst-test
os-autoinst:os-autoinst-devel-test
os-autoinst:os-autoinst-openvswitch-test
os-autoinst-distri-opensuse-deps
xterm-console
$ echo $?
0
Updated by mkittler about 2 months ago · Edited
- Status changed from In Progress to Feedback
The issue is that the cleanup of the staging project doesn't happen if the submission fails due to a build error. The log shows this as it just ends with an error message and no further cleanup:
Building openQA has failed, skipping submission
Build step 'Execute shell' marked build as failure
…
Finished: FAILURE
(from http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/2753/console)
This should be fixed by https://github.com/os-autoinst/scripts/pull/379.
Updated by mkittler about 2 months ago
The packages in https://build.opensuse.org/project/show/devel:openQA:testing are from 2 days ago. So I'm cleaning them up now to allow further submissions to see whether my change helped.
Updated by mkittler about 1 month ago
- Status changed from Feedback to Resolved
Looks like packages are submitted regularly again, e.g. https://build.opensuse.org/requests/1251567 and https://build.opensuse.org/requests/1251217. There are also no leftovers in https://build.opensuse.org/project/show/devel:openQA:testing.