action #174451
closedQA (public) - coordination #162890: [saga][epic] feature discoverability
coordination #162896: [epic] Job triggering on jobless openQA instances
openQA-in-openQA tests can get stuck with an inconsistent repository size:S
0%
Description
Observation¶
After #167395 openQA-in-openQA tests use devel:openQA:testing which is a snapshot of devel:openQA created before triggering tests. Unfortunately this snapshot can be created at a bad time where the repository is inconsistent. At least that's how it looks like on https://openqa.opensuse.org/tests/4704845#step/openqa_worker/1 where the installation of openQA-worker fails with The to be installed openQA-worker-4.6.17… requires 'openqa-common = 4.6.17… … but this requirement cannot be provided. Then it suggests do downgrade to packages from openSUSE.
The tests retries the installation but because devel:openQA:testing is snapshotted after #167395 this is in vain now.
I checked the video and it looks like the test really only adds devel:openQA:testing (and not e.g. devel:openQA in addition). So I think an inconsistent snapshot of devel:openQA:testing is the most likely explanation.
Acceptance criteria¶
- AC1: Tests are only triggered from a consistent snapshot of
devel:openQA
Suggestions¶
- Familiarize yourself with https://github.com/os-autoinst/scripts/blob/master/trigger-openqa_in_openqa and https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit and http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/ to understand the current pipeline and the involved scripts / #167395
- Ensure that the repository
devel:openQAis "consistent" (supposedly everything is in the published state) before "releasing" it ondevel:openQA:testing. - After the releasing has been done, check whether a consistent state has been released. (All split packages have the same version.)
- Ask on OBS channels about this problem of the release command and how it is usually dealt with.
Updated by mkittler 2 months ago
- Related to action #167395: Ensure only the tested revision of devel:openQA packages are submitted to openSUSE:Factory size:M added
Updated by jbaier_cz about 1 month ago
- Status changed from Workable to In Progress
- Assignee set to jbaier_cz
Updated by jbaier_cz about 1 month ago
A quick and dirty hack would be something like https://github.com/os-autoinst/scripts/pull/365; it does not however solve the entire issue as it still is not an atomic operation. If we want dig more into this, the correct procedure would probably be by creating submit requests from devel:openQA into devel:openQA:testing, let it build inside this request and then check for consistency and approve or reject if there are mismatches. That's probably the most reasonable way to have an atomic operation.
As we are probably still trying to do that without the rebuild, I will introduce a second quick hack and will try to check the versions after the release is done and delete the packages if the version is not consistent. That should solve the original issue (at the cost of delaying the tests until next iteration).
Updated by openqa_review about 1 month ago
- Due date set to 2025-01-31
Setting due date based on mean cycle time of SUSE QE Tools
Updated by jbaier_cz about 1 month ago
I was about to add the version check when I realized we only have one source package for openQA so I do not have the package to check against. The original assumption about inconsistent snapshot is maybe wrong?
Updated by jbaier_cz about 1 month ago
I refreshed the PR to copy the packages as soon as possible after the check is done and added some cleanup for cases where something goes wrong.
Updated by tinita about 1 month ago
- Status changed from In Progress to Workable
due to absence this week
Updated by jbaier_cz 30 days ago · Edited
Looking at http://jenkins.qa.suse.de/job/trigger-openQA_in_openQA-TW/lastBuild/console (build 34512):
+ echo 'Checking if all relevant packages are built and published before creating snapshot under devel:openQA:testing'
Checking if all relevant packages are built and published before creating snapshot under devel:openQA:testing
+ for package in $auto_submit_packages
+ local state
++ osc results -r openSUSE_Tumbleweed -a x86_64 --no-multibuild devel:openQA openQA
+ state='openSUSE_Tumbleweed x86_64 openQA succeeded'
+ grep -q -v 'succeeded\|disabled'
+ echo 'Creating snapshots of openQA'
Creating snapshots of openQA
+ osc release --no-delay --target-project devel:openQA:testing --target-repository=openSUSE_Tumbleweed -r openSUSE_Tumbleweed -a x86_64 devel:openQA openQA
Releasing package 'devel:openQA/openQA' repository 'openSUSE_Tumbleweed' to project 'devel:openQA:testing' repository 'openSUSE_Tumbleweed' architecture 'x86_64'
The checks are there, I spotted a different issue with OBS. I briefly discussed it in https://suse.slack.com/archives/C02BXKBMXNV/p1738240417325969 and I have another improvement in our submission script: https://github.com/os-autoinst/scripts/pull/369
Updated by jbaier_cz 30 days ago
- Related to action #176316: why is os-autoinst-distri-opensuse-deps not submitted despite diff in package content? size:S added
Updated by okurz 26 days ago
- Status changed from Feedback to In Progress
- Priority changed from Normal to High
http://jenkins.qa.suse.de/job/trigger-openQA_in_openQA-TW/34517/console still stuck since more than 2 days
Increasing priority as blocking #176316
@jbaier_cz to create OBS related upstream issue report and/or requesting help from OBS instance admins.
Updated by okurz 26 days ago
- Related to deleted (action #176316: why is os-autoinst-distri-opensuse-deps not submitted despite diff in package content? size:S)
Updated by okurz 26 days ago
- Blocks action #176316: why is os-autoinst-distri-opensuse-deps not submitted despite diff in package content? size:S added
Updated by jbaier_cz 26 days ago
- Due date deleted (
2025-02-06) - Status changed from Feedback to Resolved
The OBS issue is solved and our Jenkins pipeline is again doing the job, https://openqa.opensuse.org/tests/4827495 used the correct repository for installing packages. We should be finally fine here.
Updated by okurz about 6 hours ago
- Copied to action #178102: No automatic submission of os-autoinst+openQA as of 2025-02-28 added