action #174451
closed
QA (public) - coordination #162890: [saga][epic] feature discoverability
coordination #162896: [epic] Job triggering on jobless openQA instances
openQA-in-openQA tests can get stuck with an inconsistent repository size:S
Added by mkittler 4 months ago.
Updated about 2 months ago.
Category:
Regressions/Crashes
Description
Observation¶
After #167395 openQA-in-openQA tests use devel:openQA:testing which is a snapshot of devel:openQA created before triggering tests. Unfortunately this snapshot can be created at a bad time where the repository is inconsistent. At least that's how it looks like on https://openqa.opensuse.org/tests/4704845#step/openqa_worker/1 where the installation of openQA-worker fails with The to be installed openQA-worker-4.6.17… requires 'openqa-common = 4.6.17… … but this requirement cannot be provided. Then it suggests do downgrade to packages from openSUSE.
The tests retries the installation but because devel:openQA:testing is snapshotted after #167395 this is in vain now.
I checked the video and it looks like the test really only adds devel:openQA:testing (and not e.g. devel:openQA in addition). So I think an inconsistent snapshot of devel:openQA:testing is the most likely explanation.
Acceptance criteria¶
- AC1: Tests are only triggered from a consistent snapshot of
devel:openQA
Suggestions¶
- Related to action #167395: Ensure only the tested revision of devel:openQA packages are submitted to openSUSE:Factory size:M added
- Parent task set to #162896
- Tags set to reactive work
- Category set to Regressions/Crashes
- Target version set to Ready
- Subject changed from openQA-in-openQA tests can get stuck with an inconsistent repository to openQA-in-openQA tests can get stuck with an inconsistent repository size:S
- Description updated (diff)
- Status changed from New to Workable
- Status changed from Workable to In Progress
- Assignee set to jbaier_cz
A quick and dirty hack would be something like https://github.com/os-autoinst/scripts/pull/365; it does not however solve the entire issue as it still is not an atomic operation. If we want dig more into this, the correct procedure would probably be by creating submit requests from devel:openQA into devel:openQA:testing, let it build inside this request and then check for consistency and approve or reject if there are mismatches. That's probably the most reasonable way to have an atomic operation.
As we are probably still trying to do that without the rebuild, I will introduce a second quick hack and will try to check the versions after the release is done and delete the packages if the version is not consistent. That should solve the original issue (at the cost of delaying the tests until next iteration).
- Due date set to 2025-01-31
Setting due date based on mean cycle time of SUSE QE Tools
I was about to add the version check when I realized we only have one source package for openQA so I do not have the package to check against. The original assumption about inconsistent snapshot is maybe wrong?
I refreshed the PR to copy the packages as soon as possible after the check is done and added some cleanup for cases where something goes wrong.
- Status changed from In Progress to Workable
- Due date changed from 2025-01-31 to 2025-02-06
- Status changed from Workable to Feedback
Merged, now we need to see it in action.
Looking at http://jenkins.qa.suse.de/job/trigger-openQA_in_openQA-TW/lastBuild/console (build 34512):
+ echo 'Checking if all relevant packages are built and published before creating snapshot under devel:openQA:testing'
Checking if all relevant packages are built and published before creating snapshot under devel:openQA:testing
+ for package in $auto_submit_packages
+ local state
++ osc results -r openSUSE_Tumbleweed -a x86_64 --no-multibuild devel:openQA openQA
+ state='openSUSE_Tumbleweed x86_64 openQA succeeded'
+ grep -q -v 'succeeded\|disabled'
+ echo 'Creating snapshots of openQA'
Creating snapshots of openQA
+ osc release --no-delay --target-project devel:openQA:testing --target-repository=openSUSE_Tumbleweed -r openSUSE_Tumbleweed -a x86_64 devel:openQA openQA
Releasing package 'devel:openQA/openQA' repository 'openSUSE_Tumbleweed' to project 'devel:openQA:testing' repository 'openSUSE_Tumbleweed' architecture 'x86_64'
The checks are there, I spotted a different issue with OBS. I briefly discussed it in https://suse.slack.com/archives/C02BXKBMXNV/p1738240417325969 and I have another improvement in our submission script: https://github.com/os-autoinst/scripts/pull/369
- Related to action #176316: why is os-autoinst-distri-opensuse-deps not submitted despite diff in package content? size:S added
- Status changed from Feedback to In Progress
- Priority changed from Normal to High
- Related to deleted (action #176316: why is os-autoinst-distri-opensuse-deps not submitted despite diff in package content? size:S)
- Blocks action #176316: why is os-autoinst-distri-opensuse-deps not submitted despite diff in package content? size:S added
- Status changed from In Progress to Feedback
- Due date deleted (
2025-02-06)
- Status changed from Feedback to Resolved
- Copied to action #178102: No automatic submission of os-autoinst+openQA as of 2025-02-28 due to failing builds in openSUSE:Factory size:S added
Also available in: Atom
PDF