QA (public) » openQA Project (public)

We could do the following:

1. Save the current revision of the set of packages (auto_submit_packages=${packages:-$($osc ls "$dst_project" | grep -v '\-test$')}) via e.g. osc log devel:openQA openQA | grep/sed/… in trigger-openqa_in_openqa like we already save job_post_response.
2. Submit that revision via osc -r $rev co in os-autoinst-obs-auto-submit. We need to use the same approach for copying artifacts as in monitor-openQA_in_openQA-TW.

The problem with this is that the revision saved in 1. might not be the revision we have actually tested. Until the test is scheduled and reaches the point where packages are installed from OBS a new version might have been published on OBS.

It would probably make more sense if we do the following:

1. Keep the triggering as-is.
2. Upload the exact package versions that were installed during the test run.
3. Download the log from 2. in scripts/os-autoinst-obs-auto-submit. For this we need to know the job IDs be we have it in job_post_response and just need to restore them like in monitor-openQA_in_openQA-TW.
4. Check whether that version matches what we still have on OBS. If not, either skip the submission or try to figure out what revision had this version and submit that via osc -r $rev co.

Note that step 2 makes actually only sense for the scenarios that install packages from OBS. So within the set of jobs that we start only some jobs will contain the information about the package versions. This is presumably ok. Considering that there are multiple scenarios that install from OBS there can actually be the case when those tests did not all run on the same version of every package. Maybe we should then in step 4 simply reject the whole submission (will hopefully not happen very often).

We discussed that it would be better to look into the "branching" alternative (mentioned as 2nd suggestion in the ticket description).

So I invoked to create a "fixed" copy of some packages in our devel:openQA project under devel:openQA:testing¹:

osc linkpac --current --disable-build devel:openQA os-autoinst devel:openQA:testing
osc linkpac --current --disable-build devel:openQA openQA devel:openQA:testing

It looks like --disable-build simply sets the build flag in the config of the package to false. The publish flag is still true, though.

Unfortunately this kind of setup doesn't work. OBS says "Repository has been published" but https://download.opensuse.org/repositories/devel:/openQA:/testing/openSUSE_Tumbleweed has not been populated so far.

So also I tried:

osc branch --force --disable-build devel:openQA openQA devel:openQA:testing

But the repo is still not populated.

I guess I'll have to research how this can be done or ask for help. We could of course also download the repo manually and serve it via some HTTP server outside the scope of OBS.

EDIT: Asked on Slack: https://suse.slack.com/archives/C02BXKBMXNV/p1731605574673919

¹ Probably we should create and delete this project on the fly. The trigger script would delete it when not already there and otherwise abort. The monitoring/submission script would delete the project when done. This way the presence of the project would act as a flag to know whether testing/submissions are still pending (and be able to avoid triggering new tests/submissions if there is still one pending).

It turns out that one needs to use osc release for this (thanks to @bmwiedemann pointing this out to me).

So the following does the trick:

for package in os-autoinst openQA; do osc release --target-project devel:openQA:testing --no-delay --target-repository=openSUSE_Tumbleweed -r openSUSE_Tumbleweed -a x86_64 devel:openQA "$package"; done

This command populated https://download.opensuse.org/repositories/devel:/openQA:/testing/openSUSE_Tumbleweed/x86_64/ with no delay (so I suppose --no-delay works as advertised).

PR for being able to use the snapshot project in openQA-in-openQA tests: https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/215

I configured the submission step on Jenkins so that it'll run the scripts from Git via https://github.com/os-autoinst/scripts/pull/357.

I also added made it invoke a final cleanup of the staging repo so we never end up in the situation that a missing cleanup prevents further triggering.

The 2nd PR was merged and submissions still work, e.g. https://build.opensuse.org/requests/1226911.

I've just noticed that with my changes the throttling we put in place on http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/configure will now effectively bring all those jobs to a halt. Only if the submission job runs (with success or not) we would cleanup devel:openQA:testing and only then new tests will be triggered. Not sure how to resolve this because the whole point of this improvement is to synchronize these jobs (to avoid the problem mentioned in the ticket description). We could still let openQA-in-openQA jobs run but not overriding/using devel:openQA:testing and also not to submit anything (not sure how to prevent the submission job).

The PR has been merged and it new openQA-in-openQA tests have been triggered - without overriding devel:openQA:testing. Let's see how it behaves when the throttle for doing the next submission expires. (It'll expire only this evening.)

The only way to solve this I can currently think of is to remove the throttle on Jenkins level. Then we would check the date of last active SR within the submission job and skip if it is too young. This way the skipped submissions will not factor into the throttling.

The PR was merged and I added rm -f job_post_* before restoring artifacts to avoid leftover files in the working directory. I think it works and we got e.g. https://build.opensuse.org/requests/1228300 now (based on devel:openQA:testing).

Looks like the throttle within the script works (only " days" is missing in the output):

[submit-openQA-TW-to-oS_Fctry] $ /bin/sh -xe /tmp/jenkins13543617303560206947.sh
+ os-autoinst-obs-auto-submit
Skipping submission, there is still a pending SR younger than 2.
+ rc=0

Except when it doesn't work:

[submit-openQA-TW-to-oS_Fctry] $ /bin/sh -xe /tmp/jenkins484091042308331852.sh
+ os-autoinst-obs-auto-submit
Exception ignored in: <osc.util.safewriter.SafeWriter object at 0x7ff503457978>
BrokenPipeError: [Errno 32] Broken pipe
Retrying up to 3 more times after sleeping 3s …
Exception ignored in: <osc.util.safewriter.SafeWriter object at 0x7fe5cfb98978>
BrokenPipeError: [Errno 32] Broken pipe
Retrying up to 2 more times after sleeping 6s …
Exception ignored in: <osc.util.safewriter.SafeWriter object at 0x7f3d3176f978>
BrokenPipeError: [Errno 32] Broken pipe
Retrying up to 1 more times after sleeping 12s …
Exception ignored in: <osc.util.safewriter.SafeWriter object at 0x7f678aaaa978>
BrokenPipeError: [Errno 32] Broken pipe
ok
A    devel:openQA:testing
A    devel:openQA:testing/openQA
…

This error happened two times.

I couldn't reproduce the problem outside of Jenkins so far, e.g. martchus@jenkins:~> sudo -u jenkins osc request list --project devel:openQA:tested --type submit --state new,review --mine --days 2 works just fine.

PR: https://github.com/os-autoinst/scripts/pull/360