openSUSE admin

Dear Dr. Carsten Otto,

Thank you for contributing to openSUSE and please excuse delayed handling of the issue.

To me it looks that these errors happen because of race condition when the files are being updated on the server at the same moment while your rsync command is running.
The best reference I found is on oracle site, I am not sure why nobody else is talking about the same: https://support.oracle.com/knowledge/Oracle%20E-Business%20Suite/2315331_1.html

I'd say it is the best to retry the sync when such error happens. I will ask in the team if they have better ideas.
But you can also try syncing from stage-main-repos.opensuse.org as was mentioned recently in the list announcement and is addressed in the wiki - maybe it will bring better experience (it doesn't require authentication and should provide recent files with better speed and less contention on the main server):

https://lists.opensuse.org/archives/list/mirror@lists.opensuse.org/thread/7N5P7A4VKP6S3Y2F6U5TGSA6UZU22LG2/
https://en.opensuse.org/openSUSE:Mirror_infrastructure#stage-main-repos.opensuse.org

Let us know if any updates about the problem.

Regards,
Andrii Nikitin

Hi Andrii,

On Tue, Apr 18, 2023 at 08:47:01AM +0000, redmine@opensuse.org wrote:

To me it looks that these errors happen because of race condition when
the files are being updated on the server at the same moment while
your rsync command is running.

That's not the case, assuming your updates don't take 4 days to
complete. I saw errors related to the stuck file from April 7th (9:42
CEST) to April 11th (15:46).

Please consider this as a proper bug report, not just a temporary fluke.
Please also consider my comments in #108494. As said before, this is an
ongoing issue, and I'd like to see lasting fixes, as mentioned in said
ticket (from 2022).

But you can also try syncing from stage-main-repos.opensuse.org as was
mentioned recently in the list announcement and is addressed in the
wiki - maybe it will bring better experience (it doesn't require
authentication and should provide recent files with better speed and
less contention on the main server):

That's worth a try, thanks for the hint!

Best regards,

Carsten¶

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

Hi,

On Wed, Apr 19, 2023 at 07:43:26AM +0000, redmine@opensuse.org wrote:

Was it always the same file or always different file? It could be some
leftover from a dead process, but I cannot find any trace now.

Always the same file.

In case if you see a stuck file again - please try to reproduce the
problem with command like below, so it shows that the file is there
for significant amount of time:

I reproduced the issue using our regular rsync runs, what's the
advantage of the command you sent?

If the issue happens again - I will try to tweak some options on server side.

I already reported this issue as early as 2022, possibly before. Please
just fix it, there's no need for it to happen again.

But still switching to stage-main-repos.opensuse.org (aka
stage3.opensuse.org ) is preferred, because the main machine
stage.opensuse.org is currently very overloaded at maximum capacity.

I already did that.

Bye,

Carsten¶

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

This is still an issue.¶

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

Hi,

the issue doesn't persist, but it pops up often enough to be annoying.
To reproduce this, please either monitor the system for a longer time
period, or check any logs you might have. It might help to run
"ls -l /some/where/opensuse/miniuprepo*" every minute or so to see when
these files pop up (assuming you're able to figure out the process that
creates them).

The repository is named "opensuse-full-with-factory-dvd5" and the full
path within this repository is "/opensuse/miniuprepoXXXXX" (with random
letters in place of the Xs).

Some examples from recent log files:

"/opensuse/miniuprepoPOtZ0"
"/opensuse/miniuprepoPaYrk"
"/opensuse/miniuprepoPEmdX"
"/opensuse/miniuprepoBrUbp"
"/opensuse/miniuprepoOwJ4R"
"/opensuse/miniuprepo0vBsY"

Please make sure that the rsync server only exposes files that should be
downloaded from mirrors. In this case I believe is temporary/internal
and, as such, should not be exposed via rsync, as this causes issues.

The most recent issue occured today at 15:30 CEST
("/opensuse/miniuprepoPOtZ0").

Bye

Carsten¶

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

Hi Carsten,

otto@informatik.rwth-aachen.de wrote:

The repository is named "opensuse-full-with-factory-dvd5" and the full
path within this repository is "/opensuse/miniuprepoXXXXX" (with random
letters in place of the Xs).

I've got confirmation that issues with miniuprepo indeed happened on stage3.opensuse.org and were solved around two weeks ago.

Did you switch to stage3.opensuse.org? Are the other errors still observed?

Regards,
Andrii Nikitin

Hi,

On Wed, Jun 14, 2023 at 11:42:43AM +0000, redmine@opensuse.org wrote:

I've got confirmation that issues with miniuprepo indeed happened on
stage3.opensuse.org and were solved around two weeks ago.

I raised my issue around two months ago, I don't think this is related to
anything happening recently.

Did you switch to stage3.opensuse.org? Are the other errors still observed?

We're using stage-main-repos.opensuse.org. I haven't seen any recent
issue.

Aside from that, could you please make sure that permissions on
your server reflect the intention of files being downloadable? This
isn't the first time I've seen this issue, and suse obviously isn't able
to respond in a timely many to fix individual instances of the issue
(which is totally fine, I don't pay for support).

I just realized that I already asked you to do this, possibly more than
once. Could you please comment on this, so that I don't have to repeat
myself over and over again?

Please consider this as a proper bug report, not just a temporary fluke.
Please also consider my comments in #108494. As said before, this is an
ongoing issue, and I'd like to see lasting fixes, as mentioned in said

ticket (from 2022).¶

Thanks,

Carsten¶

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/