Project

General

Profile

Actions

tickets #127400

open

[carsten@c-otto.de: [ftpsync-opensuse-ftp@ftp] ERROR: rsync errors]

Added by otto@informatik.rwth-aachen.de 12 months ago. Updated 10 months ago.

Status:
Feedback
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
2023-04-08
Due date:
% Done:

0%

Estimated time:

Description

Hi,

there's a file stuck on stage.opensuse.org. Could you please check?

Please also consider my comments in #108494 from March 22nd 2022,
adressing the same issue. I'd love to see this fixed at the root.

/opensuse/distribution/leap/15.5/repo/oss/449.1-boot/boot/s390x/.initrd.G0Uqv3

Thanks
Carsten

----- Forwarded message from carsten@c-otto.de -----

Date: Fri, 7 Apr 2023 09:42:40 +0200 (CEST)
From: carsten@c-otto.de
To: datafetch@ftp.halifax.rwth-aachen.de
Subject: [ftpsync-opensuse-ftp@ftp] ERROR: rsync errors

rsync: [sender] send_files failed to open "/opensuse/distribution/leap/15.5/repo/oss/449.1-boot/boot/s390x/.initrd.G0Uqv3" (in opensuse-full-with-factory-dvd5): Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1819) [generator=3.2.3]

Apr 07 09:30:01 ftp ftpsync-opensuse-ftp[26323]: Mirrorsync start
Apr 07 09:30:01 ftp ftpsync-opensuse-ftp[26323]: Running mirrorsync, update is required, /pub/opensuse//Archive-Update-Required-ftp.halifax.rwth-aachen.de exists
Apr 07 09:30:01 ftp ftpsync-opensuse-ftp[26323]: Running stage2: rsync --quiet --log-file /home/datafetch/log/rsync-ftpsync-opensuse.log --filter=exclude_/Archive-Update-in-Progress-ftp.halifax.rwth-aachen.de --filter=protect_/Archive-Update-in-Progress-ftp.halifax.rwth-aachen.de --filter=exclude_/Archive-Update-Required-ftp.halifax.rwth-aachen.de --filter=protect_/Archive-Update-Required-ftp.halifax.rwth-aachen.de --filter=exclude_/project/trace/ftp.halifax.rwth-aachen.de --filter=protect_/project/trace/ftp.halifax.rwth-aachen.de --filter=exclude_/project/trace/ftp.halifax.rwth-aachen.de-stage1 --filter=protect_/project/trace/ftp.halifax.rwth-aachen.de-stage1 --filter=exclude_/project/trace/hierarchy --filter=protect/project/trace/hierarchy --filter=exclude/project/trace/traces --filter=protect/project/trace/traces --filter=include/project/ --filter=protect_/project/ --filter=include_/project/trace/ --filter=protect_/project/trace/ --filter=include_/project/trace/* --bwlimit
=0 -prltvHSB8192 --safe-links --chmod=D755,F644 --stats --no-human-readable --no-inc-recursive --exclude repositories/ --exclude=.~tmp~/ stage.opensuse.org::opensuse-full-with-factory-dvd5/opensuse/ /pub/opensuse/ --max-delete=40000 --delay-updates --delete --filter protect_debug/ --filter protect_source/
Apr 07 09:42:40 ftp ftpsync-opensuse-ftp[26323]: Back from rsync with returncode 23
Apr 07 09:42:40 ftp ftpsync-opensuse-ftp[26323]: Mirrorsync done with errors
Apr 07 09:42:40 ftp ftpsync-opensuse-ftp[26323]: bytes: 22969019430 759

----- End forwarded message -----

--
Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/


Files

signature.asc (195 Bytes) signature.asc otto@informatik.rwth-aachen.de, 2023-04-08 14:05
signature.asc (195 Bytes) signature.asc otto@informatik.rwth-aachen.de, 2023-04-18 16:43
signature.asc (195 Bytes) signature.asc otto@informatik.rwth-aachen.de, 2023-04-19 18:57
signature.asc (195 Bytes) signature.asc otto@informatik.rwth-aachen.de, 2023-04-28 19:41
signature.asc (195 Bytes) signature.asc otto@informatik.rwth-aachen.de, 2023-05-02 16:04
signature.asc (195 Bytes) signature.asc otto@informatik.rwth-aachen.de, 2023-06-14 17:42
Actions #1

Updated by pjessen 12 months ago

  • Assignee set to andriinikitin
  • Private changed from Yes to No
Actions #2

Updated by andriinikitin 12 months ago

  • Status changed from New to Feedback

Dear Dr. Carsten Otto,

Thank you for contributing to openSUSE and please excuse delayed handling of the issue.

To me it looks that these errors happen because of race condition when the files are being updated on the server at the same moment while your rsync command is running.
The best reference I found is on oracle site, I am not sure why nobody else is talking about the same: https://support.oracle.com/knowledge/Oracle%20E-Business%20Suite/2315331_1.html

I'd say it is the best to retry the sync when such error happens. I will ask in the team if they have better ideas.
But you can also try syncing from stage-main-repos.opensuse.org as was mentioned recently in the list announcement and is addressed in the wiki - maybe it will bring better experience (it doesn't require authentication and should provide recent files with better speed and less contention on the main server):

https://lists.opensuse.org/archives/list/mirror@lists.opensuse.org/thread/7N5P7A4VKP6S3Y2F6U5TGSA6UZU22LG2/
https://en.opensuse.org/openSUSE:Mirror_infrastructure#stage-main-repos.opensuse.org

Let us know if any updates about the problem.

Regards,
Andrii Nikitin

Actions #3

Updated by andriinikitin 12 months ago

I forgot to add that I checked the folders and haven't found the referenced file neither locally nor using rsync interface.

Actions #4

Updated by otto@informatik.rwth-aachen.de 12 months ago

Hi Andrii,

On Tue, Apr 18, 2023 at 08:47:01AM +0000, redmine@opensuse.org wrote:

To me it looks that these errors happen because of race condition when
the files are being updated on the server at the same moment while
your rsync command is running.

That's not the case, assuming your updates don't take 4 days to
complete. I saw errors related to the stuck file from April 7th (9:42
CEST) to April 11th (15:46).

Please consider this as a proper bug report, not just a temporary fluke.
Please also consider my comments in #108494. As said before, this is an
ongoing issue, and I'd like to see lasting fixes, as mentioned in said
ticket (from 2022).

But you can also try syncing from stage-main-repos.opensuse.org as was
mentioned recently in the list announcement and is addressed in the
wiki - maybe it will bring better experience (it doesn't require
authentication and should provide recent files with better speed and
less contention on the main server):

That's worth a try, thanks for the hint!

Best regards,

Carsten

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

Actions #5

Updated by andriinikitin 12 months ago

Hi Carset,

otto@informatik.rwth-aachen.de wrote:

That's not the case, assuming your updates don't take 4 days to
complete. I saw errors related to the stuck file from April 7th (9:42
CEST) to April 11th (15:46).

Was it always the same file or always different file? It could be some leftover from a dead process, but I cannot find any trace now.

In case if you see a stuck file again - please try to reproduce the problem with command like below, so it shows that the file is there for significant amount of time:

rsync --list-only --no-motd rsync://stage.opensuse.org/opensuse-full-with-factory-dvd5/opensuse/update/leap/15.3/sle/src/.rust*

If the issue happens again - I will try to tweak some options on server side.
But still switching to stage-main-repos.opensuse.org (aka stage3.opensuse.org ) is preferred, because the main machine stage.opensuse.org is currently very overloaded at maximum capacity.

Regards,
Andrii Nikitin

Actions #6

Updated by otto@informatik.rwth-aachen.de 11 months ago

Hi,

On Wed, Apr 19, 2023 at 07:43:26AM +0000, redmine@opensuse.org wrote:

Was it always the same file or always different file? It could be some
leftover from a dead process, but I cannot find any trace now.

Always the same file.

In case if you see a stuck file again - please try to reproduce the
problem with command like below, so it shows that the file is there
for significant amount of time:

I reproduced the issue using our regular rsync runs, what's the
advantage of the command you sent?

If the issue happens again - I will try to tweak some options on server side.

I already reported this issue as early as 2022, possibly before. Please
just fix it, there's no need for it to happen again.

But still switching to stage-main-repos.opensuse.org (aka
stage3.opensuse.org ) is preferred, because the main machine
stage.opensuse.org is currently very overloaded at maximum capacity.

I already did that.

Bye,

Carsten

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

Actions #7

Updated by otto@informatik.rwth-aachen.de 11 months ago

This is still an issue.

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

Actions #8

Updated by andriinikitin 11 months ago

otto@informatik.rwth-aachen.de wrote:

This is still an issue.

I am having hard time reproducing the issue, so I will appreciate if you send output of command below for exact path that is giving the problem, as was requested earlier:

rsync --list-only --no-motd rsync://stage.opensuse.org/opensuse-full-with-factory-dvd5/opensuse/%path_here%
Actions #9

Updated by otto@informatik.rwth-aachen.de 11 months ago

Hi,

the issue doesn't persist, but it pops up often enough to be annoying.
To reproduce this, please either monitor the system for a longer time
period, or check any logs you might have. It might help to run
"ls -l /some/where/opensuse/miniuprepo*" every minute or so to see when
these files pop up (assuming you're able to figure out the process that
creates them).

The repository is named "opensuse-full-with-factory-dvd5" and the full
path within this repository is "/opensuse/miniuprepoXXXXX" (with random
letters in place of the Xs).

Some examples from recent log files:

"/opensuse/miniuprepoPOtZ0"
"/opensuse/miniuprepoPaYrk"
"/opensuse/miniuprepoPEmdX"
"/opensuse/miniuprepoBrUbp"
"/opensuse/miniuprepoOwJ4R"
"/opensuse/miniuprepo0vBsY"

Please make sure that the rsync server only exposes files that should be
downloaded from mirrors. In this case I believe is temporary/internal
and, as such, should not be exposed via rsync, as this causes issues.

The most recent issue occured today at 15:30 CEST
("/opensuse/miniuprepoPOtZ0").

Bye

Carsten

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

Actions #10

Updated by andriinikitin 10 months ago

  • Status changed from Feedback to Workable
Actions #11

Updated by andriinikitin 10 months ago

  • Status changed from Workable to Feedback

Hi Carsten,

otto@informatik.rwth-aachen.de wrote:

The repository is named "opensuse-full-with-factory-dvd5" and the full
path within this repository is "/opensuse/miniuprepoXXXXX" (with random
letters in place of the Xs).

I've got confirmation that issues with miniuprepo indeed happened on stage3.opensuse.org and were solved around two weeks ago.

Did you switch to stage3.opensuse.org? Are the other errors still observed?

Regards,
Andrii Nikitin

Actions #12

Updated by otto@informatik.rwth-aachen.de 10 months ago

Hi,

On Wed, Jun 14, 2023 at 11:42:43AM +0000, redmine@opensuse.org wrote:

I've got confirmation that issues with miniuprepo indeed happened on
stage3.opensuse.org and were solved around two weeks ago.

I raised my issue around two months ago, I don't think this is related to
anything happening recently.

Did you switch to stage3.opensuse.org? Are the other errors still observed?

We're using stage-main-repos.opensuse.org. I haven't seen any recent
issue.

Aside from that, could you please make sure that permissions on
your server reflect the intention of files being downloadable? This
isn't the first time I've seen this issue, and suse obviously isn't able
to respond in a timely many to fix individual instances of the issue
(which is totally fine, I don't pay for support).

I just realized that I already asked you to do this, possibly more than
once. Could you please comment on this, so that I don't have to repeat
myself over and over again?


Please consider this as a proper bug report, not just a temporary fluke.
Please also consider my comments in #108494. As said before, this is an
ongoing issue, and I'd like to see lasting fixes, as mentioned in said

ticket (from 2022).

Thanks,

Carsten

Dr. Carsten Otto
http://verify.rwth-aachen.de/otto/

Actions

Also available in: Atom PDF