Project

General

Profile

tickets #100805

provo-mirror not synchronizing ?

Added by olaf@aepfle.de about 1 year ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Mirrors
Target version:
-
Start date:
2021-10-11
Due date:
2022-03-18
% Done:

100%

Estimated time:

Description

At least https://provo-mirror.opensuse.org//history/latest was no updated since a few weeks.

Perhaps there could be some monitoring of this specific file remains uptodate with something like

foreach mirror:
curl --head --location $mirror/history/latest | compare-with-Origin "last-modified:" || notify-admin $mirror

Also https://mirrors.opensuse.org/list/tumbleweed.html lists bogus entries for provo-mirror. Operator looks bogus, rsync://provo-downloadcontent.opensuse.org/repositories-scan does not seem to exist.

Olaf


Related issues

Related to openSUSE admin - tickets #107971: outdated arm64 snapshot on provo-mirror.opensuse.orgResolved2022-03-08

Related to openSUSE admin - tickets #108524: provo-mirror.opensuse.org rsync issueClosed2022-03-17

History

#1 Updated by pjessen about 1 year ago

  • Category set to Mirrors
  • Private changed from Yes to No

olaf@aepfle.de wrote:

At least https://provo-mirror.opensuse.org//history/latest was not updated since a few weeks.

Confirmed, it does not seem to have been updated for a while.

Perhaps there could be some monitoring of this specific file remains

We scan all mirrors continually, so something is not quite right with the provo mirror.

Also https://mirrors.opensuse.org/list/tumbleweed.html lists bogus entries for provo-mirror. Operator looks bogus, rsync://provo-downloadcontent.opensuse.org/repositories-scan does not seem to exist.

It works fine here, but it is for internal use only:

# rsync --list-only rsync://provo-downloadcontent.opensuse.org/repositories-scan
drwxr-xr-x            188 2018/09/06 19:22:01 .
lrwxrwxrwx             10 2017/09/06 13:23:06 factory
drwxr-xr-x             74 2017/09/11 08:40:31 debug
drwxr-xr-x             81 2021/06/02 09:45:42 distribution
drwxr-xr-x          4,096 2021/08/26 03:00:01 history
drwxr-xr-x            154 2018/05/25 18:15:30 ports
drwxr-xr-x          4,096 2021/10/13 09:21:13 repositories
drwxr-xr-x             56 2018/11/06 10:34:42 source
drwxr-xr-x             94 2020/04/06 17:54:29 tumbleweed
drwxr-xr-x            169 2021/06/02 09:46:18 update

#2 Updated by pjessen about 1 year ago

It looks like one of the Heroes is logged in on provo-mirror, running a manual rsync. I'm guessing either Lars or Christian. I won't have any time for this until Monday, I'm sure it'll be sorted out by then.

#3 Updated by pjessen about 1 year ago

  • Subject changed from stale entries in /history/latest.mirrorlist to provo-mirror not synchronizing ?
  • Status changed from New to Workable

#4 Updated by pjessen 9 months ago

  • Related to tickets #107971: outdated arm64 snapshot on provo-mirror.opensuse.org added

#5 Updated by pjessen 9 months ago

  • Status changed from Workable to In Progress
  • Assignee set to pjessen

Well, it didn't sort itself out :-(
Afaict, the sync jobs are running normally, except for maybe the weekly sync-all job. According to the log /home/mirror/log/sync_opensuse.log, since 30 December, it has been running for less than one second, which suggests a left-over lockfile. provo-mirror was rebooted about 12 hours ago and I'm just now running a manual /home/mirror/bin/sync_opensuse.sh.

#6 Updated by pjessen 9 months ago

My theory:
when rsync hits some sort of a problem, e.g.

Starting: Thu Dec 23 03:05:01 UTC 2021
rsync: [generator] write error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(829) [generator=3.1.3]rsync: connection unexpectedly closed (83614955428 bytes received so far) [receiver]
rsync error: received SIGUSR1 (code 19) at main.c(1455) [receiver=3.1.3]
Finished: Thu Dec 23 09:13:03 UTC 2021
Starting: Thu Dec 30 03:05:01 UTC 2021
Finished: Thu Dec 30 03:05:01 UTC 2021
Starting: Thu Jan  6 03:05:01 UTC 2022
Finished: Thu Jan  6 03:05:01 UTC 2022
Starting: Thu Jan 13 03:05:01 UTC 2022
Finished: Thu Jan 13 03:05:01 UTC 2022
Starting: Thu Jan 20 03:05:01 UTC 2022
Finished: Thu Jan 20 03:05:01 UTC 2022
Starting: Thu Jan 27 03:05:01 UTC 2022
Finished: Thu Jan 27 03:05:01 UTC 2022
Starting: Thu Feb  3 03:05:01 UTC 2022
Finished: Thu Feb  3 03:05:01 UTC 2022
Starting: Thu Feb 10 03:05:02 UTC 2022
Finished: Thu Feb 10 03:05:02 UTC 2022
Starting: Thu Feb 17 03:05:01 UTC 2022
Finished: Thu Feb 17 03:05:01 UTC 2022
Starting: Thu Feb 24 03:05:01 UTC 2022
Finished: Thu Feb 24 03:05:01 UTC 2022
Starting: Thu Mar  3 03:05:02 UTC 2022
Finished: Thu Mar  3 03:05:02 UTC 2022

the lockfile is not removed? According to 'man withlock':

It uses locks that are valid only while the wrapper is running, and thus will never require additional cleanup

It seems to me that is simply wrong.

#7 Updated by pjessen 9 months ago

  • Due date set to 2022-03-18
  • % Done changed from 0 to 50

Well, provo-mirror is now syncing again, albeit slowly. I'll take a look at it again next Friday.

#8 Updated by pjessen 9 months ago

I have enabled the four times a day rsync of ports.

#9 Updated by pjessen 9 months ago

#10 Updated by pjessen 4 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

I believe we have finally managed to create a mostly stable synchronization scheme on provo-mirror. There is still room for improvement, but I don't think that is a topic for this ticket. I might open a separate one to track what I think needs doing.

Also available in: Atom PDF