tickets #100805
closedprovo-mirror not synchronizing ?
100%
Description
At least https://provo-mirror.opensuse.org//history/latest was no updated since a few weeks.
Perhaps there could be some monitoring of this specific file remains uptodate with something like
foreach mirror:
curl --head --location $mirror/history/latest | compare-with-Origin "last-modified:" || notify-admin $mirror
Also https://mirrors.opensuse.org/list/tumbleweed.html lists bogus entries for provo-mirror. Operator looks bogus, rsync://provo-downloadcontent.opensuse.org/repositories-scan does not seem to exist.
Olaf
Updated by pjessen over 2 years ago
- Category set to Mirrors
- Private changed from Yes to No
olaf@aepfle.de wrote:
At least https://provo-mirror.opensuse.org//history/latest was not updated since a few weeks.
Confirmed, it does not seem to have been updated for a while.
Perhaps there could be some monitoring of this specific file remains
We scan all mirrors continually, so something is not quite right with the provo mirror.
Also https://mirrors.opensuse.org/list/tumbleweed.html lists bogus entries for provo-mirror. Operator looks bogus, rsync://provo-downloadcontent.opensuse.org/repositories-scan does not seem to exist.
It works fine here, but it is for internal use only:
# rsync --list-only rsync://provo-downloadcontent.opensuse.org/repositories-scan
drwxr-xr-x 188 2018/09/06 19:22:01 .
lrwxrwxrwx 10 2017/09/06 13:23:06 factory
drwxr-xr-x 74 2017/09/11 08:40:31 debug
drwxr-xr-x 81 2021/06/02 09:45:42 distribution
drwxr-xr-x 4,096 2021/08/26 03:00:01 history
drwxr-xr-x 154 2018/05/25 18:15:30 ports
drwxr-xr-x 4,096 2021/10/13 09:21:13 repositories
drwxr-xr-x 56 2018/11/06 10:34:42 source
drwxr-xr-x 94 2020/04/06 17:54:29 tumbleweed
drwxr-xr-x 169 2021/06/02 09:46:18 update
Updated by pjessen over 2 years ago
It looks like one of the Heroes is logged in on provo-mirror, running a manual rsync. I'm guessing either Lars or Christian. I won't have any time for this until Monday, I'm sure it'll be sorted out by then.
Updated by pjessen over 2 years ago
- Subject changed from stale entries in /history/latest.mirrorlist to provo-mirror not synchronizing ?
- Status changed from New to Workable
Updated by pjessen about 2 years ago
- Related to tickets #107971: outdated arm64 snapshot on provo-mirror.opensuse.org added
Updated by pjessen about 2 years ago
- Status changed from Workable to In Progress
- Assignee set to pjessen
Well, it didn't sort itself out :-(
Afaict, the sync jobs are running normally, except for maybe the weekly sync-all job. According to the log /home/mirror/log/sync_opensuse.log, since 30 December, it has been running for less than one second, which suggests a left-over lockfile. provo-mirror was rebooted about 12 hours ago and I'm just now running a manual /home/mirror/bin/sync_opensuse.sh.
Updated by pjessen about 2 years ago
My theory:
when rsync hits some sort of a problem, e.g.
Starting: Thu Dec 23 03:05:01 UTC 2021
rsync: [generator] write error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(829) [generator=3.1.3]rsync: connection unexpectedly closed (83614955428 bytes received so far) [receiver]
rsync error: received SIGUSR1 (code 19) at main.c(1455) [receiver=3.1.3]
Finished: Thu Dec 23 09:13:03 UTC 2021
Starting: Thu Dec 30 03:05:01 UTC 2021
Finished: Thu Dec 30 03:05:01 UTC 2021
Starting: Thu Jan 6 03:05:01 UTC 2022
Finished: Thu Jan 6 03:05:01 UTC 2022
Starting: Thu Jan 13 03:05:01 UTC 2022
Finished: Thu Jan 13 03:05:01 UTC 2022
Starting: Thu Jan 20 03:05:01 UTC 2022
Finished: Thu Jan 20 03:05:01 UTC 2022
Starting: Thu Jan 27 03:05:01 UTC 2022
Finished: Thu Jan 27 03:05:01 UTC 2022
Starting: Thu Feb 3 03:05:01 UTC 2022
Finished: Thu Feb 3 03:05:01 UTC 2022
Starting: Thu Feb 10 03:05:02 UTC 2022
Finished: Thu Feb 10 03:05:02 UTC 2022
Starting: Thu Feb 17 03:05:01 UTC 2022
Finished: Thu Feb 17 03:05:01 UTC 2022
Starting: Thu Feb 24 03:05:01 UTC 2022
Finished: Thu Feb 24 03:05:01 UTC 2022
Starting: Thu Mar 3 03:05:02 UTC 2022
Finished: Thu Mar 3 03:05:02 UTC 2022
the lockfile is not removed? According to 'man withlock':
It uses locks that are valid only while the wrapper is running, and thus will never require additional cleanup
It seems to me that is simply wrong.
Updated by pjessen about 2 years ago
- Due date set to 2022-03-18
- % Done changed from 0 to 50
Well, provo-mirror is now syncing again, albeit slowly. I'll take a look at it again next Friday.
Updated by pjessen about 2 years ago
I have enabled the four times a day rsync of ports.
Updated by pjessen about 2 years ago
- Related to tickets #108524: provo-mirror.opensuse.org rsync issue added
Updated by pjessen over 1 year ago
- Status changed from In Progress to Resolved
- % Done changed from 50 to 100
I believe we have finally managed to create a mostly stable synchronization scheme on provo-mirror. There is still room for improvement, but I don't think that is a topic for this ticket. I might open a separate one to track what I think needs doing.