tickets #102479
closedAARNet mirror - rsync blocked
100%
Description
Hi OpenSUSE team, I hope you're having a great week!
I'm just getting in touch regarding an issue with our mirror. Our rsync connections seem to be being blocked on sync operations since 2021-11-13T14:07:43Z (the timestamp of the first failed sync)
Our last successful sync completed at 2021-11-12T16:16:02Z
Nothing has changed on our end since the last successful sync as far as I'm aware, are you able to let us know if there's any additional info you need from us in order to help resolve this issue?
Regards,
Blake Peden
Systems Administrator
[cid:image001.jpg@01D7DAD3.6BD49ED0]
M: +61 413 019 220 P: +61 2 9779 6941 E: blake.peden@aarnet.edu.aublake.peden@aarnet.edu.au
zoom: https://aarnet.zoom.us/my/blake.peden
Level 7, Tower A, 799 Pacific Highway, Chatswood NSW 2067
Files
Updated by cboltz over 2 years ago
- Category set to Mirrors
- Assignee set to pjessen
Updated by pjessen over 2 years ago
- Status changed from New to In Progress
- Private changed from Yes to No
Hi Blake,
I haven't touched any of that setup for quite some time - I see there was a change in #97475 but that was in August.
'mirror.aarnet.edu.au' is on the ACL, I've checked, and the reverse mappings are good too.
However, according to the rsync logs, you were syncing from act-dksn-mirror1.aarnet.net.au [2001:388:8008:10::8] until 2021/11/14 22:21:41.
Updated by Blake.Peden@aarnet.edu.au over 2 years ago
Hi pjessen
Sorry for the delay in responding, and thanks for the information
I was just finishing up proofreading my response with the troubleshooting data I've been gathering over the past few days, but it seems the sync magically started working on the 3rd retry for the morning about 20 mins ago.
I'm unsure what triggered this change in behaviour as we haven't made any changes at all since my initial message, but hopefully, that's the end of it!
Regards,
Blake Peden
Systems Administrator
Updated by Blake.Peden@aarnet.edu.au over 2 years ago
Sorry pjessen
Unfortunately, that wasn't the end of it as I'd hoped. We got partway through a sync yesterday, but the connection was terminated:
rsync: read error: Connection reset by peer (104)
Subsequent sync attempts have the following:
This is stage.opensuse.org, restricted rsync server of openSUSE.org.
If you run a public mirror, please get in contact so we can give you
access to the this rsync server.
You'll find conditions for access and further information at
http://en.opensuse.org/Mirror_Infrastructure
Thanks!
admin@opensuse.org
@ERROR: access denied to opensuse-full from UNKNOWN (2001:388:8008:10::8)
rsync error: error starting client-server protocol (code 5) at main.c(1661) [Receiver=3.1.3]
Network connection failure, retrying in 120 seconds...¶
I've confirmed 2001:388:8008:10::8 is listed in my colleague David's initial request (https://progress.opensuse.org/issues/97475)
The exact timestamp that we received this error at is 2021-11-23T23:54:21.033Z
The sync operation was started about 20 seconds before that.
I've also tried forcing an ipv4 sync from 138.44.164.198 and I get the same result.
Regards,
Blake Peden
Updated by pjessen over 2 years ago
- Status changed from In Progress to Feedback
Blake.Peden@aarnet.edu.au wrote:
@ERROR: access denied to opensuse-full from UNKNOWN (2001:388:8008:10::8)
That suggests lack of a reverse map, but looking at it now, it looks fine:
# host vic-pmla-mirror1.aarnet.net.au
vic-pmla-mirror1.aarnet.net.au has address 138.44.71.134
vic-pmla-mirror1.aarnet.net.au has IPv6 address 2001:388:8008:11::8
# host 2001:388:8008:11::8
8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.0.8.0.0.8.8.8.3.0.1.0.0.2.ip6.arpa domain name pointer vic-pmla-mirror1.aarnet.net.au.
# host 138.44.71.134
134.71.44.138.in-addr.arpa domain name pointer vic-pmla-mirror1.aarnet.net.au.
# host act-dksn-mirror1.aarnet.net.au.
act-dksn-mirror1.aarnet.net.au has address 138.44.164.198
act-dksn-mirror1.aarnet.net.au has IPv6 address 2001:388:8008:10::8
pontifex2 (download.o.o):~ # host 138.44.164.198
198.164.44.138.in-addr.arpa domain name pointer act-dksn-mirror1.aarnet.net.au.
pontifex2 (download.o.o):~ # host 2001:388:8008:10::8
8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.8.0.0.8.8.8.3.0.1.0.0.2.ip6.arpa domain name pointer act-dksn-mirror1.aarnet.net.au.
Also, from today's log:
# grep aarnet /var/log/rsyncd-stage.log
2021/11/29 04:24:23 [28733] connect from act-dksn-mirror1.aarnet.net.au (138.44.164.198)
2021/11/29 04:24:33 [28733] rsync on opensuse-full/opensuse/ from act-dksn-mirror1.aarnet.net.au (138.44.164.198)
2021/11/29 07:50:17 [9519] connect from act-dksn-mirror1.aarnet.net.au (138.44.164.198)
2021/11/29 07:50:27 [9519] rsync on opensuse-full/opensuse/ from act-dksn-mirror1.aarnet.net.au (138.44.164.198)
All in all, everything seems to be working, seen from my end.
Updated by pjessen over 2 years ago
Looking at the logs going back to 1 November, I see regular accesses from act-dksn-mirror1.aarnet.net.au (2001:388:8008:10::8), but with three failed lookups, on 10 Nov, 23 Nov and 24 Nov.
Updated by Blake.Peden@aarnet.edu.au over 2 years ago
Hi openSUSE team,
We seem to have found a combination of settings on our end that alleviate the issue. Sync has been stable for a good amount of time now.
Thanks very much for your help!
Blake.
Updated by pjessen over 2 years ago
- Status changed from Feedback to Resolved
- % Done changed from 0 to 100
That's great, thanks for the feedback.