tickets #53267
closed
Added by bernhardout@lsmod.de almost 5 years ago.
Updated over 2 years ago.
Description
Hi,
I added an experimental public tumbleweed+history mirror in IPFS
Admin Name: Bernhard M. Wiedemann
Admin Email: opensusemirror@zq1.de
Subscribed to mirror@opensuse.org: YES
Sponsor Name: Bernhard M. Wiedemann
Sponsor URL:
https://lizards.opensuse.org/2019/04/03/experimental-opensuse-mirror-via-ipfs/
RSYNC URL: rsync://hotel.zq1.de./opensuse/
RSYNC allowed only for scans: YES
HTTP URL: https://opensuse.zq1.de./
FTP URL: NA
Limitations: worldwide, weight 1
IPs and DNS for the whitelist on stage.opensuse.org: hotel.zq1.de
IPv4=176.9.1.211 IPv6=2a01:4f8:141:31c2::2
Details on the mirror can be found at
https://lizards.opensuse.org/2019/04/03/experimental-opensuse-mirror-via-ipfs/
Great care is taken to only provide complete and consistent Tumbleweed
snapshots. Another specialty of this mirror is that it keeps a long tail
of entries in /history/
Unfortunately, due to the distributed nature of the setup
there could be ~5 minutes after a new snapshot where either the old or
the new files may be served - not reflecting what rsync reports.
That is why I propose to give it a very low weight in the
mirrorbrain config and see how it works out.
Or even better: do not add it to mirrorbrain.
It is recommended that users only use this http mirror via
download.opensuse.org redirections or via IPFS directly.
- Category set to Mirrors
- Assignee set to pjessen
- Private changed from Yes to No
update:
HTTP URL: https://opensuse.zq1.de/
(does not work with trailing dot because that is not included in the SSL cert)
- Status changed from New to In Progress
I guess I completely ignored your request, sorry.
Mirror created:
# mb show zq
identifier : opensuse.zq1.de
operatorName : Bernhard M. Wiedemann
operatorUrl :
baseurl : http://opensuse.zq1.de/
baseurlFtp :
baseurlRsync : rsync://hotel.zq1.de/opensuse/
region : na
country : us
asn : 13335
prefix : 104.16.0.0/12
lat,lng : 37.751,-97.822
regionOnly : False
countryOnly : False
asOnly : False
prefixOnly : False
ipv6Only : False
otherCountries :
fileMaxsize : 0
publicNotes :
score : 1
enabled : False
statusBaseurl : False
admin : Bernhard M. Wiedemann
adminEmail : opensusemirror@zq1.de
I have also added "hotel.zq1.de" to the stage ACL.
- Status changed from In Progress to Feedback
The scan completed, but it wasn't really happy with the rsync nor the http reply:
# mb scan -e zq
Wed Oct 2 08:06:09 2019 opensuse.zq1.de: starting
^[[BWed Oct 2 08:09:10 2019 opensuse.zq1.de: total files before scan: 0
__DIE__: (/usr/bin/scanner 1175 main::sread => /usr/bin/scanner 1144 (eval) => /usr/bin/scanner 1144 main::__ANON__)
__DIE__: (/usr/bin/scanner 1121 main::rsync_get_filelist => /usr/bin/scanner 1283 main::muxread => /usr/bin/scanner 1175 main::sread)
Wed Oct 2 08:19:10 2019 opensuse.zq1.de: no rsync, no ftp, trying http
opensuse.zq1.de: http_readdir: url=http://opensuse.zq1.de/ re=
opensuse.zq1.de: http dir: http://opensuse.zq1.de/
opensuse.zq1.de: unparseable HTML index in /
$VAR1 = '<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<style>
html{font-family:sans-serif;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a{background-color:transparent}a:active,a:hover{outline:0}strong{font-weight:700}button::-moz-focus-inner,input::-moz-focus-inner{padding:0;border:0}table{border-spacing:0;border-collapse:collapse}td{padding:0} @media print{*,:after,:before{color:#000!important;text-shadow:none!important;background:0 0!important;-webkit-box-shad';
$VAR2 = 'opensuse.zq1.de: http_readdir: unknown HTML format';
Wed Oct 2 08:19:11 2019 opensuse.zq1.de: scanned 0 files (0/s) in 601s
Wed Oct 2 08:19:11 2019 opensuse.zq1.de: purging old files
Wed Oct 2 08:19:11 2019 opensuse.zq1.de: files to be purged: 0
Wed Oct 2 08:22:06 2019 opensuse.zq1.de: total files after scan: 0 (delta: 0)
Wed Oct 2 08:22:06 2019 opensuse.zq1.de: purged old files in 175s.
Wed Oct 2 08:22:06 2019 opensuse.zq1.de: done.
Wed Oct 2 08:22:06 2019 opensuse.zq1.de: testing status of base URL...
Wed Oct 2 08:22:06 2019 opensuse.zq1.de: OK. Mirror is online now.
Completed in 16.0 minutes
- Status changed from Feedback to In Progress
- % Done changed from 0 to 50
I can imagine that it gets confused by the custom HTTP/HTML listings, but rsync has plain files. Maybe mb does not like the 0-sized dirs or adjusted mtimes?
Or did I get the IP whitelist in rsyncd.conf wrong?
hosts allow = 195.135.220.0/22 2001:067c:2178::/48 2620:113:80c0:5::/40
Worst case we leave it out of mirrorbrain. It is already serving multiple GB per day over IPFS.
- % Done changed from 50 to 0
bmwiedemann wrote:
I can imagine that it gets confused by the custom HTTP/HTML listings, but rsync has plain files. Maybe mb does not like the 0-sized dirs or adjusted mtimes?
Yes, mb couldn't parse the custom html properly. I don't know what went wrong with the rsync.
Or did I get the IP whitelist in rsyncd.conf wrong?
hosts allow = 195.135.220.0/22 2001:067c:2178::/48 2620:113:80c0:5::/40
Looks good, although I dont think "2001:067c:2178::/48" is needed anymore.
Worst case we leave it out of mirrorbrain. It is already serving multiple GB per day over IPFS.
If it doesn't work, it will eventually be disabled. It's currently listed under US mirrors.
- Status changed from In Progress to Feedback
bmwiedemann wrote:
but rsync has plain files. Maybe mb does not like the 0-sized dirs or adjusted mtimes?
The rsync problem is a timeout. I tried "rsync -avn rsync://hotel.zq1.de/opensuse ." from the command line, it ran for more than 3 hours, but it never finished.
rsync is now just being refused:
@ERROR: access denied to opensuse from UNKNOWN (2a03:7520:4c68:1:221:86ff:fe4f:8ac4)
rsync error: error starting client-server protocol (code 5) at main.c(1395) [receiver=2.6.9]
This rsync worked:
# rsync --list-only rsync://hotel.zq1.de/opensuse
dr-xr-xr-x 0 2020/01/08 13:35:43 .
dr-xr-xr-x 0 2020/01/08 13:35:43 history
dr-xr-xr-x 0 2020/01/08 13:35:43 tumbleweed
Currently, we are not seeing any content. I see that you have history (15TB) and tumbleweed (83Gb), but I wonder if the scanner recognises the listing format.
Getting a directory listing with rsync never seems to finish:
rsync --list-only rsync://hotel.zq1.de/opensuse/tumbleweed/repo//oss/x86_64/
It also seems to just hang with http:
http://opensuse.zq1.de/tumbleweed/repo/oss/x86_64
The http request ended with
Error 524 Ray ID: 55892b2c53b9cc4a • 2020-01-21 12:09:14 UTC
A timeout occurred
I restricted rsync to the IPs on the mirror setup page. Added 2a03:7520:4c00::/40 to allowed IPs now.
This timeout is from the distributed nature of the mirror - if you can skip rsync, that would be OK.
Listings of large dirs always time out, but files are there and zypper only uses repodata to find files. This mirror only ever gets updated atomically with full snapshots. So in theory, mirrorbrain only would have to fetch https://opensuse.zq1.de/tumbleweed/repo/oss/media.1/products and/or https://opensuse.zq1.de/history/list
- Status changed from Feedback to Closed
- % Done changed from 0 to 100
Hi there - and a Happy and Healthy 2022!
I'm currently closing old tickets which did not see much change.
If the main concern still exists and should be handled, please re-open by just replying to this Email.
Thanks in advance,
Lars
Also available in: Atom
PDF