Project

General

Profile

tickets #92930

New mirror - download.opensuse.net.br

Added by avicenzi over 1 year ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Mirrors
Target version:
-
Start date:
2021-05-21
Due date:
% Done:

100%

Estimated time:

Description

Hi,

I started a new mirror, it's still syncing up data from rsync.opensuse.org, but I would like to sync from stage servers.

The mirror will be maintained by openSUSE Brasil volunteers, but the VPS is sponsored by Binario Cloud.

By my calculations, it will be possible to have the repos for 15.2, 15.3, and TW, but the current sync is ignoring some archs such as ppc, i586, i686, and s390x.
I can change that later if we have free space, but this mirror will not have Factory and History data.

Not sure if required, but we use Cloudflare and I believe that CF Proxy must be disabled to work, which exposes our IP. If not needed, I would like to enable this feature.
I also set reverse IP lookup and a PTR record, if not needed let me know also.

All the configurations will be available at https://github.com/alexandrevicenzi/opensuse-mirror-docker for others to use, perhaps this could be moved to openSUSE org in the future.

Admin Name: Alexandre Vicenzi, Erico Mendonca, Igor Ferreira
Admin Email: contato@opensuse.net.br
Subscribed to mirror@opensuse.org: YES
Sponsor Name: Binario Cloud (VPS) / openSUSE Brasil (maintainers)
Sponsor URL: https://binario.cloud/
RSYNC URL: rsync.opensuse.net.br
RSYNC allowed only for scans: YES (from 195.135.220.0/22)
HTTP URL: http://download.opensuse.net.br/ and https://download.opensuse.net.br/
FTP URL: Not available
IPs and DNS for the whitelist on stage.opensuse.org: 45.225.27.127 and rsync.opensuse.net.br
Limitations:

  • 100 Mbit network, can be increased in the future
  • No data cap, but would be nice to limit to 10 TB transfer initially and limit simultaneous connections to a reasonable number (we limit 20 by IP)
  • 1 TB disc size
  • rsync only allows openSUSE servers
  • The server should also block access from Asia and Africa continents
  • The server has also some IP rules/limits to avoid DDoS
  • The mirror is deployed using Docker

History

#1 Updated by cboltz over 1 year ago

  • Category set to Mirrors
  • Assignee set to pjessen

#2 Updated by pjessen over 1 year ago

  • Subject changed from New mirror access to New mirror - download.opensuse.net.br
  • Private changed from Yes to No

#3 Updated by pjessen over 1 year ago

  • Status changed from New to In Progress

avicenzi wrote:

Not sure if required, but we use Cloudflare and I believe that CF Proxy must be disabled to work, which exposes our IP. If not needed, I would like to enable this feature.

I don't know if this has any significance for us?

I also set reverse IP lookup and a PTR record, if not needed let me know also.

It is needed, and rsync.opensuse.net.br looks good. Stage ACL updated.

  • 100 Mbit network, can be increased in the future

We can set up a lower priority for this mirror? Scale is 0-100%.

  • No data cap, but would be nice to limit to 10 TB transfer initially and limit simultaneous connections to a reasonable number (we limit 20 by IP)

We don't have any control over that - tweaking the priority might indirectly help, but that's all we can do.

  • The server should also block access from Asia and Africa continents

We have the following options:

regionOnly - only South-America
countryOnly - only Brazil.

I don't have a way of specifying 'not something' - blocking Asia and Africa is probably best achieved by using regiononly = sa, and then adding other countries permitted with 'otherCountries'. (just a list of country codes).

Wrt the mirror URL 'download.opensuse.net.br' - it currently resolves like this:

host download.opensuse.net.br
download.opensuse.net.br has address 104.21.72.251
download.opensuse.net.br has address 172.67.155.239
download.opensuse.net.br has IPv6 address 2606:4700:3036::6815:48fb
download.opensuse.net.br has IPv6 address 2606:4700:3034::ac43:9bef

I guess this is Cloudflare? I have edited the mirror setup manually to say 'sa' and 'br'.

# mb show net.br
identifier     : download.opensuse.net.br
operatorName   : Binario Cloud / openSUSE Brasil
operatorUrl    : https://binario.cloud/
baseurl        : http://download.opensuse.net.br
baseurlFtp     : 
baseurlRsync   : 
region         : sa
country        : br
asn            : 13335
prefix         : 172.67.144.0/20
lat,lng        : 37.751,-97.822
regionOnly     : True
countryOnly    : False
asOnly         : False
prefixOnly     : False
ipv6Only       : False
otherCountries : 
fileMaxsize    : 0
publicNotes    : 
score          : 100
enabled        : False
statusBaseurl  : False
admin          : mirror admin
adminEmail     : contato@opensuse.net.br

Note to self - have edited asn.py again, see #62678

#4 Updated by pjessen over 1 year ago

The initial scan fails due to the listing format:

# mb scan download.opensuse.net.br
Sun May 23 09:20:07 2021 download.opensuse.net.br: starting
Sun May 23 09:21:06 2021 download.opensuse.net.br: total files before scan: 0
download.opensuse.net.br: unparseable HTML index in /
Sun May 23 09:21:06 2021 download.opensuse.net.br: scanned 0 files (0/s) in 0s
Sun May 23 09:21:06 2021 download.opensuse.net.br: files to be purged: 0
Sun May 23 09:21:35 2021 download.opensuse.net.br: total files after scan: 0 (delta: 0)
Sun May 23 09:21:35 2021 download.opensuse.net.br: purged old files in 29s.
Sun May 23 09:21:35 2021 download.opensuse.net.br: done.
Completed in 1.5 minutes

Just using a plain apache or nginx autoindex format would help.

#5 Updated by pjessen over 1 year ago

  • Status changed from In Progress to Feedback

#6 Updated by pjessen over 1 year ago

  • Status changed from Feedback to In Progress

I have added the rsync url, which enables the scan, but I have to look into how to have an rsync url that is not publicly listed.

#7 Updated by avicenzi over 1 year ago

pjessen wrote:

avicenzi wrote:

Not sure if required, but we use Cloudflare and I believe that CF Proxy must be disabled to work, which exposes our IP. If not needed, I would like to enable this feature.

I don't know if this has any significance for us?

Yes, it masks the real IP.

The real IP for download.opensuse.net.br is 45.225.27.127, the same as rsync.opensuse.net.br.

I also set reverse IP lookup and a PTR record, if not needed let me know also.

It is needed, and rsync.opensuse.net.br looks good. Stage ACL updated.

OK

  • 100 Mbit network, can be increased in the future

We can set up a lower priority for this mirror? Scale is 0-100%.

No, leave it at 100%, if the server goes down due load we can review it.

  • No data cap, but would be nice to limit to 10 TB transfer initially and limit simultaneous connections to a reasonable number (we limit 20 by IP)

We don't have any control over that - tweaking the priority might indirectly help, but that's all we can do.

Not needed, just nice to have, let's leave it open and I'll check how much it will consume.

  • The server should also block access from Asia and Africa continents

We have the following options:

regionOnly - only South-America
countryOnly - only Brazil.

I don't have a way of specifying 'not something' - blocking Asia and Africa is probably best achieved by using regiononly = sa, and then adding other countries permitted with 'otherCountries'. (just a list of country codes).

We have some rules on our side, South America is good.

Wrt the mirror URL 'download.opensuse.net.br' - it currently resolves like this:

host download.opensuse.net.br
download.opensuse.net.br has address 104.21.72.251
download.opensuse.net.br has address 172.67.155.239
download.opensuse.net.br has IPv6 address 2606:4700:3036::6815:48fb
download.opensuse.net.br has IPv6 address 2606:4700:3034::ac43:9bef

I guess this is Cloudflare? I have edited the mirror setup manually to say 'sa' and 'br'.

Yes, we have no IPv6, the real IP is the same as rsync, but I can disable CF Proxy

# mb show net.br
identifier     : download.opensuse.net.br
operatorName   : Binario Cloud / openSUSE Brasil
operatorUrl    : https://binario.cloud/
baseurl        : http://download.opensuse.net.br
baseurlFtp     : 
baseurlRsync   : 
region         : sa
country        : br
asn            : 13335
prefix         : 172.67.144.0/20
lat,lng        : 37.751,-97.822
regionOnly     : True
countryOnly    : False
asOnly         : False
prefixOnly     : False
ipv6Only       : False
otherCountries : 
fileMaxsize    : 0
publicNotes    : 
score          : 100
enabled        : False
statusBaseurl  : False
admin          : mirror admin
adminEmail     : contato@opensuse.net.br

Note to self - have edited asn.py again, see #62678

#8 Updated by avicenzi over 1 year ago

pjessen wrote:

The initial scan fails due to the listing format:

# mb scan download.opensuse.net.br
Sun May 23 09:20:07 2021 download.opensuse.net.br: starting
Sun May 23 09:21:06 2021 download.opensuse.net.br: total files before scan: 0
download.opensuse.net.br: unparseable HTML index in /
Sun May 23 09:21:06 2021 download.opensuse.net.br: scanned 0 files (0/s) in 0s
Sun May 23 09:21:06 2021 download.opensuse.net.br: files to be purged: 0
Sun May 23 09:21:35 2021 download.opensuse.net.br: total files after scan: 0 (delta: 0)
Sun May 23 09:21:35 2021 download.opensuse.net.br: purged old files in 29s.
Sun May 23 09:21:35 2021 download.opensuse.net.br: done.
Completed in 1.5 minutes

Just using a plain apache or nginx autoindex format would help.

Not sure if I understood the scan fail, I don't expect or either want it to happen to download.opensuse.net.br, I set up rsync so the scanner can query our storage directly, which should be faster than parsing html pages.

#9 Updated by avicenzi over 1 year ago

pjessen wrote:

I have added the rsync url, which enables the scan, but I have to look into how to have an rsync url that is not publicly listed.

What do you mean by that? Do you want me to leave rsync open to the world? The mirror documentation recommends leaving it restricted to openSUSE IPs

#10 Updated by pjessen over 1 year ago

avicenzi wrote:

pjessen wrote:

avicenzi wrote:

Not sure if required, but we use Cloudflare and I believe that CF Proxy must be disabled to work, which exposes our IP. If not needed, I would like to enable this feature.

I don't know if this has any significance for us?

Yes, it masks the real IP.
The real IP for download.opensuse.net.br is 45.225.27.127, the same as rsync.opensuse.net.br.

My instinct says to use the real IP. I don't see that there is anything won by adding cloudflare to the mix.

#11 Updated by pjessen over 1 year ago

avicenzi wrote:

pjessen wrote:

I have added the rsync url, which enables the scan, but I have to look into how to have an rsync url that is not publicly listed.

What do you mean by that? Do you want me to leave rsync open to the world? The mirror documentation recommends leaving it restricted to openSUSE IPs

Yep, and so you should. My comment meant "I don't know how to add an rsync url for scanning only". There is no MirrorBrain setting that I can see.

#12 Updated by pjessen over 1 year ago

[snip 800 lines of the same]
NOTICE:  file distribution/leap/15.3/repo/oss/aarch64/libmlt++-devel-6.26.1-bp153.1.2.aarch64.rpm was just inserted by somebody else
NOTICE:  file distribution/leap/15.3/repo/oss/aarch64/libmlt++-devel-6.26.1-bp153.1.2.aarch64.rpm was just inserted by somebody else
NOTICE:  file distribution/leap/15.3/repo/oss/aarch64/libmlt++-devel-6.26.1-bp153.1.2.aarch64.rpm was just inserted by somebody else
__DIE__: (/usr/bin/scanner 1124 main::rsync_get_filelist => /usr/bin/scanner 1317 main::rsync_cb => /usr/bin/scanner 1066 main::save_file)
Wed May 26 08:59:51 2021 download.opensuse.net.br: scanned 8935 files (319/s) in 28s
__DIE__: (=> =>)
ERROR:  duplicate key value violates unique constraint "filearr_path_key"
DETAIL:  Key (path)=(distribution/leap/42.3/repo/oss/suse/x86_64/libconfig++-devel-1.4.9-10.3.x86_64.rpm) already exists.
CONTEXT:  SQL statement "UPDATE filearr 
            SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.9 minutes

#13 Updated by pjessen over 1 year ago

pjessen wrote:

NOTICE: file distribution/leap/15.3/repo/oss/aarch64/libmlt++-devel-6.26.1-bp153.1.2.aarch64.rpm was just inserted by somebody else

This is a notice issued by 'mirr_add_bypath', a stored procedure. When it hits a unique violation, it just tries again, which may explain the 800+ lines:

   WHEN unique_violation THEN
       RAISE NOTICE 'file % was just inserted by somebody else', arg_path;
       -- just update it by calling ourselves again
       SELECT into fileid mirr_add_bypath(arg_serverid, arg_path);

#14 Updated by pjessen over 1 year ago

I figured we had some sort of inconsistency in the database, and that deleting and recreating the mirror entry might work:

# mb delete download.opensuse.net.br
Traceback (most recent call last):
  File "/usr/bin/mb", line 1729, in <module>
    r = mirrordoctor.main()
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 261, in main
    return self.cmd(args)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 284, in cmd
    retval = self.onecmd(argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 422, in onecmd
    return self._dispatch_cmd(handler, argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 1123, in _dispatch_cmd
    return handler(argv[0], opts, *args)
  File "/usr/bin/mb", line 726, in do_delete
    mb.core.delete_mirror(self.conn, identifier)
  File "/usr/lib64/python2.7/site-packages/mb/core.py", line 22, in delete_mirror
    conn.Server._connection.queryAll(query)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 449, in queryAll
    return self._runWithConnection(self._queryAll, s)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 342, in _runWithConnection
    val = meth(conn, *args)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 441, in _queryAll
    self._executeRetry(conn, c, s)
  File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 244, in _executeRetry
    raise dberrors.DuplicateEntryError(msg)
sqlobject.dberrors.DuplicateEntryError: duplicate key value violates unique constraint "filearr_path_key"
DETAIL:  Key (path)=(distribution/leap/42.3/repo/oss/suse/x86_64/libconfig++-devel-1.4.9-10.3.x86_64.rpm) already exists.
CONTEXT:  SQL statement "UPDATE filearr 
            SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement

#15 Updated by pjessen over 1 year ago

pjessen wrote:

sqlobject.dberrors.DuplicateEntryError: duplicate key value violates unique constraint "filearr_path_key"
DETAIL: Key (path)=(distribution/leap/42.3/repo/oss/suse/x86_64/libconfig++-devel-1.4.9-10.3.x86_64.rpm) already exists.
CONTEXT: SQL statement "UPDATE filearr SET mirrors = arr WHERE id = arg_fileid"

That UPDATE cannot possibly cause a duplicate, so maybe a trigger ?

#16 Updated by pjessen over 1 year ago

It seems we had a number (5000+) of duplicate entries in table 'filearr', they kept gettign in the way.
After tidying up those, the scan succeeded:

# mb scan opensuse.net.br
Wed May 26 15:37:41 2021 opensuse.net.br: starting
Wed May 26 15:37:56 2021 opensuse.net.br: total files before scan: 148028
Wed May 26 15:41:23 2021 opensuse.net.br: scanned 193386 files (934/s) in 206s
Wed May 26 15:41:23 2021 opensuse.net.br: files to be purged: 0
Wed May 26 15:42:26 2021 opensuse.net.br: total files after scan: 193386 (delta: 45358)
Wed May 26 15:42:26 2021 opensuse.net.br: purged old files in 63s.
Wed May 26 15:42:26 2021 opensuse.net.br: done.
Completed in 4.8 minutes
pontifex2 (download.o.o):~ # mb enable opensuse.net.br

#17 Updated by avicenzi over 1 year ago

Excellent, I see that we had some downloads during the night.

#18 Updated by avicenzi over 1 year ago

I think we can close this one, our mirror has been running for a few days now and everything seems to be ok.

The only issue is that the scanner is not indexing 15.3 ISOs, which seems to be caused by #93621.

#19 Updated by pjessen over 1 year ago

Have just completed a full scan:

# mb scan opensuse.net.br
Wed Jun  9 11:04:58 2021 opensuse.net.br: starting
Wed Jun  9 11:08:47 2021 opensuse.net.br: total files before scan: 186873
Wed Jun  9 11:13:59 2021 opensuse.net.br: scanned 187337 files (600/s) in 311s
Wed Jun  9 11:13:59 2021 opensuse.net.br: files to be purged: 477
Wed Jun  9 11:17:40 2021 opensuse.net.br: total files after scan: 187337 (delta: 464)
Wed Jun  9 11:17:40 2021 opensuse.net.br: purged old files in 221s.
Wed Jun  9 11:17:40 2021 opensuse.net.br: done.
Completed in 12.7 minutes

When the status page is next updated, we should see the results.

#20 Updated by pjessen 9 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

All done, closing.

Also available in: Atom PDF