openSUSE admin

The initial scan fails due to the listing format:

# mb scan download.opensuse.net.br
Sun May 23 09:20:07 2021 download.opensuse.net.br: starting
Sun May 23 09:21:06 2021 download.opensuse.net.br: total files before scan: 0
download.opensuse.net.br: unparseable HTML index in /
Sun May 23 09:21:06 2021 download.opensuse.net.br: scanned 0 files (0/s) in 0s
Sun May 23 09:21:06 2021 download.opensuse.net.br: files to be purged: 0
Sun May 23 09:21:35 2021 download.opensuse.net.br: total files after scan: 0 (delta: 0)
Sun May 23 09:21:35 2021 download.opensuse.net.br: purged old files in 29s.
Sun May 23 09:21:35 2021 download.opensuse.net.br: done.
Completed in 1.5 minutes

Just using a plain apache or nginx autoindex format would help.

pjessen wrote:

avicenzi wrote:

Not sure if required, but we use Cloudflare and I believe that CF Proxy must be disabled to work, which exposes our IP. If not needed, I would like to enable this feature.

I don't know if this has any significance for us?

Yes, it masks the real IP.

The real IP for download.opensuse.net.br is 45.225.27.127, the same as rsync.opensuse.net.br.

I also set reverse IP lookup and a PTR record, if not needed let me know also.

It is needed, and rsync.opensuse.net.br looks good. Stage ACL updated.

OK

• 100 Mbit network, can be increased in the future

We can set up a lower priority for this mirror? Scale is 0-100%.

No, leave it at 100%, if the server goes down due load we can review it.

• No data cap, but would be nice to limit to 10 TB transfer initially and limit simultaneous connections to a reasonable number (we limit 20 by IP)

We don't have any control over that - tweaking the priority might indirectly help, but that's all we can do.

Not needed, just nice to have, let's leave it open and I'll check how much it will consume.

• The server should also block access from Asia and Africa continents

We have the following options:

regionOnly - only South-America
countryOnly - only Brazil.

I don't have a way of specifying 'not something' - blocking Asia and Africa is probably best achieved by using regiononly = sa, and then adding other countries permitted with 'otherCountries'. (just a list of country codes).

We have some rules on our side, South America is good.

Wrt the mirror URL 'download.opensuse.net.br' - it currently resolves like this:

host download.opensuse.net.br
download.opensuse.net.br has address 104.21.72.251
download.opensuse.net.br has address 172.67.155.239
download.opensuse.net.br has IPv6 address 2606:4700:3036::6815:48fb
download.opensuse.net.br has IPv6 address 2606:4700:3034::ac43:9bef

I guess this is Cloudflare? I have edited the mirror setup manually to say 'sa' and 'br'.

Yes, we have no IPv6, the real IP is the same as rsync, but I can disable CF Proxy

# mb show net.br
identifier     : download.opensuse.net.br
operatorName   : Binario Cloud / openSUSE Brasil
operatorUrl    : https://binario.cloud/
baseurl        : http://download.opensuse.net.br
baseurlFtp     : 
baseurlRsync   : 
region         : sa
country        : br
asn            : 13335
prefix         : 172.67.144.0/20
lat,lng        : 37.751,-97.822
regionOnly     : True
countryOnly    : False
asOnly         : False
prefixOnly     : False
ipv6Only       : False
otherCountries : 
fileMaxsize    : 0
publicNotes    : 
score          : 100
enabled        : False
statusBaseurl  : False
admin          : mirror admin
adminEmail     : contato@opensuse.net.br

Note to self - have edited asn.py again, see #62678

pjessen wrote:

The initial scan fails due to the listing format:

# mb scan download.opensuse.net.br
Sun May 23 09:20:07 2021 download.opensuse.net.br: starting
Sun May 23 09:21:06 2021 download.opensuse.net.br: total files before scan: 0
download.opensuse.net.br: unparseable HTML index in /
Sun May 23 09:21:06 2021 download.opensuse.net.br: scanned 0 files (0/s) in 0s
Sun May 23 09:21:06 2021 download.opensuse.net.br: files to be purged: 0
Sun May 23 09:21:35 2021 download.opensuse.net.br: total files after scan: 0 (delta: 0)
Sun May 23 09:21:35 2021 download.opensuse.net.br: purged old files in 29s.
Sun May 23 09:21:35 2021 download.opensuse.net.br: done.
Completed in 1.5 minutes

Just using a plain apache or nginx autoindex format would help.

Not sure if I understood the scan fail, I don't expect or either want it to happen to download.opensuse.net.br, I set up rsync so the scanner can query our storage directly, which should be faster than parsing html pages.

pjessen wrote:

I have added the rsync url, which enables the scan, but I have to look into how to have an rsync url that is not publicly listed.

What do you mean by that? Do you want me to leave rsync open to the world? The mirror documentation recommends leaving it restricted to openSUSE IPs

avicenzi wrote:

pjessen wrote:

avicenzi wrote:

Not sure if required, but we use Cloudflare and I believe that CF Proxy must be disabled to work, which exposes our IP. If not needed, I would like to enable this feature.

I don't know if this has any significance for us?

Yes, it masks the real IP.
The real IP for download.opensuse.net.br is 45.225.27.127, the same as rsync.opensuse.net.br.

My instinct says to use the real IP. I don't see that there is anything won by adding cloudflare to the mix.

avicenzi wrote:

pjessen wrote:

I have added the rsync url, which enables the scan, but I have to look into how to have an rsync url that is not publicly listed.

What do you mean by that? Do you want me to leave rsync open to the world? The mirror documentation recommends leaving it restricted to openSUSE IPs

Yep, and so you should. My comment meant "I don't know how to add an rsync url for scanning only". There is no MirrorBrain setting that I can see.

[snip 800 lines of the same]
NOTICE:  file distribution/leap/15.3/repo/oss/aarch64/libmlt++-devel-6.26.1-bp153.1.2.aarch64.rpm was just inserted by somebody else
NOTICE:  file distribution/leap/15.3/repo/oss/aarch64/libmlt++-devel-6.26.1-bp153.1.2.aarch64.rpm was just inserted by somebody else
NOTICE:  file distribution/leap/15.3/repo/oss/aarch64/libmlt++-devel-6.26.1-bp153.1.2.aarch64.rpm was just inserted by somebody else
__DIE__: (/usr/bin/scanner 1124 main::rsync_get_filelist => /usr/bin/scanner 1317 main::rsync_cb => /usr/bin/scanner 1066 main::save_file)
Wed May 26 08:59:51 2021 download.opensuse.net.br: scanned 8935 files (319/s) in 28s
__DIE__: (=> =>)
ERROR:  duplicate key value violates unique constraint "filearr_path_key"
DETAIL:  Key (path)=(distribution/leap/42.3/repo/oss/suse/x86_64/libconfig++-devel-1.4.9-10.3.x86_64.rpm) already exists.
CONTEXT:  SQL statement "UPDATE filearr 
            SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.9 minutes

pjessen wrote:

NOTICE: file distribution/leap/15.3/repo/oss/aarch64/libmlt++-devel-6.26.1-bp153.1.2.aarch64.rpm was just inserted by somebody else

This is a notice issued by 'mirr_add_bypath', a stored procedure. When it hits a unique violation, it just tries again, which may explain the 800+ lines:

   WHEN unique_violation THEN
       RAISE NOTICE 'file % was just inserted by somebody else', arg_path;
       -- just update it by calling ourselves again
       SELECT into fileid mirr_add_bypath(arg_serverid, arg_path);

I figured we had some sort of inconsistency in the database, and that deleting and recreating the mirror entry might work:

# mb delete download.opensuse.net.br
Traceback (most recent call last):
  File "/usr/bin/mb", line 1729, in <module>
    r = mirrordoctor.main()
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 261, in main
    return self.cmd(args)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 284, in cmd
    retval = self.onecmd(argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 422, in onecmd
    return self._dispatch_cmd(handler, argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 1123, in _dispatch_cmd
    return handler(argv[0], opts, *args)
  File "/usr/bin/mb", line 726, in do_delete
    mb.core.delete_mirror(self.conn, identifier)
  File "/usr/lib64/python2.7/site-packages/mb/core.py", line 22, in delete_mirror
    conn.Server._connection.queryAll(query)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 449, in queryAll
    return self._runWithConnection(self._queryAll, s)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 342, in _runWithConnection
    val = meth(conn, *args)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 441, in _queryAll
    self._executeRetry(conn, c, s)
  File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 244, in _executeRetry
    raise dberrors.DuplicateEntryError(msg)
sqlobject.dberrors.DuplicateEntryError: duplicate key value violates unique constraint "filearr_path_key"
DETAIL:  Key (path)=(distribution/leap/42.3/repo/oss/suse/x86_64/libconfig++-devel-1.4.9-10.3.x86_64.rpm) already exists.
CONTEXT:  SQL statement "UPDATE filearr 
            SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement

pjessen wrote:

sqlobject.dberrors.DuplicateEntryError: duplicate key value violates unique constraint "filearr_path_key"
DETAIL: Key (path)=(distribution/leap/42.3/repo/oss/suse/x86_64/libconfig++-devel-1.4.9-10.3.x86_64.rpm) already exists.
CONTEXT: SQL statement "UPDATE filearr SET mirrors = arr WHERE id = arg_fileid"

That UPDATE cannot possibly cause a duplicate, so maybe a trigger ?

It seems we had a number (5000+) of duplicate entries in table 'filearr', they kept gettign in the way.
After tidying up those, the scan succeeded:

# mb scan opensuse.net.br
Wed May 26 15:37:41 2021 opensuse.net.br: starting
Wed May 26 15:37:56 2021 opensuse.net.br: total files before scan: 148028
Wed May 26 15:41:23 2021 opensuse.net.br: scanned 193386 files (934/s) in 206s
Wed May 26 15:41:23 2021 opensuse.net.br: files to be purged: 0
Wed May 26 15:42:26 2021 opensuse.net.br: total files after scan: 193386 (delta: 45358)
Wed May 26 15:42:26 2021 opensuse.net.br: purged old files in 63s.
Wed May 26 15:42:26 2021 opensuse.net.br: done.
Completed in 4.8 minutes
pontifex2 (download.o.o):~ # mb enable opensuse.net.br

Excellent, I see that we had some downloads during the night.

I think we can close this one, our mirror has been running for a few days now and everything seems to be ok.

The only issue is that the scanner is not indexing 15.3 ISOs, which seems to be caused by #93621.

Have just completed a full scan:

# mb scan opensuse.net.br
Wed Jun  9 11:04:58 2021 opensuse.net.br: starting
Wed Jun  9 11:08:47 2021 opensuse.net.br: total files before scan: 186873
Wed Jun  9 11:13:59 2021 opensuse.net.br: scanned 187337 files (600/s) in 311s
Wed Jun  9 11:13:59 2021 opensuse.net.br: files to be purged: 477
Wed Jun  9 11:17:40 2021 opensuse.net.br: total files after scan: 187337 (delta: 464)
Wed Jun  9 11:17:40 2021 opensuse.net.br: purged old files in 221s.
Wed Jun  9 11:17:40 2021 opensuse.net.br: done.
Completed in 12.7 minutes

When the status page is next updated, we should see the results.