openSUSE admin

cboltz wrote:

To make things more interesting, http://mirror.linux-ia64.org redirects me to https://zero.2090000.ru/ which ... doesn't look like a mirror :-/

Can't reproduce, looks like a standard Apache directory listing.

Per, maybe disable this mirror until we know what's going on?

Yeah, good idea. The scanner should have picked up on it, so I suspect that is where we have to look for a problem. I'll disable it anyway.

pontifex2 (download.o.o):~ # mb scan mirror.ia64-linux.org
Sun Jun  6 16:42:24 2021 mirror.ia64-linux.org: starting
Sun Jun  6 16:43:33 2021 mirror.ia64-linux.org: total files before scan: 2782064
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)
Sun Jun  6 18:18:30 2021 mirror.ia64-linux.org: scanned 2541678 files (446/s) in 5697s
__DIE__: (=> =>)
ERROR:  duplicate key value violates unique constraint "filearr_path_key"
DETAIL:  Key (path)=(distribution/leap/15.2/repo/oss/x86_64/patterns-mate-mate_utilities-20170319-lp152.3.3.x86_64.rpm) already exists.
CONTEXT:  SQL statement "UPDATE filearr 
            SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.6 hours

Same problem seen when I created the latest Brazilian mirror.

pjessen wrote:

pontifex2 (download.o.o):~ # mb scan mirror.ia64-linux.org
Sun Jun  6 16:42:24 2021 mirror.ia64-linux.org: starting
Sun Jun  6 16:43:33 2021 mirror.ia64-linux.org: total files before scan: 2782064
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)
Sun Jun  6 18:18:30 2021 mirror.ia64-linux.org: scanned 2541678 files (446/s) in 5697s
__DIE__: (=> =>)
ERROR:  duplicate key value violates unique constraint "filearr_path_key"
DETAIL:  Key (path)=(distribution/leap/15.2/repo/oss/x86_64/patterns-mate-mate_utilities-20170319-lp152.3.3.x86_64.rpm) already exists.
CONTEXT:  SQL statement "UPDATE filearr 
            SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.6 hours

Same problem seen when I created the latest Brazilian mirror.

Does anyone have any idea how we can have duplicate rows in a table that has a unique index constraint? surely any duplicate insert will be refused. I have 'cleaned up' the filearr table before, deleting all the duplicates, but they keep appearing.

filearr currently has 1686 entries with 2 or more ids.

This query is useful :

select path,array_agg(id) from filearr  group by path having count(*)>1

To generate delete statements:

sed -e 's/^.*{//' filearr-duplicates-20210607 | tr -d '{}' | egrep ',' | while read ids; do printf "delete from hash where file_id in (%s);\ndelete from filearr where id in (%s);\n" $ids $ids; done

Have just deleted all duplicate rows. Now re-running scan of mirror.ia64-linux.org

# mb scan mirror.ia64-linux.org
Mon Jun  7 15:51:49 2021 mirror.ia64-linux.org: starting
Mon Jun  7 15:54:45 2021 mirror.ia64-linux.org: total files before scan: 4381122
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)
http://mirror.linux-ia64.org/opensuse/history/20210605/?F=1: error "302 Found" fetching index, skipped.
Mon Jun  7 17:44:54 2021 mirror.ia64-linux.org: scanned 2504294 files (378/s) in 6609s
__DIE__: (=> =>)
ERROR:  deadlock detected
DETAIL:  Process 6471 waits for ShareLock on transaction 1201024640; blocked by process 779.
Process 779 waits for ShareLock on transaction 1201007976; blocked by process 6471.
HINT:  See server log for query details.
CONTEXT:  while locking tuple (3020703,5) in relation "filearr"
SQL statement "UPDATE filearr 
            SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.9 hours

# mb scan mirror.ia64-linux.org
Tue Jun  8 07:05:16 2021 mirror.ia64-linux.org: starting
Tue Jun  8 07:06:45 2021 mirror.ia64-linux.org: total files before scan: 4386450
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)
Tue Jun  8 08:38:16 2021 mirror.ia64-linux.org: scanned 2536629 files (461/s) in 5490s
__DIE__: (=> =>)
ERROR:  duplicate key value violates unique constraint "filearr_path_key"
DETAIL:  Key (path)=(distribution/leap/15.2/repo/oss/x86_64/patterns-mate-mate_utilities-20170319-lp152.3.3.x86_64.rpm) already exists.
CONTEXT:  SQL statement "UPDATE filearr 
            SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.9 hours

I have just picked a random package:

repositories/systemsmanagement:/Uyuni:/Stable:/openSUSE_Leap_15-Uyuni-Client-Tools/openSUSE_Leap_15.0/x86_64/golang-github-boynux-squid_exporter-1.6-5.1.uyuni.x86_64.rpm

According to the mirrorlist offered, it is available from mirror.linux-ia64.org. However, trying to retrieve it gives a 404:

http://mirror.linux-ia64.org/opensuse/repositories/systemsmanagement:/Uyuni:/Stable:/openSUSE_Leap_15-Uyuni-Client-Tools/openSUSE_Leap_15.0/x86_64/golang-github-boynux-squid_exporter-1.6-5.1.uyuni.x86_64.rpm

In 'filearr', the following mirrors are listed as having that package:

 id  |             identifier             
-----+------------------------------------
 169 | ftp.gwdg.de
 177 | widehat.o.o
 101 | mirror.karneval.cz
 120 | ftp2.nluug.nl
 124 | ftp1.nluug.nl
 181 | opensuse.mirror.ac.za
 423 | mirror.tspu.ru
 424 | ftp.uni-stuttgart.de
 492 | opensuse.ucom.am
 495 | provo-mirror.opensuse.org
 505 | mirror.ia64-linux.org
 508 | espejito.fder.edu.uy
 566 | ftp.lysator.liu.se
 582 | provo-downloadcontent.opensuse.org

So, somehow the scan of mirror.linux-ia64.org is still not complete. Trying a manual scan.

# mb scan mirror.ia64-linux.org
Fri Jun 11 08:31:24 2021 mirror.ia64-linux.org: starting
Fri Jun 11 08:32:37 2021 mirror.ia64-linux.org: total files before scan: 4461962
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)

When the rsync scan dies above, presumably due to a time-out, it tries to scan over http. Unfortunately, that doesn't always seem to work:

http://mirror.linux-ia64.org/opensuse/debug/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/jeos/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/repo/non-oss/media.1/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/repo/non-oss/repodata/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/repo/oss/470.1-boot/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/repo/oss/boot/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.2/iso-devel/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
Fri Jun 11 08:53:44 2021 mirror.ia64-linux.org: Error 500 occured
Fri Jun 11 08:55:55 2021 mirror.ia64-linux.org: Error 500 occured
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
__DIE__: (/usr/bin/scanner 606 main::http_readdir => /usr/bin/scanner 606 main::http_readdir => /usr/bin/scanner 621 main::save_file)
mirror.ia64-linux.org: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request. at /usr/bin/scanner line 943.
Completed in 1.6 hours

Have written to the admin to inquire. In the meantime, disabling the mirror won't help, I'll need to update 'filearr' manually.