tickets #93477
closedProblem with mirror.linux-ia64.org
Added by Victor.Zhestkov@suse.com almost 3 years ago. Updated over 2 years ago.
100%
Description
Hi Team.
All the requests to the files from https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/openSUSE_Leap_15-Uyuni-Client-Tools/openSUSE_Leap_15.0/ redirecting me to http://mirror.linux-ia64.org/opensuse/repositories/systemsmanagement:/Uyuni:/Stable:/openSUSE_Leap_15-Uyuni-Client-Tools/openSUSE_Leap_15.0/ but http://mirror.linux-ia64.org/opensuse/repositories/systemsmanagement:/ is empty there.
Is it possible to fix this?
Regards,
Victor
Updated by cboltz almost 3 years ago
- Category set to Mirrors
- Assignee set to pjessen
To make things more interesting, http://mirror.linux-ia64.org redirects me to https://zero.2090000.ru/ which ... doesn't look like a mirror :-/ (The text translates to There are insufficient funds in your account. To continue using the Internet, top up your balance or activate a deferred payment. - and that looks phishy IMHO...)
Per, maybe disable this mirror until we know what's going on?
Updated by pjessen almost 3 years ago
cboltz wrote:
To make things more interesting, http://mirror.linux-ia64.org redirects me to https://zero.2090000.ru/ which ... doesn't look like a mirror :-/
Can't reproduce, looks like a standard Apache directory listing.
Per, maybe disable this mirror until we know what's going on?
Yeah, good idea. The scanner should have picked up on it, so I suspect that is where we have to look for a problem. I'll disable it anyway.
Updated by pjessen almost 3 years ago
- Subject changed from Redirection to incomplete mirror to Problem with mirror.linux-ia64.org
- Status changed from New to In Progress
- Private changed from Yes to No
Updated by pjessen almost 3 years ago
pontifex2 (download.o.o):~ # mb scan mirror.ia64-linux.org
Sun Jun 6 16:42:24 2021 mirror.ia64-linux.org: starting
Sun Jun 6 16:43:33 2021 mirror.ia64-linux.org: total files before scan: 2782064
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)
Sun Jun 6 18:18:30 2021 mirror.ia64-linux.org: scanned 2541678 files (446/s) in 5697s
__DIE__: (=> =>)
ERROR: duplicate key value violates unique constraint "filearr_path_key"
DETAIL: Key (path)=(distribution/leap/15.2/repo/oss/x86_64/patterns-mate-mate_utilities-20170319-lp152.3.3.x86_64.rpm) already exists.
CONTEXT: SQL statement "UPDATE filearr
SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.6 hours
Same problem seen when I created the latest Brazilian mirror.
Updated by pjessen almost 3 years ago
pjessen wrote:
pontifex2 (download.o.o):~ # mb scan mirror.ia64-linux.org Sun Jun 6 16:42:24 2021 mirror.ia64-linux.org: starting Sun Jun 6 16:43:33 2021 mirror.ia64-linux.org: total files before scan: 2782064 __DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist) Sun Jun 6 18:18:30 2021 mirror.ia64-linux.org: scanned 2541678 files (446/s) in 5697s __DIE__: (=> =>) ERROR: duplicate key value violates unique constraint "filearr_path_key" DETAIL: Key (path)=(distribution/leap/15.2/repo/oss/x86_64/patterns-mate-mate_utilities-20170319-lp152.3.3.x86_64.rpm) already exists. CONTEXT: SQL statement "UPDATE filearr SET mirrors = arr WHERE id = arg_fileid" PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345. Completed in 1.6 hoursSame problem seen when I created the latest Brazilian mirror.
Does anyone have any idea how we can have duplicate rows in a table that has a unique index constraint? surely any duplicate insert will be refused. I have 'cleaned up' the filearr table before, deleting all the duplicates, but they keep appearing.
Updated by pjessen almost 3 years ago
filearr currently has 1686 entries with 2 or more ids.
This query is useful :
select path,array_agg(id) from filearr group by path having count(*)>1
To generate delete statements:
sed -e 's/^.*{//' filearr-duplicates-20210607 | tr -d '{}' | egrep ',' | while read ids; do printf "delete from hash where file_id in (%s);\ndelete from filearr where id in (%s);\n" $ids $ids; done
Have just deleted all duplicate rows. Now re-running scan of mirror.ia64-linux.org
Updated by pjessen almost 3 years ago
# mb scan mirror.ia64-linux.org
Mon Jun 7 15:51:49 2021 mirror.ia64-linux.org: starting
Mon Jun 7 15:54:45 2021 mirror.ia64-linux.org: total files before scan: 4381122
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)
http://mirror.linux-ia64.org/opensuse/history/20210605/?F=1: error "302 Found" fetching index, skipped.
Mon Jun 7 17:44:54 2021 mirror.ia64-linux.org: scanned 2504294 files (378/s) in 6609s
__DIE__: (=> =>)
ERROR: deadlock detected
DETAIL: Process 6471 waits for ShareLock on transaction 1201024640; blocked by process 779.
Process 779 waits for ShareLock on transaction 1201007976; blocked by process 6471.
HINT: See server log for query details.
CONTEXT: while locking tuple (3020703,5) in relation "filearr"
SQL statement "UPDATE filearr
SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.9 hours
Updated by pjessen almost 3 years ago
# mb scan mirror.ia64-linux.org
Tue Jun 8 07:05:16 2021 mirror.ia64-linux.org: starting
Tue Jun 8 07:06:45 2021 mirror.ia64-linux.org: total files before scan: 4386450
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)
Tue Jun 8 08:38:16 2021 mirror.ia64-linux.org: scanned 2536629 files (461/s) in 5490s
__DIE__: (=> =>)
ERROR: duplicate key value violates unique constraint "filearr_path_key"
DETAIL: Key (path)=(distribution/leap/15.2/repo/oss/x86_64/patterns-mate-mate_utilities-20170319-lp152.3.3.x86_64.rpm) already exists.
CONTEXT: SQL statement "UPDATE filearr
SET mirrors = arr WHERE id = arg_fileid"
PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement at /usr/bin/scanner line 345.
Completed in 1.9 hours
Updated by pjessen almost 3 years ago
- Due date set to 2021-06-09
- % Done changed from 0 to 90
Have deleted all duplicates again, but the scan then aborted again due to a deadlock. I'm going to wait until tomorrow to see if the regular scan has been able to complete. The ticket title is actually wrong, but I'll leave it. See #93621.
Updated by pjessen almost 3 years ago
I have just picked a random package:
repositories/systemsmanagement:/Uyuni:/Stable:/openSUSE_Leap_15-Uyuni-Client-Tools/openSUSE_Leap_15.0/x86_64/golang-github-boynux-squid_exporter-1.6-5.1.uyuni.x86_64.rpm
According to the mirrorlist offered, it is available from mirror.linux-ia64.org. However, trying to retrieve it gives a 404:
In 'filearr', the following mirrors are listed as having that package:
id | identifier
-----+------------------------------------
169 | ftp.gwdg.de
177 | widehat.o.o
101 | mirror.karneval.cz
120 | ftp2.nluug.nl
124 | ftp1.nluug.nl
181 | opensuse.mirror.ac.za
423 | mirror.tspu.ru
424 | ftp.uni-stuttgart.de
492 | opensuse.ucom.am
495 | provo-mirror.opensuse.org
505 | mirror.ia64-linux.org
508 | espejito.fder.edu.uy
566 | ftp.lysator.liu.se
582 | provo-downloadcontent.opensuse.org
So, somehow the scan of mirror.linux-ia64.org is still not complete. Trying a manual scan.
Updated by pjessen almost 3 years ago
# mb scan mirror.ia64-linux.org
Fri Jun 11 08:31:24 2021 mirror.ia64-linux.org: starting
Fri Jun 11 08:32:37 2021 mirror.ia64-linux.org: total files before scan: 4461962
__DIE__: (/usr/bin/scanner 315 main::rsync_readdir => /usr/bin/scanner 1123 (eval) => /usr/bin/scanner 1124 main::rsync_get_filelist)
When the rsync scan dies above, presumably due to a time-out, it tries to scan over http. Unfortunately, that doesn't always seem to work:
http://mirror.linux-ia64.org/opensuse/debug/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/jeos/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/repo/non-oss/media.1/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/repo/non-oss/repodata/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/repo/oss/470.1-boot/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.1/repo/oss/boot/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
http://mirror.linux-ia64.org/opensuse/distribution/leap/15.2/iso-devel/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/Changes/?F=1: error "500 Can't connect to mirror.linux-ia64.org:80 (Connection timed out)" fetching index, skipped.
Fri Jun 11 08:53:44 2021 mirror.ia64-linux.org: Error 500 occured
Fri Jun 11 08:55:55 2021 mirror.ia64-linux.org: Error 500 occured
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
__DIE__: (/usr/bin/scanner 606 main::http_readdir => /usr/bin/scanner 606 main::http_readdir => /usr/bin/scanner 621 main::save_file)
mirror.ia64-linux.org: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at /usr/bin/scanner line 943.
Completed in 1.6 hours
Updated by pjessen almost 3 years ago
Have written to the admin to inquire. In the meantime, disabling the mirror won't help, I'll need to update 'filearr' manually.
Updated by pjessen almost 3 years ago
- Due date changed from 2021-06-09 to 2021-08-03
Updated by pjessen over 2 years ago
- Status changed from In Progress to Resolved
- % Done changed from 90 to 100
pjessen wrote:
I am just now running a scan of this mirror.
# mb scan mirror.ia64-linux.org
Fri Dec 3 09:38:04 2021 mirror.ia64-linux.org: starting
Fri Dec 3 09:46:29 2021 mirror.ia64-linux.org: total files before scan: 1300095
Fri Dec 3 12:17:13 2021 mirror.ia64-linux.org: scanned 2919252 files (322/s) in 9044s
Fri Dec 3 12:18:59 2021 mirror.ia64-linux.org: files to be purged: 14016
Fri Dec 3 12:40:58 2021 mirror.ia64-linux.org: total files after scan: 2919252 (delta: 1619157)
Fri Dec 3 12:40:58 2021 mirror.ia64-linux.org: purged old files in 1425s.
Fri Dec 3 12:40:58 2021 mirror.ia64-linux.org: done.
Completed in 3.0 hours
Looks good. All done.