Project

General

Profile

Actions

tickets #69334

closed

Problem with mirror replication?

Added by juliogonzalezgil about 4 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Mirrors
Target version:
-
Start date:
2020-07-24
Due date:
% Done:

0%

Estimated time:

Description

We have an Uyuni Stable 2020.07 user using the official URL for the stable
repository:

https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/
images/repo/Uyuni-Server-POOL-x86_64-Media1/

In his case, that means he uses the mirror ftp.gwdg.de and he gets this error
updating one of the Uyuni packages:

Warning: Digest verification failed for file 'bind-
formula-0.1.1573049925.b509ada-1.1.uyuni.noarch.rpm'
[/var/tmp/AP_0xniEAF0/noarch/bind-
formula-0.1.1573049925.b509ada-1.1.uyuni.noarch.rpm]

expected 87d9f5828518a721a98b7ba47234af28578e9c71ddaa4ba2e91d5490f1396b60
but got e3a476a93dfada74ef4cd863bd023efe9cd1c00101e98246d091a6c8d2d8fb2d

I could replicate it locally as well.

And the problem seems to be that zypper does not get the metadata and the
packages from the same mirrors, and some are already updated and on 2020.07,
while others are still at 2020.06

If I get rid of the mirrordirector and Use https://ftp.gwdg.de/pub/opensuse/
repositories/systemsmanagement:/Uyuni%3A/Stable/images/repo/Uyuni-Server-POOL-
x86_64-Media1/ directly

IIRC this happened once already, and you had to force a metadata refresh
somehow.

Not sure if then the problem will fix the problems on the affected mirrors.

NOTE: I don't know what the hell happened at OBS. The only difference at bind-
formula is the timestamp on the files inside the pacakge, a minute difference
at November 2019... Package didn't change since then. And I don't know why the
problem happens now, since we released several times since November 2019.

Best regards.

--
Julio González Gil
Release Engineer, SUSE Manager and Uyuni
jgonzalez@suse.com


Files

signature.asc (833 Bytes) signature.asc juliogonzalezgil, 2020-07-24 17:53

Related issues 1 (0 open1 closed)

Related to openQA Infrastructure - action #153325: osd-deployment | Failed pipeline, Digest verification failed for openQA-common size:MResolvednicksinger2024-01-10

Actions
Actions #1

Updated by juliogonzalezgil about 4 years ago

This is urgent, as it broke the Uyuni 2020.07 upgrade.

Actions #4

Updated by cboltz about 4 years ago

  • Category set to Mirrors
  • Assignee set to pjessen
  • Private changed from Yes to No

CC'ing people in the ticket system isn't as easy as you'd hope - I added mc@ as watcher (and hope that this will trigger mail notifications), but couldn't find/add the other address. Therefore I'm making this ticket public (and marked the comment with the mail addresses as private) so that everybody interested can look at it.

I'm also afraid that it might need some days to get this handled - our mirror admin is on vacation.

A quick workaround might be to do a dummy change to the affected package, so that it gets rebuild with a new revision and therefore new filename.

Actions #5

Updated by juliogonzalezgil about 4 years ago

I was able to talk to Lars, it seems he triggered a rescan of all mirrors for the /systemsmanagement:/Uyuni:/Stable directory, and some more mirrors are showing now as updated. Let's hope that will fix the issue.

Problem is the Devel (Master) project already changed, and rolling back is possible but could lead to more problems (if for some reason I make an error with the rollback), not to mention that the affected package gets its changelog from a git repository so we'd need an update there, and also get it pushed to SUSE Manager.

Actions #6

Updated by juliogonzalezgil about 4 years ago

And also not sure if that will completely fix the issue in this case. One of the mirrors had the correct information for the package (sha512 at the metadata matched the updated package), and the outdated repos where 100% outdated (none of the packages got any package updated). Let's hope that waiting a little bit more will fix the problem when the mirrors are able to sync.

Actions #7

Updated by juliogonzalezgil about 4 years ago

After the weekend, I see almost all mirrors seem to be synced: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml.mirrorlist

However I see some mirrors missing from the list at https://mirrors.opensuse.org/ for "Repositories":

opensuse.mirror.ac.za (South Africa)
ftp.twaren.net (Taiwan)
ftp.yzu.edu.tw (Taiwan)
mirror.yandex.ru (Russia)

Is it normal? Are they up?

Actions #8

Updated by pjessen about 4 years ago

juliogonzalezgil wrote:

After the weekend, I see almost all mirrors seem to be synced: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml.mirrorlist

However I see some mirrors missing from the list at https://mirrors.opensuse.org/ for "Repositories":

http://opensuse.mirror.ac.za (South Africa)
http://ftp.twaren.net (Taiwan)
http://ftp.yzu.edu.tw (Taiwan)
http://mirror.yandex.ru (11Russia)

Is it normal? Are they up?

You only have to visit them to determine if they are up. Delays in mirroring the repositories probably quite normal as it simply takes a long time to push out changes, especially when there is a lot of them.

Actions #9

Updated by juliogonzalezgil about 4 years ago

Well, you are right, in this case what I meat is if the they are correctly reporting back when synced.

The reason I am asking is because, for example, I see http://opensuse.mirror.ac.za sycned, at least for repomd.xml, as you can see at the following logs (sha256 sums are the same for the same file on a synced and unsynced mirror:

# Synced mirror according to https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml.mirrorlist
$ wget https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml
--2020-07-27 22:48:00--  https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml
Resolviendo provo-mirror.opensuse.org (provo-mirror.opensuse.org)... 91.193.113.70
Conectando con provo-mirror.opensuse.org (provo-mirror.opensuse.org)[91.193.113.70]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 11739 (11K) [text/xml]
Grabando a: “repomd.xml”

repomd.xml                                                          100%[==================================================================================================================================================================>]  11,46K  --.-KB/s    en 0s      

2020-07-27 22:48:01 (125 MB/s) - “repomd.xml” guardado [11739/11739]

$ sha256sum repomd.xml 
d943d24afba948da5971a02cb4dc68aad5beabfbcea448383aaaecc5a2eadfa2  repomd.xml
# Unsynced mirror according to https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml.mirrorlist
$ wget http://opensuse.mirror.ac.za/repositories/systemsmanagement%3A/Uyuni%3A/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml
--2020-07-27 22:50:31--  http://opensuse.mirror.ac.za/repositories/systemsmanagement%3A/Uyuni%3A/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml
Resolviendo opensuse.mirror.ac.za (opensuse.mirror.ac.za)... 155.232.191.223, 2001:4200:fffc::223
Conectando con opensuse.mirror.ac.za (opensuse.mirror.ac.za)[155.232.191.223]:80... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 11739 (11K) [text/xml]
Grabando a: “repomd.xml.1”

repomd.xml.1                                                        100%[==================================================================================================================================================================>]  11,46K  --.-KB/s    en 0s      

2020-07-27 22:50:32 (100 MB/s) - “repomd.xml.1” guardado [11739/11739]

$ sha256sum repomd.xml.1
d943d24afba948da5971a02cb4dc68aad5beabfbcea448383aaaecc5a2eadfa2  repomd.xml.1

Actions #10

Updated by juliogonzalezgil about 4 years ago

So this what I can see:

http://ftp.yzu.edu.tw/Linux/openSUSE/repositories/systemsmanagement:/Uyuni:/ (Taiwan) seems to be dead, or at least I can't connect from my machines in Spain, France or Russia.
http://ftp.twaren.net/Linux/OpenSuSE/repositories/systemsmanagement:/Uyuni:/ (Taiwan) seems never synced Stable for any reason, and the stuff inside that folder is outdated (last sync seems it happened sometime in 2019).
https://opensuse.mirror.ac.za/repositories/systemsmanagement:/Uyuni:/ (South Africa) seems to be just OK, even if the .mirrorlist files do not show it.
https://mirror.yandex.ru/opensuse/repositories/systemsmanagement:/Uyuni:/ (Russia) seems stopped syncing Stable sometime in 2018, and the rest sometime in 2019.

Not sure if the three mirrors except the one from South Africa should just get disabled for "repositories". Also I am not sure if zypper is offering them and our users could have problems in the CIS countries, Africa or Asia.

Actions #11

Updated by pjessen about 4 years ago

  • Status changed from New to In Progress

juliogonzalezgil wrote:

So this what I can see:

http://ftp.yzu.edu.tw/Linux/openSUSE/repositories/systemsmanagement:/Uyuni:/ (Taiwan) seems to be dead, or at least I can't connect from my machines in Spain, France or Russia.

Hmm, according to our status (https://mirrors.opensuse.org) is working fine, but I cannot access it either. It is possible it is only serving Taiwan or China. I have no access to our database right now, I cannot check.

http://ftp.twaren.net/Linux/OpenSuSE/repositories/systemsmanagement:/Uyuni:/ (Taiwan) seems never synced Stable for any reason, and the stuff inside that folder is outdated (last sync seems it happened sometime in 2019).

According to https://progress.opensuse.org/issues/53465 it should be working fine, but obviously something has changed. https://mirrors.opensuse.org also shows it as being inactive, so it should not be handed out by mirrorbrain. If it is handed out, that is a problem.

https://mirror.yandex.ru/opensuse/repositories/systemsmanagement:/Uyuni:/ (Russia) seems stopped syncing Stable sometime in 2018, and the rest sometime in 2019.

I'll have to see if we are pushing repos to yandex or not. The rest of it seems okay, and repositories/ is flagged as being okay which is odd.

Not sure if the three mirrors except the one from South Africa should just get disabled for "repositories".

This is automatically managed by mirrorbrain, we don't control that manually. If a mirror is not up-to-date, mirrorbrain will not (is not supposed to) hand it out to a client.

Also I am not sure if zypper is
offering them and our users could have problems in the CIS countries, Africa or Asia.

Well, that is the important part - it is important if bad mirrors are being handed out (which they shouldn't be).

Actions #12

Updated by juliogonzalezgil about 4 years ago

It seems now even the Provo mirror has now problems, according to one of the Uyuni users from US:

Package bind-formula-0.1.1573049925.b509ada-1.1.uyuni.noarch
(uyuni-server-stable) seems to be corrupted during transfer. Do you want to
retry retrieval?

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="
https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/openSUSE_Leap_15.2/noarch/bind-formula-0.1.1573049925.b509ada-1.1.uyuni.noarch.rpm
">here</a>.</p>
<hr>
<address>Apache/2.4.43 (Linux/SUSE) Server at download.opensuse.org Port
80</address>
</body></html>

According to him, the CentOS7 client tools are affected as well:

ERROR: Download failed:
https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
- [Errno 14] HTTPS Error 302 - Found.

Can someone have a look?

Actions #13

Updated by juliogonzalezgil about 4 years ago

Any news? More US users are reporting the similar problems.

Actions #14

Updated by pjessen about 4 years ago

According to him, the CentOS7 client tools are affected as well:

ERROR: Download failed:
https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
- [Errno 14] HTTPS Error 302 - Found.

Getting a 302 is the normal response, that's what mirrorbrain does. Doing a wget on the url now:

wget -nd https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
--2020-08-13 17:53:56--  https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
Resolving download.opensuse.org (download.opensuse.org)... 2001:67c:2178:8::13, 195.135.221.134
Connecting to download.opensuse.org (download.opensuse.org)|2001:67c:2178:8::13|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://ftp.gwdg.de/pub/opensuse/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm [following]
--2020-08-13 17:53:57--  https://ftp.gwdg.de/pub/opensuse/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
Resolving ftp.gwdg.de (ftp.gwdg.de)... 2001:638:60f:110::1:2, 134.76.12.6
Connecting to ftp.gwdg.de (ftp.gwdg.de)|2001:638:60f:110::1:2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22128 (22K) [application/x-redhat-package-manager]
Saving to: ‘mgr-osad-4.1.3-2.1.uyuni.noarch.rpm’

mgr-osad-4.1.3-2.1.uyuni.noarch.rpm     100%[============================================================================>]  21.61K  --.-KB/s    in 0s      

2020-08-13 17:53:57 (172 MB/s) - ‘mgr-osad-4.1.3-2.1.uyuni.noarch.rpm’ saved [22128/22128]

Getting it from the provo-mirror seems to work fine:

wget -nd https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
--2020-08-13 17:56:50--  https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
Resolving provo-mirror.opensuse.org (provo-mirror.opensuse.org)... 91.193.113.70
Connecting to provo-mirror.opensuse.org (provo-mirror.opensuse.org)|91.193.113.70|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22128 (22K) [application/x-redhat-package-manager]
Saving to: ‘mgr-osad-4.1.3-2.1.uyuni.noarch.rpm.1’

mgr-osad-4.1.3-2.1.uyuni.noarch.rpm.1   100%[============================================================================>]  21.61K  --.-KB/s    in 0s      

2020-08-13 17:56:51 (202 MB/s) - ‘mgr-osad-4.1.3-2.1.uyuni.noarch.rpm.1’ saved [22128/22128]

I'm not sure what the problem is currently, but in the last month, I see provo-mirror rsync'ing regularly, some 144000 times.

What is weird is that it seems to be only mirroring a subset:

Apache:/MirrorBrain/ 
Apache:/Modules/ 
Cloud// 
Cloud:// 
Kernel:// 
OBS:/Server:/Unstable/ 
devel:/ARM:// 
devel:/CaaSP// 
devel:/CaaSP:// 
devel:/languages:/python/ 
home:/lrupp/ 
openSUSE:/Backports:/ 
openSUSE:/Tools/ 
openSUSE:/infrastructure/ 
openSUSE:/infrastructure:// 
openSUSE:/infrastructure:/debuginfod/ 
server:/database:/postgresql/ 
server:/http/ 
server:/mail/ 
server:/monitoring/ 
server:/php:/applications/ 
server:/php:/extensions/ 
server:/php:/extensions:/php7/ 
server:/search/

I'll see if I have access to provo-mirror.

Actions #15

Updated by pjessen about 4 years ago

I don't have the access :-(

Still, we are in fact pushing out repositories to provo-mirror.

From the log:

mb -b mirrordb1 scan provo-downloadcontent.opensuse.org -d repositories/devel:/tools:/scm/SLE_12_SP5
Traceback (most recent call last):
  File "/usr/bin/mb", line 1729, in <module>
    r = mirrordoctor.main()
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 251, in main
    retval = self.postoptparse()
  File "/usr/bin/mb", line 86, in postoptparse
    debug = self.options.debug)
  File "/usr/lib64/python2.7/site-packages/mb/conn.py", line 150, in __init__
    class Version(SQLObject):
  File "/usr/lib/python2.7/site-packages/sqlobject/declarative.py", line 98, in __new__
    cls.__classinit__(cls, new_attrs)
  File "/usr/lib/python2.7/site-packages/sqlobject/main.py", line 837, in __classinit__
    sqlmeta.addColumnsFromDatabase()
  File "/usr/lib/python2.7/site-packages/sqlobject/main.py", line 476, in addColumnsFromDatabase
    for columnDef in conn.columnsFromSchema(sqlmeta.table, soClass):
  File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 386, in columnsFromSchema
    keyData = self.queryAll(keyQuery % self.sqlrepr(tableName))
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 449, in queryAll
    return self._runWithConnection(self._queryAll, s)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 340, in _runWithConnection
    conn = self.getConnection()
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 351, in getConnection
    conn = self.makeConnection()
  File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 202, in makeConnection
    ErrorMessage(e, "used connection string %r" % self.dsn))
sqlobject.dberrors.OperationalError: could not translate host name "mirrordb1.infra.opensuse.org" to address: No address associated with hostname

==== end scan ===== RC=1 ============ 20200813-164521

Do we have some sort of DNS issue ??

Actions #16

Updated by juliogonzalezgil about 4 years ago

Getting it from the provo-mirror seems to work fine:

The problem is not getting the package. That works. But for some reason the sha256sum doesn't match:

Package bind-formula-0.1.1573049925.b509ada-1.1.uyuni.noarch (uyuni-server-stable) seems to be corrupted during transfer. Do you want to retry retrieval?

Sorry for not being more clear about.

At https://mirrors.opensuse.org/ it seems there are two mirrors in Provo with the same address for HTTP and FTP, but different for rsync (don't know why):

rsync://provo-downloadcontent.opensuse.org/repositories-scan
rsync://provo-mirror.opensuse.org/opensuse/

Could be it happens like the the other mirrors, where some are fully synced, and others are not, so the user gets the metadata from one mirror with one sha256sum, while the package in the other mirror has the same name but different sha25sum.

If one works and the other doesn't, maybe it could be removed from the mirrobrain until it's fixed.

Actions #17

Updated by cboltz about 4 years ago

No idea about the different rsync addresses, but provo-mirror and provo-downloadcontent are two IPs on the same server, and should serve the same files. (Exception: if the Nuremberg download.opensuse.org is down, we can enable mirrorbrain on provo-mirror, so provo-mirror will redirect instead of serving all files itsself. provo-downloadcontent will still provide direct downloads.)

provo-mirror is currently outdated (no idea why yet, in theory it has the needed cronjobs). I manually started a full sync yesterday, but it's still at /ports, so it will take some time until it reaches /repositories - maybe tomorrow, but no promises ;-)

I just did an additional sync of systemsmanagement:Uyuni:Stable which updated lots of files in the Leap 15.1 repo. Other build targets didn't need an update.

I also just started a manual rsync for all of systemsmanagement:Uyuni - the rsync output shows that (so far - rsync is still running) Master (for several build targets) and Snapshots were outdated.

If you still see files with wrong checksums in systemsmanagement:Uyuni, can you please post the full URIs?

Actions #18

Updated by pjessen about 4 years ago

cboltz wrote:

No idea about the different rsync addresses, but provo-mirror and provo-downloadcontent are two IPs on the same server, and should serve the same files.

provo-downloadcontent is probably only used by us to push repositories/, I don't think it ought to be publicly visible.

provo-mirror is currently outdated (no idea why yet, in theory it has the needed cronjobs). I manually started a full sync yesterday, but it's still at /ports, so it will take some time until it reaches /repositories - maybe tomorrow, but no promises ;-)

ports/ is 3.2Tb - yes, it'll take a while. repositories/ is 15Tb but might/should be more less in sync.

Actions #19

Updated by pjessen about 4 years ago

  • Assignee changed from pjessen to cboltz

Any update on this, Christian, is provo-mirror getting better? I don't remember, was I supposed to do something?

Actions #20

Updated by cboltz about 4 years ago

pjessen wrote:

Any update on this, Christian, is provo-mirror getting better?

Yes, but slowly. I still have rsync running (since weeks (!) and in a loop that auto-restarts it if needed), but it seems there is a bandwidth limit (100 MBit, maybe per IP?) that makes the sync slow. Additionally, I found a bug in rsync --compress and had to disable compression (https://bugzilla.opensuse.org/show_bug.cgi?id=1175854) which makes the sync even slower.

(If you want to check what rsync is doing right now - screen -r as user mirror.)

I don't remember, was I supposed to do something?

Nothing urgent, but maybe you should check if the push really works. Given how long rsync is already working on /repositories, I have the feeling that something might be broken.

Or OBS can compile more / faster than what a 100 MBit connection can transfer ;-)

Actions #21

Updated by pjessen about 4 years ago

cboltz wrote:

Nothing urgent, but maybe you should check if the push really works. Given how long rsync is already working on /repositories, I have the feeling that something might be broken.

Or OBS can compile more / faster than what a 100 MBit connection can transfer ;-)

Judging by the push logs (pontifex:/srv/bs/logs/2020/09), the push is working. In September so far, in 4497 pushes, we have sent 297'275 files and deleted 450'897.

Since the beginning of the year, we have been slowly pushing more and more, not sure why nor if it is an indicator of anything:

January    6870
February   7846
March      7175
April      9557
May       13712
June      14798
July      37866
August    36056
September  4497
Actions #22

Updated by juliogonzalezgil about 4 years ago

Yes, it seems provo-mirror is out of sync:

https://github.com/uyuni-project/uyuni/issues/2577#issuecomment-688707267 https://github.com/uyuni-project/uyuni/issues/2577#issuecomment-688716818

Besides, it's SSL certificate is expired since Sept 4th.

Since I didn't see a report for that, I created one: https://progress.opensuse.org/issues/71104

Actions #23

Updated by juliogonzalezgil about 4 years ago

It seems widehat.opensuse.org has problems as well. I got this from one of our developers:

download.suse.org serves the right repomd.xml:

curl
http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stabl
e/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml |
sha256sum
% Total % Received % Xferd Average Speed Time Time Time
Current Dload Upload Total Spent Left Speed 100 11739 100 11739

0 0 197k 0 --:--:-- --:--:-- --:--:-- 197k
d943d24afba948da5971a02cb4dc68aad5beabfbcea448383aaaecc5a2eadfa2 -

widehat.suse.org serves a different one:

curl
http://widehat.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable
/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml |
sha256sum
% Total % Received % Xferd Average Speed Time Time Time
Current Dload Upload Total Spent Left Speed 100 11738 100 11738

0 0 143k 0 --:--:-- --:--:-- --:--:-- 143k
d0775ed7ac220563f2f3b13b7212d6266aef27a3f926597e1728783333549c29 -

and, we do get redirects from download.suse.org to widehat.suse.org for individual RPMs, eg:

curl
https://download.opensuse.org/pub/opensuse/repositories/systemsmanagement
:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/noarch/bind-fo
rmula-0.1.1573049925.b509ada-1.1.uyuni.noarch.rpm
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

302 Found

Found
The document has moved here.
Apache/2.4.43 (Linux/SUSE) Server at download.opensuse.org Port
443

So ultimately checksums on the package do not match:

curl
http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stabl
e/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml | grep
primary.xml.gz
% Total % Received % Xferd Average Speed Time Time Time
Current Dload Upload Total Spent Left Speed 0 0 0 0

0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11739 100 11739
0 0 148k 0 --:--:-- --:--:-- --:--:-- 148k

<location

href="repodata/85389bcf181674574d3a5e494ffc30bd5f2269e50fe5f5d73c6e19894d32
2ccd-primary.xml.gz"/>

curl
http://widehat.opensuse.org/pub/opensuse/repositories/systemsmanagement:/
Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/85389bc
f181674574d3a5e494ffc30bd5f2269e50fe5f5d73c6e19894d322ccd-primary.xml.gz

404 Not Found

404 Not Found
nginx

The file exists on other mirrors, eg.

curl
https://ftp.gwdg.de/pub/opensuse/repositories/systemsmanagement:/Uyuni:/S
table/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/85389bcf1816745
74d3a5e494ffc30bd5f2269e50fe5f5d73c6e19894d
322ccd-primary.xml.gz | zless

Actions #24

Updated by pjessen about 4 years ago

It does look like widehat is quite out-of-sync, I have no idea why. Since yesterday, I've been running a manual rsync to try to catch up.

Actions #25

Updated by pjessen about 4 years ago

pjessen wrote:

It does look like widehat is quite out-of-sync, I have no idea why. Since yesterday, I've been running a manual rsync to try to catch up.

Well, one issue is also user:group mapping - on pontifex we use mirror:stage (1002:192), but on widehat we use mirror:mirror (1000:1000). I think this may have caused problems in the push of repositories, at the very least.

Actions #26

Updated by juliogonzalezgil about 3 years ago

From our side this can be closed. We did not have more issues for now.

Actions #27

Updated by pjessen over 2 years ago

  • Status changed from In Progress to Resolved

juliogonzalezgil wrote:

From our side this can be closed. We did not have more issues for now.

Thanks for the feedback, closing as resolved.

Actions #28

Updated by okurz 9 months ago

  • Related to action #153325: osd-deployment | Failed pipeline, Digest verification failed for openQA-common size:M added
Actions

Also available in: Atom PDF