openSUSE admin

This is urgent, as it broke the Uyuni 2020.07 upgrade.

And an example of a mirror not updated: http://ftp1.nluug.nl/os/Linux/distr/opensuse/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/

I was able to talk to Lars, it seems he triggered a rescan of all mirrors for the /systemsmanagement:/Uyuni:/Stable directory, and some more mirrors are showing now as updated. Let's hope that will fix the issue.

Problem is the Devel (Master) project already changed, and rolling back is possible but could lead to more problems (if for some reason I make an error with the rollback), not to mention that the affected package gets its changelog from a git repository so we'd need an update there, and also get it pushed to SUSE Manager.

And also not sure if that will completely fix the issue in this case. One of the mirrors had the correct information for the package (sha512 at the metadata matched the updated package), and the outdated repos where 100% outdated (none of the packages got any package updated). Let's hope that waiting a little bit more will fix the problem when the mirrors are able to sync.

After the weekend, I see almost all mirrors seem to be synced: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml.mirrorlist

However I see some mirrors missing from the list at https://mirrors.opensuse.org/ for "Repositories":

opensuse.mirror.ac.za (South Africa)
ftp.twaren.net (Taiwan)
ftp.yzu.edu.tw (Taiwan)
mirror.yandex.ru (Russia)

Is it normal? Are they up?

juliogonzalezgil wrote:

After the weekend, I see almost all mirrors seem to be synced: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml.mirrorlist

However I see some mirrors missing from the list at https://mirrors.opensuse.org/ for "Repositories":

http://opensuse.mirror.ac.za (South Africa)
http://ftp.twaren.net (Taiwan)
http://ftp.yzu.edu.tw (Taiwan)
http://mirror.yandex.ru (11Russia)

Is it normal? Are they up?

You only have to visit them to determine if they are up. Delays in mirroring the repositories probably quite normal as it simply takes a long time to push out changes, especially when there is a lot of them.

Well, you are right, in this case what I meat is if the they are correctly reporting back when synced.

The reason I am asking is because, for example, I see http://opensuse.mirror.ac.za sycned, at least for repomd.xml, as you can see at the following logs (sha256 sums are the same for the same file on a synced and unsynced mirror:

# Synced mirror according to https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml.mirrorlist
$ wget https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml
--2020-07-27 22:48:00--  https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml
Resolviendo provo-mirror.opensuse.org (provo-mirror.opensuse.org)... 91.193.113.70
Conectando con provo-mirror.opensuse.org (provo-mirror.opensuse.org)[91.193.113.70]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 11739 (11K) [text/xml]
Grabando a: “repomd.xml”

repomd.xml                                                          100%[==================================================================================================================================================================>]  11,46K  --.-KB/s    en 0s      

2020-07-27 22:48:01 (125 MB/s) - “repomd.xml” guardado [11739/11739]

$ sha256sum repomd.xml 
d943d24afba948da5971a02cb4dc68aad5beabfbcea448383aaaecc5a2eadfa2  repomd.xml

# Unsynced mirror according to https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml.mirrorlist
$ wget http://opensuse.mirror.ac.za/repositories/systemsmanagement%3A/Uyuni%3A/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml
--2020-07-27 22:50:31--  http://opensuse.mirror.ac.za/repositories/systemsmanagement%3A/Uyuni%3A/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml
Resolviendo opensuse.mirror.ac.za (opensuse.mirror.ac.za)... 155.232.191.223, 2001:4200:fffc::223
Conectando con opensuse.mirror.ac.za (opensuse.mirror.ac.za)[155.232.191.223]:80... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 11739 (11K) [text/xml]
Grabando a: “repomd.xml.1”

repomd.xml.1                                                        100%[==================================================================================================================================================================>]  11,46K  --.-KB/s    en 0s      

2020-07-27 22:50:32 (100 MB/s) - “repomd.xml.1” guardado [11739/11739]

$ sha256sum repomd.xml.1
d943d24afba948da5971a02cb4dc68aad5beabfbcea448383aaaecc5a2eadfa2  repomd.xml.1

So this what I can see:

http://ftp.yzu.edu.tw/Linux/openSUSE/repositories/systemsmanagement:/Uyuni:/ (Taiwan) seems to be dead, or at least I can't connect from my machines in Spain, France or Russia.
http://ftp.twaren.net/Linux/OpenSuSE/repositories/systemsmanagement:/Uyuni:/ (Taiwan) seems never synced Stable for any reason, and the stuff inside that folder is outdated (last sync seems it happened sometime in 2019).
https://opensuse.mirror.ac.za/repositories/systemsmanagement:/Uyuni:/ (South Africa) seems to be just OK, even if the .mirrorlist files do not show it.
https://mirror.yandex.ru/opensuse/repositories/systemsmanagement:/Uyuni:/ (Russia) seems stopped syncing Stable sometime in 2018, and the rest sometime in 2019.

Not sure if the three mirrors except the one from South Africa should just get disabled for "repositories". Also I am not sure if zypper is offering them and our users could have problems in the CIS countries, Africa or Asia.

It seems now even the Provo mirror has now problems, according to one of the Uyuni users from US:

Package bind-formula-0.1.1573049925.b509ada-1.1.uyuni.noarch
(uyuni-server-stable) seems to be corrupted during transfer. Do you want to
retry retrieval?

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="
https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable/openSUSE_Leap_15.2/noarch/bind-formula-0.1.1573049925.b509ada-1.1.uyuni.noarch.rpm
">here</a>.</p>
<hr>
<address>Apache/2.4.43 (Linux/SUSE) Server at download.opensuse.org Port
80</address>
</body></html>

According to him, the CentOS7 client tools are affected as well:

ERROR: Download failed:
https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
- [Errno 14] HTTPS Error 302 - Found.

Can someone have a look?

Any news? More US users are reporting the similar problems.

According to him, the CentOS7 client tools are affected as well:

ERROR: Download failed:
https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
- [Errno 14] HTTPS Error 302 - Found.

Getting a 302 is the normal response, that's what mirrorbrain does. Doing a wget on the url now:

wget -nd https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
--2020-08-13 17:53:56--  https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
Resolving download.opensuse.org (download.opensuse.org)... 2001:67c:2178:8::13, 195.135.221.134
Connecting to download.opensuse.org (download.opensuse.org)|2001:67c:2178:8::13|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://ftp.gwdg.de/pub/opensuse/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm [following]
--2020-08-13 17:53:57--  https://ftp.gwdg.de/pub/opensuse/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
Resolving ftp.gwdg.de (ftp.gwdg.de)... 2001:638:60f:110::1:2, 134.76.12.6
Connecting to ftp.gwdg.de (ftp.gwdg.de)|2001:638:60f:110::1:2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22128 (22K) [application/x-redhat-package-manager]
Saving to: ‘mgr-osad-4.1.3-2.1.uyuni.noarch.rpm’

mgr-osad-4.1.3-2.1.uyuni.noarch.rpm     100%[============================================================================>]  21.61K  --.-KB/s    in 0s      

2020-08-13 17:53:57 (172 MB/s) - ‘mgr-osad-4.1.3-2.1.uyuni.noarch.rpm’ saved [22128/22128]

Getting it from the provo-mirror seems to work fine:

wget -nd https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
--2020-08-13 17:56:50--  https://provo-mirror.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/noarch/mgr-osad-4.1.3-2.1.uyuni.noarch.rpm
Resolving provo-mirror.opensuse.org (provo-mirror.opensuse.org)... 91.193.113.70
Connecting to provo-mirror.opensuse.org (provo-mirror.opensuse.org)|91.193.113.70|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22128 (22K) [application/x-redhat-package-manager]
Saving to: ‘mgr-osad-4.1.3-2.1.uyuni.noarch.rpm.1’

mgr-osad-4.1.3-2.1.uyuni.noarch.rpm.1   100%[============================================================================>]  21.61K  --.-KB/s    in 0s      

2020-08-13 17:56:51 (202 MB/s) - ‘mgr-osad-4.1.3-2.1.uyuni.noarch.rpm.1’ saved [22128/22128]

I'm not sure what the problem is currently, but in the last month, I see provo-mirror rsync'ing regularly, some 144000 times.

What is weird is that it seems to be only mirroring a subset:

Apache:/MirrorBrain/ 
Apache:/Modules/ 
Cloud// 
Cloud:// 
Kernel:// 
OBS:/Server:/Unstable/ 
devel:/ARM:// 
devel:/CaaSP// 
devel:/CaaSP:// 
devel:/languages:/python/ 
home:/lrupp/ 
openSUSE:/Backports:/ 
openSUSE:/Tools/ 
openSUSE:/infrastructure/ 
openSUSE:/infrastructure:// 
openSUSE:/infrastructure:/debuginfod/ 
server:/database:/postgresql/ 
server:/http/ 
server:/mail/ 
server:/monitoring/ 
server:/php:/applications/ 
server:/php:/extensions/ 
server:/php:/extensions:/php7/ 
server:/search/

I'll see if I have access to provo-mirror.

I don't have the access :-(

Still, we are in fact pushing out repositories to provo-mirror.

From the log:

mb -b mirrordb1 scan provo-downloadcontent.opensuse.org -d repositories/devel:/tools:/scm/SLE_12_SP5
Traceback (most recent call last):
  File "/usr/bin/mb", line 1729, in <module>
    r = mirrordoctor.main()
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 251, in main
    retval = self.postoptparse()
  File "/usr/bin/mb", line 86, in postoptparse
    debug = self.options.debug)
  File "/usr/lib64/python2.7/site-packages/mb/conn.py", line 150, in __init__
    class Version(SQLObject):
  File "/usr/lib/python2.7/site-packages/sqlobject/declarative.py", line 98, in __new__
    cls.__classinit__(cls, new_attrs)
  File "/usr/lib/python2.7/site-packages/sqlobject/main.py", line 837, in __classinit__
    sqlmeta.addColumnsFromDatabase()
  File "/usr/lib/python2.7/site-packages/sqlobject/main.py", line 476, in addColumnsFromDatabase
    for columnDef in conn.columnsFromSchema(sqlmeta.table, soClass):
  File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 386, in columnsFromSchema
    keyData = self.queryAll(keyQuery % self.sqlrepr(tableName))
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 449, in queryAll
    return self._runWithConnection(self._queryAll, s)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 340, in _runWithConnection
    conn = self.getConnection()
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 351, in getConnection
    conn = self.makeConnection()
  File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 202, in makeConnection
    ErrorMessage(e, "used connection string %r" % self.dsn))
sqlobject.dberrors.OperationalError: could not translate host name "mirrordb1.infra.opensuse.org" to address: No address associated with hostname

==== end scan ===== RC=1 ============ 20200813-164521

Do we have some sort of DNS issue ??

Getting it from the provo-mirror seems to work fine:

The problem is not getting the package. That works. But for some reason the sha256sum doesn't match:

Package bind-formula-0.1.1573049925.b509ada-1.1.uyuni.noarch (uyuni-server-stable) seems to be corrupted during transfer. Do you want to retry retrieval?

Sorry for not being more clear about.

At https://mirrors.opensuse.org/ it seems there are two mirrors in Provo with the same address for HTTP and FTP, but different for rsync (don't know why):

rsync://provo-downloadcontent.opensuse.org/repositories-scan
rsync://provo-mirror.opensuse.org/opensuse/

Could be it happens like the the other mirrors, where some are fully synced, and others are not, so the user gets the metadata from one mirror with one sha256sum, while the package in the other mirror has the same name but different sha25sum.

If one works and the other doesn't, maybe it could be removed from the mirrobrain until it's fixed.

No idea about the different rsync addresses, but provo-mirror and provo-downloadcontent are two IPs on the same server, and should serve the same files. (Exception: if the Nuremberg download.opensuse.org is down, we can enable mirrorbrain on provo-mirror, so provo-mirror will redirect instead of serving all files itsself. provo-downloadcontent will still provide direct downloads.)

provo-mirror is currently outdated (no idea why yet, in theory it has the needed cronjobs). I manually started a full sync yesterday, but it's still at /ports, so it will take some time until it reaches /repositories - maybe tomorrow, but no promises ;-)

I just did an additional sync of systemsmanagement:Uyuni:Stable which updated lots of files in the Leap 15.1 repo. Other build targets didn't need an update.

I also just started a manual rsync for all of systemsmanagement:Uyuni - the rsync output shows that (so far - rsync is still running) Master (for several build targets) and Snapshots were outdated.

If you still see files with wrong checksums in systemsmanagement:Uyuni, can you please post the full URIs?

cboltz wrote:

No idea about the different rsync addresses, but provo-mirror and provo-downloadcontent are two IPs on the same server, and should serve the same files.

provo-downloadcontent is probably only used by us to push repositories/, I don't think it ought to be publicly visible.

provo-mirror is currently outdated (no idea why yet, in theory it has the needed cronjobs). I manually started a full sync yesterday, but it's still at /ports, so it will take some time until it reaches /repositories - maybe tomorrow, but no promises ;-)

ports/ is 3.2Tb - yes, it'll take a while. repositories/ is 15Tb but might/should be more less in sync.

pjessen wrote:

Any update on this, Christian, is provo-mirror getting better?

Yes, but slowly. I still have rsync running (since weeks (!) and in a loop that auto-restarts it if needed), but it seems there is a bandwidth limit (100 MBit, maybe per IP?) that makes the sync slow. Additionally, I found a bug in rsync --compress and had to disable compression (https://bugzilla.opensuse.org/show_bug.cgi?id=1175854) which makes the sync even slower.

(If you want to check what rsync is doing right now - screen -r as user mirror.)

I don't remember, was I supposed to do something?

Nothing urgent, but maybe you should check if the push really works. Given how long rsync is already working on /repositories, I have the feeling that something might be broken.

Or OBS can compile more / faster than what a 100 MBit connection can transfer ;-)

cboltz wrote:

Nothing urgent, but maybe you should check if the push really works. Given how long rsync is already working on /repositories, I have the feeling that something might be broken.

Or OBS can compile more / faster than what a 100 MBit connection can transfer ;-)

Judging by the push logs (pontifex:/srv/bs/logs/2020/09), the push is working. In September so far, in 4497 pushes, we have sent 297'275 files and deleted 450'897.

Since the beginning of the year, we have been slowly pushing more and more, not sure why nor if it is an indicator of anything:

January    6870
February   7846
March      7175
April      9557
May       13712
June      14798
July      37866
August    36056
September  4497

Yes, it seems provo-mirror is out of sync:

https://github.com/uyuni-project/uyuni/issues/2577#issuecomment-688707267 https://github.com/uyuni-project/uyuni/issues/2577#issuecomment-688716818

Besides, it's SSL certificate is expired since Sept 4th.

Since I didn't see a report for that, I created one: https://progress.opensuse.org/issues/71104

It seems widehat.opensuse.org has problems as well. I got this from one of our developers:

download.suse.org serves the right repomd.xml:

curl
http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stabl
e/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml |
sha256sum
% Total % Received % Xferd Average Speed Time Time Time
Current Dload Upload Total Spent Left Speed 100 11739 100 11739

0 0 197k 0 --:--:-- --:--:-- --:--:-- 197k
d943d24afba948da5971a02cb4dc68aad5beabfbcea448383aaaecc5a2eadfa2 -

widehat.suse.org serves a different one:

curl
http://widehat.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable
/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml |
sha256sum
% Total % Received % Xferd Average Speed Time Time Time
Current Dload Upload Total Spent Left Speed 100 11738 100 11738

0 0 143k 0 --:--:-- --:--:-- --:--:-- 143k
d0775ed7ac220563f2f3b13b7212d6266aef27a3f926597e1728783333549c29 -

and, we do get redirects from download.suse.org to widehat.suse.org for individual RPMs, eg:

curl
https://download.opensuse.org/pub/opensuse/repositories/systemsmanagement
:/Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/noarch/bind-fo
rmula-0.1.1573049925.b509ada-1.1.uyuni.noarch.rpm
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

302 Found

Found
The document has moved here.
Apache/2.4.43 (Linux/SUSE) Server at download.opensuse.org Port
443

So ultimately checksums on the package do not match:

curl
http://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stabl
e/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/repomd.xml | grep
primary.xml.gz
% Total % Received % Xferd Average Speed Time Time Time
Current Dload Upload Total Spent Left Speed 0 0 0 0

0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11739 100 11739
0 0 148k 0 --:--:-- --:--:-- --:--:-- 148k

<location

href="repodata/85389bcf181674574d3a5e494ffc30bd5f2269e50fe5f5d73c6e19894d32
2ccd-primary.xml.gz"/>

curl
http://widehat.opensuse.org/pub/opensuse/repositories/systemsmanagement:/
Uyuni:/Stable/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/85389bc
f181674574d3a5e494ffc30bd5f2269e50fe5f5d73c6e19894d322ccd-primary.xml.gz

404 Not Found

404 Not Found
nginx

The file exists on other mirrors, eg.

curl
https://ftp.gwdg.de/pub/opensuse/repositories/systemsmanagement:/Uyuni:/S
table/images/repo/Uyuni-Server-POOL-x86_64-Media1/repodata/85389bcf1816745
74d3a5e494ffc30bd5f2269e50fe5f5d73c6e19894d
322ccd-primary.xml.gz | zless

It does look like widehat is quite out-of-sync, I have no idea why. Since yesterday, I've been running a manual rsync to try to catch up.

pjessen wrote:

It does look like widehat is quite out-of-sync, I have no idea why. Since yesterday, I've been running a manual rsync to try to catch up.

Well, one issue is also user:group mapping - on pontifex we use mirror:stage (1002:192), but on widehat we use mirror:mirror (1000:1000). I think this may have caused problems in the push of repositories, at the very least.

From our side this can be closed. We did not have more issues for now.