openSUSE admin

Regarding the potential workaround of using http:, my repository is configured as http:, and I'm also having trouble, albeit with slightly different symptoms. Just today, I've gotten several errors of the following form from a "zypper dup":

Retrieving: gnome-shell-classic-41.1-1.1.noarch.rpm
...........[error (974 B/s)]
Download (curl) error for
'http://download.opensuse.org/ports/aarch64/tumbleweed/repo/oss/noarch/gnome-shell-classic-41.1-1.1.noarch.rpm':

Error code: Curl error 55
Error message: Connection died, tried 5 times before giving up

Abort, retry, ignore? [a/r/i/...? shows all options] (a): r

It recovers after I respond "r" to its question about what to do.

I wonder if you use mirrorcache.opensuse.org with http or with https - do you see the errors?

The problem is also discussed here https://github.com/openSUSE/zypper/issues/399#issuecomment-998678390

I've reverted some config changes on download.opensuse.org from last Thursday, (when the issue triggered with high frequency).
My guess is that both errors 16 and 55 were related to that change and should gone now (or at least do not occur with the same frequency), but let me know if it you still see it (often).

dhwalker wrote:

Regarding the potential workaround of using http:, my repository is configured as http:, and I'm also having trouble, albeit with slightly different symptoms. Just today, I've gotten several errors of the following form from a "zypper dup":

Retrieving: gnome-shell-classic-41.1-1.1.noarch.rpm
...........[error (974 B/s)]
Download (curl) error for
'http://download.opensuse.org/ports/aarch64/tumbleweed/repo/oss/noarch/gnome-shell-classic-41.1-1.1.noarch.rpm':

For some reason, mirrorbrain thinks that file is not mirrored, so you're given a direct feed from download.o.o - depending on your location, the quality of that will vary. However, I do see that file at e.g. ftp.gwdg.de/opensuse, so why mirrorbrain does not see it there .....

pjessen wrote:

dhwalker wrote:

Regarding the potential workaround of using http:, my repository is configured as http:, and I'm also having trouble, albeit with slightly different symptoms. Just today, I've gotten several errors of the following form from a "zypper dup":

Retrieving: gnome-shell-classic-41.1-1.1.noarch.rpm
...........[error (974 B/s)]
Download (curl) error for
'http://download.opensuse.org/ports/aarch64/tumbleweed/repo/oss/noarch/gnome-shell-classic-41.1-1.1.noarch.rpm':

For some reason, mirrorbrain thinks that file is not mirrored, so you're given a direct feed from download.o.o - depending on your location, the quality of that will vary. However, I do see that file at e.g. ftp.gwdg.de/opensuse, so why mirrorbrain does not see it there .....

"some reason" being: that file has yet to be mirrored, i.e. the rsync has yet to catch up.

The same goes for many others, the newer they are. I see files changed on 16 Dec still having no mirrors. Older files/packages have 2-15 mirrors.

Update to TW20211220 went through without any hiccups. Thx.

That particular file has at least made it out to one mirror, http://ftp.lysator.liu.se/pub/opensuse, but it looks to me as if download.o.o remains under heavy load. I'll open a separate issue on that.

fkrueger wrote:

Update to TW20211220 went through without any hiccups. Thx.

Unfortunately, with today's update to TW20211221 the "curl error 16" is back.

pjessen wrote:

That particular file has at least made it out to one mirror, http://ftp.lysator.liu.se/pub/opensuse, but it looks to me as if download.o.o remains under heavy load. I'll open a separate issue on that.

It may be due to those curl errors. You people in Europe must be running into this issue now but I'm getting them since October (I should've reported it). I'm located in Brazil.

They started shy, just every now and then, but now in Dec they started to appear more regularly and I've seen quite some people on openSUSE Support channel complaining about them too. I, myself, have some share of blame on my end since I edited my repos to use https, and from my understanding mirrorbrain+https are not in good terms with each other.

Yesterday, on a Tumbleweed VM, I tried zypper ref and every official repo gave me that curl error so download.o.o must be under heavy load, at least partially, because of this, people try to update/upgrade their system and are running into this problem so they try and try again.

The sad thing is that we're not even sure what's causing those curl errors right now, as seen in this zypper issue https://github.com/openSUSE/zypper/issues/399.

luc14n0 wrote:

pjessen wrote:

That particular file has at least made it out to one mirror, http://ftp.lysator.liu.se/pub/opensuse, but it looks to me as if download.o.o remains under heavy load. I'll open a separate issue on that.

It may be due to those curl errors. You people in Europe must be running into this issue now but I'm getting them since October (I should've reported it). I'm located in Brazil.

All of South America have five mirrors, none of which carry ports/ - I guess the next option would be North America but again no mirrors that have ports/ :-(

They started shy, just every now and then, but now in Dec they started to appear more regularly and I've seen quite some people on openSUSE Support channel complaining about them too. I, myself, have some share of blame on my end since I edited my repos to use https, and from my understanding mirrorbrain+https are not in good terms with each other.

https requests are all being redirected to mirrorcache, but it won't make any difference in this situation.

pjessen wrote:

All of South America have five mirrors, none of which carry ports/ - I guess the next option would be North America but again no mirrors that have ports/ :-(
...
https requests are all being redirected to mirrorcache, but it won't make any difference in this situation.

Would you care to explain to me your reference to ports/, please? How that would help me in this situation?

Thanks in advance.

andriinikitin wrote:

This was discussed in several places and we don't have fully reliable solution to it yet.
But the situation should have improved since the issue was reported. Could you confirm if you still see such errors or they are (almost) gone?
If you can collect any related statistics or describe your personal view on the problem over last month or two - it might help as well.

Regards,
Andrii Nikitin

I haven't encountered any problems related to the current issue for a long time, so feel free to close it. Works fine for me. Thx.

andriinikitin wrote:

But the situation should have improved since the issue was reported.

Some improvements were achieved in the meantime.

But I do get another questionable impression from my attempt to load over 9200 software packages for a running distribution upgrade.

Involved components:

• zypper 1.14.55-1.3
• curl 7.84.0-1.2

Could you confirm if you still see such errors or …?

Example:

Retrieving: xrefresh-debuginfo-1.0.7-2.2.x86_64.rpm ................................................................................................................[error]
Download (curl) error for 'https://download.opensuse.org/debug/tumbleweed/repo/oss/x86_64/xrefresh-debuginfo-1.0.7-2.2.x86_64.rpm':
Error code: Curl error 16
Error message: 

Abort, retry, ignore? [a/r/i/...? shows all options] (a): r
Retrieving: xrefresh-debuginfo-1.0.7-2.2.x86_64.rpm ................................................................................................................[error]
Timeout exceeded when accessing 'https://download.opensuse.org/debug/tumbleweed/repo/oss/x86_64/xrefresh-debuginfo-1.0.7-2.2.x86_64.rpm'.
Abort, retry, ignore? [a/r/i] (r): 
Trying again...
Retrieving: xrefresh-debuginfo-1.0.7-2.2.x86_64.rpm ................................................................................................................[error]
Download (curl) error for 'https://download.opensuse.org/debug/tumbleweed/repo/oss/x86_64/xrefresh-debuginfo-1.0.7-2.2.x86_64.rpm':
Error code: Curl error 92
Error message: HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

Abort, retry, ignore? [a/r/i/...? shows all options] (a): 
Problem occurred during or after installation or removal of packages:
Installation has been aborted as directed.
Please see the above error message for a hint.

Today's update to TW20220831 with almost 2,600 packages is really a pain. In fact, after almost every 5th package download stops with the error message

Fehler beim Download (curl) für 'https://download.opensuse.org/tumbleweed/repo/oss/noarch/ibus-dict-emoji-1.5.27-2.1.noarch.rpm':
Fehlercode: 'Curl error 16'
Fehlermeldung:

So no improvements at all. Awful!

error 16? here it's 55.

Retrieving: libvpx7-32bit-1.12.0-1.2.x86_64.rpm ................................................................[error]
Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/libvpx7-32bit-1.12.0-1.2.x86_64.rpm':
Error code: Curl error 55
Error message: Connection died, tried 5 times before giving up

Abort, retry, ignore? [a/r/i/...? shows all options] (a): r

ping -A download.opensuse.org
...
64 bytes from download.opensuse.org (2001:67c:2178:8::13): icmp_seq=313 ttl=54 time=49.1 ms
...
^C64 bytes from 2001:67c:2178:8::13: icmp_seq=314 ttl=54 time=45.6 ms

--- download.opensuse.org ping statistics ---
314 packets transmitted, 196 received, 37.5796% packet loss, time 18319ms
rtt min/avg/max/mdev = 31.286/55.767/105.103/14.524 ms, pipe 2, ipg/ewma 58.527/52.613 ms

which could also explain the slow connection establishment speed (which makes a 2600 package download take longer than normal).

jengelh wrote:

error 16? here it's 55.
error 16 and 92 :-)

jengelh wrote:

ping -A download.opensuse.org
...
64 bytes from download.opensuse.org (2001:67c:2178:8::13): icmp_seq=313 ttl=54 time=49.1 ms
...
^C64 bytes from 2001:67c:2178:8::13: icmp_seq=314 ttl=54 time=45.6 ms

--- download.opensuse.org ping statistics ---
314 packets transmitted, 196 received, 37.5796% packet loss, time 18319ms
rtt min/avg/max/mdev = 31.286/55.767/105.103/14.524 ms, pipe 2, ipg/ewma 58.527/52.613 ms

which could also explain the slow connection establishment speed (which makes a 2600 package download take longer than normal).

FWIW - for me, in Zurich, I see about 50ms pings on ipv4, even less on ipv6. Some packets being dropped.

fkrueger wrote:

Today's update to TW20220831 with almost 2,600 packages is really a pain. In fact, after almost every 5th package download stops with the error message

Fehler beim Download (curl) für 'https://download.opensuse.org/tumbleweed/repo/oss/noarch/ibus-dict-emoji-1.5.27-2.1.noarch.rpm':
Fehlercode: 'Curl error 16'
Fehlermeldung:

So no improvements at all. Awful!

Saying that is a bit too harsh, you're depreciating the work of many people who are trying and are improving things.

Someone correct me if I'm wrong here but it seems this is happening because there were some complications during yesterday's (Thursday) maintenance where all openSUSE's VMs were recreated. However, that only aggravated things that weren't working as smooth as they could. The root cause appears to be Rsync not being able to keep up with the amount of changes that OBS generates so it's a bottleneck.

luc14n0 wrote:

fkrueger wrote:

Today's update to TW20220831 with almost 2,600 packages is really a pain. In fact, after almost every 5th package download stops with the error message

Fehler beim Download (curl) für 'https://download.opensuse.org/tumbleweed/repo/oss/noarch/ibus-dict-emoji-1.5.27-2.1.noarch.rpm':
Fehlercode: 'Curl error 16'
Fehlermeldung:

So no improvements at all. Awful!

you're depreciating the work of many people who are trying and are improving things.

Your remark is insinuating! I always apprecitate the work done here. However, "curl error 16" (or any other figure) is still alive.

JFYI: It took 2.5 hours to update to TW20220831 given a 1 Gbit/s connection.

fkrueger wrote:

JFYI: It took 2.5 hours to update to TW20220831 given a 1 Gbit/s connection.

@luc14n0: Any idea/suggestion/solution?

luc14n0 wrote:

The root cause appears to be Rsync not being able to keep up with the amount of changes that OBS generates so it's a bottleneck.

That does cause problems for synch'ing repositories/, but it should not really affect the distribution (leap, tumbleweed) as such. Of course, any major tw update will take time to distribute, but that's life.

fkrueger wrote:

Your remark is insinuating! I always apprecitate the work done here. However, "curl error 16" (or any other figure) is still alive.

Sorry about that, I should've chosen my words more carefully. What I meant was that the words you used made it look like depreciation.

fkrueger wrote:

JFYI: It took 2.5 hours to update to TW20220831 given a 1 Gbit/s connection.

Sadly having a high speed and reliable connection is only half of the puzzle. I never see Zypper downloading packages at meaningful high speeds, maybe it has something to do with this default in our /etc/zypp/zypp.conf files:

##
## Sets the minimum download speed (bytes per second)
## until the connection is dropped
## This can be useful to prevent security attacks on hosts by
## providing updates at very low speeds.
##
## 0 means no limit
##
# download.min_download_speed = 0

I never played with this before so I can't say whether or not Zypper honors it. But in my mind (I'm probably wrong), from my personal experience watching dups, if by default Zypper allows very low speeds it means that for many people Zypper will be "dragging its feet" when downloading packages. If many connections are "dragging their feet" to download packages rather than using some more sensible speed, this doesn't sound good for me because the longer it takes for packages to download the more are the chances of problems arising. I hope someone can provide some insights on this matter for me.

fkrueger wrote:

fkrueger wrote:

JFYI: It took 2.5 hours to update to TW20220831 given a 1 Gbit/s connection.

@luc14n0: Any idea/suggestion/solution?

My copping mechanism has been splitting the distribution upgrade in two major steps. In step one I leave Zypper downloading packages in "the background", so when it's appropriate I can do step two, which is the real zypper dup, without major headaches.

You could make use of a Bash/Zsh/Fish alias like zdown='zypper dup --download-only --no-confirm --force-resolution' and while you are going about doing your things Zypper will be downloading packages, saying "yes" to any yes/no questions and, if any conflict arises, it forces its way through (for --download-only it doesn't matter what is the answer for conflicts, as long as it keeps downloading packages, when you dup for real you can properly manage those conflicts).

Other option would be to use DNF. At the moment DNF is smarter than Zypper regarding to make new attempts when it bump into some issue (like those Curl ones) that prevents it from downloading stuff. But depending whether or not you use development repos which generates some kinds of conflicts from time to time, DNF won't be able to do it's distro-sync (equivalent to Zypper's dist-upgrade, a.k.a. dup). In general, for upgrading to a new TW snapshot, DNF can do the job even faster than Zypper because of download concurrency.

pjessen wrote:

That does cause problems for synch'ing repositories/, but it should not really affect the distribution (leap, tumbleweed) as such. Of course, any major tw update will take time to distribute, but that's life.

Oh, really?! I never managed a mirror before so excuse my ignorance. In mind I kind of thought (about this issue many of us are facing) that if mirrors can't keep up with updates, for whatever reason be it the source can't provide it fast enough or the mirror itself can't download fast enough or whatnot. If they can't keep up and are most of the time too busy trying to keep up with the content users who get redirected to those particularly "busy" mirrors are not going to have a good time.

luc14n0 wrote:

In mind I kind of thought (about this issue many of us are facing) that if mirrors can't keep up with updates, for whatever reason be it the source can't provide it fast enough or the mirror itself can't download fast enough or whatnot. If they can't keep up and are most of the time too busy trying to keep up with the content users who get redirected to those particularly "busy" mirrors are not going to have a good time.

It is primarily an issue with the build service repositories, but a large tumbleweed update (300Gb+) will also take time to propagate.
Users should not be given any out-of-date mirror, so mirrors currently updating will not receive much user traffic.

pjessen wrote:

luc14n0 wrote:
It is primarily an issue with the build service repositories, but a large tumbleweed update (300Gb+) will also take time to propagate.
Users should not be given any out-of-date mirror, so mirrors currently updating will not receive much user traffic.

OK, thanks for the insight. So the pool of available mirrors shrink in such conditions huh.