Project

General

Profile

Actions
Copy link

tickets #108656

closed

Super slow speed on Leap repos specifically /backports and update/sle repository downloads

Added by lkocman about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
andriinikitin
Category:
Mirrors
Target version:
-
Start date:
2022-03-21
Due date:
% Done:

0%

Estimated time:

Description

Hello openSUSE admins,

I'd like to share consern regarding consuming updates for Leap. The
backports and SLE directories have very poor donwload speeds. Check out
oss vs backports and slee (MiB/s vs KiB/s). Keep in mind that Leap
system needs to have all of these repositories enabled. The SLE repo
itself takes about two minutes to fetch just the repodata.

Reporter (in cc) is from CZ and I can confirm this (althought I
normally use my private mirror).

Is this something we can affect or shall this go directly to mirror@?

(from zypper -vv after clean--all)

Retrieving:
http://download.opensuse.org/update/leap/15.4/backports/repodata/ff71bd55d6b805c32471bbe115dc3cbbd3622e57f2575e31d0721962dcea8eb9-primary.xml.gz
[....done (10.1 KiB/s)]
Retrieving:

http://download.opensuse.org/distribution/leap/15.4/repo/oss/repodata/9c6ff3ef1b3eb5cc07c3d3cd926d0cdf73e579f09edf6c37d814d6c6a9c18d3e-primary.xml.gz
[......done (10.1 MiB/s)]

Retrieving:
http://download.opensuse.org/update/leap/15.4/sle/repodata/a0fa5255fb09df27d83d53a92977e44b7266c9b22e3adbcb40a1e7da34d75b19-primary.xml.gz
[......................................................................
........................................done (110.8 KiB/s)]

Thank you

--
Best regards

Lubos Kocman
openSUSE Leap Release Manager

Actions
Copy link
 #1

Updated by pjessen about 2 years ago

  • Category set to Mirrors
  • Status changed from New to Feedback
  • Assignee set to pjessen
  • Private changed from Yes to No

Initial observation - there is no reason why any repo or directories should be slower than any other. However, mirrors of those might differ a lot.
To enable me to tell which mirrors are being allocated/proposed, please provide the IP address of the client.

Actions
Copy link
 #2

Updated by pjessen about 2 years ago

lkocman wrote:

Retrieving:
http://download.opensuse.org/update/leap/15.4/backports/repodata/ff71bd55d6b805c32471bbe115dc3cbbd3622e57f2575e31d0721962dcea8eb9-primary.xml.gz
[....done (10.1 KiB/s)]

That particular file is only 1797bytes, so maybe not a good one for measuring download speed. Still, download speed in Germany from mirror.braun-software-solutions.de is reported as 121Mb/sec :-)

Retrieving:

http://download.opensuse.org/distribution/leap/15.4/repo/oss/repodata/9c6ff3ef1b3eb5cc07c3d3cd926d0cdf73e579f09edf6c37d814d6c6a9c18d3e-primary.xml.gz
[......done (10.1 MiB/s)]

Hmm, I'm getting a 404 right now, but surely 10Mb/sec is okay.

Retrieving:
http://download.opensuse.org/update/leap/15.4/sle/repodata/a0fa5255fb09df27d83d53a92977e44b7266c9b22e3adbcb40a1e7da34d75b19-primary.xml.gz
[......................................................................
........................................done (110.8 KiB/s)]

Also a 404, but I tried http://download.opensuse.org/update/leap/15.4/sle/repodata/c6933fa0c741bc6a36b5810d349faf918c89fbbf503ba3be971bdf2f435cb003-primary.xml.gz instead - downloading from garr.it at 1.5Mb/sec

Actions
Copy link
 #3

Updated by msvec about 2 years ago

FTR, the download speed is super-slow again.

wget http://download.opensuse.org/update/leap/15.4/sle/repodata/83c1a43da05d5115eccbc5c4c7dd7bad7808fa3627d0fae4ad4aeff47d6d1380-primary.xml.gz

wget gives 212KB/s, but zypper gets stuck on this forever:
"Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata "

Michal

Actions
Copy link
 #4

Updated by pjessen about 2 years ago

msvec wrote:

FTR, the download speed is super-slow again.

wget http://download.opensuse.org/update/leap/15.4/sle/repodata/83c1a43da05d5115eccbc5c4c7dd7bad7808fa3627d0fae4ad4aeff47d6d1380-primary.xml.gz

wget gives 212KB/s, but zypper gets stuck on this forever:
"Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata "

Michal

In order to do anything at all about this, I really have to know from where, i.e from which client address. Alternatively, post which exact mirror it is that is being slow.

Regardless, it is unlikely we can do anything, it is a matter been the client and the mirror. For myself, I get 18Mb/s for that file, from downloadcontent.o.o.

Actions
Copy link
 #6

Updated by pjessen about 2 years ago

  • Status changed from Feedback to New
wget -vvv http://downloadcontent.opensuse.org/update/leap/15.3/sle/repodata/fd627cc03e1dc5092c31ad2cd102988dc530e41bedfc4588681dfe707b473c43-primary.xml.gz
--2022-03-26 10:46:14--  http://downloadcontent.opensuse.org/update/leap/15.3/sle/repodata/fd627cc03e1dc5092c31ad2cd102988dc530e41bedfc4588681dfe707b473c43-primary.xml.gz
Resolving downloadcontent.opensuse.org (downloadcontent.opensuse.org)... 2001:67c:2178:8::27
Connecting to downloadcontent.opensuse.org (downloadcontent.opensuse.org)|2001:67c:2178:8::27|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 74688428 (71M) [application/octet-stream]
Saving to: ‘fd627cc03e1dc5092c31ad2cd102988dc530e41bedfc4588681dfe707b473c43-primary.xml.gz’
fd627cc03e1dc5092c31ad2cd102988dc530e41 100%[============================================================================>]  71.23M  22.0MB/s    in 3.3s    
2022-03-26 10:46:17 (21.3 MB/s) - ‘fd627cc03e1dc5092c31ad2cd102988dc530e41bedfc4588681dfe707b473c43-primary.xml.gz’ saved [74688428/74688428]
ml.gz’ saved [74688428/74688428]

Currently running at ~150-200KB/s

It is possible that pontifex was very heavily loaded when you tried this yesterday, but it is not easy to say. From my download above, it is clear that pontifex has the ability to serve 22MB/sec. I tried another client machine, and got almost 50Mb/sec.

Actions
Copy link
 #7

Updated by pjessen about 2 years ago

I have no idea if this might be involved, but I'll add it anyway. For rsync.o.o, updates are being pushed to us, but for some reason, backports/ and sle/ are not being pushed. See #100506. Regardless, that is only one mirror, but we simply cannot get updates/ sync'ed so quickly, so sometimes downloads have to revert to downloadcontent.o.o.

Actions
Copy link
 #9

Updated by pjessen about 2 years ago

msvec wrote:

Another case just now:

wget https://rsync.opensuse.org/update/leap/15.3/sle/repodata/b140e87f2402a357f3ee5b2fa134d24eef08383f902721511f22f81e63d0c1fe-primary.xml.gz
Currently running at 300KB/s

I have no idea how you are accessing that - on rsync.o.o, I have three requests only: from 192.168.87.1 (??), from a German address and from my own IPv6 address. Time-wise, the only one that fits is 192.168.87.1.

Actions
Copy link
 #11

Updated by andriinikitin about 2 years ago

I am evaluating (almost) complete switch from MirrorBrain to MirrorCache today, so it may work better (or worse).

Actions
Copy link
 #12

Updated by pjessen about 2 years ago

andriinikitin wrote:

I am evaluating (almost) complete switch from MirrorBrain to MirrorCache today, so it may work better (or worse).

I wonder how that is going to affect the download speed from rsync.o.o :-)

Actions
Copy link
 #13

Updated by pjessen about 2 years ago

  • Assignee changed from pjessen to andriinikitin
Actions
Copy link
 #15

Updated by andriinikitin about 2 years ago

pjessen wrote:

I wonder how that is going to affect the download speed from rsync.o.o :-)

So I'd say 3 problems can be considered in this ticket:

  1. download speed from rsync.o.o
  2. slowness (of zypper) when downloading *primary.xml.gz from http://download.opensuse.org/update/leap/15.3/sle/repodata/
  3. the fact that http://download.opensuse.org/update/leap/15.3/sle/repodata/ is released several times per day, requiring every user to download (and for zypper to process it) ~90M in almost every zypper refresh command.

I'd say that it is unlikely that we can do something about 3. in short run, but imho the community definitely should escalate it and maybe challenge release managers(?). (Although I heard that zypper team is working on allowing download of delta for -primary.xml.gz instead of full download, which definitely can address this problem, unless I am mistaken here).

Regarding 2. : Today I enabled MirrorCache for all .rpm and .xml.gz requests and my testing shows good speed - can anybody confirm it?

Actions
Copy link
 #16

Updated by andriinikitin almost 2 years ago

  • Status changed from New to Resolved

It looks this is resolved now , can it be closed?

Actions
Copy link

Also available in: Atom PDF