openSUSE admin

a) the listing for leap/15.1/repo/oss/x86_64/ is around 8Mb.
b) the listing for leap/15.2/repo/oss/x86_64/ is much longer (47Mb) because we have extra versions of each package.

(b) is clearly the real problem, but the situation could be improved a lot by just compressing the output. The 15.1 listing could be reduced to ~500K, the 15.2 listing to ~2Mb.

Well, I thought that was going to be easy. I have added the following to the download macro - I have also tested that it takes, but I'm not getting any compressed output.

<directory /srv/ftp/pub/opensuse>
.
.
<ifModule mod_deflate.c>
Addoutputfilterbytype  deflate text/html text/plain text/xml text/css
</ifmodule>

</directory>

I have also tried some variations with SetOutputFilter and AddOutputfilter, without success.

New styles implemented on d.o.o. http://download.opensuse.org/distribution/leap/15.2/repo/oss/x86_64/ saved as 8,796,400 bytes, a big decrease in size compared to 22 Jan. It loaded in about 30 seconds in SeaMonkey 2.49.5, much improved.

The double-spaced line items are most unwelcome though. :(

Loading http://download.opensuse.org/distribution/leap/15.3/repo/oss/x86_64/ in SeaMonkey minutes ago pegged CPU for about 50 seconds. Afterward, loading http://download.opensuse.org/distribution/leap/15.4/repo/oss/x86_64/ in SeaMonkey pegged CPU for about 250 seconds. Saved with wget it was 38,660,984 bytes. http://ftp5.gwdg.de/pub/opensuse/repositories/home:/, which loads in SM too quickly to measure, was a mere 1,430,655, bytes, 3.7% of D.O.O. OTOH, https://download.opensuse.org/repositories/home:/ loaded in about 30 seconds, much faster than I remember in previous times. By pegged CPU I mean both literally, and SM being completely non-responsive. Most mirror pages used directly use different markup and CSS, and load much faster than those showing as d.o.o. Solved? No way 4 minute load time or 38MB page size is OK for a list of package URIs.

The problem isn't how many files are there. The problem is page size 27 times as big as it needs to be to do what needs doing (1 / .037). That doesn't even count the ridiculously overabundant 249,218 bytes of CSS to download and process. Works good should be the top priority, not looks good. They're just lists of URIs. https://archives.fedoraproject.org/ package list pages somehow manage with no CSS at all, as does mirrors.kernel.org for Mageia Cauldron x86_64, which loads its 28,759 line 3,887,437 x86_64/media/core/release/ bytes too quickly to measure.

mrmazda wrote:

The problem isn't how many files are there. The problem is page size 27 times as big as it needs to be to do what needs doing (1 / .037). That doesn't even count the ridiculously overabundant 249,218 bytes of CSS to download and process. Works good should be the top priority, not looks good. They're just lists of URIs. https://archives.fedoraproject.org/ package list pages somehow manage with no CSS at all, as does mirrors.kernel.org for Mageia Cauldron x86_64, which loads its 28,759 line 3,887,437 x86_64/media/core/release/ bytes too quickly to measure.

Hi, I've been reassigned to this ticket because the UI will probably change to what is currently rendered with http://mirrorchache.opensuse.org/distribution/leap/15.3/repo/oss/x86_64/
(Please note that you may receive a bit delayed reply from mirrorcache.o.o because it currently doesn't have enough hardware resources behind it. The goal is that download.o.o should be used instead).

While overall your concern is valid, below are important notes to consider:

• currently 15.4 has too many outdated files. Normally download.o.o provides only the latest version of a package, sometimes the previous version as well. 15.4 has several versions for each package, which is not intentional, because we just don't have enough disk space to provide that in general case. So this was noticed recently and should be addressed soon.
• In my tests mirrorcache shows a bit better size than current one for 15.3 - 7.0 Mb vs 9.0 Mb
• We must render .metalink , (which alsmost doubles the size) so you cannot simply compare to other websites with listings like ftp5.gwdg.de
• 250K of CSS shouldn't be a problem if you render few MB of page.
• It worth to notice that it will be an option to load data in json file. That is usable for using in other tools , e.g. http://mirrorcache.opensuse.org/distribution/leap/15.3/repo/oss/x86_64/?json (2.1 Mb instead of 7.3 Mb).
• In my understanding proper solution would be for huge folders to provide an alternative for rendering a paged view with filter, something like in OBS. E.g. it only shows 25 packages out of 14K https://build.opensuse.org/project/show/openSUSE:Factory . We have something like this in mind, just it waits its priority. Feel free to contribute to the project if you have chance.

If you wish, I can create a request regarding paged view at https://github.com/openSUSE/MirrorCache and mark this call as waiting for implementation in MirrorCache.
Or let me know if you need any better ideas.