openSUSE admin

I did some investigation today.

When I run:

curl http://cache.opensuse.net.br/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz.metalink

What I get is:

<url type="http" location="BR" preference="100">http://download.opensuse.net.br/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="BR" preference="99">http://mirror.uepg.br/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<!-- File origin location:  -->
<url type="http" location="" preference="98">http://mirrorcache-us.opensuse.org/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>

In this case, if the Brazilian mirrorcache has no file, it sends to US.
If I query mirrorcache-us, it finds the file in Provo.

The problem is when I query opensuse.org, when I run:

curl http://download.opensuse.org/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz.metalink

What I get is:

<resources>
  <!-- Mirrors which handle this country (BR):  -->
  <url type="http" location="BR" preference="100">http://mirror.uepg.br/opensuse//update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="BR" preference="99">http://download.opensuse.net.br//update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <!-- Mirrors in the same continent (SA):  -->
  <!-- Mirrors in other parts of the world:  -->
  <url type="http" location="UA" preference="98">http://fastmirror.pp.ua/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="UA" preference="97">http://opensuse.ip-connect.info/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="PT" preference="96">http://ftp.rnl.tecnico.ulisboa.pt/pub/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="FR" preference="95">http://fr2.rpmfind.net/linux/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="GB" preference="94">http://mirrors.coreix.net/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="CH" preference="93">http://opensusemirror.lihaso.com//update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="CH" preference="92">http://pkg.adfinis.com/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="CH" preference="91">http://mirror.hostsuisse.com/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="DE" preference="90">http://mirror1.hs-esslingen.de/pub/Mirrors/ftp.opensuse.org/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <url type="http" location="NL" preference="89">http://mirror.nl.leaseweb.net/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
  <!-- File origin location:  -->
  <url type="http" location="" preference="88">http://downloadcontent2.opensuse.org/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
</resources>

download.o.o does not know that it should send South America requests to US, or rather, send to a closer place, but instead, it sends to whatever place it finds in Europe.

Things get even worse for two reasons:

1. Not many mirrors sync 15.3 in LATAM because it's EOL and takes a lot of space, and worse, they may exclude SLE repo because it's huge in 15.3 (~200 GB)
2. Mirrorcache sync of what is mirrored is slow, it took a while to show our own mirror (download.opensuse.net.br) which had the file, this can cause the metadata to be out of sync

In my understanding, when LATAM requests download.o.o it should send that request to Brazil mirrorcache (cache.opensuse.net.br), but that does not seem to be the case.
If that would be the case, it would fetch the file from Provo, which is close to South America, but since the response for the request is coming from mirrorcache in EU, it sends it to a random place in Europe that has the file.

There are two solutions I think:

1. download.opensuse.org should return responses from cache.opensuse.net.br if the request comes from South America
2. if 1 is not possible, we need to re-add all mirrors from SA into the cache in Europe, this way, I think EU mirrorcache would be able to send it back to Brazil

Andrii, I hope that this helps to clarify the problem.

avicenzi wrote:

download.o.o does not know that it should send South America requests to US, or rather, send to a closer place, but instead, it sends to whatever place it finds in Europe.

IIRC, with mirrorbrain, when a file was not found on any mirror on the same continent, any mirror would be good enough, but ordered by network "proximity", ie. ASN.

pjessen wrote:

avicenzi wrote:

download.o.o does not know that it should send South America requests to US, or rather, send to a closer place, but instead, it sends to whatever place it finds in Europe.

IIRC, with mirrorbrain, when a file was not found on any mirror on the same continent, any mirror would be good enough, but ordered by network "proximity", ie. ASN.

In this case, it seems that the problem is with download.o.o, since SA has its own mirrorcache now.
Querying SA mirrorcache returns the file if available, or it sends to Provo if not.

Maybe this is related to #115136, the SA mirror does not have an opensuse.org URL yet.