tickets #114953
closedRequests from LATAM being redirected to Ukrainian mirror?
100%
Description
While mirroring the SLE directory from download.org, I noticed that quite often it is being redirected to fastmirror.pp.ua (Ukraine) while I'm located in Brazil(!).
Is this correct?
GET http://download.opensuse.org/update/leap/15.3/sle/repodata/d7dbbdf8a3c5ae7c7b1df0393f8625ac2922d294a38d565344a3482456d6aff4-other.xml.gz HTTP/1.1
Host: download.opensuse.org
Proxy-Authorization: Basic *************
Accept: /
Accept-Encoding: gzip
Proxy-Connection: Keep-Alive
User-Agent: RMT/2.6.5
...
HTTP/1.1 302 Found
Date: Wed, 03 Aug 2022 18:49:21 GMT
Server: Mojolicious (Perl)
Expires: Wed, 03 Aug 2022 18:49:21 GMT
Location: http://fastmirror.pp.ua/opensuse/update/leap/15.3/sle/repodata/d7dbbdf8a3c5ae7c7b1df0393f8625ac2922d294a38d565344a3482456d6aff4-other.xml.gz
Content-Type: application/x-gzip
Cache-Control: must-revalidate
Content-Length: 0
X-Cache: MISS from pxl0squid103
X-Cache-Lookup: MISS from pxl0squid103:4001
Via: 1.1 pxl0squid103 (squid/3.5.20)
Connection: keep-alive
Connection #2 to host 127.0.0.1 left intact
Issue another request to this URL: 'http://fastmirror.pp.ua/opensuse/update/leap/15.3/sle/repodata/d7dbbdf8a3c5ae7c7b1df0393f8625ac2922d294a38d565344a3482456d6aff4-other.xml.gz'
Uses proxy env variable no_proxy == 'localhost,127.0.0.1,.bb,.bancobrasil.com.br,.bb.com.br'
Found bundle for host fastmirror.pp.ua: 0x5569a1b3e8e0 [can pipeline]
Re-using existing connection! (#2) with proxy 127.0.0.1
Connected to 127.0.0.1 (127.0.0.1) port 40000 (#2)
Proxy auth using Basic with user 'c1210271'
GET http://fastmirror.pp.ua/opensuse/update/leap/15.3/sle/repodata/d7dbbdf8a3c5ae7c7b1df0393f8625ac2922d294a38d565344a3482456d6aff4-other.xml.gz HTTP/1.1
Host: fastmirror.pp.ua
Proxy-Authorization: Basic ***********
Accept: /
Accept-Encoding: gzip
Proxy-Connection: Keep-Alive
User-Agent: RMT/2.6.5
HTTP/1.1 403 Forbidden
Date: Wed, 03 Aug 2022 18:49:21 GMT
Content-Type: text/html
Cache-Control: no-cache
Content-Length: 3037
X-Frame-Options: deny
X-Cache: MISS from pxl0squid103
X-Cache-Lookup: MISS from pxl0squid103:4001
Via: 1.1 pxl0squid103 (squid/3.5.20)
Connection: keep-alive
Updated by cboltz over 1 year ago
- Category set to Mirrors
- Assignee set to andriinikitin
Updated by avicenzi over 1 year ago
I did some investigation today.
When I run:
What I get is:
<url type="http" location="BR" preference="100">http://download.opensuse.net.br/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="BR" preference="99">http://mirror.uepg.br/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<!-- File origin location: -->
<url type="http" location="" preference="98">http://mirrorcache-us.opensuse.org/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
In this case, if the Brazilian mirrorcache has no file, it sends to US.
If I query mirrorcache-us, it finds the file in Provo.
The problem is when I query opensuse.org, when I run:
What I get is:
<resources>
<!-- Mirrors which handle this country (BR): -->
<url type="http" location="BR" preference="100">http://mirror.uepg.br/opensuse//update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="BR" preference="99">http://download.opensuse.net.br//update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<!-- Mirrors in the same continent (SA): -->
<!-- Mirrors in other parts of the world: -->
<url type="http" location="UA" preference="98">http://fastmirror.pp.ua/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="UA" preference="97">http://opensuse.ip-connect.info/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="PT" preference="96">http://ftp.rnl.tecnico.ulisboa.pt/pub/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="FR" preference="95">http://fr2.rpmfind.net/linux/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="GB" preference="94">http://mirrors.coreix.net/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="CH" preference="93">http://opensusemirror.lihaso.com//update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="CH" preference="92">http://pkg.adfinis.com/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="CH" preference="91">http://mirror.hostsuisse.com/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="DE" preference="90">http://mirror1.hs-esslingen.de/pub/Mirrors/ftp.opensuse.org/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<url type="http" location="NL" preference="89">http://mirror.nl.leaseweb.net/opensuse/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
<!-- File origin location: -->
<url type="http" location="" preference="88">http://downloadcontent2.opensuse.org/update/leap/15.3/sle/repodata/e6921fde2a121c785266cd84b0f2fbe998c50771e2744ad92ecccf020ffbc629-filelists.xml.gz</url>
</resources>
download.o.o does not know that it should send South America requests to US, or rather, send to a closer place, but instead, it sends to whatever place it finds in Europe.
Things get even worse for two reasons:
- Not many mirrors sync 15.3 in LATAM because it's EOL and takes a lot of space, and worse, they may exclude SLE repo because it's huge in 15.3 (~200 GB)
- Mirrorcache sync of what is mirrored is slow, it took a while to show our own mirror (download.opensuse.net.br) which had the file, this can cause the metadata to be out of sync
In my understanding, when LATAM requests download.o.o it should send that request to Brazil mirrorcache (cache.opensuse.net.br), but that does not seem to be the case.
If that would be the case, it would fetch the file from Provo, which is close to South America, but since the response for the request is coming from mirrorcache in EU, it sends it to a random place in Europe that has the file.
There are two solutions I think:
- download.opensuse.org should return responses from cache.opensuse.net.br if the request comes from South America
- if 1 is not possible, we need to re-add all mirrors from SA into the cache in Europe, this way, I think EU mirrorcache would be able to send it back to Brazil
Andrii, I hope that this helps to clarify the problem.
Updated by pjessen over 1 year ago
avicenzi wrote:
download.o.o does not know that it should send South America requests to US, or rather, send to a closer place, but instead, it sends to whatever place it finds in Europe.
IIRC, with mirrorbrain, when a file was not found on any mirror on the same continent, any mirror would be good enough, but ordered by network "proximity", ie. ASN.
Updated by avicenzi over 1 year ago
pjessen wrote:
avicenzi wrote:
download.o.o does not know that it should send South America requests to US, or rather, send to a closer place, but instead, it sends to whatever place it finds in Europe.
IIRC, with mirrorbrain, when a file was not found on any mirror on the same continent, any mirror would be good enough, but ordered by network "proximity", ie. ASN.
In this case, it seems that the problem is with download.o.o, since SA has its own mirrorcache now.
Querying SA mirrorcache returns the file if available, or it sends to Provo if not.
Maybe this is related to #115136, the SA mirror does not have an opensuse.org URL yet.
Updated by andriinikitin over 1 year ago
- Status changed from Workable to Resolved
- % Done changed from 0 to 100
It may take some time before new files reach mirrors in South America and some time in addition before download.opensuse.org is aware about that (i.e. when scanner registers the new files on those mirrors).
Until that happens, all requests will be redirected to whatever mirrors are known so far.
It looks the UA mirror was the fastest to get updates, so download.o.o redirected you there.
/update/leap/15.3/sle/repodata is being published several times per day, I don't see what can be fixed or improved in infrastructure in short run.
At the same time South America has got own instance of Mirror Redirector (MirrorCache) - cache.opensuse.net.br , so most of requests from South America to download.opensuse.org will be redirected there. With this change the behavior a bit and requests will not be redirected outside Americas and Nuremberg anymore.
I am closing this call, but let me know if anything needs assistance.
Regards,
Andrii
Nikitin