tickets #125615
closedProblem with our mirror - espejito.fder.edu.uy
Added by asabigue@fder.edu.uy about 1 year ago. Updated 4 months ago.
100%
Description
Hello!
We are receiving almost no traffic to our mirror, but we don't see any
trouble with the data, update scripts or other working details. I
checked at https://mirrors.opensuse.org/ where we are still listed, yet,
few things on green.
Could you please let us know if there is something not working as expected?
Our mirror is espejito.fder.edu.uy
regards
ariel
Updated by pjessen about 1 year ago
- Category set to Mirrors
- Private changed from Yes to No
Updated by avicenzi about 1 year ago
- Assignee set to andriinikitin
Hi Ariel,
My theory is that we now have many mirrors in South America, more precisely, in Brazil.
The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily.
The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay.
This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror.
I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh.
To solve that you should sync more frequently, maybe every 2h or even a shorter period.
This is just a theory for now, but Andrii may be able to dig some metrics from our servers.
Best,
Alex
Updated by pjessen about 1 year ago
- Related to tickets #114811: repo push to espejito.fder.edu.uy is stuck. added
Updated by pjessen about 1 year ago
- Subject changed from Problem with our mirror to Problem with our mirror - espejito.fder.edu.uy
avicenzi wrote:
I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh.
Note: this mirror was initially set up for repositories push, but it stopped working at some point. See #114811 - I guess our mirror admin has been to busy to do something about it.
Updated by asabigue@fder.edu.uy about 1 year ago
With repositories there was a space problem: we only have 45TB of mirror
space, 22TB available and repositories swallowed all that space and we
needed to stop rsyncing it. I think I never saw you pushing data there.
Right now we allocata 9,2TB of space for opensuse mirroring without
repositories.
but those 9,2TB are up to date (cron job running hourly since today) and
no traffic redirection.... sort of a waste of BW+disk
Updated by carlos@fisica.ufpr.br about 1 year ago
redmine@opensuse.org (redmine@opensuse.org) wrote on Wed, Mar 08, 2023 at 08:45:16AM -03:
[openSUSE Tracker]
Issue #125615 has been updated by avicenzi.Assignee set to andriinikitin
Hi Ariel,
My theory is that we now have many mirrors in South America, more precisely, in Brazil.
The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily.
The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay.
You didn't mention the main Brazilian mirror: opensuse.c3sl.ufpr.br!
Updated by avicenzi about 1 year ago
There's an issue somewhere, MirrorCache does not seem to return Espejito for Uruguay.
I tried a couple of files, see for example:
curl -L https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-aarch64-Media.iso.metalink?COUNTRY=uy
Espejito is registered and enabled in BR and EU instances of MirrorCache, but none of these return Espejito when I ask for a specific country.
If you have space Ariel, I suggest syncing everything from stage-main-repos.opensuse.org.
As mentioned on the mailing list, this should require 5 TB and will host most things users need.
Updated by avicenzi about 1 year ago
carlos@fisica.ufpr.br wrote:
redmine@opensuse.org (redmine@opensuse.org) wrote on Wed, Mar 08, 2023 at 08:45:16AM -03:
[openSUSE Tracker]
Issue #125615 has been updated by avicenzi.Assignee set to andriinikitin
Hi Ariel,
My theory is that we now have many mirrors in South America, more precisely, in Brazil.
The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily.
The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay.You didn't mention the main Brazilian mirror: opensuse.c3sl.ufpr.br!
I did in the mailing list.
We have 9 mirrors in Brazil, the closest ones (by distance) to Uruguay are UEPG and POP/SC, but UY users can also be sent to UFPR, or any other mirror in SA.
Updated by avicenzi about 1 year ago
It seems that today MirrorCache is returning Espejito for Uruguay.
This is odd, maybe MirrorCache was stuck scanning this mirror?
Or it marked the mirror as down for some reason?
I think only Andrii can answer what really happened.
Updated by avicenzi about 1 year ago
- Status changed from New to In Progress
- % Done changed from 0 to 70
Updated by andriinikitin 11 months ago
Sorry I was overwhelmed with various things and missed this ticket somehow.
Currently the report shows that some traffic is being sent to the mirror:
https://mirrorcache-br-2.opensuse.org/report/download?group=mirror
But internal health check reports quite many problems for the server, this must be the reason why redirection is stopped periodically (note that only problems are reported here: the check is done every few minutes and the mirror is marked back as available, just with less stability and less chance to get redirects):
# sudo -u mirrorcache psql -P pager=off -c 'select * from server_capability_check where server_id = 4 order by dt;'
server_id | capability | dt | extra
-----------+------------+-------------------------+--------------------------------------------------------------------------------------
4 | http | 2023-05-11 12:12:09.163 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-11 12:12:12.187 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-11 13:27:09.686 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-11 13:27:12.705 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-11 15:59:58.655 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-11 16:00:01.674 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-11 16:01:55.86 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-11 16:01:58.879 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 00:20:40.005 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 12:20:36.139 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-12 12:20:39.162 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 14:59:36.931 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 18:13:49.025 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-12 18:13:52.051 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 18:30:06.291 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 18:56:21.163 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-12 18:56:24.181 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 21:19:11.299 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 22:02:16.8 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-12 22:39:27.591 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-12 22:39:30.606 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-13 01:43:59.399 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-13 01:44:02.418 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-13 14:50:17.505 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-13 14:50:20.545 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-13 19:32:32.327 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-14 00:00:11.695 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-14 13:00:08.024 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-14 20:01:27.548 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-15 11:02:37.693 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-15 22:45:06.627 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-15 22:45:09.646 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-16 15:07:42.36 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-16 15:49:41.727 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-16 16:44:31.411 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-16 16:44:34.431 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-16 23:05:11.857 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-17 00:29:17.372 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-17 00:29:20.382 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-17 20:49:30.397 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-17 20:49:33.414 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-18 12:05:50.986 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-18 12:05:54.011 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-18 14:38:35.179 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-19 01:46:59.94 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-19 01:47:02.96 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-19 03:20:20.104 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-19 14:30:34.03 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-19 14:30:37.05 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-19 17:12:13.361 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-19 17:12:16.38 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-19 23:05:16.334 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-19 23:05:19.34 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-20 10:15:12.794 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-20 12:20:09.496 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-22 13:34:28.152 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-22 13:34:31.17 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-22 22:50:03.997 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-23 16:14:51.11 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-23 16:14:54.129 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-23 18:18:20.464 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-23 20:51:39.216 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-23 20:51:42.235 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-24 00:02:21.584 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-24 00:02:24.603 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-24 09:06:11.111 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-24 09:06:14.13 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-24 13:02:32.836 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-24 13:09:02.231 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-24 15:34:09.581 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-24 15:34:12.602 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-24 17:07:25.65 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | http | 2023-05-25 00:40:14.131 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
4 | https | 2023-05-25 00:40:17.153 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
| | |
(74 rows)
Updated by andriinikitin 11 months ago
- Assignee changed from andriinikitin to avicenzi
At the same time other mirrors are not perfect as well, below is error count per mirror. Maybe something for Alex to look at:
# sudo -u mirrorcache psql -P pager=off -c 'select server.id, server.hostname, count(*) from server left join server_capability_check on server.id = server_id where enabled group by server.id, server.hostname order by server.id;'
id | hostname | count
----+---------------------------------+-------
1 | opensuse.c3sl.ufpr.br | 20
2 | ftp.unicamp.br | 23
3 | mirror.cedia.org.ec | 18
4 | espejito.fder.edu.uy | 74
5 | download.opensuse.net.br | 120
6 | linorg.usp.br | 48
7 | mirror.uepg.br | 30
8 | opensuse.ic.ufmt.br | 187
9 | opensuse.unc.edu.ar | 103
10 | mirror1.cl.netactuate.com | 42
12 | opensuse.redlibre.cl | 9
13 | opensuse.mirrors.locaweb.com.br | 308
14 | mirrors.ucr.ac.cr | 126
15 | mirror-br.opensuse.org | 18
16 | mirror.pop-sc.rnp.br | 18
(15 rows)
Updated by asabigue@fder.edu.uy 4 months ago
Happy new year!
I believe you can close it....
regards
ariel
On 12/1/24 11:39, avicenzi wrote:
[openSUSE Tracker]
Issue #125615 has been updated by avicenzi.Hi Ariel,
Do you still have issues or can we close this ticket?
Best,
Alex
tickets #125615: Problem with our mirror - espejito.fder.edu.uy
https://progress.opensuse.org/issues/125615#change-751492
- Author: asabigue@fder.edu.uy
- Status: In Progress
- Priority: Normal
- Assignee: avicenzi
- Category: Mirrors
* Start date: 2023-03-08¶
Hello!
We are receiving almost no traffic to our mirror, but we don't see any
trouble with the data, update scripts or other working details. I
checked at https://mirrors.opensuse.org/ where we are still listed, yet,
few things on green.Could you please let us know if there is something not working as expected?
Our mirror is espejito.fder.edu.uy
regards
ariel