Project

General

Profile

Actions

tickets #125615

closed

Problem with our mirror - espejito.fder.edu.uy

Added by asabigue@fder.edu.uy about 1 year ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Mirrors
Target version:
-
Start date:
2023-03-08
Due date:
% Done:

100%

Estimated time:

Description

Hello!

We are receiving almost no traffic to our mirror, but we don't see any
trouble with the data, update scripts or other working details. I
checked at https://mirrors.opensuse.org/ where we are still listed, yet,
few things on green.

Could you please let us know if there is something not working as expected?

Our mirror is espejito.fder.edu.uy

regards

ariel


Related issues 1 (1 open0 closed)

Related to openSUSE admin - tickets #114811: repo push to espejito.fder.edu.uy is stuck. Newandriinikitin2022-07-29

Actions
Actions #1

Updated by pjessen about 1 year ago

  • Category set to Mirrors
  • Private changed from Yes to No
Actions #2

Updated by avicenzi about 1 year ago

  • Assignee set to andriinikitin

Hi Ariel,

My theory is that we now have many mirrors in South America, more precisely, in Brazil.
The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily.
The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay.

This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror.

I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh.
To solve that you should sync more frequently, maybe every 2h or even a shorter period.

This is just a theory for now, but Andrii may be able to dig some metrics from our servers.

Best,

Alex

Actions #3

Updated by pjessen about 1 year ago

  • Related to tickets #114811: repo push to espejito.fder.edu.uy is stuck. added
Actions #4

Updated by pjessen about 1 year ago

  • Subject changed from Problem with our mirror to Problem with our mirror - espejito.fder.edu.uy

avicenzi wrote:

I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh.

Note: this mirror was initially set up for repositories push, but it stopped working at some point. See #114811 - I guess our mirror admin has been to busy to do something about it.

Actions #5

Updated by asabigue@fder.edu.uy about 1 year ago

With repositories there was a space problem: we only have 45TB of mirror
space, 22TB available and repositories swallowed all that space and we
needed to stop rsyncing it. I think I never saw you pushing data there.
Right now we allocata 9,2TB of space  for opensuse mirroring without
repositories.

but those 9,2TB are up to date (cron job running hourly since today) and
no traffic redirection.... sort of a waste of BW+disk

Actions #6

Updated by carlos@fisica.ufpr.br about 1 year ago

redmine@opensuse.org (redmine@opensuse.org) wrote on Wed, Mar 08, 2023 at 08:45:16AM -03:

[openSUSE Tracker]
Issue #125615 has been updated by avicenzi.

Assignee set to andriinikitin

Hi Ariel,

My theory is that we now have many mirrors in South America, more precisely, in Brazil.
The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily.
The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay.

You didn't mention the main Brazilian mirror: opensuse.c3sl.ufpr.br!

Actions #7

Updated by avicenzi about 1 year ago

There's an issue somewhere, MirrorCache does not seem to return Espejito for Uruguay.

I tried a couple of files, see for example:

curl -L https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-aarch64-Media.iso.metalink?COUNTRY=uy

Espejito is registered and enabled in BR and EU instances of MirrorCache, but none of these return Espejito when I ask for a specific country.

If you have space Ariel, I suggest syncing everything from stage-main-repos.opensuse.org.
As mentioned on the mailing list, this should require 5 TB and will host most things users need.

Actions #8

Updated by avicenzi about 1 year ago

carlos@fisica.ufpr.br wrote:

redmine@opensuse.org (redmine@opensuse.org) wrote on Wed, Mar 08, 2023 at 08:45:16AM -03:

[openSUSE Tracker]
Issue #125615 has been updated by avicenzi.

Assignee set to andriinikitin

Hi Ariel,

My theory is that we now have many mirrors in South America, more precisely, in Brazil.
The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily.
The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay.

You didn't mention the main Brazilian mirror: opensuse.c3sl.ufpr.br!

I did in the mailing list.

We have 9 mirrors in Brazil, the closest ones (by distance) to Uruguay are UEPG and POP/SC, but UY users can also be sent to UFPR, or any other mirror in SA.

Actions #9

Updated by avicenzi about 1 year ago

It seems that today MirrorCache is returning Espejito for Uruguay.

This is odd, maybe MirrorCache was stuck scanning this mirror?
Or it marked the mirror as down for some reason?

I think only Andrii can answer what really happened.

Actions #10

Updated by avicenzi about 1 year ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 70
Actions #11

Updated by andriinikitin 10 months ago

Sorry I was overwhelmed with various things and missed this ticket somehow.

Currently the report shows that some traffic is being sent to the mirror:

https://mirrorcache-br-2.opensuse.org/report/download?group=mirror

But internal health check reports quite many problems for the server, this must be the reason why redirection is stopped periodically (note that only problems are reported here: the check is done every few minutes and the mirror is marked back as available, just with less stability and less chance to get redirects):

# sudo -u mirrorcache psql -P pager=off -c 'select * from server_capability_check where server_id = 4 order by dt;'
 server_id | capability |           dt            |                                        extra                                         
-----------+------------+-------------------------+--------------------------------------------------------------------------------------
         4 | http       | 2023-05-11 12:12:09.163 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-11 12:12:12.187 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-11 13:27:09.686 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-11 13:27:12.705 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-11 15:59:58.655 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-11 16:00:01.674 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-11 16:01:55.86  | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-11 16:01:58.879 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 00:20:40.005 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 12:20:36.139 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-12 12:20:39.162 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 14:59:36.931 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 18:13:49.025 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-12 18:13:52.051 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 18:30:06.291 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 18:56:21.163 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-12 18:56:24.181 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 21:19:11.299 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 22:02:16.8   | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-12 22:39:27.591 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-12 22:39:30.606 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-13 01:43:59.399 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-13 01:44:02.418 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-13 14:50:17.505 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-13 14:50:20.545 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-13 19:32:32.327 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-14 00:00:11.695 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-14 13:00:08.024 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-14 20:01:27.548 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-15 11:02:37.693 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-15 22:45:06.627 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-15 22:45:09.646 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-16 15:07:42.36  | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-16 15:49:41.727 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-16 16:44:31.411 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-16 16:44:34.431 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-16 23:05:11.857 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-17 00:29:17.372 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-17 00:29:20.382 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-17 20:49:30.397 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-17 20:49:33.414 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-18 12:05:50.986 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-18 12:05:54.011 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-18 14:38:35.179 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-19 01:46:59.94  | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-19 01:47:02.96  | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-19 03:20:20.104 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-19 14:30:34.03  | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-19 14:30:37.05  | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-19 17:12:13.361 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-19 17:12:16.38  | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-19 23:05:16.334 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-19 23:05:19.34  | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-20 10:15:12.794 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-20 12:20:09.496 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-22 13:34:28.152 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-22 13:34:31.17  | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-22 22:50:03.997 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-23 16:14:51.11  | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-23 16:14:54.129 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-23 18:18:20.464 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-23 20:51:39.216 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-23 20:51:42.235 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-24 00:02:21.584 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-24 00:02:24.603 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-24 09:06:11.111 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-24 09:06:14.13  | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-24 13:02:32.836 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-24 13:09:02.231 | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-24 15:34:09.581 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-24 15:34:12.602 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-24 17:07:25.65  | Connect timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | http       | 2023-05-25 00:40:14.131 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
         4 | https      | 2023-05-25 00:40:17.153 | Request timeout at /usr/share/mirrorcache/script/../lib/Net/URIProtocols.pm line 83.+
           |            |                         | 
(74 rows)
Actions #12

Updated by andriinikitin 10 months ago

  • Assignee changed from andriinikitin to avicenzi

At the same time other mirrors are not perfect as well, below is error count per mirror. Maybe something for Alex to look at:

# sudo -u mirrorcache psql -P pager=off -c 'select server.id, server.hostname, count(*) from server left join server_capability_check on server.id = server_id where enabled group by server.id, server.hostname order by server.id;'
 id |            hostname             | count 
----+---------------------------------+-------
  1 | opensuse.c3sl.ufpr.br           |    20
  2 | ftp.unicamp.br                  |    23
  3 | mirror.cedia.org.ec             |    18
  4 | espejito.fder.edu.uy            |    74
  5 | download.opensuse.net.br        |   120
  6 | linorg.usp.br                   |    48
  7 | mirror.uepg.br                  |    30
  8 | opensuse.ic.ufmt.br             |   187
  9 | opensuse.unc.edu.ar             |   103
 10 | mirror1.cl.netactuate.com       |    42
 12 | opensuse.redlibre.cl            |     9
 13 | opensuse.mirrors.locaweb.com.br |   308
 14 | mirrors.ucr.ac.cr               |   126
 15 | mirror-br.opensuse.org          |    18
 16 | mirror.pop-sc.rnp.br            |    18
(15 rows)
Actions #13

Updated by avicenzi 3 months ago

Hi Ariel,

Do you still have issues or can we close this ticket?

Best,

Alex

Actions #14

Updated by avicenzi 3 months ago

  • Status changed from In Progress to Feedback
  • % Done changed from 70 to 100
Actions #15

Updated by asabigue@fder.edu.uy 3 months ago

Happy new year!

I believe you can close it....

regards

ariel

On 12/1/24 11:39, avicenzi wrote:

[openSUSE Tracker]
Issue #125615 has been updated by avicenzi.

Hi Ariel,

Do you still have issues or can we close this ticket?

Best,

Alex


tickets #125615: Problem with our mirror - espejito.fder.edu.uy
https://progress.opensuse.org/issues/125615#change-751492

  • Author: asabigue@fder.edu.uy
  • Status: In Progress
  • Priority: Normal
  • Assignee: avicenzi
  • Category: Mirrors

* Start date: 2023-03-08

Hello!

We are receiving almost no traffic to our mirror, but we don't see any
trouble with the data, update scripts or other working details. I
checked at https://mirrors.opensuse.org/ where we are still listed, yet,
few things on green.

Could you please let us know if there is something not working as expected?

Our mirror is espejito.fder.edu.uy

regards

ariel

Actions #16

Updated by avicenzi 3 months ago

  • Status changed from Feedback to Closed
Actions

Also available in: Atom PDF