tickets #92173
closedpoor quality of mirrors in Brazil
Added by avicenzi almost 3 years ago. Updated over 1 year ago.
100%
Description
Over the past few weeks, we started to notice some problems in the quality of mirrors in Brazil, reports from the community, and also from SUSE employees.
There are two main reasons for the problem:
- Low number of mirrors in Brazil
- Requests being redirected to DE and back to BR (mirror cache)
There are three possible solutions:
- Increase the number of mirrors and mirrored content
- Route BR to US, not DE, the ping and latency will be lower in most cases
- Add a mirrorcache in Brazil
Updated by pjessen almost 3 years ago
- Private changed from Yes to No
Just a couple of minor comments -
- many countries have few or no mirrors, in South America as well as in Europe.
- requests always go to download.o.o (mirrorbrain) in Nürnberg first, then they are redirected.
To improve on the level or quality of national mirrors, we are 100% reliant on the respective national communities promoting it.
"Route BR to US, not DE" ? I'm not sure if I understand what that means. For the time being, we only have one mirrorbrain instance. Technically, there is nothing preventing us from setting up a few instances, strategically placed around the world, with anycast DNS - it is only a matter of resources.
Updated by avicenzi almost 3 years ago
Sure, a lot of countries do not have the size and problems that Brazil has with infra.
The "Route BR to US, not DE" would be to send BR requests (metadata and download) to US and not Nuremberg, but I'm not sure if that is possible.
One of the biggest issues that I heard from some people is either, the content is not available in Brazil, there's only one TW mirror, and the other is that this routing to DE is costly in Brazil, due to latency and distancy, which will happen very frequently if the content is not in Brazil.
On the "we are 100% reliant on the respective national communities promoting it", that's not a major problem, I started to promote it and I already got two possible sponsors.
I strongly believe that we will have two new mirrors in Brazil by the end of May, and I will try to also host a mirrorcache instance in one of these.
Updated by pjessen almost 3 years ago
Space needed, from https://mirrors.opensuse.org/list/rsyncinfo-stage.o.o.txt
rsync modules and size of stage.opensuse.org:
opensuse-ports 4.40T
opensuse-full 3.53T
opensuse-full-with-factory 3.06T
opensuse-full-with-factory-dvd5 3.06T
opensuse-full-really-everything 8.62T
opensuse-full-really-everything-including-repositories 28.20T
opensuse-updates 596.85G
buildservice-repos 6.57T
buildservice-repos-main 3.16T
opensuse-source 518.15G
opensuse-debug 1.18T
opensuse-history 1.92T
Updated by pjessen almost 3 years ago
avicenzi wrote:
The "Route BR to US, not DE" would be to send BR requests (metadata and download) to US and not Nuremberg, but I'm not sure if that is possible.
Metadata - without multiple mirrorbrains, that isn't possible.
Download - if Brazilian clients are being given German mirrors, maybe that is something in the mirrorbrain selection algorithm. I don't know if mirrorbrain sees one continent as being different to another. I think mirrorbrain can decide available mirrors by "network proximity", so if US mirrors are better connected to Brazil, that ought to work.
On the "we are 100% reliant on the respective national communities promoting it", that's not a major problem, I started to promote it and I already got two possible sponsors.
That's very cool, more mirrors are always welcome. I think we used to have more in South America, but they slowly went inactive.
Updated by avicenzi almost 3 years ago
Binario Cloud agreed to give a server and space to host an openSUSE mirror.
The VM is 1 cpu 2 gb RAM, 1TB storage, 100mbit transfer, no hard data cap on transfer.
They agreed to monitor how we grow and adjust to our needs if needed.
This will be valid for 1 year, with the possibility to extend.
UFSC (University of Santa Catarina), also agreed to host a mirror, but I don't have details on how much storage it will be available.
UFSC is waiting for a containerized solution that can run on Swarm.
I started to work on it and I'll use Binario mirror as a test for my docker images.
Once it's working I'll ask UFSC to deploy to their infra.
Updated by pjessen almost 3 years ago
avicenzi wrote:
Binario Cloud agreed to give a server and space to host an openSUSE mirror.
The VM is 1 cpu 2 gb RAM, 1TB storage, 100mbit transfer, no hard data cap on transfer.
They agreed to monitor how we grow and adjust to our needs if needed.
This will be valid for 1 year, with the possibility to extend.UFSC (University of Santa Catarina), also agreed to host a mirror, but I don't have details on how much storage it will be available.
UFSC is waiting for a containerized solution that can run on Swarm.
I started to work on it and I'll use Binario mirror as a test for my docker images.
Once it's working I'll ask UFSC to deploy to their infra.
That's very cool - when you're ready, I'll just need some details for creating them, see
https://en.opensuse.org/openSUSE:Mirror_infrastructure#Registering_your_mirror
One comment - 100Mbit isn't very much, but it depends on the local demand.
Updated by avicenzi almost 3 years ago
pjessen wrote:
One comment - 100Mbit isn't very much, but it depends on the local demand.
Well, it will be limited by disc speeds, as I said, they are willing to upgrade if needed, but first, we need metrics to know how bad or good the current setup is.
Updated by pjessen almost 3 years ago
avicenzi wrote:
pjessen wrote:
One comment - 100Mbit isn't very much, but it depends on the local demand.
Well, it will be limited by disc speeds, as I said, they are willing to upgrade if needed, but first, we need metrics to know how bad or good the current setup is.
We can probably get some data from the download.o.o logs, at least to see what sort of demand we are looking at.
Updated by avicenzi over 1 year ago
- Status changed from New to Resolved
- % Done changed from 0 to 100
Over the last year, we made many improvements in Brazil's infra. The most significant changes were announced here 1.
Brazil now has two mirrorcache instances, one downloadcontent for proxy cache, and a few new mirrors.
I consider the original problem solved, and the community seems happy with the recent changes.