openSUSE admin

Hi,

Is this still an issue?

crameleon wrote:

Hi,

Is this still an issue?

No. This is no longer an issue and can be closed. I don't know how to close tickets myself.

Regards,
Srinidhi.

crameleon wrote:

Alright! Thanks for getting back.

Hello @crameleon,

This issue is back and we are very badly impacted:

# mtr api.opensuse.org -4 -r 20
Start: 2023-04-06T12:08:48+0530
HOST: blrcooker                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- firewall.blr.build.net     0.0%    10    0.3   0.3   0.2   0.3   0.0
  2.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- 164.99.203.246             0.0%    10    0.9   0.9   0.7   1.1   0.1
  4.|-- 192.31.114.253             0.0%    10    1.4   2.9   1.1  18.0   5.3
  5.|-- 32.114.83.145              0.0%    10    9.9  10.1   9.9  10.4   0.2
  6.|-- 32.109.34.101              0.0%    10    8.9  11.0   8.9  12.6   1.3
  7.|-- 165.87.139.85              0.0%    10   12.6  11.2   9.1  12.6   1.1
  8.|-- 32.119.109.109             0.0%    10   12.0  11.5   9.9  12.9   0.8
  9.|-- 165.87.76.34               0.0%    10    8.4   8.5   8.3   8.7   0.1
 10.|-- 32.109.50.94               0.0%    10  128.3 128.3 128.1 128.5   0.1
 11.|-- ae10-1.fra30.core-backbon  0.0%    10  128.1 128.2 128.0 128.4   0.1
 12.|-- ae2-2001.nbg30.core-backb  0.0%    10  131.3 131.3 131.1 131.4   0.1
 13.|-- core-backbone.microfocus.  0.0%    10  131.1 131.2 131.1 131.3   0.1
 14.|-- 195.135.221.26             0.0%    10  131.3 132.1 131.2 137.6   2.0
 15.|-- 195.135.221.162            0.0%    10  131.5 131.5 131.5 131.7   0.1
mtr: udp socket connect failed: Invalid argument

# curl 'https://api.opensuse.org/public/build/SUSE:SLE-15:Update/pool/i586?view=binaryversionscode' > /dev/null                                                                                                                                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 45.8M  100 45.8M    0     0  47286      0  0:16:56  0:16:56 --:--:-- 44674

Looks like the problem is once again with the core-backbone servers. Could you please help us out here?

Regards,
Srinidhi.

Hi,

this is a tough one. The network is managed by SUSE, but there are currently no openSUSE volunteers in the SUSE network team. I will try to forward it and reference this and your last ticket.

Cheers,
Georg

Internal ticket: https://sd.suse.com/browse/SD-117744.

Hi,

I received feedback:

• your last MTR output looks normal
• can you provide MTR and tracepath output with both IPv4 and IPv6 to further investigate the issue?

crameleon wrote:

Hi,

I received feedback:

• your last MTR output looks normal

Oh, okay! That is interesting!

• can you provide MTR and tracepath output with both IPv4 and IPv6 to further investigate the issue?

Unfortunately, I can only provide data for IPv4 because we do not have an IPv6 address assigned to our Build Service backend.

MTR¶

# mtr api.opensuse.org -4 -r 20
Start: 2023-04-06T20:53:33+0530
HOST: blrcooker                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- firewall.blr.build.net     0.0%    10    0.2   0.2   0.2   0.4   0.1
  2.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- 164.99.203.246             0.0%    10    1.0   1.0   0.7   1.2   0.1
  4.|-- 192.31.114.253             0.0%    10    1.3   1.2   1.0   1.4   0.1
  5.|-- 32.114.83.145              0.0%    10   10.3  10.1   9.9  10.3   0.1
  6.|-- 32.109.34.101              0.0%    10    9.9  10.9   9.1  12.2   1.0
  7.|-- 165.87.139.85              0.0%    10   12.7  11.2   9.3  12.7   1.1
  8.|-- 32.119.109.109            10.0%    10   10.2  11.4  10.2  12.3   0.7
  9.|-- 165.87.76.34               0.0%    10    8.5   8.5   8.3   8.6   0.1
 10.|-- 32.109.50.94               0.0%    10  128.1 128.3 128.1 128.5   0.1
 11.|-- ae10-1.fra30.core-backbon  0.0%    10  128.1 128.3 128.1 128.5   0.1
 12.|-- ae2-2001.nbg30.core-backb  0.0%    10  131.3 131.3 131.1 131.4   0.1
 13.|-- core-backbone.microfocus.  0.0%    10  132.4 131.3 131.1 132.4   0.4
 14.|-- 195.135.221.26             0.0%    10  131.6 131.4 131.3 131.6   0.1
 15.|-- 195.135.221.162            0.0%    10  131.6 131.6 131.4 131.7   0.1
mtr: udp socket connect failed: Invalid argument

Traceroute¶

# traceroute api.opensuse.org 
traceroute to api.opensuse.org (195.135.221.162), 30 hops max, 60 byte packets
 1  firewall.blr.build.net (192.168.10.1)  0.190 ms  0.264 ms  0.231 ms
 2  * * *
 3  164.99.203.246 (164.99.203.246)  0.819 ms  0.871 ms  0.756 ms
 4  192.31.114.253 (192.31.114.253)  1.030 ms  1.017 ms  0.965 ms
 5  32.114.83.145 (32.114.83.145)  9.714 ms  9.708 ms  9.695 ms
 6  32.109.34.101 (32.109.34.101)  11.806 ms  11.739 ms  11.709 ms
 7  165.87.139.85 (165.87.139.85)  8.683 ms  12.098 ms  12.104 ms
 8  32.119.109.109 (32.119.109.109)  30.684 ms  30.423 ms  30.435 ms
 9  165.87.76.34 (165.87.76.34)  8.325 ms  8.662 ms  8.308 ms
10  32.109.50.94 (32.109.50.94)  128.300 ms  128.308 ms  128.294 ms
11  ae3-1.fra20.core-backbone.com (80.81.192.187)  128.653 ms  128.631 ms  128.584 ms
12  ae2-2001.nbg30.core-backbone.com (81.95.15.161)  131.367 ms  131.360 ms  131.262 ms
13  core-backbone.microfocus.com (5.56.18.210)  131.280 ms  131.245 ms  131.239 ms
14  195.135.221.26 (195.135.221.26)  131.349 ms  131.728 ms  131.675 ms
15  195.135.221.162 (195.135.221.162)  131.549 ms !X  131.858 ms !X  131.861 ms !X

Thanks for the quick reply.
Could you provide tracepath output as well, to check for possible MTU problems?

crameleon wrote:

Thanks for the quick reply.

I'm desperately refreshing this page every 2-5 minutes :-) as our Private Build Service instance has been almost unusable for the entire working day today! :-(

Could you provide tracepath output as well, to check for possible MTU problems?

# tracepath api.opensuse.org 
 1?: [LOCALHOST]                                         pmtu 1500
 1:  firewall.blr.build.net                                0.183ms 
 1:  firewall.blr.build.net                                0.109ms 
 2:  no reply
 3:  164.99.203.246                                        7.238ms 
 4:  192.31.114.253                                        1.028ms 
 5:  32.114.83.145                                        16.434ms 
 6:  32.109.34.101                                        15.548ms asymm  9 
 7:  165.87.139.85                                        18.080ms asymm  8 
 8:  32.119.109.109                                       18.462ms 
 9:  165.87.76.34                                         14.828ms asymm  8 
10:  32.109.50.94                                        134.602ms asymm 11 
11:  ae3-1.fra20.core-backbone.com                       134.942ms asymm 12 
12:  ae2-2001.nbg30.core-backbone.com                    137.651ms asymm 14 
13:  core-backbone.microfocus.com                        137.952ms asymm 14 
14:  195.135.221.26                                      142.212ms asymm 15 
15:  195.135.221.162                                     141.957ms !H
     Resume: pmtu 1500

Regards,
Srinidhi.

Hi,

crameleon wrote:

Thanks, that unfortunately doesn't show an issue either. My network colleague suggests it is likely an issue on the destination server as he receives flaky behavior when downloading from api.opensuse.org as well and suggests one should capture tcpdumps from both the source and destination. Since the destination seems to not be managed by regular openSUSE Heroes but by the team operating the Build Service, I will assign the ticket to them. I hope they can help you further. Keep in mind that many are from Germany and are likely to be enjoying their easter holidays already. Sorry to not have any better news for now.

Any updates for me, please?

The network throughput has worsened since Sunday!

# curl -4 'https://api.opensuse.org/public/build/SUSE:SLE-15:Update/pool/i586?view=binaryversionscode' > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 45.9M  100 45.9M    0     0  35117      0  0:22:52  0:22:52 --:--:-- 36741

Regards,
Srinidhi.

Thanks, that unfortunately doesn't show an issue either. My network colleague suggests it is likely an issue on the destination server as he receives flaky behavior when downloading > from api.opensuse.org as well and suggests one should capture tcpdumps from both the source and destination. Since the destination seems to not be managed by regular openSUSE Heroes > but by the team operating the Build Service, I will assign the ticket to them. I hope they can help you further. Keep in mind that many are from Germany and are likely to be
enjoying their easter holidays already. Sorry to not have any better news for now.

Sorry, but this really looks like a problem in front of any OBS managed systems.

The referenced URL can be requested within 0.018 seconds on the login proxy, while it takes more then 6 seconds when asking via public IP. The login proxy itself has also free slots.

Feel free to reach out to me, when you want to debug this together.

Will get Adrian and my colleague in touch.

Hi,

crameleon wrote:

Will get Adrian and my colleague in touch.

I wanted to know my options now. It has been more than 9 days since our private OBS instance has been impacted.

How can I help in debugging this problem further?

Regards,
Srinidhi.

Copying Robert's comment from the SD ticket (4/12/23, 20:52):

I've tried to add this to the opensuse.org ticket, but didn't find a way to update it, so I'll add it below
I've run that curl command for over 20 times, got download speeds between 500k and 4000k (downloads times between 11 sec and 90 sec), mostly under 30 sec.
I've started the downloads from Provo DC, SUSE internet line. The user seems to be in the same location, Microfocus internet line
Is the user still experiencing the slow download ? From the ticket it looks like 35k download speed. I didn't manage to reproduce that.
Are specific times when this happens ?
Does it happens for consecutive downloads (if the download is very slow, stopping it and restarting makes any difference ?)
When this happens please ask the user to
stop the download
start a "tcpdump -i any host api.opensuse.org -w api-opensuse.pcap"
run the curl command.
When it is over, stop the tcpdump and provide the pcap.
If Adrian is online at that time, he could take a tcpdump on api.opensuse.org frontend, filtering for user's public IP

crameleon wrote in #note-24:

Hi,

has this been resolved in the meanwhile?

Yes. But differently - I had to backport (read: cherry-pick) all of the IPv6 support code from the master over to our private OBS 2.10 and switch to using IPv6 to avoid getting into these problems every 4-5 months.

Thank you for checking! I couldn't close this ticket. Please feel free to close this ticket.

Regards,
Srinidhi.