Actions
action #174304
closedSlow network connection between PowerVM/HMC SUT and OpenQA controller host size:S
Status:
Resolved
Priority:
Normal
Assignee:
Category:
Support
Target version:
Start date:
2024-12-12
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Some LTP tests trigger large amounts of kernel log messages and then upload journalctl logfile to OpenQA. In this example, shutdown_ltp-journalctl.txt log file contains 18MB of data and the upload took 111 seconds (upload speed: ~168KB/s, upload timeout: 100s).
https://openqa.suse.de/tests/16073659#step/shutdown_ltp/55
The PowerVM SUT (upload source) is nessberry-2, the OpenQA controller host (upload destination) is worker29.oqa.prg2.suse.org. What is the expected bandwidth between the machines?
Acceptance criteria¶
- AC1: A sufficiently high bandwidth is ensured between usual OSD involved testing machines, e.g. ~1GBit/s
Suggestions¶
- Follow and respond on https://sd.suse.com/servicedesk/customer/portal/1/SD-175581
- Check bandwidth between various hosts, e.g. with iperf, and comment back in the SD ticket
- And use
-4and-6to check IPv4 vs. IPv6
- And use
- Depending on how the problem can be reproduced, e.g. which hosts are affected, investigate/mitigate/solve the underlying issue, e.g. anything in the range from "just trigger a reboot to the HMC or switch" to work in a task force with IT to identify misbehaving systems severly impacting the available bandwidth
- Also be aware about #133700 and shove it in the face of IT
- Don't just bump timeouts here
Updated by okurz 3 months ago
- Category set to Support
- Assignee set to okurz
- Target version set to Ready
hi, as suggested in https://suse.slack.com/archives/C02CANHLANP/p1734001300419849?thread_ts=1733757997.878699&cid=C02CANHLANP
I doubt we have the necessary observables accessible to us to help here. I suggest you start a discussion with IT who are responsible for the underlying network infrastructure. As we all know about the limitations of Jira SD tickets I suggest you create a ticket in https://progress.opensuse.org/projects/openqa-infrastructure/ , create an SD ticket and link back to that progress ticket
so this is the progress ticket. Do you plan to create the Jira SD ticket accordingly or how do you recommend to proceed?
Updated by okurz 3 months ago
- Status changed from New to Blocked
nevermind, you reported the ticket https://sd.suse.com/servicedesk/customer/portal/1/SD-175581
Updated by okurz 3 months ago
- Priority changed from Normal to Urgent
we should handle the open point from SD ticket in https://sd.suse.com/servicedesk/customer/portal/1/SD-175581
Could you please check connectivity with the servers
in the same segments as nessberry and worker29.oqa.prg2.suse.org
Those servers are in different network segments and must check speeds without routers and firewalls.
Could you please check it?
If speed isn't acceptable we will ask Network team's help.
Updated by okurz 3 months ago
- Subject changed from Slow network connection between PowerVM/HMC SUT and OpenQA controller host to Slow network connection between PowerVM/HMC SUT and OpenQA controller host size:S
- Description updated (diff)
- Status changed from New to Workable
- Assignee set to nicksinger
Updated by nicksinger 3 months ago
- Priority changed from Urgent to High
I've conducted the wanted tests and found no issues between several hosts:
worker29:~ # iperf3 -4 -c worker30.oqa.prg2.suse.org
Connecting to host worker30.oqa.prg2.suse.org, port 5201
[ 5] local 10.145.10.2 port 53104 connected to 10.145.10.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.10 GBytes 9.43 Gbits/sec 45 1.62 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.93 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.41 Gbits/sec 0 2.12 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.41 Gbits/sec 0 2.39 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 2.42 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.42 Gbits/sec 0 2.42 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.41 Gbits/sec 0 2.42 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.41 Gbits/sec 0 2.43 MBytes
[ 5] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 2.74 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec 144 1.96 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec 189 sender
[ 5] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec receiver
iperf Done.
worker29:~ # iperf3 -6 -c worker30.oqa.prg2.suse.org
Connecting to host worker30.oqa.prg2.suse.org, port 5201
[ 5] local 2a07:de40:b203:12:7ec2:55ff:fe24:de9c port 43506 connected to 2a07:de40:b203:12:7ec2:55ff:fe24:dc34 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.08 GBytes 9.28 Gbits/sec 12 1.65 MBytes
[ 5] 1.00-2.00 sec 1.08 GBytes 9.27 Gbits/sec 15 1.81 MBytes
[ 5] 2.00-3.00 sec 1.08 GBytes 9.28 Gbits/sec 0 1.90 MBytes
[ 5] 3.00-4.00 sec 1.08 GBytes 9.27 Gbits/sec 0 2.28 MBytes
[ 5] 4.00-5.00 sec 1.08 GBytes 9.29 Gbits/sec 0 2.49 MBytes
[ 5] 5.00-6.00 sec 1.08 GBytes 9.27 Gbits/sec 0 2.50 MBytes
[ 5] 6.00-7.00 sec 1.08 GBytes 9.28 Gbits/sec 0 2.50 MBytes
[ 5] 7.00-8.00 sec 1.08 GBytes 9.29 Gbits/sec 0 2.50 MBytes
[ 5] 8.00-9.00 sec 1.08 GBytes 9.28 Gbits/sec 4 2.50 MBytes
[ 5] 9.00-10.00 sec 1.08 GBytes 9.28 Gbits/sec 0 2.50 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.8 GBytes 9.28 Gbits/sec 31 sender
[ 5] 0.00-10.00 sec 10.8 GBytes 9.28 Gbits/sec receiver
iperf Done.
worker29:~ # iperf3 -R -4 -c worker30.oqa.prg2.suse.org
Connecting to host worker30.oqa.prg2.suse.org, port 5201
Reverse mode, remote host worker30.oqa.prg2.suse.org is sending
[ 5] local 10.145.10.2 port 49700 connected to 10.145.10.3 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.06 GBytes 9.08 Gbits/sec
[ 5] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 2.00-3.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 3.00-4.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec
[ 5] 5.00-6.00 sec 1.10 GBytes 9.42 Gbits/sec
[ 5] 6.00-7.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 7.00-8.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 9.00-10.00 sec 1.10 GBytes 9.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec 294 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiver
iperf Done.
worker29:~ # iperf3 -R -6 -c worker30.oqa.prg2.suse.org
Connecting to host worker30.oqa.prg2.suse.org, port 5201
Reverse mode, remote host worker30.oqa.prg2.suse.org is sending
[ 5] local 2a07:de40:b203:12:7ec2:55ff:fe24:de9c port 44980 connected to 2a07:de40:b203:12:7ec2:55ff:fe24:dc34 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.07 GBytes 9.15 Gbits/sec
[ 5] 1.00-2.00 sec 1.08 GBytes 9.25 Gbits/sec
[ 5] 2.00-3.00 sec 1.07 GBytes 9.21 Gbits/sec
[ 5] 3.00-4.00 sec 1.07 GBytes 9.21 Gbits/sec
[ 5] 4.00-5.00 sec 1.07 GBytes 9.20 Gbits/sec
[ 5] 5.00-6.00 sec 1.07 GBytes 9.20 Gbits/sec
[ 5] 6.00-7.00 sec 1.08 GBytes 9.24 Gbits/sec
[ 5] 7.00-8.00 sec 1.08 GBytes 9.25 Gbits/sec
[ 5] 8.00-9.00 sec 1.08 GBytes 9.29 Gbits/sec
[ 5] 9.00-10.00 sec 1.08 GBytes 9.28 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.7 GBytes 9.23 Gbits/sec 84 sender
[ 5] 0.00-10.00 sec 10.7 GBytes 9.23 Gbits/sec receiver
iperf Done.
Looks fine. The machines are in the same chassis so I tested with storage as well and archived ~1Gbit/s - so all good here. Moving on into the qe.prg2.suse.org-domain I used ada to verify the speed to worker29:
ada:~ # iperf3 -4 -c worker29.oqa.prg2.suse.org
Connecting to host worker29.oqa.prg2.suse.org, port 5201
[ 5] local 10.146.4.1 port 45332 connected to 10.145.10.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 953 Mbits/sec 6 399 KBytes
[ 5] 1.00-2.00 sec 113 MBytes 946 Mbits/sec 5 488 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 943 Mbits/sec 2 489 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 943 Mbits/sec 2 491 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 934 Mbits/sec 4 492 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 2 492 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 943 Mbits/sec 1 492 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 943 Mbits/sec 0 492 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 934 Mbits/sec 0 492 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 0 492 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 22 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 940 Mbits/sec receiver
iperf Done.
ada:~ # iperf3 -6 -c worker29.oqa.prg2.suse.org
Connecting to host worker29.oqa.prg2.suse.org, port 5201
[ 5] local 2a07:de40:b230:1:3eec:efff:feb9:b830 port 50866 connected to 2a07:de40:b203:12:7ec2:55ff:fe24:de9c port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 944 Mbits/sec 10 434 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 931 Mbits/sec 5 483 KBytes
[ 5] 2.00-3.00 sec 111 MBytes 930 Mbits/sec 3 483 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 930 Mbits/sec 0 483 KBytes
[ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 8 483 KBytes
[ 5] 5.00-6.00 sec 111 MBytes 929 Mbits/sec 3 483 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 930 Mbits/sec 2 483 KBytes
[ 5] 7.00-8.00 sec 111 MBytes 929 Mbits/sec 1 483 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 930 Mbits/sec 0 484 KBytes
[ 5] 9.00-10.00 sec 110 MBytes 926 Mbits/sec 0 484 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.08 GBytes 930 Mbits/sec 32 sender
[ 5] 0.00-10.00 sec 1.08 GBytes 928 Mbits/sec receiver
iperf Done.
ada:~ # iperf3 -R -4 -c worker29.oqa.prg2.suse.org
Connecting to host worker29.oqa.prg2.suse.org, port 5201
Reverse mode, remote host worker29.oqa.prg2.suse.org is sending
[ 5] local 10.146.4.1 port 36584 connected to 10.145.10.2 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 112 MBytes 935 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 942 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 942 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 942 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 182 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.
ada:~ # iperf3 -R -6 -c worker29.oqa.prg2.suse.org
Connecting to host worker29.oqa.prg2.suse.org, port 5201
Reverse mode, remote host worker29.oqa.prg2.suse.org is sending
[ 5] local 2a07:de40:b230:1:3eec:efff:feb9:b830 port 40908 connected to 2a07:de40:b203:12:7ec2:55ff:fe24:de9c port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 111 MBytes 927 Mbits/sec
[ 5] 1.00-2.00 sec 111 MBytes 929 Mbits/sec
[ 5] 2.00-3.00 sec 111 MBytes 928 Mbits/sec
[ 5] 3.00-4.00 sec 111 MBytes 929 Mbits/sec
[ 5] 4.00-5.00 sec 111 MBytes 928 Mbits/sec
[ 5] 5.00-6.00 sec 111 MBytes 928 Mbits/sec
[ 5] 6.00-7.00 sec 111 MBytes 929 Mbits/sec
[ 5] 7.00-8.00 sec 111 MBytes 929 Mbits/sec
[ 5] 8.00-9.00 sec 111 MBytes 928 Mbits/sec
[ 5] 9.00-10.00 sec 111 MBytes 928 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.08 GBytes 930 Mbits/sec 169 sender
[ 5] 0.00-10.00 sec 1.08 GBytes 928 Mbits/sec receiver
iperf Done.
Again everything seems fine into and out over this network-segment-border. My last try was with the actual 2 systems:
nessberry-2:~ # iperf3 -4 -c worker29.oqa.prg2.suse.org
Connecting to host worker29.oqa.prg2.suse.org, port 5201
[ 5] local 10.146.4.126 port 50830 connected to 10.145.10.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 115 MBytes 967 Mbits/sec 2 625 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 938 Mbits/sec 6 732 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 943 Mbits/sec 1 841 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 2 928 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 943 Mbits/sec 0 1018 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 3 1.07 MBytes
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 1.14 MBytes
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 1.21 MBytes
[ 5] 8.00-9.00 sec 112 MBytes 943 Mbits/sec 1 1.28 MBytes
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 0 1.34 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec 15 sender
[ 5] 0.00-10.01 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.
nessberry-2:~ # iperf3 -6 -c worker29.oqa.prg2.suse.org
Connecting to host worker29.oqa.prg2.suse.org, port 5201
[ 5] local 2a07:de40:b230:1:9877:a7ff:fec1:5f02 port 59546 connected to 2a07:de40:b203:12:7ec2:55ff:fe24:de9c port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 33 2.79 KBytes
[ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 36 2.79 KBytes
[ 5] 2.00-3.00 sec 256 KBytes 2.10 Mbits/sec 54 2.79 KBytes
[ 5] 3.00-4.00 sec 256 KBytes 2.10 Mbits/sec 32 2.79 KBytes
[ 5] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes
[ 5] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 33 2.79 KBytes
[ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes
[ 5] 8.00-9.00 sec 256 KBytes 2.10 Mbits/sec 24 2.79 KBytes
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 28 2.79 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.62 MBytes 1.36 Mbits/sec 312 sender
[ 5] 0.00-10.00 sec 1.50 MBytes 1.26 Mbits/sec receiver
iperf Done.
nessberry-2:~ # iperf3 -R -4 -c worker29.oqa.prg2.suse.org
Connecting to host worker29.oqa.prg2.suse.org, port 5201
Reverse mode, remote host worker29.oqa.prg2.suse.org is sending
[ 5] local 10.146.4.126 port 49298 connected to 10.145.10.2 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 112 MBytes 941 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 942 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 942 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 108 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.
nessberry-2:~ # iperf3 -R -6 -c worker29.oqa.prg2.suse.org
Connecting to host worker29.oqa.prg2.suse.org, port 5201
Reverse mode, remote host worker29.oqa.prg2.suse.org is sending
[ 5] local 2a07:de40:b230:1:9877:a7ff:fec1:5f02 port 37022 connected to 2a07:de40:b203:12:7ec2:55ff:fe24:de9c port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 111 MBytes 928 Mbits/sec
[ 5] 1.00-2.00 sec 111 MBytes 928 Mbits/sec
[ 5] 2.00-3.00 sec 111 MBytes 928 Mbits/sec
[ 5] 3.00-4.00 sec 111 MBytes 929 Mbits/sec
[ 5] 4.00-5.00 sec 111 MBytes 928 Mbits/sec
[ 5] 5.00-6.00 sec 111 MBytes 929 Mbits/sec
[ 5] 6.00-7.00 sec 111 MBytes 928 Mbits/sec
[ 5] 7.00-8.00 sec 111 MBytes 928 Mbits/sec
[ 5] 8.00-9.00 sec 111 MBytes 928 Mbits/sec
[ 5] 9.00-10.00 sec 111 MBytes 928 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.08 GBytes 930 Mbits/sec 117 sender
[ 5] 0.00-10.00 sec 1.08 GBytes 928 Mbits/sec receiver
iperf Done.
Updated by MDoucha 3 months ago · Edited
I think it should be noted that nessberry-2 -> worker29 transfer over IPv6 is almost 1000 times slower than all the other speed tests:
nicksinger wrote in #note-7:
nessberry-2:~ # iperf3 -6 -c worker29.oqa.prg2.suse.org Connecting to host worker29.oqa.prg2.suse.org, port 5201 [ 5] local 2a07:de40:b230:1:9877:a7ff:fec1:5f02 port 59546 connected to 2a07:de40:b203:12:7ec2:55ff:fe24:de9c port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 33 2.79 KBytes [ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 36 2.79 KBytes [ 5] 2.00-3.00 sec 256 KBytes 2.10 Mbits/sec 54 2.79 KBytes [ 5] 3.00-4.00 sec 256 KBytes 2.10 Mbits/sec 32 2.79 KBytes [ 5] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes [ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes [ 5] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 33 2.79 KBytes [ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes [ 5] 8.00-9.00 sec 256 KBytes 2.10 Mbits/sec 24 2.79 KBytes [ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 28 2.79 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.62 MBytes 1.36 Mbits/sec 312 sender [ 5] 0.00-10.00 sec 1.50 MBytes 1.26 Mbits/sec receiver
Updated by nicksinger 3 months ago
- Status changed from Workable to In Progress
MDoucha wrote in #note-8:
I think it should be noted that
nessberry-2 -> worker29transfer over IPv6 is almost 1000 times slower than all the other speed tests:nicksinger wrote in #note-7:
nessberry-2:~ # iperf3 -6 -c worker29.oqa.prg2.suse.org Connecting to host worker29.oqa.prg2.suse.org, port 5201 [ 5] local 2a07:de40:b230:1:9877:a7ff:fec1:5f02 port 59546 connected to 2a07:de40:b203:12:7ec2:55ff:fe24:de9c port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 33 2.79 KBytes [ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 36 2.79 KBytes [ 5] 2.00-3.00 sec 256 KBytes 2.10 Mbits/sec 54 2.79 KBytes [ 5] 3.00-4.00 sec 256 KBytes 2.10 Mbits/sec 32 2.79 KBytes [ 5] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes [ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes [ 5] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 33 2.79 KBytes [ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 24 2.79 KBytes [ 5] 8.00-9.00 sec 256 KBytes 2.10 Mbits/sec 24 2.79 KBytes [ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 28 2.79 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.62 MBytes 1.36 Mbits/sec 312 sender [ 5] 0.00-10.00 sec 1.50 MBytes 1.26 Mbits/sec receiver
Oh yes you're absolutely right. I will look into that one and try to add another commend in SD before they close it. Thanks for catching that!
Updated by openqa_review 3 months ago
- Due date set to 2025-01-01
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger 2 months ago
- Status changed from In Progress to Blocked
I'm currently waiting for some feedback in SD (https://sd.suse.com/servicedesk/customer/portal/1/SD-175581) if we can easily adjust the switch port settings before hacking around too much (since the webui of the HMC currently doesn't allow us to change MTU easily).
Updated by livdywan about 2 months ago
nicksinger wrote in #note-11:
I'm currently waiting for some feedback in SD (https://sd.suse.com/servicedesk/customer/portal/1/SD-175581) if we can easily adjust the switch port settings before hacking around too much (since the webui of the HMC currently doesn't allow us to change MTU easily).
Still waiting on SD-175581. Added another comment. I'm guessing Viktor may not have seen the message (Viktor is available according to Slack).
Updated by livdywan about 2 months ago
livdywan wrote in #note-13:
nicksinger wrote in #note-11:
I'm currently waiting for some feedback in SD (https://sd.suse.com/servicedesk/customer/portal/1/SD-175581) if we can easily adjust the switch port settings before hacking around too much (since the webui of the HMC currently doesn't allow us to change MTU easily).
Still waiting on SD-175581. Added another comment. I'm guessing Viktor may not have seen the message (Viktor is available according to Slack).
@nicksinger Did you see the response?
MTU on switch side has been lowered to 1500 on nessberry switch port
now I can see that server is sending large (jumbo) frames that being dropped due to size
please try to disable the large packets on egress (TX)
Updated by nicksinger about 2 months ago
- Status changed from Blocked to In Progress
livdywan wrote in #note-14:
livdywan wrote in #note-13:
nicksinger wrote in #note-11:
I'm currently waiting for some feedback in SD (https://sd.suse.com/servicedesk/customer/portal/1/SD-175581) if we can easily adjust the switch port settings before hacking around too much (since the webui of the HMC currently doesn't allow us to change MTU easily).
Still waiting on SD-175581. Added another comment. I'm guessing Viktor may not have seen the message (Viktor is available according to Slack).
@nicksinger Did you see the response?
MTU on switch side has been lowered to 1500 on nessberry switch port
now I can see that server is sending large (jumbo) frames that being dropped due to size
please try to disable the large packets on egress (TX)
Yes, I've conducted some more speed tests and worked together with Jacek to reduce the MTU for nessberry. Unfortunately v6 speeds are still slow so basically no change. I managed to recover the VIOS password and try to check what I can change there.
Currently I'm looking into the general setup and how I can understand the different interface (en1, ent1, ent4, sea5, etc…) and found https://www.ibm.com/support/pages/shared-ethernet-adapter-sea-simplification-no-dedicated-control-channel-adapter which explains a lot already.
Trying to adjust each and every device now to properly reflect our current configuration. Eventual findings and general advice to reproduce/debug I will add to https://gitlab.suse.de/suse/wiki/-/blob/main/qe_infrastructure.md
Updated by openqa_review about 2 months ago
- Due date set to 2025-01-28
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger about 1 month ago
- Status changed from In Progress to Feedback
Updated by nicksinger about 1 month ago
- Status changed from Feedback to Workable
- Priority changed from High to Normal
According to the feedback I received in Slack, I think we can lower the prio of the ticket. I will continue to debug this and if it produces more problems in OSD we can reconsider taking it out of production again.
Updated by nicksinger about 1 month ago
- Status changed from Workable to In Progress
iperf3 from "AIX Toolbox" complains about missing requirements. So tried to install "DNF" following https://www.ibm.com/support/pages/node/6585774 which requires me to install/upgrade to openssl 1.1.x.
# lslpp -Lc openssl.base
#Package Name:Fileset:Level:State:PTF Id:Fix State:Type:Description:Destination Dir.:Uninstaller:Message Catalog:Message Set:Message Number:Parent:Automatic:EFIX Locked:Install Path:Build Date
openssl.base:openssl.base:1.0.2.1500: : :C: :Open Secure Socket Layer: : : : : : :0:0:/:
confirms only version 1.0 installed, after following https://www.ibm.com/support/pages/downloading-and-installing-or-upgrading-openssl-and-openssh up until step 4:
# lslpp -Lc openssl.base
#Package Name:Fileset:Level:State:PTF Id:Fix State:Type:Description:Destination Dir.:Uninstaller:Message Catalog:Message Set:Message Number:Parent:Automatic:EFIX Locked:Install Path:Build Date
openssl.base:openssl.base:1.1.2.2401: : :C: :Open Secure Socket Layer: : : : : : :0:0:/:
But the script still failed (differently) to access the required resource. I had to manually download dnf_bundle_aix_71_72.tar, scp it onto nessberry-vios and modify dnf_aixtoolbox.sh to not invoke lwp-download but rather just move it into its own temp directory. After this the script finally completed:
# ./dl.sh -d
Attempting download of dnf_bundle_aix_71_72.tar ...
Extracting dnf_bundle_aix_71_72.tar ...
x ca-certificates-2023.2.60-0.aix7.1.ppc.rpm, 991206 bytes, 1936 tape blocks
x dnf-4.2.17-32_4.aix7.1.noarch.rpm, 11135 bytes, 22 tape blocks
x dnf-automatic-4.2.17-32_4.aix7.1.noarch.rpm, 12450 bytes, 25 tape blocks
x dnf-data-4.2.17-32_4.aix7.1.noarch.rpm, 23693 bytes, 47 tape blocks
x extra_list, 384 bytes, 1 tape blocks
x install_dnf.sh, 13552 bytes, 27 tape blocks
x libcomps-0.1.15-32_1.aix7.1.ppc.rpm, 1273688 bytes, 2488 tape blocks
x libdnf-0.39.1-32_3.aix7.1.ppc.rpm, 15816971 bytes, 30893 tape blocks
x libmodulemd-1.5.2-32_2.aix7.1.ppc.rpm, 3433754 bytes, 6707 tape blocks
x librepo-1.11.0-32_2.aix7.1.ppc.rpm, 3653390 bytes, 7136 tape blocks
x libsmartcols-2.34-32_1.aix7.1.ppc.rpm, 183412 bytes, 359 tape blocks
x libsolv-0.7.9-32_3.aix7.1.ppc.rpm, 1533540 bytes, 2996 tape blocks
x libzstd-1.4.4-32_1.aix7.1.ppc.rpm, 469194 bytes, 917 tape blocks
x p11-kit-0.23.22-0.aix7.1.ppc.rpm, 2148539 bytes, 4197 tape blocks
x p11-kit-tools-0.23.22-0.aix7.1.ppc.rpm, 274086 bytes, 536 tape blocks
x python3-3.9.16-0.aix7.1.ppc.rpm, 8920 bytes, 18 tape blocks
x python3-dateutil-2.8.0-2.aix7.1.noarch.rpm, 6128 bytes, 12 tape blocks
x python3-devel-3.9.16-0.aix7.1.ppc.rpm, 7699 bytes, 16 tape blocks
x python3-dnf-4.2.17-32_4.aix7.1.noarch.rpm, 7598 bytes, 15 tape blocks
x python3-dnf-plugin-migrate-4.0.16-32_2.aix7.1.noarch.rpm, 6568 bytes, 13 tape blocks
x python3-dnf-plugins-core-4.0.16-32_2.aix7.1.noarch.rpm, 6776 bytes, 14 tape blocks
x python3-gpg-1.13.1-32_3.aix7.1.ppc.rpm, 7040 bytes, 14 tape blocks
x python3-hawkey-0.39.1-32_3.aix7.1.ppc.rpm, 23264 bytes, 46 tape blocks
x python3-idle-3.9.16-0.aix7.1.ppc.rpm, 7283 bytes, 15 tape blocks
x python3-libcomps-0.1.15-32_1.aix7.1.ppc.rpm, 13360 bytes, 27 tape blocks
x python3-libdnf-0.39.1-32_3.aix7.1.ppc.rpm, 23264 bytes, 46 tape blocks
x python3-librepo-1.11.0-32_2.aix7.1.ppc.rpm, 12860 bytes, 26 tape blocks
x python3-six-1.13.0-2.aix7.1.noarch.rpm, 5820 bytes, 12 tape blocks
x python3-test-3.9.16-0.aix7.1.ppc.rpm, 6228 bytes, 13 tape blocks
x python3-tkinter-3.9.16-0.aix7.1.ppc.rpm, 6120 bytes, 12 tape blocks
x python3.9-3.9.16-0.aix7.1.ppc.rpm, 56782290 bytes, 110903 tape blocks
x python3.9-dateutil-2.8.0-2.aix7.1.noarch.rpm, 310485 bytes, 607 tape blocks
x python3.9-devel-3.9.16-0.aix7.1.ppc.rpm, 19947643 bytes, 38961 tape blocks
x python3.9-dnf-4.2.17-32_4.aix7.1.noarch.rpm, 196779 bytes, 385 tape blocks
x python3.9-dnf-plugin-migrate-4.0.16-32_2.aix7.1.noarch.rpm, 8242 bytes, 17 tape blocks
x python3.9-dnf-plugins-core-4.0.16-32_2.aix7.1.noarch.rpm, 33639 bytes, 66 tape blocks
x python3.9-gpg-1.13.1-32_3.aix7.1.ppc.rpm, 1192531 bytes, 2330 tape blocks
x python3.9-hawkey-0.39.1-32_3.aix7.1.ppc.rpm, 3116653 bytes, 6088 tape blocks
x python3.9-idle-3.9.16-0.aix7.1.ppc.rpm, 2328852 bytes, 4549 tape blocks
x python3.9-libcomps-0.1.15-32_1.aix7.1.ppc.rpm, 1049812 bytes, 2051 tape blocks
x python3.9-libdnf-0.39.1-32_3.aix7.1.ppc.rpm, 22550948 bytes, 44045 tape blocks
x python3.9-librepo-1.11.0-32_2.aix7.1.ppc.rpm, 2293000 bytes, 4479 tape blocks
x python3.9-six-1.13.0-2.aix7.1.noarch.rpm, 45630 bytes, 90 tape blocks
x python3.9-test-3.9.16-0.aix7.1.ppc.rpm, 33159212 bytes, 64765 tape blocks
x python3.9-tkinter-3.9.16-0.aix7.1.ppc.rpm, 1162787 bytes, 2272 tape blocks
x rpm-python3-4.15.1-32_2.aix7.1.ppc.rpm, 9496 bytes, 19 tape blocks
x rpm-python3.9-4.15.1-32_2.aix7.1.ppc.rpm, 98982 bytes, 194 tape blocks
x rpm.rte, 43051008 bytes, 84084 tape blocks
x yum-4.2.17-32_4.aix7.1.noarch.rpm, 8634 bytes, 17 tape blocks
x zchunk-1.1.4-32_2.aix7.1.ppc.rpm, 554294 bytes, 1083 tape blocks
x zchunk-devel-1.1.4-32_2.aix7.1.ppc.rpm, 19256 bytes, 38 tape blocks
x zchunk-libs-1.1.4-32_2.aix7.1.ppc.rpm, 534032 bytes, 1044 tape blocks
Installing rpm.rte at the latest version ...
This may take several minutes depending on the number of rpms installed...
+-----------------------------------------------------------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...
SUCCESSES
---------
Filesets listed in this section passed pre-installation verification
and will be installed.
Selected Filesets
-----------------
rpm.rte 4.15.1.1011 # RPM Package Manager
<< End of Success Section >>
+-----------------------------------------------------------------------------+
BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
1 Selected to be installed, of which:
1 Passed pre-installation verification
----
1 Total to be installed
+-----------------------------------------------------------------------------+
Installing Software...
+-----------------------------------------------------------------------------+
installp: APPLYING software for:
rpm.rte 4.15.1.1011
Checking if rpm package is already owning
some of the libraries and binaries..
. . . . . << Copyright notice for rpm.rte >> . . . . . . .
Licensed Materials - Property of IBM
5765H4000
Copyright International Business Machines Corp. 2014, 2020-2023.
Copyright "Cowboy" Ben Alman, 2010.
Copyright Steven Benner (http://stevenbenner.com/), 2013.
Copyright The Regents of the University of California, 1990, 1993, 1994.
Copyright Ralf Corsepius <corsepiu@fedoraproject.org>, 2007.
Copyright Vasil Dinkov, Vadikom Web Ltd., 2016.
Copyright The Dojo Foundation, 2011.
Copyright Ulrich Drepper <drepper@gnu.ai.mit.edu>, 1995-1997, 2000-2006.
Copyright Tom Dyas <tdyas@eden.rutgers.edu>, 1998.
Copyright W. L. Estes <wlestes@uncg.edu>, 1998.
Copyright Free Software Foundation, Inc., 1989, 1991-2018.
Copyright Dave Furfero, 2011-2014.
Copyright Neal Gompa <ngompa13@gmail.com>, 2015.
Copyright Dimitri van Heesch, 1997-2017.
Copyright The Internet Society (1998).
Copyright Per Oeyvind Karlsen <proyvind@moondrake.org>, 2010.
Copyright Dmitry V. Levin <ldv@altlinux.org>, 2019.
Copyright Dan Nicholson <dbn.lists@gmail.com>, 2012-2015.
Copyright Red Hat, Inc., 1998-2003, 2005, 2007, 2009-2011, 2016-2017.
Copyright Red Hat Software, 1995-2002.
Copyright Scott James Remnant <scott@netsplit.com>, 2004.
Copyright John Resig, 2011.
Copyright Enrico Scholz <enrico.scholz@informatik.tu-chemnitz.de>, 2004.
Copyright System Design and Research, 2003.
Copyright Julian Seward et al., 2002-2015.
Copyright Mark J. Wielaard <mark@klomp.org>, 2016, 2019.
Copyright X Consortium, 1994.
All rights reserved.
US Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for rpm.rte >>. . . .
##################################################################
This version of rpm installs files under the path /usr/opt/rpm.
More detailed information on new changes can be found in
/usr/opt/rpm/README file.
##################################################################
Please wait...
Successfully updated the Kernel Authorization Table.
Successfully updated the Kernel Role Table.
Successfully updated the Kernel Command Table.
Successfully updated the Kernel Device Table.
Successfully updated the Kernel Object Domain Table.
Successfully updated the Kernel Domains Table.
Successfully updated the Kernel RBAC log level.
Successfully updated the Kernel Authorization Table.
Successfully updated the Kernel Role Table.
Successfully updated the Kernel Command Table.
Successfully updated the Kernel Device Table.
Successfully updated the Kernel Object Domain Table.
Successfully updated the Kernel Domains Table.
Successfully updated the Kernel RBAC log level.
Finished processing all filesets. (Total time: 33 secs).
+-----------------------------------------------------------------------------+
Summaries:
+-----------------------------------------------------------------------------+
Installation Summary
--------------------
Name Level Part Event Result
-------------------------------------------------------------------------------
rpm.rte 4.15.1.1011 USR APPLY SUCCESS
rpm.rte 4.15.1.1011 ROOT APPLY SUCCESS
Checking whether any of the rpms from dnf_bundle are already installed...
Takes couple of minutes to process.
ca-certificates-2023.2.60-0 will be installed
dnf-4.2.17-32_4 will be installed
dnf-automatic-4.2.17-32_4 will be installed
dnf-data-4.2.17-32_4 will be installed
libcomps-0.1.15-32_1 will be installed
libdnf-0.39.1-32_3 will be installed
libmodulemd-1.5.2-32_2 will be installed
librepo-1.11.0-32_2 will be installed
libsmartcols-2.34-32_1 will be installed
libsolv-0.7.9-32_3 will be installed
libzstd-1.4.4-32_1 will be installed
p11-kit-0.23.22-0 will be installed
p11-kit-tools-0.23.22-0 will be installed
python3-3.9.16-0 will be installed
python3-dnf-4.2.17-32_4 will be installed
python3-gpg-1.13.1-32_3 will be installed
python3-hawkey-0.39.1-32_3 will be installed
python3-libcomps-0.1.15-32_1 will be installed
python3-libdnf-0.39.1-32_3 will be installed
python3-librepo-1.11.0-32_2 will be installed
python3.9-3.9.16-0 will be installed
python3.9-dnf-4.2.17-32_4 will be installed
python3.9-gpg-1.13.1-32_3 will be installed
python3.9-hawkey-0.39.1-32_3 will be installed
python3.9-libcomps-0.1.15-32_1 will be installed
python3.9-libdnf-0.39.1-32_3 will be installed
python3.9-librepo-1.11.0-32_2 will be installed
rpm-python3-4.15.1-32_2 will be installed
rpm-python3.9-4.15.1-32_2 will be installed
zchunk-libs-1.1.4-32_2 will be installed
Installing the packages...
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...
1:python3.9-3.9.16-0 ################################# [ 3%]
2:python3-3.9.16-0 ################################# [ 7%]
3:libzstd-1.4.4-32_1 ################################# [ 10%]
4:libmodulemd-1.5.2-32_2 ################################# [ 13%]
5:libsolv-0.7.9-32_3 ################################# [ 17%]
6:rpm-python3.9-4.15.1-32_2 ################################# [ 20%]
7:libsmartcols-2.34-32_1 ################################# [ 23%]
8:zchunk-libs-1.1.4-32_2 ################################# [ 27%]
9:librepo-1.11.0-32_2 ################################# [ 30%]
10:libdnf-0.39.1-32_3 ################################# [ 33%]
11:python3.9-libdnf-0.39.1-32_3 ################################# [ 37%]
12:python3.9-hawkey-0.39.1-32_3 ################################# [ 40%]
13:python3.9-gpg-1.13.1-32_3 ################################# [ 43%]
14:python3.9-librepo-1.11.0-32_2 ################################# [ 47%]
15:p11-kit-0.23.22-0 ################################# [ 50%]
16:p11-kit-tools-0.23.22-0 ################################# [ 53%]
17:libcomps-0.1.15-32_1 ################################# [ 57%]
18:python3.9-libcomps-0.1.15-32_1 ################################# [ 60%]
19:dnf-data-4.2.17-32_4 ################################# [ 63%]
20:python3.9-dnf-4.2.17-32_4 ################################# [ 67%]
21:python3-dnf-4.2.17-32_4 ################################# [ 70%]
22:dnf-4.2.17-32_4 ################################# [ 73%]
23:dnf-automatic-4.2.17-32_4 ################################# [ 77%]
24:python3-libcomps-0.1.15-32_1 ################################# [ 80%]
25:ca-certificates-2023.2.60-0 ################################# [ 83%]
26:python3-librepo-1.11.0-32_2 ################################# [ 87%]
27:python3-gpg-1.13.1-32_3 ################################# [ 90%]
28:python3-hawkey-0.39.1-32_3 ################################# [ 93%]
29:python3-libdnf-0.39.1-32_3 ################################# [ 97%]
30:rpm-python3-4.15.1-32_2 ################################# [100%]
dnf installed successfully.
Please run 'dnf update' to update packages to the latest level.
Please note, RPM packages are downloaded in dnf cache /var/cache/dnf.
RPM packages install files go under the path /opt.
Hence it is recommended to always keep at least 512MB of free space in /var & /opt
to avoid any download and installation/update failures.
I followed the recommended step and updated all packages:
# /opt/freeware/bin/dnf update
AIX generic repository 1.8 MB/s | 21 MB 00:11
AIX noarch repository 1.6 MB/s | 3.5 MB 00:02
AIX 7.2 specific repository 1.3 MB/s | 1.4 MB 00:01
Last metadata expiration check: 0:00:01 ago on Tue Jan 21 21:28:12 NFT 2025.
Dependencies resolved.
================================================================================================================================================================
Package Architecture Version Repository Size
================================================================================================================================================================
Upgrading:
ca-certificates ppc 2024.2.66-1 AIX_Toolbox 1.0 M
libcomps ppc 0.1.15-101 AIX_Toolbox 622 k
libmodulemd ppc 1.5.2-100 AIX_Toolbox 1.2 M
librepo ppc 1.11.0-103 AIX_Toolbox 336 k
libsmartcols ppc 2.34-101 AIX_Toolbox 614 k
libzstd ppc 1.5.2-3 AIX_Toolbox 1.5 M
p11-kit ppc 0.25.5-2 AIX_Toolbox 4.6 M
p11-kit-tools ppc 0.25.5-2 AIX_Toolbox 321 k
python3 ppc 3.9.20-1 AIX_Toolbox 9.0 k
python3-gpg ppc 1.13.1-101 AIX_Toolbox 7.1 k
python3-libcomps ppc 0.1.15-101 AIX_Toolbox 15 k
python3-librepo ppc 1.11.0-103 AIX_Toolbox 13 k
python3.9 ppc 3.9.20-1 AIX_Toolbox 40 M
python3.9-gpg ppc 1.13.1-101 AIX_Toolbox 1.7 M
python3.9-libcomps ppc 0.1.15-101 AIX_Toolbox 589 k
python3.9-librepo ppc 1.11.0-103 AIX_Toolbox 223 k
zchunk-libs ppc 1.1.4-103 AIX_Toolbox 824 k
dnf ppc 4.2.17-32_51 AIX_Toolbox_72 14 k
dnf-automatic ppc 4.2.17-32_51 AIX_Toolbox_72 12 k
dnf-data ppc 4.2.17-32_51 AIX_Toolbox_72 24 k
libdnf ppc 0.39.1-32_52 AIX_Toolbox_72 2.3 M
libsolv ppc 0.7.9-32_53 AIX_Toolbox_72 672 k
python3-dnf ppc 4.2.17-32_51 AIX_Toolbox_72 8.0 k
python3-hawkey ppc 0.39.1-32_52 AIX_Toolbox_72 23 k
python3-libdnf ppc 0.39.1-32_52 AIX_Toolbox_72 23 k
python3.9-dnf ppc 4.2.17-32_51 AIX_Toolbox_72 193 k
python3.9-hawkey ppc 0.39.1-32_52 AIX_Toolbox_72 409 k
python3.9-libdnf ppc 0.39.1-32_52 AIX_Toolbox_72 2.2 M
Installing dependencies:
bash ppc 5.2.21-1 AIX_Toolbox 3.0 M
bzip2 ppc 1.0.8-2 AIX_Toolbox 238 k
check ppc 0.13.0-1 AIX_Toolbox 93 k
curl ppc 8.11.1-1 AIX_Toolbox 2.0 M
cyrus-sasl ppc 2.1.28-1 AIX_Toolbox 1.2 M
db ppc 1:5.3.28-1 AIX_Toolbox 17 M
expat ppc 2.6.3-1 AIX_Toolbox 855 k
gdbm ppc 1.23-1 AIX_Toolbox 283 k
gettext ppc 0.21-2 AIX_Toolbox 14 M
glib2 ppc 2.81.0-1 AIX_Toolbox 16 M
gmp ppc 6.3.0-1 AIX_Toolbox 1.0 M
gnupg2 ppc 2.4.3-1 AIX_Toolbox 9.9 M
gnutls ppc 3.8.7-1 AIX_Toolbox 4.3 M
gpgme ppc 1.13.1-101 AIX_Toolbox 518 k
info ppc 7.0.2-1 AIX_Toolbox 643 k
json-c ppc 0.17-1 AIX_Toolbox 381 k
krb5-libs ppc 1.21.3-1 AIX_Toolbox 7.5 M
libassuan ppc 2.5.6-1 AIX_Toolbox 332 k
libffi ppc 3.4.4-2 AIX_Toolbox 73 k
libgcrypt ppc 1.10.3-1 AIX_Toolbox 2.7 M
libgpg-error ppc 1.49-1 AIX_Toolbox 515 k
libiconv ppc 1.17-1 AIX_Toolbox 1.6 M
libksba ppc 1.6.3-1 AIX_Toolbox 932 k
libnghttp2 ppc 1.62.1-1 AIX_Toolbox 311 k
libssh2 ppc 1.11.0-1 AIX_Toolbox 975 k
libtasn1 ppc 4.19.0-1 AIX_Toolbox 227 k
libtextstyle ppc 0.21-2 AIX_Toolbox 1.1 M
libunistring ppc 1.1-1 AIX_Toolbox 3.0 M
libxml2 ppc 2.12.9-1 AIX_Toolbox 3.1 M
libyaml ppc 0.2.5-1 AIX_Toolbox 509 k
ncurses ppc 6.4-1 AIX_Toolbox 3.8 M
nettle ppc 3.9.1-1 AIX_Toolbox 3.1 M
npth ppc 1.5-1 AIX_Toolbox 220 k
openldap ppc 2.5.16-2 AIX_Toolbox 4.0 M
pcre2 ppc 10.40-1 AIX_Toolbox 1.3 M
readline ppc 8.2-1 AIX_Toolbox 2.3 M
sqlite ppc 3.45.3-1 AIX_Toolbox 11 M
xz-libs ppc 5.4.3-1 AIX_Toolbox 575 k
zlib ppc 1.2.13-1 AIX_Toolbox 345 k
libgcc ppc 1:10-2 AIX_Toolbox_72 15 k
libgcc10 ppc 10.3.0-6 AIX_Toolbox_72 793 k
libgomp ppc 1:10-2 AIX_Toolbox_72 14 k
libgomp10 ppc 10.3.0-6 AIX_Toolbox_72 2.0 M
libstdc++ ppc 1:10-2 AIX_Toolbox_72 14 k
libstdc++10 ppc 10.3.0-6 AIX_Toolbox_72 19 M
Installing weak dependencies:
pinentry ppc 1.0.0-1 AIX_Toolbox 115 k
Transaction Summary
================================================================================================================================================================
Install 46 Packages
Upgrade 28 Packages
Total download size: 202 M
Is this ok [y/N]: y
Downloading Packages:
(1/74): check-0.13.0-1.aix7.1.ppc.rpm 355 kB/s | 93 kB 00:00
(2/74): bzip2-1.0.8-2.aix6.1.ppc.rpm 778 kB/s | 238 kB 00:00
(3/74): cyrus-sasl-2.1.28-1.aix6.1.ppc.rpm 1.7 MB/s | 1.2 MB 00:00
(4/74): curl-8.11.1-1.aix7.1.ppc.rpm 1.6 MB/s | 2.0 MB 00:01
(5/74): bash-5.2.21-1.aix7.1.ppc.rpm 1.5 MB/s | 3.0 MB 00:02
(6/74): expat-2.6.3-1.aix7.1.ppc.rpm 1.5 MB/s | 855 kB 00:00
(7/74): gdbm-1.23-1.aix7.1.ppc.rpm 1.8 MB/s | 283 kB 00:00
(8/74): db-5.3.28-1.aix6.1.ppc.rpm 1.9 MB/s | 17 MB 00:08
(9/74): gettext-0.21-2.aix7.1.ppc.rpm 1.7 MB/s | 14 MB 00:08
(10/74): gmp-6.3.0-1.aix7.1.ppc.rpm 1.6 MB/s | 1.0 MB 00:00
(11/74): glib2-2.81.0-1.aix7.1.ppc.rpm 1.7 MB/s | 16 MB 00:09
(12/74): gpgme-1.13.1-101.aix7.1.ppc.rpm 1.6 MB/s | 518 kB 00:00
(13/74): info-7.0.2-1.aix7.1.ppc.rpm 1.5 MB/s | 643 kB 00:00
(14/74): json-c-0.17-1.aix7.1.ppc.rpm 1.5 MB/s | 381 kB 00:00
(15/74): gnutls-3.8.7-1.aix7.1.ppc.rpm 1.8 MB/s | 4.3 MB 00:02
(16/74): libassuan-2.5.6-1.aix7.1.ppc.rpm 1.6 MB/s | 332 kB 00:00
(17/74): libffi-3.4.4-2.aix7.1.ppc.rpm 810 kB/s | 73 kB 00:00
(18/74): libgcrypt-1.10.3-1.aix7.1.ppc.rpm 1.8 MB/s | 2.7 MB 00:01
(19/74): libgpg-error-1.49-1.aix7.1.ppc.rpm 1.6 MB/s | 515 kB 00:00
(20/74): gnupg2-2.4.3-1.aix7.1.ppc.rpm 1.7 MB/s | 9.9 MB 00:05
(21/74): libiconv-1.17-1.aix7.1.ppc.rpm 1.7 MB/s | 1.6 MB 00:00
(22/74): libnghttp2-1.62.1-1.aix7.1.ppc.rpm 1.5 MB/s | 311 kB 00:00
(23/74): libksba-1.6.3-1.aix7.1.ppc.rpm 1.6 MB/s | 932 kB 00:00
(24/74): libtasn1-4.19.0-1.aix7.1.ppc.rpm 1.2 MB/s | 227 kB 00:00
(25/74): libssh2-1.11.0-1.aix7.1.ppc.rpm 1.8 MB/s | 975 kB 00:00
(26/74): krb5-libs-1.21.3-1.aix7.1.ppc.rpm 1.7 MB/s | 7.5 MB 00:04
(27/74): libtextstyle-0.21-2.aix7.1.ppc.rpm 1.6 MB/s | 1.1 MB 00:00
(28/74): libyaml-0.2.5-1.aix7.1.ppc.rpm 1.5 MB/s | 509 kB 00:00
(29/74): libunistring-1.1-1.aix7.1.ppc.rpm 1.8 MB/s | 3.0 MB 00:01
(30/74): libxml2-2.12.9-1.aix7.1.ppc.rpm 1.6 MB/s | 3.1 MB 00:01
(31/74): npth-1.5-1.aix6.1.ppc.rpm 1.4 MB/s | 220 kB 00:00
(32/74): ncurses-6.4-1.aix7.1.ppc.rpm 1.6 MB/s | 3.8 MB 00:02
(33/74): nettle-3.9.1-1.aix7.1.ppc.rpm 1.8 MB/s | 3.1 MB 00:01
(34/74): pinentry-1.0.0-1.aix6.1.ppc.rpm 1.3 MB/s | 115 kB 00:00
(35/74): pcre2-10.40-1.aix7.1.ppc.rpm 1.7 MB/s | 1.3 MB 00:00
(36/74): openldap-2.5.16-2.aix7.1.ppc.rpm 1.7 MB/s | 4.0 MB 00:02
(37/74): readline-8.2-1.aix7.1.ppc.rpm 1.7 MB/s | 2.3 MB 00:01
(38/74): zlib-1.2.13-1.aix7.1.ppc.rpm 1.7 MB/s | 345 kB 00:00
(39/74): xz-libs-5.4.3-1.aix7.1.ppc.rpm 1.7 MB/s | 575 kB 00:00
(40/74): libgcc-10-2.aix7.2.ppc.rpm 481 kB/s | 15 kB 00:00
(41/74): libgomp-10-2.aix7.2.ppc.rpm 479 kB/s | 14 kB 00:00
(42/74): libgcc10-10.3.0-6.aix7.2.ppc.rpm 1.5 MB/s | 793 kB 00:00
(43/74): libstdcplusplus-10-2.aix7.2.ppc.rpm 449 kB/s | 14 kB 00:00
(44/74): libgomp10-10.3.0-6.aix7.2.ppc.rpm 1.8 MB/s | 2.0 MB 00:01
(45/74): ca-certificates-2024.2.66-1.aix7.1.ppc.rpm 1.9 MB/s | 1.0 MB 00:00
(46/74): libcomps-0.1.15-101.aix7.1.ppc.rpm 1.7 MB/s | 622 kB 00:00
(47/74): libmodulemd-1.5.2-100.aix7.1.ppc.rpm 1.7 MB/s | 1.2 MB 00:00
(48/74): librepo-1.11.0-103.aix7.1.ppc.rpm 1.7 MB/s | 336 kB 00:00
(49/74): libsmartcols-2.34-101.aix7.1.ppc.rpm 1.6 MB/s | 614 kB 00:00
(50/74): libzstd-1.5.2-3.aix7.1.ppc.rpm 1.6 MB/s | 1.5 MB 00:00
(51/74): sqlite-3.45.3-1.aix7.1.ppc.rpm 1.7 MB/s | 11 MB 00:06
(52/74): p11-kit-tools-0.25.5-2.aix7.1.ppc.rpm 1.3 MB/s | 321 kB 00:00
(53/74): python3-3.9.20-1.aix7.1.ppc.rpm 280 kB/s | 9.0 kB 00:00
(54/74): python3-gpg-1.13.1-101.aix7.1.ppc.rpm 191 kB/s | 7.1 kB 00:00
(55/74): python3-libcomps-0.1.15-101.aix7.1.ppc.rpm 475 kB/s | 15 kB 00:00
(56/74): python3-librepo-1.11.0-103.aix7.1.ppc.rpm 431 kB/s | 13 kB 00:00
(57/74): p11-kit-0.25.5-2.aix7.1.ppc.rpm 1.8 MB/s | 4.6 MB 00:02
(58/74): python3.9-gpg-1.13.1-101.aix7.1.ppc.rpm 1.7 MB/s | 1.7 MB 00:00
(59/74): python3.9-libcomps-0.1.15-101.aix7.1.ppc.rpm 1.6 MB/s | 589 kB 00:00
(60/74): python3.9-librepo-1.11.0-103.aix7.1.ppc.rpm 1.5 MB/s | 223 kB 00:00
(61/74): zchunk-libs-1.1.4-103.aix7.1.ppc.rpm 1.7 MB/s | 824 kB 00:00
(62/74): dnf-4.2.17-32_51.aix7.2.ppc.rpm 444 kB/s | 14 kB 00:00
(63/74): dnf-automatic-4.2.17-32_51.aix7.2.ppc.rpm 403 kB/s | 12 kB 00:00
(64/74): dnf-data-4.2.17-32_51.aix7.2.ppc.rpm 806 kB/s | 24 kB 00:00
(65/74): libdnf-0.39.1-32_52.aix7.2.ppc.rpm 1.8 MB/s | 2.3 MB 00:01
(66/74): libsolv-0.7.9-32_53.aix7.2.ppc.rpm 1.7 MB/s | 672 kB 00:00
(67/74): python3-dnf-4.2.17-32_51.aix7.2.ppc.rpm 275 kB/s | 8.0 kB 00:00
(68/74): python3-hawkey-0.39.1-32_52.aix7.2.ppc.rpm 757 kB/s | 23 kB 00:00
(69/74): python3-libdnf-0.39.1-32_52.aix7.2.ppc.rpm 787 kB/s | 23 kB 00:00
(70/74): python3.9-dnf-4.2.17-32_51.aix7.2.ppc.rpm 1.7 MB/s | 193 kB 00:00
(71/74): python3.9-hawkey-0.39.1-32_52.aix7.2.ppc.rpm 1.6 MB/s | 409 kB 00:00
(72/74): libstdcplusplus10-10.3.0-6.aix7.2.ppc.rpm 1.7 MB/s | 19 MB 00:11
(73/74): python3.9-libdnf-0.39.1-32_52.aix7.2.ppc.rpm 1.7 MB/s | 2.2 MB 00:01
(74/74): python3.9-3.9.20-1.aix7.1.ppc.rpm 1.8 MB/s | 40 MB 00:22
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 4.0 MB/s | 202 MB 00:50
Running transaction check
Transaction check succeeded.
Running transaction test
Error: Transaction test error:
installing package db-1:5.3.28-1.ppc needs 32MB on the /opt filesystem
installing package libgomp10-10.3.0-6.ppc needs 41MB on the /opt filesystem
installing package libgomp-1:10-2.ppc needs 41MB on the /opt filesystem
installing package cyrus-sasl-2.1.28-1.ppc needs 45MB on the /opt filesystem
installing package openldap-2.5.16-2.ppc needs 59MB on the /opt filesystem
installing package libssh2-1.11.0-1.ppc needs 62MB on the /opt filesystem
installing package libnghttp2-1.62.1-1.ppc needs 63MB on the /opt filesystem
installing package libyaml-0.2.5-1.ppc needs 65MB on the /opt filesystem
installing package npth-1.5-1.ppc needs 66MB on the /opt filesystem
installing package pcre2-10.40-1.ppc needs 70MB on the /opt filesystem
installing package glib2-2.81.0-1.ppc needs 136MB on the /opt filesystem
installing package libtextstyle-0.21-2.ppc needs 140MB on the /opt filesystem
installing package libxml2-2.12.9-1.ppc needs 151MB on the /opt filesystem
installing package libunistring-1.1-1.ppc needs 159MB on the /opt filesystem
installing package libiconv-1.17-1.ppc needs 162MB on the /opt filesystem
installing package gettext-0.21-2.ppc needs 206MB on the /opt filesystem
installing package bash-5.2.21-1.ppc needs 216MB on the /opt filesystem
installing package info-7.0.2-1.ppc needs 218MB on the /opt filesystem
installing package libgpg-error-1.49-1.ppc needs 220MB on the /opt filesystem
installing package libmodulemd-1.5.2-100.ppc needs 225MB on the /opt filesystem
installing package libassuan-2.5.6-1.ppc needs 216MB on the /opt filesystem
installing package readline-8.2-1.ppc needs 224MB on the /opt filesystem
installing package sqlite-3.45.3-1.ppc needs 272MB on the /opt filesystem
installing package libsmartcols-2.34-101.ppc needs 275MB on the /opt filesystem
installing package libgcrypt-1.10.3-1.ppc needs 285MB on the /opt filesystem
installing package p11-kit-0.25.5-2.ppc needs 313MB on the /opt filesystem
installing package p11-kit-tools-0.25.5-2.ppc needs 303MB on the /opt filesystem
installing package ca-certificates-2024.2.66-1.ppc needs 305MB on the /opt filesystem
installing package libksba-1.6.3-1.ppc needs 307MB on the /opt filesystem
installing package pinentry-1.0.0-1.ppc needs 307MB on the /opt filesystem
installing package nettle-3.9.1-1.ppc needs 318MB on the /opt filesystem
installing package gnutls-3.8.7-1.ppc needs 334MB on the /opt filesystem
installing package gdbm-1.23-1.ppc needs 336MB on the /opt filesystem
installing package python3.9-3.9.20-1.ppc needs 504MB on the /opt filesystem
installing package python3-3.9.20-1.ppc needs 303MB on the /opt filesystem
installing package krb5-libs-1.21.3-1.ppc needs 331MB on the /opt filesystem
installing package curl-8.11.1-1.ppc needs 337MB on the /opt filesystem
installing package zchunk-libs-1.1.4-103.ppc needs 341MB on the /opt filesystem
installing package libsolv-0.7.9-32_53.ppc needs 342MB on the /opt filesystem
installing package gnupg2-2.4.3-1.ppc needs 367MB on the /opt filesystem
installing package gpgme-1.13.1-101.ppc needs 369MB on the /opt filesystem
installing package librepo-1.11.0-103.ppc needs 370MB on the /opt filesystem
installing package libdnf-0.39.1-32_52.ppc needs 373MB on the /opt filesystem
installing package python3.9-libdnf-0.39.1-32_52.ppc needs 346MB on the /opt filesystem
installing package python3.9-hawkey-0.39.1-32_52.ppc needs 275MB on the /opt filesystem
installing package python3.9-gpg-1.13.1-101.ppc needs 275MB on the /opt filesystem
installing package python3.9-librepo-1.11.0-103.ppc needs 270MB on the /opt filesystem
installing package libcomps-0.1.15-101.ppc needs 267MB on the /opt filesystem
installing package python3.9-libcomps-0.1.15-101.ppc needs 265MB on the /opt filesystem
installing package dnf-data-4.2.17-32_51.ppc needs 263MB on the /opt filesystem
installing package python3.9-dnf-4.2.17-32_51.ppc needs 264MB on the /opt filesystem
installing package python3-dnf-4.2.17-32_51.ppc needs 263MB on the /opt filesystem
installing package dnf-4.2.17-32_51.ppc needs 263MB on the /opt filesystem
installing package dnf-automatic-4.2.17-32_51.ppc needs 263MB on the /opt filesystem
installing package python3-libcomps-0.1.15-101.ppc needs 263MB on the /opt filesystem
installing package python3-librepo-1.11.0-103.ppc needs 263MB on the /opt filesystem
installing package python3-gpg-1.13.1-101.ppc needs 263MB on the /opt filesystem
installing package python3-hawkey-0.39.1-32_52.ppc needs 263MB on the /opt filesystem
installing package python3-libdnf-0.39.1-32_52.ppc needs 263MB on the /opt filesystem
Error Summary
-------------
Disk Requirements:
At least 504MB more space needed on the /opt filesystem.
not sure how to do this. But installing iperf3 with dnf worked:
# /opt/freeware/bin/dnf install iperf3
Last metadata expiration check: 0:02:08 ago on Tue Jan 21 21:28:12 NFT 2025.
Dependencies resolved.
================================================================================================================================================================
Package Architecture Version Repository Size
================================================================================================================================================================
Installing:
iperf3 ppc 3.16-1 AIX_Toolbox 505 k
Installing dependencies:
libgcc ppc 1:10-2 AIX_Toolbox_72 15 k
libgcc10 ppc 10.3.0-6 AIX_Toolbox_72 793 k
Transaction Summary
================================================================================================================================================================
Install 3 Packages
Total size: 1.3 M
Total download size: 505 k
Installed size: 4.3 M
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] libgcc-10-2.aix7.2.ppc.rpm: Already downloaded
[SKIPPED] libgcc10-10.3.0-6.aix7.2.ppc.rpm: Already downloaded
(3/3): iperf3-3.16-1.aix7.1.ppc.rpm 1.0 MB/s | 505 kB 00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 981 kB/s | 505 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : libgcc10-10.3.0-6.ppc 1/3
Installing : libgcc-1:10-2.ppc 2/3
Installing : iperf3-3.16-1.ppc 3/3
Verifying : iperf3-3.16-1.ppc 1/3
Verifying : libgcc-1:10-2.ppc 2/3
Verifying : libgcc10-10.3.0-6.ppc 3/3
Installed:
iperf3-3.16-1.ppc libgcc-1:10-2.ppc libgcc10-10.3.0-6.ppc
Complete!
now trying to conduct all speed tests again.
Updated by nicksinger about 1 month ago
worker29:~ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.146.4.134, port 61783
[ 5] local 10.145.10.2 port 5201 connected to 10.146.4.134 port 61784
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 66.6 MBytes 558 Mbits/sec
[ 5] 1.00-2.00 sec 111 MBytes 929 Mbits/sec
[ 5] 2.00-3.00 sec 108 MBytes 902 Mbits/sec
[ 5] 3.00-4.00 sec 111 MBytes 928 Mbits/sec
[ 5] 4.00-5.00 sec 110 MBytes 919 Mbits/sec
[ 5] 5.00-6.00 sec 106 MBytes 893 Mbits/sec
[ 5] 6.00-7.00 sec 107 MBytes 898 Mbits/sec
[ 5] 7.00-8.00 sec 106 MBytes 891 Mbits/sec
[ 5] 8.00-9.00 sec 104 MBytes 876 Mbits/sec
[ 5] 9.00-10.00 sec 110 MBytes 919 Mbits/sec
[ 5] 10.00-10.38 sec 42.2 MBytes 924 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.38 sec 1.06 GBytes 873 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
Accepted connection from 10.146.4.134, port 61788
[ 5] local 10.145.10.2 port 5201 connected to 10.146.4.134 port 61789
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 70.2 MBytes 589 Mbits/sec 0 133 KBytes
[ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec 0 133 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 937 Mbits/sec 0 133 KBytes
[ 5] 3.00-4.00 sec 113 MBytes 948 Mbits/sec 0 133 KBytes
[ 5] 4.00-5.00 sec 113 MBytes 948 Mbits/sec 0 133 KBytes
[ 5] 5.00-6.00 sec 113 MBytes 949 Mbits/sec 0 133 KBytes
[ 5] 6.00-7.00 sec 113 MBytes 948 Mbits/sec 0 133 KBytes
[ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec 0 133 KBytes
[ 5] 8.00-9.00 sec 113 MBytes 948 Mbits/sec 0 133 KBytes
[ 5] 9.00-10.00 sec 113 MBytes 950 Mbits/sec 0 133 KBytes
[ 5] 10.00-10.38 sec 43.2 MBytes 946 Mbits/sec 0 133 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.38 sec 1.10 GBytes 913 Mbits/sec 0 sender
-----------------------------------------------------------
Server listening on 5201 (test #3)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
while trying to do the same for v6, I broke the network for the VIOS completely… trying to repair it now
Updated by gpathak about 1 month ago
- Due date changed from 2025-01-28 to 2025-02-05
Updated by nicksinger about 1 month ago
- Status changed from Workable to In Progress
Updated by openqa_review about 1 month ago
- Due date set to 2025-02-15
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger 28 days ago
- Status changed from In Progress to Workable
So https://sd.suse.com/servicedesk/customer/portal/1/SD-175581 is now closed basically reducing the machines MTU down to 1500 and confirming that the network side up until reaching the machine (into the VIOS) is fine.
https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/5964 is opened and last Friday I managed to get all pipelines succeeding. This resolves a IP conflict between nessberry-vios and nessberry-9 but as nessberry-9 doesn't even exist I don't think this will bring much of an improvement.
I was just able to reproduce terrible uploads again from nessberry-3. Testing a direct connection to the VIOS yields results > 1Gbit/s in both directions. While searching for ideas what to try next I found https://www.ibm.com/support/pages/powervm-virtual-ethernet-speed-often-confused-vios-sea-speed explaining how we get "routed" to the outside. It shows the "Vitual Ethernet run by Hypervisor" and currently the only thing looking like it could be our bottleneck (as the VIOS is fine, I would assume the SEA is fine as well).
I currently only see a way forward in trying to debug these single components involved. As I don't know how this is done and the previous steps already took a very long time, I don't think this is currently feasible for me to archive. Maybe somebody else should have a different perspective on this issue at hand.
Updated by okurz 28 days ago
I suggest you consult with PowerPC domain experts outside our squad, e.g. in https://app.slack.com/client/T02863RC2AC/C04K6388YUX "discuss-powerpc-architecture"
Updated by nicksinger 27 days ago
okurz wrote in #note-27:
I suggest you consult with PowerPC domain experts outside our squad, e.g. in https://app.slack.com/client/T02863RC2AC/C04K6388YUX "discuss-powerpc-architecture"
Did so with https://suse.slack.com/archives/C04K6388YUX/p1738669205334109
Running an continuous iperf on nessberry-3 while checking the "Performance Monitoring" in the HMC shows no significant resource usage hinting to a too slow VIOS. Lets see if other experts have other ideas
Updated by okurz 27 days ago
- Related to action #133700: Network bandwidth graphs per switch, like https://mrtg.suse.de/qanet13nue, for all current top-of-rack switches (TORs) that we are connected to size:M added
Updated by nicksinger 27 days ago
- Status changed from Workable to In Progress
While collecting some more information I took another look at the running iperf traffic in wireshark. I noticed some checksum errors in some packages and wondered if this can be seen in Linux as well. awk '$1 ~ "Tcp:" { print $13 }' /proc/net/snmp; on nessberry-3 showed >30k retransmissions growing fast if running iperf and stopping if I stop iperf. This prompted me look closer into our hardware configuration on the VIOS (all commands executed in after oem_setup_env and after my changes). We have:
# lsdev | grep ^en
en0 Defined 03-00 Standard Ethernet Network Interface
en1 Defined 03-01 Standard Ethernet Network Interface
en2 Defined 03-02 Standard Ethernet Network Interface
en3 Defined 03-03 Standard Ethernet Network Interface
en4 Defined Standard Ethernet Network Interface
en5 Available Standard Ethernet Network Interface
ent0 Available 03-00 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent1 Available 03-01 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent2 Available 03-02 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent3 Available 03-03 4-Port Gigabit Ethernet PCI-Express Adapter (e414571614102004)
ent4 Available Virtual I/O Ethernet Adapter (l-lan)
ent5 Available Shared Ethernet Adapter
The VIOS itself has:
# ifconfig -a
en5: flags=1e084863,18c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 10.146.4.133 netmask 0xfffffe00 broadcast 10.146.5.255
I'm not sure which physical port this corresponds to but it could be a different one used by the LPARs. Given we only have one "Shared Ethernet Adapter" and it shows up in the "Virtual Network Diagram" inside the HMC, I decided to look closer into it:
# lsattr -El ent5
accounting disabled Enable per-client accounting of network statistics True
adapter_reset no Reset real adapter on HA takeover True
ctl_chan Control Channel adapter for SEA failover True
fb_delay 30 Delay before failback occurs (seconds) True
ff_action recover Action to take for SEA flipflop True
ff_detect disabled Enable flipflop detection True
gvrp no Enable GARP VLAN Registration Protocol (GVRP) True
ha_mode disabled High Availability Mode True
hash_algo 0 Hash algorithm used to select a SEA thread True
health_time 60 Time in seconds required until the SEA is deemed healthy True
jumbo_frames no Enable Gigabit Ethernet Jumbo Frames True
large_receive no Enable receive TCP segment aggregation True
largesend 1 Enable Hardware Transmit TCP Resegmentation True
link_time 60 Time in seconds required for the link to be declared healthy after a status change True
lldpsvc no Enable IEEE 802.1qbg services True
netaddr 0 Address to ping True
noauto_failback disabled Disable auto failback True
nthreads 7 Number of SEA threads in Thread mode True
plso_bridge yes Enable Platform Large Send bridge mode True
pvid 1 PVID to use for the SEA device True
pvid_adapter ent4 Default virtual adapter to use for non-VLAN-tagged packets True
qos_mode disabled Adapters to use when the primary channel fails True
queue_size 8192 Queue size for a SEA thread True
real_adapter ent1 Physical adapter associated with the SEA True
send_RARP yes Transmit Reverse ARP after HA takeover True
thread 1 Thread mode enabled (1) or disabled (0) True
virt_adapters ent4 List of virtual adapters associated with the SEA (comma separated) True
virt_adapters and real_adapter look interesting and ent1 indeed also corresponds to a physical interface on the network card. Attributes of that port show:
# lsattr -El ent1
alt_addr 0x000000000000 Alternate ethernet address True
busintr 25 Bus interrupt level False
busmem 0xa0180000 Bus memory address False
chksum_offload yes Enable hardware transmit and receive checksum True
delay_open no Enable delay of open until link state is known True
flow_ctrl yes Enable Transmit and Receive Flow Control True
gen_intr no Enable additional interrupt by hardware if there are packets to process True
intr_priority 3 Interrupt priority False
ipv6_offload yes Enable large send and checksum offload for IPv6 packets True
jumbo_frames no Transmit jumbo frames True
large_receive yes Enable receive TCP segment aggregation True
large_send yes Enable hardware transmit TCP resegmentation True
media_speed Auto_Negotiation Media speed True
receive_proc 16 RX packets processed before PIO to adapter True
rx_bds_done 5 RX packets before RX interrupt True
rx_coalesce 16 Receive packet coalesce count True
rx_hog 1000 RX buffers processed per RX interrupt True
rx_queues 1 Number of Receive queues True
rx_ticks_done 50 Clock ticks before RX interrupt True
rxbuf_pool_sz 4096 Rcv buffer pool size True
rxdesc_que_sz 2048 RX descriptor queue size True
slih_hog 10 Max Interrupt events processed per interrupt True
tx_bds_done 0 TX buffers used before TX done interrupt True
tx_que_sz 8192 Software transmit queue size True
tx_queues 1 Number of Transmit queues True
tx_ticks_done 1023 Clock ticks before TX interrupt True
txdesc_que_sz 512 TX descriptor queue size True
udp_rss_hash no Enable UDP RSS hash functionality True
use_alt_addr no Enable alternate ethernet address True
Despite the use of the word "large", the description of large_send and large_receive mention "TCP segment" and "resegmentation" which sounded related. Also "jumbo_frames" is listed as separate option. So I tried to enable them on ent1 but also some other useful sounding options on ent4:
chdev -l ent4 -a chksum_offload=yes -P
chdev -l ent4 -a ipv6_offload=yes -P
chdev -l ent1 -a large_send=yes -P
chdev -l ent1 -a large_receive=yes -P
"-P" and a reboot of the VIOS is the easiest way to apply these settings without getting any "device in use" errors. With these changes applied, I get 942 Mbits/sec in iperf to from and to worker29. I'm going to restart the whole machine now to make sure I made these changes persistent and don't miss any transient changes in configuration.
Updated by nicksinger 27 days ago
- Status changed from In Progress to Resolved
nessberry LPARs now have gigabit performance after a full machine reboot so everything is configured and permanent.
Actions