Project

General

Profile

Actions
Copy link

action #118885

closed

No cable connection or link to the 2nd network card in amd-zen3-gpu-sut1-1

Added by Julie_CAO over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
2022-10-14
Due date:
2022-10-28
% Done:

0%

Estimated time:
Tags:
next-office-day

Description

Hi Nick or Oliver or tools team,

amd-zen3-gpu-sut1-1 in OSD ipmi worker pool should have had 3 network connections as described in hcpd.conf, 1st for ipmi, 2nd for the master embeded network card 'em1', 3th for the secondary netword card 'p3p1'.

but we found 'p3p1' has no link detected. Could you please help to check the cable connection?

  host amd-zen3-gpu-sut1-sp                          { hardware ethernet ec:2a:72:0c:25:4c; fixed-address 10.162.2.131; option host-name "amd-zen3-gpu-sut1-sp"; }
  host amd-zen3-gpu-sut1-1                           { hardware ethernet ec:2a:72:02:84:20; fixed-address 10.162.2.132; option host-name "amd-zen3-gpu-sut1-1"; filename "pxelinux.0"; }
  host amd-zen3-gpu-sut1-2                           { hardware ethernet b4:96:91:9c:5a:d4; fixed-address 10.162.2.133; option host-name "amd-zen3-gpu-sut1-2"; filename "pxelinux.0"; }

amd-zen3-gpu-sut1-1:~ # ip a
1: lo: ...
2: p3p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b4:96:91:9c:5a:d4 brd ff:ff:ff:ff:ff:ff
    altname enp65s0f0
3: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether ec:2a:72:02:84:20 brd ff:ff:ff:ff:ff:ff
    altname eno8303
    altname enp225s0f0
4: em2: ...
5: p3p2: ...
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ec:2a:72:02:84:20 brd ff:ff:ff:ff:ff:ff
    inet 10.162.2.132/18 brd 10.162.63.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::ee2a:72ff:fe02:8420/64 scope link 
       valid_lft forever preferred_lft forever

amd-zen3-gpu-sut1-1:~ # ethtool p3p1
Settings for p3p1:
    Supported ports: [  ]
    ...
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: Other
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
    **Link detected: no**
Actions
Copy link
 #1

Updated by okurz over 1 year ago

  • Priority changed from Normal to High
  • Target version set to Ready
Actions
Copy link
 #2

Updated by okurz over 1 year ago

  • Assignee set to okurz

I plan to visit the server location

Actions
Copy link
 #3

Updated by okurz over 1 year ago

  • Tags set to next-office-day
  • Due date set to 2022-10-28
  • Status changed from New to Feedback

I connected p3p1, MAC b4:96:91:9c:5a:d4, to the port gi30 of the switch qanet02nue.qa.suse.de. I updated racktables in https://racktables.nue.suse.com/index.php?page=object&tab=ports&object_id=16390 with the connection and confirmed within the switch that the connection works as well as within the machine as I could login over SSH to amd-zen3-gpu-sut1-1.

EDIT: ethtool still shows no link detected, likely because the interface is not configured for network usage within the OS:

# ethtool p3p1
…
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Twisted Pair
…
    Link detected: no

The command ip link set dev p3p1 up changes that to "yes"

The journal shows:

Oct 17 04:25:57 amd-zen3-gpu-sut1-1 kernel: i40e 0000:41:00.0 p3p1: NIC Link is Down
Oct 17 04:26:01 amd-zen3-gpu-sut1-1 kernel: i40e 0000:41:00.0 p3p1: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None

so the device should be usable now.

@Julie_CAO can you confirm this works for you?

Actions
Copy link
 #4

Updated by Julie_CAO over 1 year ago

  • Status changed from Feedback to Resolved

Thank you, @okurz .

I just verified p3p1 has been connected and gotten it IP. Thanks for the quick fix.

2: p3p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b4:96:91:9c:5a:d4 brd ff:ff:ff:ff:ff:ff
    altname enp65s0f0
    inet 10.162.2.133/18 brd 10.162.63.255 scope global p3p1
       valid_lft forever preferred_lft forever
    inet6 2620:113:80c0:80a0:10:162:31:a4b6/64 scope global dynamic noprefixroute 
       valid_lft 2591975sec preferred_lft 1619975sec
    inet6 fe80::b696:91ff:fe9c:5ad4/64 scope link 
       valid_lft forever preferred_lft forever
Actions
Copy link

Also available in: Atom PDF