action #128030: IPMI of ix64ph1075.qa.suse.de is not accessible size:M - openQA Infrastructure - openSUSE Project Management Tool

Actions

Copy link

action #128030

closed

IPMI of ix64ph1075.qa.suse.de is not accessible size:M

Added by Julie_CAO over 1 year ago. Updated over 1 year ago.

Status:

Resolved

Priority:

High

Assignee:

okurz

Category:

Target version:

openQA Project - Ready

Start date:

2023-04-20

Due date:

% Done:

Estimated time:

Tags:

infra

Description

The baremental machine in OSD worker pool began not to work yesterday. Its IPMI is not ping'able.

     16:
        WORKER_CLASS: 64bit-ipmi,ix64ph1075
        IPMI_HOSTNAME: 10.162.28.200
        IPMI_PASSWORD: xxx
        IPMI_USER: admin
        SUT_IP: ix64ph1075.qa.suse.de

ix64ph1087:~ # ping 10.162.28.200
PING 10.162.28.200 (10.162.28.200) 56(84) bytes of data.
From 10.162.2.87 icmp_seq=1 Destination Host Unreachable
From 10.162.2.87 icmp_seq=2 Destination Host Unreachable
From 10.162.2.87 icmp_seq=3 Destination Host Unreachable

Could you help with it or shall I open an infra ticket? @osd-admins

Actions

Copy link

Updated by okurz over 1 year ago

Tags set to infra
Status changed from New to In Progress
Assignee set to okurz
Priority changed from Normal to High
Target version set to Ready

sorry about that, my mistake in https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/54

Actions

Copy link

Updated by okurz over 1 year ago

Status changed from In Progress to Blocked

https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/56 and https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/521

I could not identify the current IP addresses of the machine or management interface due to #128045 and https://racktables.nue.suse.com/index.php?page=object&tab=default&object_id=1534 does not list a remote switchable power supply.

$ host ix64ph1075.qa.suse.de; host ix64hm1200.qa.suse.de; ping -c1 -4 ix64hm1200.qa.suse.de; ping -c1 ix64hm1200.qa.suse.de
ix64ph1075.qa.suse.de has address 10.162.2.75
ix64ph1075.qa.suse.de has IPv6 address 2620:113:80c0:80a0:10:162:2:75
ix64hm1200.qa.suse.de has address 10.162.28.200
ix64hm1200.qa.suse.de has IPv6 address 2620:113:80c0:80a0:10:162:28:200
PING  (10.162.28.200) 56(84) bytes of data.
64 bytes from ix64hm1200.qa.suse.de (10.162.28.200): icmp_seq=1 ttl=62 time=24.3 ms

---  ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 24.349/24.349/24.349/0.000 ms
PING ix64hm1200.qa.suse.de(2620:113:80c0:80a0:10:162:28:200 (2620:113:80c0:80a0:10:162:28:200)) 56 data bytes
From 2620:113:80c0:8120::6 (2620:113:80c0:8120::6) icmp_seq=1 Destination unreachable: Address unreachable

--- ix64hm1200.qa.suse.de ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

so the A+AAAA records are fine and IPv4 works but IPv6 not or not yet. Maybe the machine will eventually show up with the correct address so for now I reverted the change to DNS record in worker conf with https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/522. Until then blocked by #128045

Actions

Copy link

Updated by Julie_CAO over 1 year ago

FYI, we have another SUT which was not available yesterday, quinn-sp.qa.suse.de. I don't know if it is related to changes to the links you mentioned as the gitlab is temparialy inaccessible for now from my side.

0:kermit-1:~ # ping quinn-sp.qa.suse.de
ping: quinn-sp.qa.suse.de: Name or service not known
0:kermit-1:~ # ping 10.162.3.87
PING 10.162.3.87 (10.162.3.87) 56(84) bytes of data.
^C
--- 10.162.3.87 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3054ms

Actions

Copy link

Updated by okurz over 1 year ago

Due date set to 2023-05-05
Status changed from Blocked to Feedback

Thanks for letting me know. The DNS entry for "quinn" was deleted by me by mistake. Fixed in https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/58

Actions

Copy link

Updated by okurz over 1 year ago

#128045 resolved. quinn and ix64ph1075 are reachable again with exception of ix64hm1200.qa.suse.de over IPv6 which does not work yet. The configuration in http://10.162.28.200/ looks ok, maybe need to change to "DHCPv6 Stateful"?

Actions

Copy link

Updated by Julie_CAO over 1 year ago

ix64ph1075 is reachable, but quinn-sp.qa.suse.de is still not available. The resolving path looks weired.

susetest:~ # ping quinn-sp.qa.suse.de
PING ix64mm1087.qa.suse.de (10.162.3.87) 56(84) bytes of data.
From ix64ph1075.qa.suse.de (10.162.2.75) icmp_seq=1 Destination Host Unreachable
From ix64ph1075.qa.suse.de (10.162.2.75) icmp_seq=2 Destination Host Unreachable
From ix64ph1075.qa.suse.de (10.162.2.75) icmp_seq=3 Destination Host Unreachable

Actions

Copy link

Updated by okurz over 1 year ago

Subject changed from IPMI of ix64ph1075.qa.suse.de is not accessible to IPMI of ix64ph1075.qa.suse.de is not accessible size:M
Due date deleted (~~2023-05-05~~)

Julie_CAO wrote: