Project

General

Profile

Actions
Copy link

action #128030

closed

IPMI of ix64ph1075.qa.suse.de is not accessible size:M

Added by Julie_CAO over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project (public) - Ready
Start date:
2023-04-20
Due date:
% Done:

0%

Estimated time:
Tags:
infra

Description

The baremental machine in OSD worker pool began not to work yesterday. Its IPMI is not ping'able.

     16:
        WORKER_CLASS: 64bit-ipmi,ix64ph1075
        IPMI_HOSTNAME: 10.162.28.200
        IPMI_PASSWORD: xxx
        IPMI_USER: admin
        SUT_IP: ix64ph1075.qa.suse.de

ix64ph1087:~ # ping 10.162.28.200
PING 10.162.28.200 (10.162.28.200) 56(84) bytes of data.
From 10.162.2.87 icmp_seq=1 Destination Host Unreachable
From 10.162.2.87 icmp_seq=2 Destination Host Unreachable
From 10.162.2.87 icmp_seq=3 Destination Host Unreachable

Could you help with it or shall I open an infra ticket? @osd-admins

Actions
Copy link
 #1

Updated by okurz over 1 year ago

  • Tags set to infra
  • Status changed from New to In Progress
  • Assignee set to okurz
  • Priority changed from Normal to High
  • Target version set to Ready
Actions
Copy link
 #2

Updated by okurz over 1 year ago

  • Status changed from In Progress to Blocked

https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/56 and https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/521

I could not identify the current IP addresses of the machine or management interface due to #128045 and https://racktables.nue.suse.com/index.php?page=object&tab=default&object_id=1534 does not list a remote switchable power supply.

$ host ix64ph1075.qa.suse.de; host ix64hm1200.qa.suse.de; ping -c1 -4 ix64hm1200.qa.suse.de; ping -c1 ix64hm1200.qa.suse.de
ix64ph1075.qa.suse.de has address 10.162.2.75
ix64ph1075.qa.suse.de has IPv6 address 2620:113:80c0:80a0:10:162:2:75
ix64hm1200.qa.suse.de has address 10.162.28.200
ix64hm1200.qa.suse.de has IPv6 address 2620:113:80c0:80a0:10:162:28:200
PING  (10.162.28.200) 56(84) bytes of data.
64 bytes from ix64hm1200.qa.suse.de (10.162.28.200): icmp_seq=1 ttl=62 time=24.3 ms

---  ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 24.349/24.349/24.349/0.000 ms
PING ix64hm1200.qa.suse.de(2620:113:80c0:80a0:10:162:28:200 (2620:113:80c0:80a0:10:162:28:200)) 56 data bytes
From 2620:113:80c0:8120::6 (2620:113:80c0:8120::6) icmp_seq=1 Destination unreachable: Address unreachable

--- ix64hm1200.qa.suse.de ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

so the A+AAAA records are fine and IPv4 works but IPv6 not or not yet. Maybe the machine will eventually show up with the correct address so for now I reverted the change to DNS record in worker conf with https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/522. Until then blocked by #128045

Actions
Copy link
 #3

Updated by Julie_CAO over 1 year ago

FYI, we have another SUT which was not available yesterday, quinn-sp.qa.suse.de. I don't know if it is related to changes to the links you mentioned as the gitlab is temparialy inaccessible for now from my side.

0:kermit-1:~ # ping quinn-sp.qa.suse.de
ping: quinn-sp.qa.suse.de: Name or service not known
0:kermit-1:~ # ping 10.162.3.87
PING 10.162.3.87 (10.162.3.87) 56(84) bytes of data.
^C
--- 10.162.3.87 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3054ms
Actions
Copy link
 #4

Updated by okurz over 1 year ago

  • Due date set to 2023-05-05
  • Status changed from Blocked to Feedback

Thanks for letting me know. The DNS entry for "quinn" was deleted by me by mistake. Fixed in https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/58

Actions
Copy link
 #5

Updated by okurz over 1 year ago

#128045 resolved. quinn and ix64ph1075 are reachable again with exception of ix64hm1200.qa.suse.de over IPv6 which does not work yet. The configuration in http://10.162.28.200/ looks ok, maybe need to change to "DHCPv6 Stateful"?

Actions
Copy link
 #6

Updated by Julie_CAO over 1 year ago

ix64ph1075 is reachable, but quinn-sp.qa.suse.de is still not available. The resolving path looks weired.

susetest:~ # ping quinn-sp.qa.suse.de
PING ix64mm1087.qa.suse.de (10.162.3.87) 56(84) bytes of data.
From ix64ph1075.qa.suse.de (10.162.2.75) icmp_seq=1 Destination Host Unreachable
From ix64ph1075.qa.suse.de (10.162.2.75) icmp_seq=2 Destination Host Unreachable
From ix64ph1075.qa.suse.de (10.162.2.75) icmp_seq=3 Destination Host Unreachable
Actions
Copy link
 #7

Updated by okurz over 1 year ago

  • Subject changed from IPMI of ix64ph1075.qa.suse.de is not accessible to IPMI of ix64ph1075.qa.suse.de is not accessible size:M
  • Due date deleted (2023-05-05)

Julie_CAO wrote:

ix64ph1075 is reachable, but quinn-sp.qa.suse.de is still not available. The resolving path looks weired.

Fixed with https://gitlab.suse.de/qa-sle/qanet-configs/-/commit/586c80f8e9f85ddd362fb78311e8ebfb285fdfd3

For ix64hm1200 deleted the AAAA record as we couldn't get the firmware to use IPv6 properly, https://gitlab.suse.de/qa-sle/qanet-configs/-/commit/e2169f204bfcc237930e81d458ea5a6b59e159f4

enabled to use hostname again in workerconf: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/525

Actions
Copy link
 #8

Updated by okurz over 1 year ago

  • Status changed from Feedback to Resolved

Crosschecked ix64ph1075 and quinn again, looks good.

Actions
Copy link

Also available in: Atom PDF