QA » openQA Project » openQA Infrastructure

chan-1 is not recovered yet.
Ticket https://sd.suse.com/servicedesk/customer/portal/1/SD-70018

The corresponding worker is disabled
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/374

Suggest infra guys to contact vendor for further support.

This issue has associated bugzilla
https://bugzilla.suse.com/show_bug.cgi?id=1194105

okurz wrote:

the ticket was blocked on https://bugzilla.suse.com/show_bug.cgi?id=1194105 which is VERIFIED FIXED so we can move (back) to "New" and clarify what needs to be done next.

@nicksinger https://racktables.nue.suse.com/?page=search&last_page=index&last_tab=default&q=chan-1 does not resolve anything. What is this machine?

I previously updated the description to include the most recent and relevant ticket. There is also information what was done and what we're waiting (feedback from vendor). The machine in question is https://racktables.nue.suse.com/index.php?page=object&object_id=13554

nicksinger and me checked the machine together. Using ssh qanet14nue.qa.suse.de, disconnecting and reconnecting the BMC LAN cable we could verify that the machine's BMC is connected to gi17 on qanet14nue.qa.suse.de. show mac address-table interface GigabitEthernet 17 showed the MAC address to be e0:d5:5e:a7:e8:34. ssh qanet and tail -n 10000 /var/log/dhcpd.log | grep -i e0:d5:5e:a7:e8:34 revealed the IPv4 address of the BMC 10.162.3.68 . Over browser we could connect to the BMC and found the machine to be reported to be generally ok. The presence of two CPUs was reported. Surprisingly "Hardware > Memory" reported DIMM slots to be occupied but we did remove all. Maybe the BMC needs to have such information from at least one successful POST of the system to know such information. We have then checked all different kind of memory configurations. In no case could we hear any tones from the on-board buzzer and no content showed on the VGA output. The BMC seems to be fully operational. We did unscrew again the heatsink on CPU1 (secondary) and could confirm that the CPU is soldered into the socket and not unpluggable. Support by the manufacturer is advised.

I asked for further information in racktables by email, see email on osd-admins@suse.de

Server informaton from Invoice:
PH-R150-T62
Phoenics 1U 8-bay Server
S/N :GIG3N7612A0015
A/T :SU10468

Detailed specifications from order:
Motherboard
System on Chip (SoC)
Cavium® ThunderXTM ARM processor
64bit ARMv8 architecture, BGA 2601, 28nm
8 DIMM Slots (max. 1024GB DDR4)
1 x 40GbE QSFP+ LAN Port and
4 x 10GbE SFP+ LAN Ports
4x SATA3 (6Gb/s) Ports (RAID 0,1,5,10)

PCI Slot:
1x PCI-E 3.0 x8 Slot (FHHL)
Integrated IPMI 2.0 with Dedicated LAN
Integrated Aspeed AST2400 BMC
2x USB 3.0

CPU
1 x Cavium ARM ThunderX
48 Cores pro CPU, 2,0 GHz

RAM
4 x Micron MTA9ASF1G72PZ-2G9
= 64 GB, 8 x 8 GB

Disk
2 x Intel S4610 SSDSC2KG480G801
480 GB, SSD, 3,42 DWPD

Official website for this model:
https://www.gigabyte.com/Enterprise/ARM-Server/R150-T62-rev-100

Actually both chan-1 and chow-1 are down now in NUE. Probably due to the same issue. I already added this information in SD-70018 months ago. So I think @nsinger is already aware of this.

the ticket came up in our "SLO high" query as it was not updated within 30 days. It is still important to handle the one or both machines but also tasks like labs move are more important for now.

okurz wrote:

codec responded so you need to ping gigabyte directly

It is pleasing to here this good news ? Could you help share more info if there is ?

And, by the way, this ticket https://sd.suse.com/servicedesk/customer/portal/1/SD-70018 had been closed due to there is already internal discussion in the mean time. I think it points to the discussion here within QA.

nicksinger wrote:

I've created an account on the gigabyte "eSupport" website and raised our problems there and mentioned that codec cannot provide us help.
The link to the ticket is https://esupport.gigabyte.com/Default/#id:1374724 but most likely you cannot see this without being logged in. I will keep it monitored.

I can not see it even logged in. "No record".

@nicksinger, as discussed, just send the hardware to "them" – I don't know who "them" is, you decide :)

From https://sd.suse.com/servicedesk/customer/portal/1/SD-106628
Martina Markova 2023-01-26 16:03 wrote

Follow up email sent to Nick.

Today I just plugged in my notebook with a dhcp directly attached to the BMC and was able to access the webinterface as well as SOL. However, the machine still does not boot, reports very little over sol and shows no output on the connected VGA screen. I send over this information to codec by mail (visible on the osd-admins ML) today.

I mounted the machine in NUE-FC-B:5 connected to PDU outlet A9 and mgmt ethernet on switch port 22 . As we don't the MAC address because there was incomplete information on racktables for the machine so far I created https://sd.suse.com/servicedesk/customer/portal/1/SD-113959) for Eng-Infra to provide us log or admin access to walter1.qe.nue2.suse.org

As workaround

I did nmap -p 443 10.168.192.0/22 > with with chan mgmt patchcable not plugged in, then again with it plugged in and diffed it. Result:

+Nmap scan report for 10.168.193.122
+Host is up (0.00051s latency).
+
+PORT    STATE SERVICE
+443/tcp open  https
+MAC Address: E0:D5:5E:A7:E8:34 (Giga-byte Technology)
+

We attempted a firmware upgrade as suggested. This succeeded but no change in behaviour in the boot process. Ongoing email conversation, CC'ed to osd-admins@suse.de. Latest message at time of writing https://mailman.suse.de/mlarch/SuSE/osd-admins/2023/osd-admins.2023.03/msg00043.html

The last email I saw was https://mailman.suse.de/mlarch/SuSE/osd-admins/2023/osd-admins.2023.03/msg00185.html on 21 Mar 2023 . No related message in https://mailman.suse.de/mlarch/SuSE/osd-admins/2023/osd-admins.2023.04/threads.html

@nicksinger please make sure that our relevant communication partners do the next step and then bump the due date accordingly.