QA

I picked up an adapter from NUE1-SRV2-A as discussed with bmwiedemann and brought that adapter to FC Basement. I connected cloud4 qa4-node7 with the adapter to a VGA screen and USB keyboard. I powered on the system and could see a Supermicro system booting up. The system actually showed the PXE boot menu from qa-jump.qe.nue2.suse.org. I reset the machine and entered the system setup and selected "IPMI" which showed a static configuration for mac 0c-c4-7a-4c-f3-f4 IPv4 10.162.65.51. The address is mentioned in https://gitlab.suse.de/qa-sle/qanet-configs/-/commit/1c46c3531b6c6b456210485e29e55e280ba655e2 "Bulk indentation change" where the entry is actually removed, what a lie!

The removed block from the dhcpd config:

-  #qa4 nodes IPMI 10.162.65.49 to 10.162.65.57
-  group {
-#     option routers 10.162.65.254;
-#     option subnet-mask 255.255.255.0;
-#     option domain-name "bmc.cloud4adm.qa.suse.de";
-#     host gate { hardware ethernet 0c:c4:7a:4d:03:29; fixed-address 10.162.65.49; option host-name "gate"; }
-#     host n1 { hardware ethernet 0c:c4:7a:4c:f3:f4; fixed-address 10.162.65.51; option host-name "n1"; }
-#     host n2 { hardware ethernet 0c:c4:7a:4c:f1:7f; fixed-address 10.162.65.52; option host-name "n2"; }
-#     host n3 { hardware ethernet 0c:c4:7a:4d:03:24; fixed-address 10.162.65.53; option host-name "n3"; }
-#     host n4 { hardware ethernet 0c:c4:7a:4d:03:23; fixed-address 10.162.65.54; option host-name "n4"; }
-#     host n5 { hardware ethernet 0c:c4:7a:4c:f2:cc; fixed-address 10.162.65.55; option host-name "n5"; }
-#     host n6 { hardware ethernet 0c:c4:7a:4d:01:d7; fixed-address 10.162.65.56; option host-name "n6"; }
-#     host n7 { hardware ethernet 0c:c4:7a:4d:03:1f; fixed-address 10.162.65.57; option host-name "n7"; }

I could configure to use DHCP or a different address from the 10.168 range but instead on walter1 I did

ip addr add 10.162.65.252/24 dev eth0

but ping does not reveal something. And the IPMI status page in system setup says "IPMI Network Link Status: No Connect". Let's see. I exited the setup and saw the system ending up in the PXE boot menu. On qa-jump.qe.nue2.suse.org I could see in journalctl -e that the request can actually from 10.168.193.58. Trying to select any installation entry ends up in an error. The menu just blinks and is back where it was. journal on qa-jump shows:

May 17 13:33:45 qa-jump in.tftpd[19657]: RRQ from 10.168.193.58 filename /mounts/dist/install/SLP/SLE-15-SP5-Full-LATEST/x86_64/DVD1/boot/x86_64/loader/linux
May 17 13:33:45 qa-jump in.tftpd[19657]: sending NAK (1, File not found) to 10.168.193.58

walter1 says that the mac is 0c:c4:7a:51:e9:f4

Checking PXE on qa-jump. I am not sure what is off here. /mnt/dist is empty so is /mounts/dist. I added to /etc/fstab

/mnt/dist                                       /mounts/dist                  none     defaults,bind    0 0

and then unmounted /mnt/dist and all dist.suse.de related shares and then mount -a would bring back a proper content in /mnt/dist. Now PXE on qa4-node7 does something useful. I booted into the Leap 15.4 installer. After "Probing EDD … ok" there is just a blinking cursor but alt-f2 shows tty2 from the installer system just fine.

In the meantime the switch shows the above mac …e9:f4 on ge-5/0/45.0 so top right blue cable corresponding to the racktables entry we already have. But no further response to the IPMI interface.

I could do

vgchange -a y
mount /dev/system/root /mnt
chroot /mnt /bin/bash

and enter a SLE15SP1 which also has ipmitool but ipmitool says that no IPMI device was found. I could continue with the actual installation. Or set IPMI to DHCP in system setup.

on walter1 arping -I eth0 10.162.65.51 actually works, ping does not

ARPING 10.162.65.51 from 10.162.65.252 eth0
Unicast reply from 10.162.65.51 [0C:C4:7A:4C:F3:F4]  0.930ms
Unicast reply from 10.162.65.51 [0C:C4:7A:4C:F3:F4]  0.889ms
…

same experiment on seth, same rack as tgt

ip addr add 10.162.65.251/24 dev br0
ping -I br0 10.162.65.51
# (nothing)
arping 10.162.65.51
# good response

also no response using ipmitool -4 -I lanplus -C 3 -H 10.162.65.51 -U root -P [masked] on both walter1 and seth. Maybe the IPMI interface does simply not respond to ICMP requests.

I did not manage to login into the ssh installer system remotely so I just called "yast.ssh" from the local console. In the installer I set default old SUSE root password, static hostname "qa4-node7", dhcp, select ipmitool for installation and ssh server.

After installation I rebooted and found that grub showing the SLE15SP1 entry still. I let the system continue to boot ending up in a "SUSE Manager Server 4.0" and IPv4 10.162.66.50, apparently statically configure. This time at least with an address in that range I could ping from walter1 but I could not find a working root password.

Now I reset the machine and configured IPMI to use DHCP but I did not see a related DHCP request on walter1. I entered the boot menu and selected the second hard drive and that booted the installed Leap 15.4. Using ssh root@10.168.193.58 I could login.
Now I could call ipmitool lan print which gives

Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD 
Auth Type Enable        : Callback : MD2 MD5 PASSWORD 
                        : User     : MD2 MD5 PASSWORD 
                        : Operator : MD2 MD5 PASSWORD 
                        : Admin    : MD2 MD5 PASSWORD 
                        : OEM      : MD2 MD5 PASSWORD 
IP Address Source       : DHCP Address
IP Address              : 0.0.0.0
Subnet Mask             : 0.0.0.0
MAC Address             : 0c:c4:7a:4c:f3:f4
SNMP Community String   : public
IP Header               : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled

so no IP address received. Using

ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipsrc dhcp

I tried to explicitly make it request an address but there is nothing.

trying static with the next free IPv4 address in 10.168.192.0

ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 10.168.192.141
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 10.168.195.254

and I set default ADMIN:ADMIN credentials for now

ipmitool user set name 2 ADMIN
ipmitool user set password 2
ipmitool channel setaccess 1 2 link=on ipmi=on callin=on privilege=4
ipmitool user enable 2

following https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool

Called watch -n 0.1 ipmitool lan stats get 1 and could see RX and TX packets as long as the cable is connected but stopping as soon as I pull out the cable.

I also crosschecked with a different cable on different switch port.

Interestingly nmap finds the IPMI node but no open ports

qa4-node7:~ # nmap 10.168.192.141
Starting Nmap 7.92 ( https://nmap.org ) at 2023-05-17 15:11 UTC
Nmap scan report for 10.168.192.141
Host is up (0.00035s latency).
All 1000 scanned ports on 10.168.192.141 are in ignored states.
Not shown: 1000 filtered tcp ports (no-response)
MAC Address: 0C:C4:7A:4C:F3:F4 (Super Micro Computer)

Nmap done: 1 IP address (1 host up) scanned in 21.18 seconds

https://serverfault.com/a/994962 to the rescue:

ipmitool lan set 1 vlan id off

and suddenly everything works including DHCP. I could also authenticate using the "root" user. So that means that likely I need to physically access every node, get access to a linux with ipmitool, delete the vlan assignment, set to dhcp and enable authentication with known credentials.

The current installation can be accessed with ssh root@10.168.193.40, old SUSE default root password.

Setting due date based on mean cycle time of SUSE QE Tools

Connected qa4-node6 with my personal usb thumbdrive, monitor, keyboard. Switched on, press F11 on start, select USB thumbdrive in BIOS mode (not UEFI) as harddisks are installed with BIOS mode as well. Booted grml64, selected terminal, ran:

ipmitool lan set 1 vlan id off
ipmitool lan set 1 ipsrc dhcp
ipmitool user set name 2 ADMIN
ipmitool user set password 2
ipmitool channel setaccess 1 2 link=on ipmi=on callin=on privilege=4
ipmitool user enable 2

there were error messages about "IANA PEN registry open failed: No such file or directory" which I found upstream in https://github.com/ipmitool/ipmitool/issues/379 and I shouldn't worry about. But then there were other errors like "Get Device ID command failed". Also with retries and waiting I could not set the password.
But ipmitool -I lanplus -C 3 -H 10.168.193.165 -U root -P [masked] at least sometimes. grml has ipmitool 1.8.19 so newer upstream version than openSUSE or SLE with 1.8.18. No difference when using ipmitool from installed SLE. Going to next machine. On qa4-node5 trying just vlan id off and ipsrc dhcp. After setting to dhcp eventually the machine would get an IPv4 address, 10.168.193.185, but then the IPMI also for local access behaved quite weird and unstable, same errors as on node node6. Maybe DHCP is problematic at least for the firmware version.

So setting static address for next node, qa4-node4:

ipmitool lan set 1 vlan id off
ipmitool lan set 1 ipaddr 10.168.192.148
ipmitool lan set 1 netmask 255.255.252.0
ipmitool lan set 1 defgw ipaddr 10.168.195.254

remote access works. Leaving that for now, continuing with next. Did the next on qa4-node3 with 10.168.192.146. Executing the ipmitool commands takes very long, the complete chain multiple minutes, remote access also with errors. Works with multiple retries. Maybe BMC firmware update helps. Found https://www.supermicro.com/en/support/resources/downloadcenter/firmware/MBD-X10SLD-F/BMC . For previous node https://10.168.192.148/cgi/url_redirect.cgi?url_name=mainmenu says Firmware Revision : 03.72, download is 3.88. Updating firmware with REDFISH_X10_388_20200221_unsigned.bin , worked ok.

In the meantime configured node1,2,3. node1 is reachable just fine, node2 isn't, node3 unreliable. Maybe problem with switch? network? Configured all nodes with static address for now.

TODO:

1. Try to set .148 to DHCP and check IPMI stability with new firmware
2. Update other node's firmware
3. Create specific tickets for problematic nodes
4. Consider renaming in racktables and DHCP the other way around as in before meaning that node1 would be right-most
5. Submit MR for DHCP+DNS config
6. Try remote installation once more with instructions for users
7. Suggest the solution to others

I set .148 to DHCP and the system reappeared as https://10.168.193.132 . I also set the hostname in the BMC web menu to quake4-sp.qe.nue2.suse.org.
Same procedure for quake6+7, also upgraded firmware. Same for quake1 on https://10.168.192.142 and quake8 on https://10.168.192.156/ . Couldn't reach 2,3,5 right now.
Was running runs=10000 count_fail_ratio ipmitool -I lanplus -C 3 -H 10.168.193.132 -U root -P 'XXX' mc guid on ok1.qe.nue2.suse.org. Result: count_fail_ratio: Run: 10000. Fails: 0. Fail ratio 0±0%. No fails, computed failure probability < .03%, same for quake6+7, super fast and super stable. I created #129868 for the not reachable nodes.

Added some info on https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs#Machines_available_for_reservation, should wait for https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3536

I updated https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs#Machines_available_for_reservation with IPMI user+password instructions. Installation is definitely possible even though autoyast as described above is not without caveats.