QA

https://jira.suse.com/browse/ENGINFRA-4030 is still open now but times on both suttner1+2 are good and in sync and dhcpd also seems to be in sync. I can also now see that https://powerhmc1.oqa.prg2.suse.org/ shows grenache and – as expected – a red lock mentioning that HMC is not controlling the machine as grenache uses novalink. But I can't ping any of grenache-vios.oqa.prg2.suse.org or grenache-fsp1.oqa.prg2.suse.org or grenache-1.oqa.prg2.suse.org

The Jira task https://jira.suse.com/browse/ENGINFRA-3750 provided some very helpful pictures of the operators display. We should keep this for future installations.
The HMC for grenache already looked very good after the cabling was done. VIOS was able to start without any issues and with it the NovaLink LPAR. However, none of the LPARs had network. Gerhard from IT organized that we connect all 4 ports on the C11 network card in that system. After finding and resolving a typo in the VLAN configuration and restarting the systems, all mentioned LPARs had network. Still no RMC connection from either NovaLink or VIOS but this might not be needed after all for our use-case.

Currently ssh padmin@grenache.oqa.prg2.suse.org works and can be used to start LPARs like e.g. grenache-1 with lpar power-on -i name=grenache-1 - this is also what openQA does to start and stop LPARs.
Also grenache-1 can be reached via ssh grenache-1.oqa.prg2.suse.org.

My next steps are to validate PXE in the other LPARs (grenache-{2..8}) and bring the machine back into openQA.

PXE unfortunately does not work. I can ping qa-jump but the bootp request times out after 5 tries. I have yet to find out whats the culprit.

On a positive note: I was able to establish a fully working RMC connection with the VIOS after logging into it via a vterm (mkvterm -m grenache -p vios1 on HMC). On the VIOS I discovered with ifconfig -a that the hmc facing network port (en14) still had an old 10.162.x.x-IP. After elevating privileges with oem_setup_env I could execute smitty to get an interactive configuration menu. In there I went to "Communications Applications and Services" -> "TCP/IP" -> "Use DHCP for TCPIP Configuration & Startup" -> select en14 from the list -> "Use DHCP starting".

Next I will try to understand why PXE is not working. First test already shows that qa-jump can provide the requested "ppc64le/grub2" binary. I tested this on my local workstation with:

workstation ~ » atftp 10.168.192.10 69
tftp> get ppc64le/grub2
tftp>

workstation ~ » file grub2
grub2: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), statically linked, stripped

I will now try to understand if the LPAR cannot access the server or if we have some more misconfiguration anywhere.

Discussed with nicksinger in daily infra call. With https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/740 we enabled pre-production worker classes to be used with openQA again.

Yesterday we found out that the PXE/bootp-process is different then expected. The "Server" one has to enter in the SMS menu is not the TFTP-Server-IP but rather the IP of the server handing out DHCP leases. With that I managed to get a successful openQA run: https://openqa.suse.de/tests/13762881 - however, TIMEOUT_SCALE=4 was needed because loading kernel+initrd takes quite some time.