QA » openQA Project » openQA Infrastructure

I noticed that there's indeed a slot occupied with an additional network card. I enabled some BIOS initialization feature for that device in the hope I could then boot from it. Unfortunately I now cannot even enter the setup menu anymore as the boot is stuck at:

Press <ESC> or <DEL> to enter setup, <F10> Display Boot Menu, <F12> to Network P
XE Boot                                                                         
Entering Setup...                                                               


BIOS Date: 09/29/2022 10:23:43 Ver: F31L                                        
BMC Firmware Version: 12.60.38                                                  

BMC IP : 192.168.204.18                                                         

Processor CPU0: Ampere(R) Altra(R) Processor Q80-33                             
Processor Speed: 3200 MHz                                                       
Processor Max Speed: 3300 MHz                                                   
Total Memory: 512 GB                                                            
USB Devices total: 0 KBDs, 0 MICE, 0 MASS, 2 HUBs                               

NVMe Device0:  SAMSUNG MZPLJ6T4HALA-00007 , 6.4TB                               

Detected ATA/ATAPI Devices...                                                   
SATA Hard Device Number : 0                                                     




                                                                             92

After a BIOS update (https://www.gigabyte.com/de/Enterprise/Rack-Server/R272-P30-rev-100#Support-Bios) it arm21 works again. After enabling x86 emulation the other ethernet device shows up during early boot (as some broadcom device) and can now in fact be used.

I've put the mac address of that interfaced into the o3 dnsmasq config and with the following config the iPXE boot works with my locally cross-compiled binary (that I have put on o3):

# UEFI tftp boot aarch64
dhcp-option=tag:efi-aarch64,option:bootfile-name,ipxe/ipxe-arm64.efi

# UEFI http boot aarch64
dhcp-option=tag:efi-aarch64-http,option:bootfile-name,http://10.150.1.11/tftpboot/ipxe/ipxe-arm64.efi
dhcp-option=tag:efi-aarch64-http,option:vendor-class,HTTPClient

I'll create a PR for further documentation of this. EDIT: https://github.com/os-autoinst/scripts/pull/255

For now I manually changed the command line to the aarch64 one documented under https://wiki.suse.net/index.php/OpenQA#Remote_management_with_IPMI_and_BMC_tools. Unfortunately I couldn't get "Hardware detection". Likely the serial device was wrong.

I cannot boot via autoyast. The systems always hang on "Hardware detection". When enabling ssh one gets additional error messages and it doesn't work:

Loading basic drivers
Hardware detection
[   90.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 26s! [wickedd-dhcp4:1243]
[  114.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 48s! [wickedd-dhcp4:1243]
[  138.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 71s! [wickedd-dhcp4:1243]
[  162.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 93s! [wickedd-dhcp4:1243]
[  186.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 116s! [wickedd-dhcp4:1243]
[  210.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 138s! [wickedd-dhcp4:1243]
[  234.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 160s! [wickedd-dhcp4:1243]
[  258.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 183s! [wickedd-dhcp4:1243]
[  282.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 205s! [wickedd-dhcp4:1243]
[  306.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 227s! [wickedd-dhcp4:1243]

On arm22 I could at least boot without autoyast. On arm21 the system unfortunately still hangs after hardware initialization. Both systems are on the same firmware versions and I'm using the exact same way to boot so I'm not sure what makes the difference.

I suppose the firmware update for the nic "BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller" would be available to download on https://www.broadcom.com/products/ethernet-connectivity/network-adapters/bcm57416-1gbase-t-ic (according to https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/updating-the-firmware-online.html) but I couldn't find any firmware downloads on that page. Maybe the firmware from this page is relevant: https://www.broadcom.com/products/ethernet-connectivity/network-adapters/p210tp

To summarize: I was able to setup arm22 (just installed the OS, not configured yet) but arm1 hangs as mentioned in the previous comment. The problem on arm21 must be more severe and is likely not caused by booting from network because Antony said in the chate "I'm trying to install tumbleweed on the one with the issue, so that I can apply the fw update, but I can't even install tumbleweed from a usb...".

I've been creating an infra ticket about the problematic arm worker: https://sd.suse.com/servicedesk/customer/portal/1/SD-130480

I saw that openqaworker-arm22's 32 workers were sitting idle while the other aarch64 workers were rather busy with jobs.
A test job revealed that some packages were missing on the worker, hugepages had to be set up and openqa-auto-update.timer as well as openqa-continuous-update.timer had to be enabled. I did all of that and ran a test job, which passed. After a reboot I added the production class (64bit only, Neoverse-N1 cannot run 32bit kernels) to see what's missing in practice.

mkittler wrote in #note-8:

I cannot boot via autoyast. The systems always hang on "Hardware detection". When enabling ssh one gets additional error messages and it doesn't work:

Loading basic drivers
Hardware detection
[   90.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 26s! [wickedd-dhcp4:1243]
[  114.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 48s! [wickedd-dhcp4:1243]
[  138.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 71s! [wickedd-dhcp4:1243]
[  162.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 93s! [wickedd-dhcp4:1243]
[  186.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 116s! [wickedd-dhcp4:1243]
[  210.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 138s! [wickedd-dhcp4:1243]
[  234.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 160s! [wickedd-dhcp4:1243]
[  258.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 183s! [wickedd-dhcp4:1243]
[  282.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 205s! [wickedd-dhcp4:1243]
[  306.877330][    C7] watchdog: BUG: soft lockup - CPU#7 stuck for 227s! [wickedd-dhcp4:1243]

On arm22 I could at least boot without autoyast. On arm21 the system unfortunately still hangs after hardware initialization. Both systems are on the same firmware versions and I'm using the exact same way to boot so I'm not sure what makes the difference.

I guess CPU core #7 is broken in some way. You could confirm that by booting with maxcpus=6 and then getting cores 8-80 online with for i in {8..80}; do echo 1 > /sys/devices/system/cpu/cpu${i}/online; done.

Now tracking https://jira.suse.com/browse/ENGINFRA-2676

okurz wrote in #note-17:

Now tracking https://jira.suse.com/browse/ENGINFRA-2676

still valid block

okurz wrote in #note-19:

I suggest to re-estimate. The suggestions still have "TBD". Also I added a point regarding monitoring.

Last comment on Jira from 2023-10-11. Note that we're excluding blocked tickets in our go-to query so this won't be estimated until that is resolved or we re-consider blocking on that.

livdywan wrote in #note-20:

okurz wrote in #note-19:

I suggest to re-estimate. The suggestions still have "TBD". Also I added a point regarding monitoring.

Note that we're excluding blocked tickets in our go-to query so this won't be estimated until that is resolved or we re-consider blocking on that.

yes, that's fine. Only the latest after the SD ticket was resolved we should re-estimate.

livdywan wrote in #note-22:

Update from ENGINFRA-2676

Fedex lost the incoming server package from Happyware

No news yet. Checking on the ticket.

I adjusted both entries in /etc/dnsmasq.d/openqa.conf according to the new IP addresses:

dhcp-host=14:23:f2:20:fe:c0,openqaworker-arm21
dhcp-host=14:23:f2:20:fe:c1,openqaworker-arm21-iface2

After restarting dnsmasq (reload wasn't sufficient) I get the iPXE binary served now an could select the AY entry but something seems wrong:

http://download.opensuse.org/distribution/leap/15.5/repo/oss/boot/x86_64/loader/linux.................. Connection timed out (https://ipxe.org/4c0a6092)

something is odd. Now it can reach d.o.o but hangs at 70%

nicksinger tried the installation again, it progressed a little bit but still initrd download then failed or timed out. Please try out just wget http:/… or tftp on another machine, e.g. openqaworker-arm22

After I fixed ariel to properly resolve download.opensuse.org (by moving away a static config from /etc/dnsmasq.d/doo.conf) I ran a 1000 wget downloads for the kernel and initrd each - a 100% success rate and really fast downloads. But for the love of god, I cannot get a system loaded because iPXE is always stuck at a different percentage of either initrd or the kernel. Interestingly enough iPXE itself seems to be working because ctrl+c aborts the download and just cycles the boot process again. If somebody wants to try, feel free - I can take over once the installation is started.

Maybe it makes sense to cross check whether iPXE works on another host like the other arm worker. We don't need to re-install it but just check whether we could boot an installation. If that works then maybe the new arm worker itself is still to blame.

a similar issue as we face is described here: https://github.com/ipxe/ipxe/issues/1023#issuecomment-1708614076 - the broadcom NIC seems to cause problems and I will try now how to make use of the NII-fallback driver.

You could also try the EFI shell with HTTP boot or HTTP downloads onto a FAT partition.

Compiling and using the "snponly" version of iPXE allowed me to boot reproducible on both machines. As such, I added the according binary to ariel and adjusted the pxeboot.conf of dnsmasq on it. I also created https://github.com/os-autoinst/scripts/pull/273 to at least somehow document my findings.

nicksinger wrote in #note-44:

As written previously I was able to continue the installation but it hang at "detecting hardware". Will have to try different serial settings as I observed the same on worker22

When you say "worker22" do you mean openqaworker22.openqanet.opensuse.org or openqaworker-arm22.openqanet.opensuse.org? w22 aka. openqaworker22.openqanet.opensuse.org has console=tty console=ttyS1,115200, warm22 aka. openqaworker-arm22.openqanet.opensuse.org has no console setting in /proc/cmdline, probably should have /dev/ttyAMA0 or /dev/ttyAMA1

When you say "worker22" do you mean openqaworker22.openqanet.opensuse.org or openqaworker-arm22.openqanet.opensuse.org? w22 aka. openqaworker22.openqanet.opensuse.org has console=tty console=ttyS1,115200, warm22 aka. openqaworker-arm22.openqanet.opensuse.org has no console setting in /proc/cmdline, probably should have /dev/ttyAMA0 or /dev/ttyAMA1

I mean arm22. I tried it with console=ttyAMA0,115200 and can now see the yast installer via SOL. It seems like the linked AY profile cannot handle the disk situation on that machine:

   -││Important issues were detected:                                     ││
   -││                                                                    ││
   -││ *  Unable to propose a partitioning layout                         ││
   -││ *  partitioning,0:                                                 ││
   -││     +  Disk '/dev/nvme1n1' was not found                           ││
   -││ *  partitioning,1:                                                 ││
   -││     +  Disk '/dev/nvme2n1' was not found                           ││
   -││                                                                    ││
   -││                                                                    ││
   -││Please, correct these problems and try again.

in all ticket comments please consider our best practices from progress.opensuse.org/projects/qa/wiki/tools to make it clear what you plan to do as next step or also when you don't plan anything

now facing problems with starting qemu because of the huge pages setup. Found some hints on https://progress.opensuse.org/issues/53234#note-12 which I followed and also issued echo 128 > /proc/sys/vm/nr_hugepages which was present on arm22 (not sure what sets it and how) but qemu still refuses to start up properly: https://openqa.opensuse.org/tests/3800270#details

I had to adjust /etc/default/grub for the correct huge pages table config. Now I managed to validate the worker with https://openqa.opensuse.org/tests/3805306
I will now enable it for production and monitor it a little bit more before considering this ticket here as resolved.