Project

General

Profile

Actions
Copy link

action #178627

closed

[openQA][sut][ipmi][ipxe] Machine bare-metal1.oqa.prg2.suse.org fails to boot from ipxe size:S

Added by waynechen55 about 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
Regressions/Crashes
Target version:
openQA Project (public) - Ready
Start date:
2025-03-11
Due date:
% Done:

0%

Estimated time:
Tags:
infra, reactive work, bare-metal

Description

Observation

Actual iPXE boot process is as below:

>>Checking Media Presence......
>>Media Present......
>>Start PXE over IPv4 on MAC: 3C-EC-EF-FF-16-A6.
  Station IP address is 10.145.10.14
  Server IP address is 10.144.110.161
  NBP filename is kernelqa/ipxe.efi
  NBP filesize is 207360 Bytes
>>Checking Media Presence......
>>Media Present......
 Downloading NBP file...
  NBP file downloaded successfully.
iPXE initialising devices...
autoexec.ipxe... Not found (https://ipxe.org/2d12618e)
iPXE 1.21.1+ (g8fc11) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP iSCSI TFTP VLAN AoE EFI Menu
Chainloading the bootscript now from baremetal-support.qe.prg2.suse.org
Configuring (net0 3c:ec:ef:ff:16:a6).................. ok
http://baremetal-support.qe.prg2.suse.org/script.ipxe... ok
++++++++++++++++++++++++++++++++++++++++++
++++++++++++ openQA ipxe boot ++++++++++++
+ Host: bare-metal1.oqa.prg2.suse.org
++++++++++++++++++++++++++++++++++++++++++
http://openqa.suse.de/assets/repo/SL-Micro.x86_64-6.2-Default-SelfInstall-Build5.5.install/pxeboot.SL-Micro.x86_64-6.2.kernel... Not found (https://ipxe.org/2d0c618e)
Could not boot: Not found (https://ipxe.org/2d0c618e)
>>Checking Media Presence......
>>Media Present......
>>Start PXE over IPv4 on MAC: 3C-EC-EF-FF-16-A7.
  Station IP address is 10.145.10.142
  Server IP address is 10.144.110.161
  NBP filename is /ipxe/ipxe.efi
  NBP filesize is 208384 Bytes
>>Checking Media Presence......
>>Media Present......
 Downloading NBP file...
  NBP file downloaded successfully.
iPXE initialising devices...
autoexec.ipxe... Not found (https://ipxe.org/2d12618e)
iPXE 1.21.1+ (g8fc11) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP iSCSI TFTP VLAN AoE EFI Menu
Welcome to qe iPXE
Configuring (net0 3c:ec:ef:ff:16:a7).................. ok
http://10.144.110.161/tftpboot/ipxe/menu.ipxe... ok
iPXE boot menu for netboot.qe.prg2.suse.org        -- OS Inst (You will be prompted to edit cmdline after selecting an os) --    -------------------------------- x86_64 ----------------------------------    (L)Boot openSUSE Leap15.6 from download.opensuse.org / ttyS1 (http)           (E)Boot openSUSE Leap15.6 from download.opensuse.org / ttyS2 (http)           (A)Boot openSUSE Leap15.6 / ttyS1 autoyast=https://is.gd/oqaay4               (K)Boot openSUSE Tumbleweed KVM server from O3 repo with autoyast/ ttyS1 (    (Z)Boot openSUSE Tumbleweed XEN server from O3 repo with autoyast/ ttyS1 (    (T)Boot openSUSE Tumbleweed from download.opensuse.org / ttyS1 (http)         (S)Boot openSUSE Tumbleweed from download.opensuse.org / ttyS2 (http)         (M)Memtest 64 EFI                                                             -------------------------------- AArch64 ---------------------------------    (o)Boot openSUSE Leap15.6 / ttyS1 autoyast=https://is.gd/oqaay4               (O)Boot openSUSE Leap15.6                                                     ------------------------- Advanced options -------------------------------    (N)Netboot.xyz (x86_64 / aarch64)                                             Configure settings                                                            (i)Drop to iPXE shell                                                         Reboot computer                                                                                                                                             Exit iPXE and continue BIOS boot                                      (30)    Exit iPXE and continue BIOS boot                                      (30)    Exit iPXE and continue BIOS boot                                      (29)    Exit iPXE and continue BIOS boot                                      (29)    Exit iPXE and continue BIOS boot                                      (28)    Exit iPXE and continue BIOS boot                                      (28)   
 Exit iPXE and continue BIOS boot                                      (27)    Exit iPXE and continue BIOS boot                                      (27)    Exit iPXE and continue BIOS boot                                      (26)    Exit iPXE and continue BIOS boot                                      (26)    Exit iPXE and continue BIOS boot                                      (25)    Exit iPXE and continue BIOS boot                                      (25)    Exit iPXE and continue BIOS boot                                      (24)    Exit iPXE and continue BIOS boot                                      (24)    Exit iPXE and continue BIOS boot                                      (23)    Exit iPXE and continue BIOS boot                                      (23)    Exit iPXE and continue BIOS boot                                      (22)    Exit iPXE and continue BIOS boot                                      (22)    Exit iPXE and continue BIOS boot                                      (21)    Exit iPXE and continue BIOS boot                                      (21)    Exit iPXE and continue BIOS boot                                      (20)    Exit iPXE and continue BIOS boot                                      (20)    Exit iPXE and continue BIOS boot                                      (19)    Exit iPXE and continue BIOS boot                                      (19)    Exit iPXE and continue BIOS boot                                      (18)    Exit iPXE and continue BIOS boot                                      (18)    Exit iPXE and continue BIOS boot                                      (17)    Exit iPXE and continue BIOS boot                                      (17)    Exit iPXE and continue BIOS boot                                      (16)    Exit iPXE and continue BIOS boot                                      (16)    Exit iPXE and continue BIOS boot                                      (15)    Exit iPXE and continue BIOS boot                 
                     (15)    Exit iPXE and continue BIOS boot                                      (14)    Exit iPXE and continue BIOS boot                                      (14)    Exit iPXE and continue BIOS boot                                      (13)    Exit iPXE and continue BIOS boot                                      (13)    Exit iPXE and continue BIOS boot                                      (12)    Exit iPXE and continue BIOS boot                                      (12)    Exit iPXE and continue BIOS boot                                      (11)    Exit iPXE and continue BIOS boot                                      (11)    Exit iPXE and continue BIOS boot                                      (10)    Exit iPXE and continue BIOS boot                                      (10)    Exit iPXE and continue BIOS boot                                       (9)    Exit iPXE and continue BIOS boot                                       (9)    Exit iPXE and continue BIOS boot                                       (8)    Exit iPXE and continue BIOS boot                                       (8)    Exit iPXE and continue BIOS boot                                       (7)    Exit iPXE and continue BIOS boot                                       (7)    Exit iPXE and continue BIOS boot                                       (6)    Exit iPXE and continue BIOS boot                                       (6)    Exit iPXE and continue BIOS boot                                       (5)    Exit iPXE and continue BIOS boot                                       (5)    Exit iPXE and continue BIOS boot                                       (4)    Exit iPXE and continue BIOS boot                                       (4)    Exit iPXE and continue BIOS boot                                       (3)    Exit iPXE and continue BIOS boot                                       (3)    Exit iPXE and continue BIOS boot                                       (2)    Exit iPXE and continu
e BIOS boot                                       (2)    Exit iPXE and continue BIOS boot                                       (1)    Exit iPXE and continue BIOS boot                                       (1)    Exit iPXE and continue BIOS boot                                              Exit iPXE and continue BIOS boot                                            
Please press 't' to show the boot menu on this console

Corresponding screenshot is as below:

Failed openqa jobs:
https://openqa.suse.de/tests/17012442#step/selfinstall/2
https://openqa.suse.de/tests/17012443#step/selfinstall/2
https://openqa.suse.de/tests/17012444#step/selfinstall/2

Latest: https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle-micro&flavor=Default-SelfInstall-pxe&machine=virt-mm-baremetal-ipmi&test=slem62_guest_ignition%2Bcombustion_on_slem62_host%40waynechen55%2Fos-autoinst-distri-opensuse%23wayne%2Fautomate_guest_installation_using_agama&version=6.2

Steps to reproduce

  • Set ipxe boot script
  • Boot from ipxe
  • Or simply clone one of the failing jobs

Impact

Can not run ipmi backend test with this machine

Problem

Looks like infra issue, including dhcp config and etc.

Suggestions

  • Check dhcp config
  • Check ipxe config
  • Confirm that this is

Workaround

n/a

Files

bare_metal1_fails_boot_from_ipxe.png (33.8 KB) bare_metal1_fails_boot_from_ipxe.png waynechen55, 2025-03-11 06:08
bare_metal1_fails_boot_from_ipxe.png
Actions
Copy link
 #1

Updated by waynechen55 about 2 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by livdywan about 2 months ago

  • Tags set to infra, reactive work, bare-metal
  • Description updated (diff)
  • Priority changed from Normal to High
  • Target version set to Ready

I'm guessing this might be https://openqa.suse.de/tests/17012444#step/selfinstall/2 since the needle looks like the one included here?

Actions
Copy link
 #3

Updated by livdywan about 2 months ago

  • Description updated (diff)
  • Category set to Regressions/Crashes
Actions
Copy link
 #4

Updated by livdywan about 2 months ago

  • Subject changed from [openQA][sut][ipmi][ipxe] Machine bare-metal1.oqa.prg2.suse.org fails to boot from ipxe to [openQA][sut][ipmi][ipxe] Machine bare-metal1.oqa.prg2.suse.org fails to boot from ipxe size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #5

Updated by waynechen55 about 2 months ago

livdywan wrote in #note-2:

I'm guessing this might be https://openqa.suse.de/tests/17012444#step/selfinstall/2 since the needle looks like the one included here?

It does not boot normally. The screenshot like this one should not been seen.

Actions
Copy link
 #6

Updated by waynechen55 about 2 months ago

It offers wrong ipxe script file:

iPXE 1.21.1+ (g8fc11) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP iSCSI TFTP VLAN AoE EFI Menu
Welcome to qe iPXE
Configuring (net0 3c:ec:ef:ff:16:b1).................. ok
http://10.144.110.161/tftpboot/ipxe/menu.ipxe... ok

The correct should looks like:

iPXE 1.21.1+ (g8fc11) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP iSCSI TFTP VLAN AoE EFI Menu
Chainloading the bootscript now from baremetal-support.qe.prg2.suse.org
Configuring (net0 3c:ec:ef:ff:09:40).................. ok
http://baremetal-support.qe.prg2.suse.org/script.ipxe... ok
Actions
Copy link
 #7

Updated by waynechen55 about 2 months ago

There should be something wrong with configuration ???

Actions
Copy link
 #8

Updated by okurz about 2 months ago

  • Due date set to 2025-03-21
  • Status changed from Workable to Feedback
  • Assignee set to okurz

I can confirm that from https://gitlab.suse.de/OPS-Service/salt/-/blob/production/pillar/domain/oqa_prg2_suse_org/hosts.yaml?ref_type=heads#L61 the interface that is configured for booting iPXE is 3c:ec:ef:ff:09:40 as you wrote regarding the "correct" one.
Also https://netbox.suse.de/dcim/devices/7115/interfaces/ shows that. The other interface shouldn't even be connected. You can try to disable the second interface in the firmware of the machine or work with IT DC Ops to have the second cable disconnected.

Actions
Copy link
 #9

Updated by nanzhang about 2 months ago

Build5.5 was deleted somehow in last weekend which caused the ipxe boot failure, and now it was recovered. Please retry it.
https://openqa.suse.de/assets/repo/SL-Micro.x86_64-6.2-Default-SelfInstall-Build5.5.install/

Actions
Copy link
 #10

Updated by okurz about 2 months ago

  • Due date changed from 2025-03-21 to 2025-04-04
  • Priority changed from High to Normal
  • Target version changed from Ready to Tools - Next

@waynechen55 as I wrote in #178627-8 IMHO the second interface shouldn't be connected. Please try to disable the second interface in the firmware of the machine or work with IT DC Ops to have the second cable disconnected.

Actions
Copy link
 #11

Updated by okurz about 1 month ago

  • Due date deleted (2025-04-04)
  • Status changed from Feedback to Resolved
  • Priority changed from Normal to High
  • Target version changed from Tools - Next to Ready

https://openqa.suse.de/tests/17145480 is a passed job from yesterday so I assume the problem fixed itself.

Actions
Copy link

Also available in: Atom PDF