QA » openQA Project » openQA Infrastructure

I also created SD ticket https://sd.suse.com/servicedesk/customer/portal/1/SD-166375

I found this

When booting a Linux kernel, iPXE will construct a “magic initrd” by injecting downloaded files into the initial RAM filesystem image. Any argument supplied to the initrd command will be used as the pathname for that image within the initrd.magic initial RAM filesystem.

in https://ipxe.org/cmd/imgfetch

So this really has something to do with iPXE boot image I think.

waynechen55 wrote in #note-4:

I found this

When booting a Linux kernel, iPXE will construct a “magic initrd” by injecting downloaded files into the initial RAM filesystem image. Any argument supplied to the initrd command will be used as the pathname for that image within the initrd.magic initial RAM filesystem.

in https://ipxe.org/cmd/imgfetch

So this really has something to do with iPXE boot image I think.

FYI @okurz @nicksinger @livdywan

From IT guy:

host  kermit-1.qe.nue2.suse.org.
kermit-1.qe.nue2.suse.org has address 10.168.192.89
kermit-1.qe.nue2.suse.org has IPv6 address 2a07:de40:a102:5:ae1f:6bff:fe47:326

DHCP:

kermit-1:
  mac: 'ac:1f:6b:47:03:26'
  ip4: kermit-1.qe.nue2.suse.org
  dhcp_filename: 'kernelqa/ipxe.efi'

PXE server: 10.168.192.10 -> qa-jump.qe.nue2.suse.org.

qa-jump:~ #  l /srv/tftpboot/kernelqa/ipxe.efi
-rw-r--r-- 1 tftp tftp 945216 Mar 14  2023 /srv/tftpboot/kernelqa/ipxe.efi


host scooter-1.qe.nue2.suse.org.
scooter-1.qe.nue2.suse.org has address 10.168.192.87
scooter-1.qe.nue2.suse.org has IPv6 address 2a07:de40:a102:5:ae1f:6bff:fe47:7338

DHCP:
scooter-1:
  mac: 'ac:1f:6b:47:73:38'
  ip4: scooter-1.qe.nue2.suse.org
  dhcp_filename: 'kernelqa/ipxe.efi'


host squiddlydiddly.qe.nue2.suse.org
squiddlydiddly.qe.nue2.suse.org has address 10.168.193.18


  squiddlydiddly-1:
  mac: '7c:c2:55:85:f9:a4'
  ip4: squiddlydiddly.qe.nue2.suse.org
  hostname: squiddlydiddly
  dhcp_filename: 'kernelqa/ipxe_aarch64.efi'

  qa-jump:~ # l /srv/tftpboot/kernelqa/ipxe_aarch64.efi 
-rw-r--r-- 1 tftp tftp 912032 Mar 14  2023 /srv/tftpboot/kernelqa/ipxe_aarch64.efi

That means all hosts have the same PXE server 10.168.192.10
kermit-1.qe.nue2.suse.org. and scooter-1.qe.nue2.suse.org. boot the file kernelqa/ipxe.efi

The sever squiddlydiddly.qe.nue2.suse.org boot the file kernelqa/ipxe_aarch64.efi

both files ipxe_aarch64.efi and ipxe.efi are NOT symlink to any RPM it just a binary files.
I did not found any records how they ware copy to the server - that the source of them.
and also its not salt because the server qa-jump.qe.nue2.suse.org is not our official PXE server.

I think its something what was done by QA itself in the past.

@okurz @nicksinger @livdywan or kernel qa guy can help ???

I think it would be better to take a small step forward to start solve the problem:

1. Based on current situation, there is no record of when and who copied this ipxe.efi, let alone the source of current ipxe.efi. So who should be responsible for ipxe.efi maintenance ? And what could be the source of ipxe.efi on our iPXE server ?

2. IT suggests

We (as Infra team) should managed this by our salt I would prefer to be an RPM in an official SUSE repo, or at least a project inside IBS / build.suse.de. Take a random code from web, compiled it and run does not sound good to me. Somebody should review the code and confirm that does not contains  back-door or any other spyware.

So is this feasible or can be improved ?

@okurz @nicksinger @livdywan @kernelqa

Hi Kernel QA,

@MMoese @pcervinka

Tools team think kernel will be helpful on next step to help solve the issue.

• The original issue is

UEFI iPXE x86_64 machines, like kermit-1.qe.nue2.suse.org and scooter-1.qe.nue2.suse.org, and aarch64 machine, like squiddlydiddly, fail to load SL Micro 6.1 initrd due to following error:

http://server_ip_address/ipxe/x86_64/pxeboot.SL-Micro.x86_64-6.1.kernel... ok
http://server_ip_address/ipxe/x86_64/pxeboot.SL-Micro.x86_64-6.1.initrd... 61%     ok
EFI stub: ERROR: Failed to open file: initrd
EFI stub: ERROR: Failed to load initrd: 0x800000000000000e
EFI stub: ERROR: efi_stub_entry() failed!
Could not boot: Error 0x7f04828e (http://ipxe.org/7f04828e)
Could not boot: Error 0x7f04828e (http://ipxe.org/7f04828e)

pxeboot.SL-Micro.x86_64-6.1.kernel and pxeboot.SL-Micro.x86_64-6.1.initrd come from SL Micro 6.1 Build12.2 pxe install tar https://download.suse.de/ibs/SUSE:/SLFO:/Products:/SL-Micro:/6.1:/ToTest/images/SL-Micro.x86_64-6.1-Default-SelfInstall-Build12.2.install.tar

But all UEFI iPXE x86_64 and aarch64 machines in Beijing lab can boot successfully with the same kernel and initird.

• I found

When booting a Linux kernel, iPXE will construct a “magic initrd” by injecting downloaded files into the initial RAM filesystem image. Any argument supplied to the initrd command will be used as the pathname for that image within the initrd.magic initial RAM filesystem.

in https://ipxe.org/cmd/imgfetch. So this really has something to do with iPXE boot image I think.

• According what IT guys found, ipxe binary is not maintained by them and it seems that it is done by QA in the past. It turns out to be Kernel QA judging by

host  kermit-1.qe.nue2.suse.org.
kermit-1.qe.nue2.suse.org has address 10.168.192.89
kermit-1.qe.nue2.suse.org has IPv6 address 2a07:de40:a102:5:ae1f:6bff:fe47:326

DHCP:

kermit-1:
  mac: 'ac:1f:6b:47:03:26'
  ip4: kermit-1.qe.nue2.suse.org
  dhcp_filename: 'kernelqa/ipxe.efi'

PXE server: 10.168.192.10 -> qa-jump.qe.nue2.suse.org.

qa-jump:~ #  l /srv/tftpboot/kernelqa/ipxe.efi
-rw-r--r-- 1 tftp tftp 945216 Mar 14  2023 /srv/tftpboot/kernelqa/ipxe.efi


host scooter-1.qe.nue2.suse.org.
scooter-1.qe.nue2.suse.org has address 10.168.192.87
scooter-1.qe.nue2.suse.org has IPv6 address 2a07:de40:a102:5:ae1f:6bff:fe47:7338

DHCP:
scooter-1:
  mac: 'ac:1f:6b:47:73:38'
  ip4: scooter-1.qe.nue2.suse.org
  dhcp_filename: 'kernelqa/ipxe.efi'


host squiddlydiddly.qe.nue2.suse.org
squiddlydiddly.qe.nue2.suse.org has address 10.168.193.18


  squiddlydiddly-1:
  mac: '7c:c2:55:85:f9:a4'
  ip4: squiddlydiddly.qe.nue2.suse.org
  hostname: squiddlydiddly
  dhcp_filename: 'kernelqa/ipxe_aarch64.efi'

  qa-jump:~ # l /srv/tftpboot/kernelqa/ipxe_aarch64.efi 
-rw-r--r-- 1 tftp tftp 912032 Mar 14  2023 /srv/tftpboot/kernelqa/ipxe_aarch64.efi

• @MMoese already drafted a page about how to build ipxe binary, see https://confluence.suse.com/display/qasle/iPXE

• So do you think you can help update ipxe binary image to the latest ??? I think it is necessary. And DON NOT FORGET BACKUP OLD ONE .

I think you have your bootscript incorrectly configured for these machines. I would rather not replace the file we have for our machines with something different - as this might also break our machines that work perfectly well.
Rather I would suggest you build and maintain your own iPXE file, they can safely reside somewhere else and you could update the dhcp configs to point to this file. Building such a binary is really easy, and our confluence pages have everything you need for this. But I'm willing to help you if you have issues with that, just ping me on Slack then. But please don't overwrite whatever files we have in kernelqa/ subdirectory.

MMoese wrote in #note-10:

I think you have your bootscript incorrectly configured for these machines. I would rather not replace the file we have for our machines with something different - as this might also break our machines that work perfectly well.
Rather I would suggest you build and maintain your own iPXE file, they can safely reside somewhere else and you could update the dhcp configs to point to this file. Building such a binary is really easy, and our confluence pages have everything you need for this. But I'm willing to help you if you have issues with that, just ping me on Slack then. But please don't overwrite whatever files we have in kernelqa/ subdirectory.

I would rather say I opened this ticket after many tries with different configurations, but nothing works with these machines on OSD. On the contrary, all machines in Beijing can easily and successfully boot. And ipxe binary image really has something to do with kernel/initrd loading. So I think:

1. If you think bootscript for these machines are not correctly configured, could you help let me know which setting is incorrect, which setting is missing or which setting should be removed ?
2. If you do not know either, I can continue to explore. If, at the last, updating ipxe binary is needed, then it has to be done.

@MMoese

I think I solved the problem to some extent, but I think:

1. Current ipxe boot image may not handle SL-Micro 6.1 pxe-capable intird downloading well. So no one knows what the downloaded initrd will be converted to. I tried many possibilities, but nothing works.

2. I solved the problem to some extent by using:

kernel .......... initrd="TO BE USED INITRD NAME"..........
initrd --name "TO BE USED INITRD NAME" INITRD_URL

Current ipxe_install.pm module does not pass --name option to initrd command. I just used "initrd" as the name. It looks like workaround ???

But I did not give it a full installation/configuration due to downloading issue from host outside.

1. What do you mean with conversion? The image just gets downloaded to RAM?
2. Feel free to add a PR to add the name to the initrd in ipxe_install.pm

Not sure if this is a workaround or actually needed like this - but in doubt better have a workaround than non-working machines :)

MMoese wrote in #note-15:

1. What do you mean with conversion? The image just gets downloaded to RAM?
2. Feel free to add a PR to add the name to the initrd in ipxe_install.pm

Not sure if this is a workaround or actually needed like this - but in doubt better have a workaround than non-working machines :)

I think current ipxe image does not convert the downloaded initrd image into a well-known file, for example, "initrd", "initrd.magic" or others. This might be caused by this ipex image does not treat file pxeboot.SL-Micro.x86_64-6.1.initrd the same as file initrd.