QA » openQA Project » openQA Tests

I used interactive mode to manually select "Boot from Harddisk".

This is what I got: https://openqa.suse.de/tests/386068/modules/boot_to_desktop/steps/1
Looks like some kind of BIOS.

Maybe there is some UEFI Boot entry missing - just using the HDD image doesn't seem to be enough.

dheidler wrote:

I used interactive mode to manually select "Boot from Harddisk".

This is what I got: https://openqa.suse.de/tests/386068/modules/boot_to_desktop/steps/1
Looks like some kind of BIOS.

Maybe there is some UEFI Boot entry missing - just using the HDD image doesn't seem to be enough.

How does this work on x86 with uEFI? Usually with a uEFI installation we generate a grub (or shim) efi file on the ESP (the EFI FAT partition) in /EFI/sles/something.efi. A pointer to that path then gets added to the EFI boot order which gets saved in NVRAM (-pflash with kvm).

So it's perfectly normal that you can't just say "boot from hard disk", since EFI doesn't know what exactly it should boot from there. For all it knows, you could have 500 different efi bootloaders installed on a single hard disk.

on x86 with UEFI when starting up a VM image we get the grub2 boot menu https://openqa.suse.de/tests/408315/modules/boot_to_desktop/steps/1

This is because qemu is being loaded with -boot order=c
https://openqa.suse.de/tests/408315/file/autoinst-log.txt

on aarch64 we are getting the installation media boot menu instead, probably because -boot order=c is missing from the qemu commandline
https://openqa.suse.de/tests/408560/file/autoinst-log.txt

https://github.com/os-autoinst/os-autoinst/blob/c8b866a180c8f44d377cbb349cf933adbce50db4/backend/qemu.pm#L562-L572 only sets the -boot order parameter when the $arch_supports_boot_order is true

https://github.com/os-autoinst/os-autoinst/blob/c8b866a180c8f44d377cbb349cf933adbce50db4/backend/qemu.pm#L324-L331 explicitly sets $arch_support_boot_order as false for aarch64

Does this still have to be the case, or has qemu on aarch64 now support changing the boot order? We could certainly do with it, because the only alternative is very nasty cludgy test hacks

RBrownSUSE wrote:

https://github.com/os-autoinst/os-autoinst/blob/c8b866a180c8f44d377cbb349cf933adbce50db4/backend/qemu.pm#L562-L572 only sets the -boot order parameter when the $arch_supports_boot_order is true

So -boot has been deprecated a couple years ago. It's still somewhat supported because people got used to it and it's easy to use, but it's nothing I would recommend. It has a few shortcomings:

1) "disk" can be anything if you attach multiple disks
2) what is a "CD"? Can we guarantee that we also take cd-over-virtio-scsi into account for example?
3) If you have 2 NICs, how do you define which one goes first?
4) What if you want to boot from SD?
5) s390x doesn't support it

The "proper" way to pass the boot order these days is to use the bootindex= parameter in the respective -device options. See https://github.com/qemu/qemu/blob/master/docs/bootindex.txt for details.

https://github.com/os-autoinst/os-autoinst/blob/c8b866a180c8f44d377cbb349cf933adbce50db4/backend/qemu.pm#L324-L331 explicitly sets $arch_support_boot_order as false for aarch64

Does this still have to be the case, or has qemu on aarch64 now support changing the boot order? We could certainly do with it, because the only alternative is very nasty cludgy test hacks

I just tried and it looks like it doesn't take -boot into account. Bootindex however works like a charm. I would recommend to just switch to that instead. Then you get s390x boot order setting for free as well ;).

aarch64 is ignorig bootindex too
Command for aarche64 bellow doesn't work, but for x86_64 does work.

aarch64 not working, iso is always booted first
/usr/bin/qemu-system-aarch64 -serial file:serial0 -soundhw ac97 -global isa-fdc.driveA= -device VGA -m 4096 -machine virt,usb=off -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56,bootindex=1 -device virtio-scsi-pci,id=scsi0 -device virtio-blk-device,drive=hd1,bootindex=2 -drive file=1,cache=unsafe,if=none,id=hd1,format=qcow2 -drive media=cdrom,if=none,id=cd0,format=raw,file=/var/lib/openqa/share/factory/iso/SLE-12-SP2-Server-DVD-aarch64-Build2050-Media1.iso,snapshot=on -device scsi-cd,drive=cd0,bus=scsi0.0,bootindex=3 -bios /usr/share/qemu/qemu-uefi-aarch64.bin -device nec-usb-xhci -device usb-tablet -device usb-kbd -smp 1 -enable-kvm -no-shutdown -vnc :99,share=force-shared

x86_64 is working, network is first same as legacy -boot
/usr/bin/qemu-system-x86_64 -global isa-fdc.driveA= -vga cirrus -m 1024 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56,bootindex=1 -device virtio-scsi-pci,id=scsi0 -device virtio-blk,drive=hd1,bootindex=2 -drive file=/var/lib/openqa/share/factory/hdd/test2.qcow2,cache=unsafe,if=none,id=hd1,format=qcow2 -drive media=cdrom,if=none,id=cd0,format=raw,file=/var/lib/openqa/share/factory/iso/SLE-12-SP2-Server-DVD-x86_64-Build2043-Media1.iso -device scsi-cd,drive=cd0,bus=scsi0.0,bootindex=3 -boot once=d,menu=on,splash-time=5000 -device usb-ehci -device usb-tablet -smp 1 -enable-kvm

See #11948#note-4 for a proposal

@okurz read https://progress.opensuse.org/issues/11948#note-7 again

oh, ok. so maybe we need an updated package for qemu on aarch64 or something?

dzedro wrote:

aarch64 is ignorig bootindex too
Command for aarche64 bellow doesn't work, but for x86_64 does work.

aarch64 not working, iso is always booted first
/usr/bin/qemu-system-aarch64 -serial file:serial0 -soundhw ac97 -global isa-fdc.driveA= -device VGA -m 4096 -machine virt,usb=off -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56,bootindex=1 -device virtio-scsi-pci,id=scsi0 -device virtio-blk-device,drive=hd1,bootindex=2 -drive file=1,cache=unsafe,if=none,id=hd1,format=qcow2 -drive media=cdrom,if=none,id=cd0,format=raw,file=/var/lib/openqa/share/factory/iso/SLE-12-SP2-Server-DVD-aarch64-Build2050-Media1.iso,snapshot=on -device scsi-cd,drive=cd0,bus=scsi0.0,bootindex=3 -bios /usr/share/qemu/qemu-uefi-aarch64.bin -device nec-usb-xhci -device usb-tablet -device usb-kbd -smp 1 -enable-kvm -no-shutdown -vnc :99,share=force-shared

I used that command line and it booted from network (which is bootindex=1, so first on the list). If I put network as 3 and scsi-disk as 1, it boots from iso. I'm not quite sure about the virtio-blk-device. Could you please try virtio-blk-pci instead?

I tried on server overdrive2 virtio-blk-device and virtio-blk-pci no change,bootindex=1 on net or disk but ISO was always booted first.

/usr/bin/qemu-system-aarch64 -serial file:serial0 -soundhw ac97 -global isa-fdc.driveA= -device VGA -m 4096 -machine virt,usb=off -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56,bootindex=2 -device virtio-scsi-pci,id=scsi0 -device virtio-blk-device,drive=hd1,bootindex=1 -drive file=1,cache=unsafe,if=none,id=hd1,format=qcow2 -drive media=cdrom,if=none,id=cd0,format=raw,file=/var/lib/openqa/share/factory/iso/SLE-12-SP2-Server-DVD-aarch64-Build2052-Media1.iso,snapshot=on -device scsi-cd,drive=cd0,bus=scsi0.0,bootindex=3 -bios /usr/share/qemu/qemu-uefi-aarch64.bin -device nec-usb-xhci -device usb-tablet -device usb-kbd -smp 1 -enable-kvm -no-shutdown -vnc :99,share=force-shared

overdrive2:~ # rpm -qa|grep qemu
qemu-linux-user-2.3.1-5.1.aarch64
qemu-uefi-aarch64-0.1+svn17700-3.6.noarch
qemu-tools-2.5.0-4.1.aarch64
qemu-ipxe-1.0.0-25.1.noarch
qemu-vgabios-1.8.1-25.1.noarch
qemu-arm-2.5.0-4.1.aarch64
qemu-2.5.0-4.1.aarch64

So this works for me on thunderx7 - I get the iso boot prompt rather than network dots:

/usr/bin/qemu-system-aarch64 -nographic -soundhw ac97 -global isa-fdc.driveA= -m 4096 -machine virt,usb=off,gic-version=3 -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56,bootindex=2 -device virtio-scsi-pci,id=scsi0 -device virtio-blk-device,drive=hd1,bootindex=3 -drive file=1,cache=unsafe,if=none,id=hd1,format=qcow2 -drive media=cdrom,if=none,id=cd0,format=raw,file=/mounts/dist/install/SLE-12-SP2-Server-RC1/SLE-12-SP2-Server-DVD-aarch64-RC1-DVD1.iso,snapshot=on -device scsi-cd,drive=cd0,bus=scsi0.0,bootindex=1 -bios /usr/share/qemu/qemu-uefi-aarch64.bin -device nec-usb-xhci -device usb-tablet -device usb-kbd -smp 1 -enable-kvm -no-shutdown -vnc :99,share=force-shared

I'll try again on overdrive2.

algraf wrote:

I'll try again on overdrive2.

It's a virtual firmware bug. Firmware (/usr/share/qemu/qemu-uefi-aarch64.bin) on overdrive2 doesn't honor the bootindex. If instead you try with /dev/shm/QEMU_EFI.fd you will see it work.

algraf wrote:

algraf wrote:

I'll try again on overdrive2.

It's a virtual firmware bug. Firmware (/usr/share/qemu/qemu-uefi-aarch64.bin) on overdrive2 doesn't honor the bootindex. If instead you try with /dev/shm/QEMU_EFI.fd you will see it work.

If I use the latest OVMF from SLES, it also works:

http://download.suse.de/ibs/SUSE:/SLE-12-SP2:/GA/standard/noarch/qemu-uefi-aarch64-2015+git1462940744.321151f-6.6.noarch.rpm

However keep in mind that with this OVMF version, bootup takes ~10-20 seconds until anything happens at all, so you will run into the initial 15 second timeout to search for the grub screen.

Any updates here?

With qemu-uefi-aarch64-2015+git1462940744.321151f-6.6.noarch.rpm it worked on overdrive2, but I was not able to successfully run it on another aarch64 server from orthos
I will try it again and ask Alexander for help

In SLEnkins we are using EFI fw "qemu-uefi-aarch64" from http://download.opensuse.org/ports/aarch64/factory/repo/oss/ with following workaround to boot from hardrive.

When installation from DVD is finished we just write "fs0:\EFI\sles\grubaa64.efi" into /boot/efi/startup.nsh - it acts like autoexec.bat for EFI, it tells EFI about grub location and boots from it.

echo "fs0:\EFI\sles\grubaa64.efi" > /boot/efi/startup.nsh

Then the EFI for aarch64 is able to boot even without bootindex/order, but you have to wait approx. 10 sec for EFI timeout to load startup.nsh from FS0: partition.

Another working method is usage of nvram template for EFI from the same repo above with <boot order='1'/> for a disk device. But it applies only to virsh. Strip for aarch64 xml virsh domain definition follows:

<loader readonly='yes' type='pflash'>/usr/share/qemu/aavmf-aarch64-code.bin</loader>
<nvram template='/usr/share/qemu/aavmf-aarch64-vars.bin'>/var/tmp/slenkins/workspace/iso2qcow2/nvram.bin</nvram>

When I run the virsh domain it runs following qemu command (maybe you can grab some qemu parameters from that):

/usr/bin/qemu-system-aarch64 -name guest=basebox-iso2qcow2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-19-basebox-iso2qcow2/master-key.aes -machine virt-2.6,accel=tcg,usb=off,dump-guest-core=off -cpu cortex-a57 -drive file=/usr/share/qemu/aavmf-aarch64-code.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/tmp/slenkins/workspace/iso2qcow2/nvram.bin,if=pflash,format=raw,unit=1 -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1a31ea64-b32e-45c3-5d94-89a0df4d88d2 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-19-basebox-iso2qcow2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device virtio-scsi-device,id=scsi0 -device virtio-serial-device,id=virtio-serial0 -drive file=/var/tmp/slenkins/workspace/iso2qcow2/hard-disk.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2 -drive file=/var/tmp/slenkins/workspace/iso2qcow2/cdrom.iso,format=raw,if=none,id=drive-scsi0-0-0-1,readonly=on -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1,bootindex=1 -drive file=/var/tmp/slenkins/workspace/iso2qcow2/installation.raw,format=raw,if=none,id=drive-virtio-disk0 -device virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=29,id=hostnet0 -device virtio-net-device,netdev=hostnet0,id=net0,mac=52:54:00:5b:82:93 -netdev tap,fd=31,id=hostnet1 -device virtio-net-device,netdev=hostnet1,id=net1,mac=52:54:00:5b:82:94 -netdev tap,fd=32,id=hostnet2 -device virtio-net-device,netdev=hostnet2,id=net2,mac=52:54:00:5b:82:95 -netdev tap,fd=33,id=hostnet3 -device virtio-net-device,netdev=hostnet3,id=net3,mac=52:54:00:5b:82:96 -serial pty -device virtio-balloon-device,id=balloon0 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-device,rng=objrng0,id=rng0 -msg timestamp=on

This works for me:

cp /usr/share/qemu/aavmf-aarch64-vars.bin /home/thehejik/nvram.bin (do a copy of nvram efi template)

sudo qemu-system-aarch64 -machine virt-2.6 -cpu cortex-a53 -nographic -drive file=/usr/share/qemu/aavmf-aarch64-code.bin,if=pflash,format=raw,unit=0 -drive file=/home/thehejik/nvram.bin,if=pflash,format=raw,unit=1

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: sles+extratests
http://openqa.suse.de/tests/627155

Created bug and PR it's not fixing the issue but at least showing what is happening

Remove extratest-textmode and extratest-gnome from ARM till this issue will be fixed. Need to return them back afterwards

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: sles+extratests
http://openqa.suse.de/tests/627155

There is a similar problem that I'm going to try to address by adding a post fail hook on grub_test, as it seems that somehow the uefi boot order is being lost somewhere over the ranbow here... (Might not be the same issue, but might be the same... we'll see)

I triggered a job to boot from HDD manually

openqa_clone_job_osd 878193 _GROUP=0 EXTRATEST=1 HDD_1=sle-12-SP3-aarch64-0333-textmode@aarch64.qcow2 PUBLISH_HDD_1= TEST=extra_tests_in_textmode BOOT_HDD_IMAGE=1

which fails in an UEFI menue: https://openqa.suse.de/tests/882027#step/boot_to_desktop/1 so I guess we are not done here.

The according qemu command line is

/usr/bin/qemu-system-aarch64 -serial file:serial0 -soundhw ac97 -global isa-fdc.driveA= -device virtio-gpu-pci -m 1024 -machine virt,usb=off,gic-version=host -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -device virtio-scsi-pci,id=scsi0 -device virtio-blk-device,drive=hd1,bootindex=0 -drive file=raid/l1,cache=unsafe,if=none,id=hd1,format=qcow2 -drive media=cdrom,if=none,id=cd0,format=raw,file=/abuild/openqa/openqa/pool/1/SLE-12-SP3-Server-DVD-aarch64-Build0333-Media1.iso,snapshot=on -device scsi-cd,drive=cd0,bus=scsi0.0 -bios /usr/share/qemu/qemu-uefi-aarch64.bin -device nec-usb-xhci -device usb-tablet -device usb-kbd -smp 2 -enable-kvm -no-shutdown -vnc :91,share=force-shared -qmp unix:qmp_socket,server,nowait -monitor unix:hmp_socket,server,nowait -S -monitor telnet:127.0.0.1:20012,server,nowait

PR merged, now we can test extra tests on aarch64. Added 'extra_tests_in_textmode' and 'extra_tests_on_gnome' in test development and triggered once:

$ openqa_client_osd isos post _NOOBSOLETEBUILD=1 ISO=SLE-12-SP3-Server-DVD-aarch64-Build063-Media1.iso BUILD=0363 ARCH=aarch64 BUILD_SLE=0363 DISTRI=sle FLAVOR=Server-DVD VERSION=12-SP3 BUILD_SDK=0185 TEST=extra_tests_in_textmode,extra_tests_on_gnome _GROUP=0
{ count => 4, ids => [910688 .. 910691] }

failed with

Attemping to download: https://openqa.suse.de /var/lib/openqa/cache/SLE-12-SP3-Server-DVD-aarch64-Build063-Media1.iso, iso, 910689
CACHE: download of SLE-12-SP3-Server-DVD-aarch64-Build063-Media1.iso failed with 404: Not Found

in https://openqa.suse.de/tests/910689/file/autoinst-log.txt for whatever reason again. Great. Can someone explain? Apparently I mistyped the ISO parameter "063" instead of "0363", stupid.

Next try

$ openqa_client_osd isos post _NOOBSOLETEBUILD=1 ISO=SLE-12-SP3-Server-DVD-aarch64-Build0363-Media1.iso BUILD=0363 ARCH=aarch64 BUILD_SLE=0363 DISTRI=sle FLAVOR=Server-DVD VERSION=12-SP3 BUILD_SDK=0185 TEST=extra_tests_in_textmode,extra_tests_on_gnome _GROUP=0
{ count => 4, ids => [910692 .. 910695] }

-> waiting for extra_tests_in_textmode, extra_tests_on_gnome

extra_tests_on_gnome fails in user_defined_snapshot which reboots but does not seem to use "wait_boot".

extra_tests_in_textmode: https://openqa.suse.de/tests/910694#step/snapper_cleanup/4 shows that an image on aarch64 has no snapshots enabled. Should I expect any warning in the installer job in e.g. https://openqa.suse.de/tests/910692#step/partitioning/1 ?

I assume that there are no snapshots because snapper says there is no root-config in https://openqa.suse.de/tests/910694#step/snapper_cleanup/4 which so far often seemed to be the case when there aren't any snapshots.
There would be only a warning when using the expert partitioner and defining a root volume which is too small. We have one according test https://openqa.suse.de/tests/909167#step/partitioning_warnings/10 showing the warning works as expected on aarch64. Looking on https://openqa.suse.de/tests/909645#step/partitioning/1 FS is 18.54 GB big, and that is enough for snapshots. https://openqa.suse.de/tests/909162#step/boot_into_snapshot/9 shows that there are snapshots so maybe different problem. Anyway, the first module btrfs_qgroups fails with exactly the same bug we saw on ppc64le so it's already helpful.

Problem was that the other openQA machine definitions, e.g. 64bit for x86_64 set HDDSIZEGB=20 already but aarch64 does not. I don't know why this is true and I consider it somehow wrong by default to do this on the machine. I now set it on the test suite as it relies on snapshots being present which is only the case when the HDD is big enough. Adjusted test suites and retriggered with

$ openqa_client_osd isos post _NOOBSOLETEBUILD=1 ISO=SLE-12-SP3-Server-DVD-aarch64-Build0363-Media1.iso BUILD=0363 ARCH=aarch64 BUILD_SLE=0363 DISTRI=sle FLAVOR=Server-DVD VERSION=12-SP3 BUILD_SDK=0185 TEST=extra_tests_in_textmode _GROUP=0
{ count => 2, ids => [911010, 911011] }

-> https://openqa.suse.de/tests/911010#live

I can't seem to understand what image https://openqa.suse.de/tests/911018#step/boot_to_desktop/21 boots that it does not find snapshots. The image creation job shows https://openqa.suse.de/tests/911010#step/grub_test/1 "Start bootloader from read-only snapshot" and also when I start the qcow image manually I find that boot entry and snapshots but somehow the job wants to pickup a different file…

scheduled 1 creation job for sle12sp2 image: https://openqa.suse.de/tests/918749
if it done successfully, will restart this online migration job: https://openqa.suse.de/tests/918720

ok, the problem in #11948#note-50 is that the current implementation of caching is just looking at the filename. This means that when a downstream job is started, it will download the image from the web UI but not re-do this whenever the asset changes.

Current workaround: Delete the file from the cache directory on the corresponding worker.

mitiao wrote:

scheduled 1 creation job for sle12sp2 image: https://openqa.suse.de/tests/918749
if it done successfully, will restart this online migration job: https://openqa.suse.de/tests/918720

On 12sp2, the bootloader looks in text(not graphic) which is different when booting 12sp3:
https://openqa.suse.de/tests/918749#step/bootloader_uefi/4

12sp3:
https://openqa.suse.de/tests/918583#step/bootloader_uefi/1

Is it expected? what can I do to make sp2 booting be consistent with sp3?
If not, should we have new needles for sp2 uefi bootloader?

mitiao wrote:

On 12sp2, the bootloader looks in text(not graphic) which is different when booting 12sp3:
https://openqa.suse.de/tests/918749#step/bootloader_uefi/4
[…]
Is it expected?

not at all expected. You can see in the SLE 12 SP3 GM result that the graphical installer boot menu looks exactly look on other architectures.

what can I do to make sp2 booting be consistent with sp3?

Maybe the change of UEFI firmware and/or aarch64 worker and qemu caused a change.

If not, should we have new needles for sp2 uefi bootloader?

If already the bootloader screen differs I fear more will be different. So at best crosscheck manually e.g. with qemu on aarch64 how it looks like or if it's really the only screen that is different. However I don't see this related to "can't boot pre-installed image". I guess still the story for migration on aarch64 could be a slight challenge as we might have the problem that old sles only runs on old qemu+firmware on overdrive2 and new sles only on new qemu+firmware. But check the job history on overdrive2 to find maintenance tests for sles 12 sp2 on arm as a start.

moved sles12_minimal_base+sdk_create_hdd to SLES functional group as it's pretty stable.

extra_tests_in_textmode seems to incomplete often because the source media (Media2) is already deleted by GRU cleanup task when the job starts. That smells like a implementation problem in the cleanup problem. Workaround might be to just move it to the acceptance group with higher quota. extra_tests_on_gnome fails in user_defined_snapshot asserting the tianocore menue when grub for a snapshot is shown correctly. Need to check and change that.

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2855 for extra_tests_on_gnome

https://openqa.suse.de/tests/924843 fails now with the same symptom as ppc64le, moved to sles functional group

same for extra_tests_in_textmode after it always incompletes in the test development group because I assume Media2 is prematurely deleted because of too low job group.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: sles+extratests
http://openqa.suse.de/tests/627155