Project

General

Profile

Actions

action #113794

closed

Use prepared OVMF image with expected settings size:M

Added by livdywan over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2022-06-03
Due date:
% Done:

20%

Estimated time:

Description

Observation

With QEMU 7, the default video resolution was changed which caused #111992 since tests end up with an unexpected resolution. #111992#note-16 suggests using a prepared NVRAM image.

This might also be useful in preparation for #113357.

Acceptance criteria

  • AC1: Tests can boot with a custom NVRAM image

Suggestions

  • Maybe nvram tooling can be used to change the settings (efitools)
  • The following can be used to manually generate a file. Create ovmf code+vars file, spawn VM, enter menu manually, then terminate the VM to save the file
/usr/bin/qemu-img create -f qcow2 -F raw -b /usr/share/qemu/ovmf-x86_64-ms-code.bin pflash-code-overlay0 1966080
/usr/bin/qemu-img create -f qcow2 -F raw -b /usr/share/qemu/ovmf-x86_64-ms-vars.bin pflash-vars-overlay0 131072
/usr/bin/qemu-system-x86_64 -vga qxl -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 2048 -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -device qemu-xhci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :101,share=force-shared -drive id=pflash-code-overlay0,if=pflash,file=pflash-code-overlay0,unit=0,readonly=on -drive id=pflash-vars-overlay0,if=pflash,file=pflash-vars-overlay0,unit=1
  • See if we can generate the image in openQA or GitLab and automatically upload it as an asset to openQA's asset storage. Maybe small changes are necessary how we handle those special UEFI variables.

Related issues 5 (3 open2 closed)

Related to openQA Infrastructure - action #114523: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64 size:MWorkable2022-06-03

Actions
Related to openQA Tests - action #116914: [qe-core] recent uefi changes makes test failWorkable2022-09-21

Actions
Related to openQA Tests - action #116812: [qe-core] Leap 15.5 uefi console switch fail size:MBlockedokurz2022-09-19

Actions
Related to openQA Tests - action #116704: [qe-core] nvme@uefi: boot from DVD post install instead of from HDDResolvedtinita2022-09-16

Actions
Copied from openQA Project - action #111992: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl size:MResolvedtinita2022-06-03

Actions
Actions #1

Updated by livdywan over 1 year ago

  • Copied from action #111992: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl size:M added
Actions #2

Updated by livdywan over 1 year ago

  • Subject changed from Use prepared OVMF image with expected settings to Use prepared OVMF image with expected settings size:M
  • Description updated (diff)
  • Status changed from New to Workable
  • Assignee deleted (nicksinger)
Actions #3

Updated by mkittler over 1 year ago

  • Assignee set to mkittler
Actions #4

Updated by okurz over 1 year ago

  • Related to action #114523: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64 size:M added
Actions #5

Updated by mkittler over 1 year ago

  • Assignee deleted (mkittler)

With all the worker recovery going on I haven't had a chance to work on this ticket further. On Monday I'll also not be able to work on it the first half on the day. So anybody is welcome to pickup the ticket meanwhile.

Yesterday I explored the possibility to use efivar or tools from efitools to update the pflash-vars file without having to spawn a VM. However, it didn't work (e.g. efivar --fromfile=… didn't allow to edit read/write to the file). So likely we cannot avoid spawning a VM for doing the settings.

By the way, to enter the boot menu manually without VNC, one can use the following command (after creating the images as shown in the description):

/usr/bin/qemu-system-x86_64 -vga qxl -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 2048 -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -device qemu-xhci -device usb-tablet -smp 1 -enable-kvm -no-shutdown  -drive id=pflash-code-overlay0,if=pflash,file=pflash-code-overlay0,unit=0,readonly=on -drive id=pflash-vars-overlay0,if=pflash,file=pflash-vars-overlay0,unit=1 -boot menu=on
Actions #6

Updated by livdywan over 1 year ago

mkittler wrote:

Yesterday I explored the possibility to use efivar or tools from efitools to update the pflash-vars file without having to spawn a VM. However, it didn't work (e.g. efivar --fromfile=… didn't allow to edit read/write to the file). So likely we cannot avoid spawning a VM for doing the settings.

Would you mind clarifying what didn't work? Which commands did you try? Did you get an error?

Actions #7

Updated by mkittler over 1 year ago

I tried efivar --fromfile=pflash-vars-overlay0 --list and it just returned my normal EFI vars (and not the ones from the QEMU image). Specifying /usr/share/qemu/ovmf-x86_64-ms-vars.bin directly didn't work either. I've checked the binaries from efitools but their --help-text didn't look like specifying a file would be possible at all.

Actions #8

Updated by okurz over 1 year ago

  • Priority changed from High to Normal
  • Target version changed from Ready to future

We have to focus on other urgent tasks with reduced capacity so moving this out of backlog

Actions #9

Updated by okurz over 1 year ago

  • Target version changed from future to Ready

adding back to the backlog. I think I overlooked that we have two blocked tickets in the backlog

Actions #10

Updated by tinita over 1 year ago

  • Priority changed from Normal to High

We need this as a prerequisite for #111992

Actions #11

Updated by tinita over 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita
Actions #12

Updated by openqa_review over 1 year ago

  • Due date set to 2022-09-08

Setting due date based on mean cycle time of SUSE QE Tools

Actions #13

Updated by tinita over 1 year ago

I created a job which sets the resolution in the tianocore menu, then enters grub and types "halt".
It publishes the modified pflash file via PUBLISH_PFLASH_VARS=ovmf-x86_64-ms-vars-800x600.qcow2.
Now I need to test if I can use that in a second job.

openqa-clone-job --host localhost:9526 https://openqa.opensuse.org/tests/2536831 QEMUVGA=qxl UEFI=1 UEFI_PFLASH_CODE=/usr/share/qemu/ovmf-x86_64-ms-code.bin UEFI_PFLASH_VARS=/usr/share/qemu/ovmf-x86_64-ms-vars.bin  DUALBOOT=1 BOOT_MENU=1 PAUSE_ON_SCREEN_MISMATCH=1 _SKIP_POST_FAIL_HOOKS=1 RETRY=0 SCHEDULE=tests/boot/boot_to_desktop _TRIGGER_JOB_DONE_HOOK=0 ISO=openSUSE-Tumbleweed-DVD-x86_64-Snapshot20220824-Media.iso HDD_1= PUBLISH_PFLASH_VARS=ovmf-x86_64-ms-vars-800x600.qcow2

...

Actions #14

Updated by livdywan over 1 year ago

Discussed in the Unblock. Marius suggested this could be done by moving the logic into a test module, come up with a jobs post command line and e.g. add this to a (masnual) gitlab pipeline.

Actions #16

Updated by tinita over 1 year ago

I created a verification run:
https://openqa.opensuse.org/tests/2600931
Interestingly it worked with the openSUSE-Tumbleweed-NET-x86_64-Snapshot20220829-Media.iso while locally only DVD worked.
Next step: use the generated pflash vars on o3

Actions #17

Updated by tinita over 1 year ago

This PR is needed as a first step, so that existing code does not fail if we use the prepared pflash vars image: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15471 - Allow already preselected resolution 800x600

Actions #19

Updated by tinita over 1 year ago

Verification run: https://openqa.opensuse.org/tests/2636971#step/bootloader_uefi/19

Setting: UEFI_PFLASH_VARS ovmf-x86_64-ms-vars-800x600.qcow2

Confirming that it is accepeted that the correct resolution is already set.

Actions #20

Updated by tinita over 1 year ago

Verification run on openqaworker1 with an upgraded qemu-ovmf-x86_64-202202-150400.3.3.noarch: https://openqa.opensuse.org/tests/2637658

Downgraded and added zypper lock again

Actions #23

Updated by livdywan over 1 year ago

  • Due date changed from 2022-09-08 to 2022-09-14
  • Status changed from In Progress to Feedback

Changes merged, let's see if this works as expected

Actions #24

Updated by tinita over 1 year ago

  • Status changed from Feedback to In Progress
Actions #25

Updated by tinita over 1 year ago

I'm trying to figure out the right openqa-cli api -X POST jobs commandline
Marius pointed my to
https://github.com/os-autoinst/scripts/blob/master/trigger-openqa_in_openqa#L30

Then I need to create a new repo in gitlab and create a schedule.

Actions #26

Updated by tinita over 1 year ago

Current working command:

openqa-cli api -X POST jobs --host https://openqa.opensuse.org
 TEST=ovmf-resolution QEMUVGA=qxl UEFI=1
 UEFI_PFLASH_CODE=/usr/share/qemu/ovmf-x86_64-ms-code.bin UEFI_PFLASH_VARS=/usr/share/qemu/ovmf-x86_64-ms-vars.bin
 SCHEDULE=tests/boot/tianocore_set_resolution.pm
 PUBLISH_PFLASH_VARS=ovmf-x86_64-ms-vars-800x600.qcow2
 DISTRI=openSUSE VERSION=Tumbleweed FLAVOR=NET
 ISO=openSUSE-Tumbleweed-NET-x86_64-Snapshot20220911-Media.iso

https://openqa.opensuse.org/tests/2647994

Actions #27

Updated by tinita over 1 year ago

I moved the find_latest_published_tumbleweed_image from trigger-openqa_in_openqa into _common in smaller functions in order to test and reuse it. So far I pushed to a branch:
https://github.com/perlpunk/scripts/tree/trigger-pflash-test

Then I can use it in a new script to schedule the pflash job with the latest iso image.

Actions #28

Updated by tinita over 1 year ago

  • Due date changed from 2022-09-14 to 2022-09-17

I created a trigger-pflash-test and a github workflow.
TODO:

  • Change workflow to run once a day
  • Create a dedicated user plus api key on o3 for scheduling the job
Actions #29

Updated by okurz over 1 year ago

We triggered a manual job from github actions which created an ovmf vars image with resolution 800x600. This created an image "ovmf-x86_64-ms-vars-800x600.qcow2" on o3 which I hard-linked to the "fixed" dir /var/lib/openqa/share/factory/hdd/fixed . We still plan to create an update on a weekly base.

tinita changed the definitions of openQA machines "uefi_win" and win10+win11 variants to use that file. I created a verification run with

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org/tests/2664073 _GROUP=0 BUILD= TEST=gnome_dual_windows10_svga_ovmf_image

Created job #2669479: opensuse-Tumbleweed-DVD-x86_64-Build20220914-gnome_dual_windows10@uefi_win -> https://openqa.opensuse.org/t2669479

tinita removed the package locks on o3 workers and upgraded the package. I created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15537 to only apply the resolution setting when necessary.

I downloaded the pflash vars image to OSD once manually with

cd /var/lib/openqa/factory/hdd/fixed
wget https://openqa.opensuse.org/tests/2669542/asset/hdd/ovmf-x86_64-ms-vars-800x600.qcow2

and tinita removed the locks from both o3+osd

EDIT: whoopsie, the above test job is missing the actual image setting. Retriggered with

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org/tests/2664073 _GROUP=0 BUILD= TEST=gnome_dual_windows10_svga_ovmf_image UEFI_PFLASH_VARS=ovmf-x86_64-ms-vars-800x600.qcow2

Created job #2670480: opensuse-Tumbleweed-DVD-x86_64-Build20220914-gnome_dual_windows10@uefi_win -> https://openqa.opensuse.org/t2670480

Actions #30

Updated by tinita over 1 year ago

https://github.com/os-autoinst/scripts/pull/187 - Trigger test to generate pflash vars image (merged)

Actions #31

Updated by tinita over 1 year ago

  • Status changed from In Progress to Feedback
Actions #32

Updated by okurz over 1 year ago

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15537 merged. https://openqa.opensuse.org/t2670542 shows that tests work now with the new image without setting any different resolution in the firmware menu. I suggest to wait until tomorrow to resolve the ticket unless objections or regressions have been met.

Actions #33

Updated by tinita over 1 year ago

  • Status changed from Feedback to Resolved
Actions #34

Updated by okurz over 1 year ago

  • Due date deleted (2022-09-17)

https://openqa.suse.de/tests/9526141#settings is using QEMUVGA=virtio. I suggested to jlausuch who saw that to try if the resolution will be correct if we use the updated pflash vars image.

Actions #35

Updated by jlausuch over 1 year ago

@okurz @tinita
I see most uefi jobs failing now, specially in JeOS and SLE Micro... I tried restarting some of them with UEFI_PFLASH_VARS=ovmf-x86_64-ms-vars-800x600.qcow2 and seems to work... should we update this in uefi machine in OSD or should I specify this variable in all uefi tests that are broken?

Actions #36

Updated by okurz over 1 year ago

Use the most downstream place where UEFI+QEMUVGA=qxl/virtio is set. So e.g. if UEFI is set by the machine and QEMUVGA in a job template then set in the job template. If QEMUVGA is set in a machine but UEFI in a job template (I advise against that combination setting) then in the job template. If UEFI+QEMUVGA are set in a machine definition (preferred solution) then please also set in the machine.

Please use the still open #111992 for any remaining problems.

Actions #38

Updated by jlausuch over 1 year ago

Hi, I have updated the machine uefi-virtio-vga in OSD (which already had QEMUVGA=virtio) with UEFI_PFLASH_CODE=ovmf-x86_64-ms-vars-800x600.qcow2 .
Those are used in JeOS tests where we had the most bunch of tests failing in the bootloader.

Actions #40

Updated by pstivanin over 1 year ago

@tinita:
I see that there still is a lock on the following worker:

pstivanin@openqa:~> sudo salt 'openqaworker-arm-3.suse.de' cmd.run 'zypper ll'
openqaworker-arm-3.suse.de:

    # | Name              | Type    | Repository | Comment
    --+-------------------+---------+------------+--------
    1 | qemu-uefi-aarch64 | package | (any)      |

is this expected or can we remove it?

Thanks a lot!

Actions #41

Updated by okurz over 1 year ago

pstivanin wrote:

@tinita:
I see that there still is a lock on the following worker:

pstivanin@openqa:~> sudo salt 'openqaworker-arm-3.suse.de' cmd.run 'zypper ll'
openqaworker-arm-3.suse.de:

    # | Name              | Type    | Repository | Comment
    --+-------------------+---------+------------+--------
    1 | qemu-uefi-aarch64 | package | (any)      |

is this expected or can we remove it?

This is still expected, see #114523 for details regarding aarch64.

Actions #42

Updated by okurz over 1 year ago

  • Related to action #116914: [qe-core] recent uefi changes makes test fail added
Actions #43

Updated by okurz over 1 year ago

  • Related to action #116812: [qe-core] Leap 15.5 uefi console switch fail size:M added
Actions #44

Updated by okurz over 1 year ago

  • Related to action #116704: [qe-core] nvme@uefi: boot from DVD post install instead of from HDD added
Actions

Also available in: Atom PDF