QA (public) » openQA Project (public)

The following PR should fix the issue: https://github.com/os-autoinst/os-autoinst/pull/2083 It doesn't work, see @favogt's comment on GitHub.

cdywan wrote:

Maybe what we want to do is as simple as not using qxl and failing early if somebody tries to do that: https://github.com/os-autoinst/os-autoinst/pull/2082

I count at least two votes against not supporting qxl, hence closing my PR.

So this probably means adapting the tests to use 1280x800 needles in the first steps.

BOOT_MENU=1 helps:

script/openqa-clone-job --host localhost:9526 https://openqa.opensuse.org/tests/2449859 QEMUVGA=qxl UEFI=1 UEFI_PFLASH_CODE=/usr/share/qemu/ovmf-x86_64-ms-code.bin UEFI_PFLASH_VARS=/usr/share/qemu/ovmf-x86_64-ms-vars.bin DUALBOOT=1 BOOT_MENU=1

I was able to enter the tianocore-devicemanager and select a different resolution. Created some needles.
Next: trying to enter the openSUSE bootmenu

tinita wrote:

I was able to enter the tianocore-devicemanager and select a different resolution. Created some needles.
Next: trying to enter the openSUSE bootmenu

What's the plan on how to use this in production? Change the resolution on each (initial) boot or run this module once to generate a vars image and use that in all tests?

favogt wrote:

tinita wrote:

I was able to enter the tianocore-devicemanager and select a different resolution. Created some needles.
Next: trying to enter the openSUSE bootmenu

What's the plan on how to use this in production? Change the resolution on each (initial) boot or run this module once to generate a vars image and use that in all tests?

I would say we should have an openQA test code implementation which is able to change the resolution first. And the configuration can be saved in an uefi-pflash-vars file that we could upload from openQA and reuse in other or all tests.

What I did locally to select the resolution manually and save that in the vars file:

/usr/bin/qemu-img create -f qcow2 -F raw -b /usr/share/qemu/ovmf-x86_64-ms-code.bin pflash-code-overlay0 1966080
/usr/bin/qemu-img create -f qcow2 -F raw -b /usr/share/qemu/ovmf-x86_64-ms-vars.bin pflash-vars-overlay0 131072
/usr/bin/qemu-system-x86_64 -vga qxl -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 2048 -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -device qemu-xhci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :101,share=force-shared -drive id=pflash-code-overlay0,if=pflash,file=pflash-code-overlay0,unit=0,readonly=on -drive id=pflash-vars-overlay0,if=pflash,file=pflash-vars-overlay0,unit=1

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15184 - Select resolution in Tianocore bootloader
https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/774 - Select resolution in Tianocore bootloader

Both PRs merged. I will upgrade ovmf on one o3 worker and trigger tests specifically there to crosscheck if this is enough.

EDIT: Removed zypper lock on openqaworker4 and cloned a specific test:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org/tests/2462733 WORKER_CLASS=openqaworker4 _GROUP=0 BUILD= TEST=okurz-gnome_dual_windows10_poo111992

Created job #2462909: opensuse-Tumbleweed-DVD-x86_64-Build20220710-gnome_dual_windows10@uefi_win -> https://openqa.opensuse.org/t2462909

https://openqa.opensuse.org/tests/2462909#step/bootloader_uefi/40 failed so not good enough.

As I could not find older versions of qemu-ovmf-x86_64 I rolled back to an older snapshot:

# snapper ls
  # | Type   | Pre # | Date                     | User | Used Space | Cleanup | Description            | Userdata    
----+--------+-------+--------------------------+------+------------+---------+------------------------+-------------
 0  | single |       |                          | root |            |         | current                |             
21  | single |       | 2022-06-23T00:02:59 CEST | root | 116.28 MiB | number  | Snapshot Update of #20 |             
22  | single |       | 2022-06-24T00:59:13 CEST | root | 105.86 MiB | number  | Snapshot Update of #21 |             
23  | single |       | 2022-07-05T01:26:00 CEST | root |  97.38 MiB | number  | Snapshot Update of #22 |             
24  | pre    |       | 2022-07-06T14:28:10 CEST | root |   2.58 MiB | number  | zypp(zypper)           | important=no
25  | post   |    24 | 2022-07-06T14:28:23 CEST | root |   3.06 MiB | number  |                        | important=no
26  | single |       | 2022-07-07T00:57:17 CEST | root |  96.66 MiB | number  | Snapshot Update of #23 |             
27  | pre    |       | 2022-07-07T03:00:06 CEST | root | 560.00 KiB | number  | zypp(zypper)           | important=no
28  | post   |    27 | 2022-07-07T03:00:52 CEST | root |   4.07 MiB | number  |                        | important=no
29* | single |       | 2022-07-08T01:46:38 CEST | root |  57.60 MiB |         | Snapshot Update of #26 |             
30  | pre    |       | 2022-07-08T03:15:53 CEST | root | 448.00 KiB | number  | zypp(zypper)           | important=no
31  | post   |    30 | 2022-07-08T03:15:57 CEST | root |   3.40 MiB | number  |                        | important=no
32  | pre    |       | 2022-07-11T11:48:19 CEST | root | 240.00 KiB | number  | zypp(zypper)           | important=no
33  | post   |    32 | 2022-07-11T11:48:25 CEST | root | 708.00 KiB | number  |                        | important=no
# snapper rollback 31 && reboot

https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/775 - Select 800x600 resolution in Tianocore bootloader
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15209 - Select 800x600 resolution in Tianocore bootloader

Verification run on openqaworker4 with upgraded qemu-ovmf-x86_64 package: https://openqa.opensuse.org/tests/2467536

Looks good

tinita wrote:

Is there anything left to do? Remove some zypper locks elsewhere?

Yes, remove the according zypper lock on all o3 machines and upgrade.

Comparing opensuse-Tumbleweed-DVD-x86_64-Build20221003-upgrade_Leap_15.2_cryptlvm@uefi failing in bootloader_uefi with the custom OVMF vars file to opensuse-Tumbleweed-DVD-x86_64-cryptlvm_vanilla_upgraded_ovmf_okurz@uefi not failing in bootloader_uefi. It seems like the custom OVMF vars image is remembering "something" about boot selections hence preventing the system to behave as we expect it.

Considering @okurz's last findings we need to either tweak how we create the custom OVMF vars file (so it acts like the vanilla file and only changes the resolution) or we find a way to deal with the different resolution. Not sure what we can tweak when creating the custom vars file, though. Dealing with the different resolution would maybe be possible by creating specific needles but at least in the installer we should not need any specific needles anymore. We actually could look into setting XRES=1280 and YRES=800 in accordance to the new defaults for qxl tests. With a bit of luck it doesn't need to many needle changes (at least less than dealing with distorted/resized images).

Note that the only really problematic case was the case where GRUB then booted with a resolution of 1280x800 and even the installer continued using that wrong resolution. I suppose these problematic cases are specific to the qxl backend. Considering only qxl is the really problematic case we should also restrict the use of a custom OVMF vars file (or any other workarounds) to that scenario in any case. I suppose dealing with GRUB booting at 1024x768 (as we see it in other graphics backends) by editing some needles is acceptable (we don't need to force GRUB to 800x600).

Just copying from #116704:

I created this product bug report: https://bugzilla.opensuse.org/show_bug.cgi?id=1204067

I ran a test with the downgraded and with the current package:

current: https://openqa.opensuse.org/tests/2778306#step/bootloader_start/2
downgraded: https://openqa.opensuse.org/tests/2778580#step/bootloader_start/2
I tested with the branch that removes all old grub2-tw needles:
NEEDLES_DIR=https://github.com/perlpunk/os-autoinst-needles-opensuse.git#remove-bootloader-needles

My newly created needles are matching 100% in both cases.

PR: https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/784

I downgraded qemu-ovmf-x86_64 on all workers again.

https://gitlab.com/kraxel/virt-firmware has some support for modifying EFI variable store files. With that it's possible to see the change in the file itself:

name=PlatformConfig guid=7235c51c-0c80-4cab-87ac-3b084a6304b1 size=8
  qword: 0x0000025800000320

The format is defined here: https://github.com/tianocore/edk2/blob/c05a218a9758225ddf94eedb365633f2154551da/OvmfPkg/PlatformDxe/PlatformConfig.h#L21
A uint32_t of the horizontal resolution (800 -> 0x320) followed by an uint32_t of the vertical resolution (600 -> 0x258).

With some modification to also support writing PlatformConfig alongside bootorder and certificates, it should be possible to set the preferred resolution in var images directly.

tinita wrote:

I downgraded qemu-ovmf-x86_64 on all workers again.

Looks like ow1 had the new one still, I downgraded it.

That's weird, I did

for i in openqaworker1 openqaworker4 openqaworker7 openqaworker19 rebel; do echo $i; ssh root@$i "zypper in --oldpackage http://download.opensuse.org/update/leap/15.3/sle/noarch/qemu-ovmf-x86_64-202008-150300.10.14.1.noarch.rpm; zypper al -m 'poo#116914' qemu-ovmf-x86_64; zypper locks"; done

Maybe someone upgraded it again?

I also now downgraded all osd workers (that didn't already have the downgrade).

https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/784 merged

tinita wrote:

That's weird, I did

for i in openqaworker1 openqaworker4 openqaworker7 openqaworker19 rebel; do echo $i; ssh root@$i "zypper in --oldpackage http://download.opensuse.org/update/leap/15.3/sle/noarch/qemu-ovmf-x86_64-202008-150300.10.14.1.noarch.rpm; zypper al -m 'poo#116914' qemu-ovmf-x86_64; zypper locks"; done

Maybe someone upgraded it again?

And today I also had to downgrade it on ow4 and ow7. /var/log/zypp/history does not show your downgrade attempt, which is strange.

I did few tests on aarch64 o3 workers and found that:

• qemu 6.2.0 + qemu-uefi-aarch64 2022.02: is broken with distorted image, visible on fonts of grub (known already)
• qemu 6.2.0 + qemu-uefi-aarch64 2022.05 (from virt repo): works, but grub is displayed with a higher resolution
• qemu 7.1.0 (from TW) + qemu-uefi-aarch64 2022.02: is broken with distorted image, visible on fonts of grub
• qemu 7.1.0 (from TW) + qemu-uefi-aarch64 2022.05 (from virt repo): works, but grub is displayed with a higher resolution

So, I updated qemu-uefi-aarch64 and qemu-uefi-aarch32 to version 2022.05 from virt repo for all aarch64 o3 workers. (It should also be safe for boo#1197458)

Concrete steps:

• Pick an o3 worker
• remove the zypper lock
• Install the package
• Check if rpm -q --changelog qemuvmfx86_64 contains the change from comment #1
• check/re-run the test from https://bugzilla.opensuse.org/show_bug.cgi?id=1204067

cdywan wrote:

Concrete steps:

• Pick an o3 worker
• remove the zypper lock
• Install the package
• Check if rpm -q --changelog qemuvmfx86_64 contains the change from comment #1

I just checked on a fresh 15.4 container that the patch is there:

% cat /etc/os-release 
NAME="openSUSE Leap"
VERSION="15.4"
...
% zypper in qemu-ovmf-x86_64
...
% zypper info qemu-ovmf-x86_64
...
Information for package qemu-ovmf-x86_64:
-----------------------------------------
Repository     : Update repository with updates from SUSE Linux Enterprise 15
Name           : qemu-ovmf-x86_64
Version        : 202202-150400.5.5.1
Arch           : noarch
Vendor         : SUSE LLC <https://www.suse.com/>
Installed Size : 59.0 MiB
Installed      : Yes
Status         : up-to-date
Source package : ovmf-202202-150400.5.5.1.src
Upstream URL   : https://github.com/tianocore/edk2
Summary        : Open Virtual Machine Firmware - QEMU rom images (x86_64)
...
% rpm -ql qemu-ovmf-x86_64 --changelog | head
* Fri Oct 07 2022 jlee@suse.com
- Add patches to fix detection issue of NVME controller (bsc#1203825)
  - ovmf-MdeModulePkg-NvmExpressDxe-fix-check-for-Cap.Css.patch
  - ovmf-MdeModulePkg-NvmExpressPei-fix-check-for-NVM-command.

@favogt could you run the tests that were previously failing, on openqaworker19?
I upgraded there to ovmf-202202-150400.5.5.1.src which has several patches that might fix the issues.

I'm not sure how to run tests because all the linked tests are deleted by now. The only one I ran was https://openqa.opensuse.org/tests/3189827

@Oliver: I see that on both arm-1 and arm-3 there are still some locks:

pstivanin@openqa:~> sudo salt 'openqaworker-arm-1.suse.de' cmd.run 'zypper ll'
openqaworker-arm-1.suse.de:

    # | Name              | Type    | Repository | Comment
    --+-------------------+---------+------------+-----------
    1 | qemu-ovmf-x86_64  | package | (any)      | poo#116812
    2 | qemu-uefi-aarch64 | package | (any)  

pstivanin@openqa:~> sudo salt 'openqaworker-arm-3.suse.de' cmd.run 'zypper ll'
openqaworker-arm-3.suse.de:

    # | Name              | Type    | Repository | Comment
    --+-------------------+---------+------------+-----------
    1 | qemu-ovmf-x86_64  | package | (any)      | poo#116812
    2 | qemu-uefi-aarch64 | package | (any)      |

is this expected?

Thanks

pstivanin wrote:

@Oliver: I see that on both arm-1 and arm-3 there are still some locks:
[...]
is this expected?

Yes, see #114523 and #116812

favogt wrote in #note-49:

https://gitlab.com/kraxel/virt-firmware has some support for modifying EFI variable store files. With that it's possible to see the change in the file itself:

name=PlatformConfig guid=7235c51c-0c80-4cab-87ac-3b084a6304b1 size=8
  qword: 0x0000025800000320

The format is defined here: https://github.com/tianocore/edk2/blob/c05a218a9758225ddf94eedb365633f2154551da/OvmfPkg/PlatformDxe/PlatformConfig.h#L21
A uint32_t of the horizontal resolution (800 -> 0x320) followed by an uint32_t of the vertical resolution (600 -> 0x258).

With some modification to also support writing PlatformConfig alongside bootorder and certificates, it should be possible to set the preferred resolution in var images directly.

I packaged virt-firmware a while ago and it's possible to set this directly using

virt-fw-vars --set-json /dev/stdin -i vars.bin -o vars-800x600.bin <<EOF
{
    "version": 2,
    "variables": [
        {
            "name": "PlatformConfig",
            "guid": "7235c51c-0c80-4cab-87ac-3b084a6304b1",
            "attr": 7,
            "data": "2003000058020000"
        }
    ]
}
EOF