QA » openQA Project » openQA Tests

Seems like the screen resolution is distorted, as in the "video" parameter ineffective, e.g. see https://openqa.opensuse.org/tests/973405#step/boot_to_desktop/6 of "external_iso" vs. last good https://openqa.opensuse.org/tests/972393#step/boot_to_desktop/4

crosschecking with cirrus

openqa_clone_job_o3 --skip-chained-deps 973405 QEMUVGA=cirrus _GROUP=0 BUILD=test TEST=okurz_poo53729_cirrus

Created job #973806: opensuse-Tumbleweed-DVD-x86_64-Build20190702-external_iso@64bit -> https://openqa.opensuse.org/t973806, passed booting step.

Maybe we can patch the boot parameters by overriding video:

openqa_clone_job_o3 --skip-chained-deps 973405 _GROUP=0 BUILD=test TEST=okurz_poo53729_external_iso_boot_with_video_override EXTRABOOTPARAMS_BOOT_LOCAL="video=1024x768"

Created job #973816: opensuse-Tumbleweed-DVD-x86_64-Build20190702-external_iso@64bit -> https://openqa.opensuse.org/t973816

failed because there is a missing space: https://openqa.opensuse.org/tests/973816#step/boot_to_desktop/4

openqa_clone_job_o3 --skip-chained-deps 973405 _GROUP=0 BUILD=test TEST=okurz_poo53729_external_iso_boot_with_video_override EXTRABOOTPARAMS_BOOT_LOCAL=" video=1024x768"

Created job #973822: opensuse-Tumbleweed-DVD-x86_64-Build20190702-external_iso@64bit -> https://openqa.opensuse.org/t973822 fails as well, seems like later boot parameters do not overwrite first. I would need to carefully delete previous parameters. Seems to be a safer choice when we hardcode "QEMUVGA=cirrus" on upgrade jobs for now.

Maybe I can succeed to delete some characters:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 973405 _GROUP=0 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/boot_std_cirrus EXTRABOOTPARAMS_DELETE_CHARACTERS=20 EXTRABOOTPARAMS_DELETE_NEEDLE_TARGET=grub-prompt-extra_cmdline

Created job #973826: opensuse-Tumbleweed-DVD-x86_64-Buildtest-okurz_poo53729_external_iso_boot_with_video_override@64bit -> https://openqa.opensuse.org/t973826

failed because of wrong product-/needledir, trying again with proper clone from original:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 973405 _GROUP=0 TEST=external_iso@okurz/os-autoinst-distri-opensuse#fix/boot_std_cirrus BUILD=okurz/os-autoinst-distri-opensuse#7788 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/boot_std_cirrus PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse NEEDLES_DIR=/var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles EXTRABOOTPARAMS_BOOT_LOCAL=" video=1024x768" EXTRABOOTPARAMS_DELETE_CHARACTERS=20 EXTRABOOTPARAMS_DELETE_NEEDLE_TARGET=grub-prompt-extra_cmdline

Created job #974042: opensuse-Tumbleweed-DVD-x86_64-Build20190702-external_iso@64bit -> https://openqa.opensuse.org/t974042

Look into my most lazy way to fix the video boot parameters on old qcow images :) -> https://openqa.opensuse.org/tests/974150#step/boot_to_desktop/108

Actually it should not be too hard to just delete the -16 suffix in the right position:

$ openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 973405 _GROUP=0 TEST=external_iso@okurz/os-autoinst-distri-opensuse#fix/boot_std_cirrus BUILD=okurz/os-autoinst-distri-opensuse#7788 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/boot_std_cirrus PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse NEEDLES_DIR=/var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles EXTRABOOTPARAMS_DELETE_CHARACTERS=3 EXTRABOOTPARAMS_DELETE_NEEDLE_TARGET=grub-prompt-video_16bit

Created job #974425: opensuse-Tumbleweed-DVD-x86_64-Build20190702-external_iso@64bit -> https://openqa.opensuse.org/t974425

Created new needle grub-prompt-video_16bit-20190703 from https://openqa.opensuse.org/tests/974150#step/boot_to_desktop/90

and retriggered as https://openqa.opensuse.org/tests/974426

that did not retrigger the branch on "EXTRABOOTPARAMS_BOOT_LOCAL" at all so setting an empty value:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 973405 _GROUP=0 TEST=external_iso@okurz/os-autoinst-distri-opensuse#fix/boot_std_cirrus BUILD=okurz/os-autoinst-distri-opensuse#7788 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/boot_std_cirrus PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse NEEDLES_DIR=/var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles EXTRABOOTPARAMS_DELETE_CHARACTERS=3 EXTRABOOTPARAMS_BOOT_LOCAL=' ' EXTRABOOTPARAMS_DELETE_NEEDLE_TARGET=grub-prompt-video_16bit

https://openqa.opensuse.org/tests/974429#step/boot_to_desktop/110 shows where we successfully delete the "-16" suffix but I fear we need to delete the "vga=791" part as well. Maybe it's just easier to mount the qcow images and replace the boot parameters statically in a copied file?

On openqaworker4:

cp -a /var/lib/openqa/share/factory/hdd/opensuse-42.1-x86_64-GM-gnome@64bit.qcow2 /tmp/
modprobe nbd max_part=8
qemu-nbd --connect=/dev/nbd0 /tmp/opensuse-42.1-x86_64-GM-gnome@64bit.qcow2 
mkdir -p mnt
mount /dev/nbd0p2 /tmp/mnt
grep '1024x768-16' /tmp/mnt/boot/grub2/grub.cfg 
sed -i -e 's/\(video=1024x768\)-16/\1/' -e 's/ vga=791//' /tmp/mnt/boot/grub2/grub.cfg
sed -i -e 's/\(video=1024x768\)-16/\1/' -e 's/ vga=791//' /tmp/mnt/boot/grub2/grub.cfg /tmp/mnt/etc/default/grub
grep '1024x768' /tmp/mnt/boot/grub2/grub.cfg /tmp/mnt/etc/default/grub
umount /tmp/mnt
qemu-nbd --disconnect /dev/nbd0

and then copied over to o3, on o3:

cd /var/lib/openqa/share/factory/hdd/fixed
mv opensuse-42.1-x86_64-GM-gnome@64bit{,_cirrus}.qcow2
scp root@openqaworker4:/tmp/opensuse-42.1-x86_64-GM-gnome@64bit.qcow2 .

openqa_clone_job_o3 --skip-chained-deps 973405 TEST=okurz_poo_changed_image_for_non_cirrus_external_iso_poo53729 _GROUP=0 BUILD=test

Created job #974471: opensuse-Tumbleweed-DVD-x86_64-Build20190702-external_iso@64bit -> https://openqa.opensuse.org/t974471

failed with no visible change to the parameters, probably because the cache did not detect the qcow image as a new one.

so I hardlinked the qcow image to …64bit_std on o3 and try again:

openqa_clone_job_o3 --skip-chained-deps 973405 TEST=okurz_poo_changed_image_for_non_cirrus_external_iso_poo53729 _GROUP=0 BUILD=test HDD_1=opensuse-42.1-x86_64-GM-gnome@64bit_std.qcow2

Created job #974472: opensuse-Tumbleweed-DVD-x86_64-Build20190702-external_iso@64bit -> https://openqa.opensuse.org/t974472

https://openqa.opensuse.org/tests/974472/file/serial0.txt shows that we picked up the right image however it did not yet seem to have an effect. Or did I prematurely abort? Retriggered as https://openqa.opensuse.org/tests/974473 no, failed the same.

In https://openqa.opensuse.org/tests/974471/file/serial0.txt I can see

[    2.794179] [drm] Initialized drm 1.1.0 20060810
[    2.798376] parport_pc 00:04: reported by Plug and Play ACPI
[    2.799464] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
[    2.825202] fb: switching to bochsdrmfb from EFI VGA
[    2.826243] Console: switching to colour dummy device 80x25
[    2.860081] bochs-drm 0000:00:02.0: BAR 0: can't reserve [mem 0xfd000000-0xfdffffff pref]
[    2.860924] [drm:bochs_hw_init [bochs_drm]] *ERROR* Cannot request framebuffer
[    2.861657] Trying to free nonexistent resource <00000000fd000000-00000000fdffffff>
[    2.920318] ppdev: user-space parallel port driver
[    2.921419] bochs-drm: probe of 0000:00:02.0 failed with error -16

vs. from the last job on cirrus:

[    1.703436] fb: switching to cirrusdrmfb from EFI VGA
[    1.704390] Console: switching to colour dummy device 80x25
[    1.708032] [TTM] Zone  kernel: Available graphics memory: 766560 kiB
[    1.708781] [TTM] Initializing pool allocator
[    1.709286] [TTM] Initializing DMA pool allocator
[    1.710451] [drm] fb mappable at 0xFC000000
[    1.710941] [drm] vram aper at 0xFC000000
[    1.711408] [drm] size 33554432
[    1.711772] [drm] fb depth is 16
[    1.716010] [drm]    pitch is 2048
[    1.716479] fbcon: cirrusdrmfb (fb0) is primary device
[    1.717419] Console: switching to colour frame buffer device 128x48
[    1.735883] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device
[    1.740023] cirrus 0000:00:02.0: registered panic notifier
[    1.744159] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0

so the first does not look good. One more point for "keep migration jobs on cirrus" for now. However the one job I started with all boot parameters deleted except video=1024x768 also has qemu 3.1 from Leap 15.1 (https://openqa.opensuse.org/tests/974150/file/serial0.txt) whereas the failed ones are on qemu 2.11 so maybe only the combination of new qemu and tweaked video parameters is the fix?

So for now I did on o3:

ariel:/var/lib/openqa/share/factory/hdd/fixed # for i in $(ls *64bit.qcow2) ; do ln $i ${i%.qcow2}_cirrus.qcow2; done
ariel:/var/lib/openqa/share/factory/hdd/fixed # ls *64bit_cirrus.qcow2
opensuse-15.0-x86_64-GM-gnome@64bit_cirrus.qcow2
opensuse-15.0-x86_64-GM-kde@64bit_cirrus.qcow2
opensuse-42.1-x86_64-GM-gnome@64bit_cirrus.qcow2
opensuse-42.1-x86_64-Updates-20170213-1-gnome@64bit_cirrus.qcow2
opensuse-42.1-x86_64-Updates-20170213-1-kde@64bit_cirrus.qcow2
opensuse-42.2-x86_64-GM-gnome@64bit_cirrus.qcow2
opensuse-42.2-x86_64-GM-kde@64bit_cirrus.qcow2
opensuse-42.2-x86_64-Updates-20180313-2-gnome@64bit_cirrus.qcow2
opensuse-42.3-x86_64-GM-gnome@64bit_cirrus.qcow2
opensuse-42.3-x86_64-GM-kde@64bit_cirrus.qcow2

The problematic test scenarios in the current build seem to be:

• opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.3-kde@64bit using opensuse-42.3-x86_64-GM-kde@64bit.qcow2
• opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.3-gnome@64bit using opensuse-42.3-x86_64-GM-gnome@64bit.qcow2
• opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.2-kde@64bit using opensuse-42.2-x86_64-GM-kde@64bit.qcow2
• https://openqa.opensuse.org/tests/973325
• https://openqa.opensuse.org/tests/973612
• https://openqa.opensuse.org/tests/973324
• https://openqa.opensuse.org/tests/973314
• https://openqa.opensuse.org/tests/973614
• https://openqa.opensuse.org/tests/973356
• https://openqa.opensuse.org/tests/973525
• https://openqa.opensuse.org/tests/973513
• https://openqa.opensuse.org/tests/973502
• https://openqa.opensuse.org/tests/973608
• https://openqa.opensuse.org/tests/973427
• https://openqa.opensuse.org/tests/973428
• https://openqa.opensuse.org/tests/973405

for each I can try to clone with "QEMUVGA=cirrus" hardcoded for now, then schedule the "…@64bit_cirrus" variant instead, move the "…@64bit" variant to development and see what can be done about the qcow images:

for i in 973340 973334 973609 973325 973612 973324 973314 973614 973356 973525 973513 973502 973608 973427 973428 973405; do openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org $i QEMUVGA=cirrus; done

Created job #974524: opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.3-kde@64bit -> https://openqa.opensuse.org/t974524
Created job #974525: opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.3-gnome@64bit -> https://openqa.opensuse.org/t974525
Created job #974526: opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.2-kde@64bit -> https://openqa.opensuse.org/t974526
Created job #974527: opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.2-gnome@64bit -> https://openqa.opensuse.org/t974527
Created job #974528: opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.1-kde@64bit -> https://openqa.opensuse.org/t974528
Created job #974529: opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-Leap-42.1-gnome@64bit -> https://openqa.opensuse.org/t974529
Created job #974530: opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-13.2@64bit -> https://openqa.opensuse.org/t974530
Created job #974531: opensuse-Tumbleweed-NET-x86_64-Build20190702-zdup-13.1-gnome@64bit -> https://openqa.opensuse.org/t974531
Created job #974532: opensuse-Tumbleweed-DVD-x86_64-Build20190702-zdup-Leap-42.3-kde@64bit -> https://openqa.opensuse.org/t974532
Created job #974533: opensuse-Tumbleweed-DVD-x86_64-Build20190702-zdup-Leap-42.3-gnome@64bit -> https://openqa.opensuse.org/t974533
Created job #974534: opensuse-Tumbleweed-DVD-x86_64-Build20190702-zdup-Leap-42.2-gnome@64bit -> https://openqa.opensuse.org/t974534
Created job #974535: opensuse-Tumbleweed-DVD-x86_64-Build20190702-zdup-Leap-42.1-gnome@64bit -> https://openqa.opensuse.org/t974535
Created job #974536: opensuse-Tumbleweed-DVD-x86_64-Build20190702-zdup-Leap-15.0-gnome@64bit -> https://openqa.opensuse.org/t974536
Created job #974537: opensuse-Tumbleweed-DVD-x86_64-Build20190702-gnome+import_ssh_keys@64bit -> https://openqa.opensuse.org/t974537
Created job #974538: opensuse-Tumbleweed-DVD-x86_64-Build20190702-gnome+do_not_import_ssh_keys@64bit -> https://openqa.opensuse.org/t974538
Created job #974539: opensuse-Tumbleweed-DVD-x86_64-Build20190702-external_iso@64bit -> https://openqa.opensuse.org/t974539

Moved all problematic scenarios from job group Tumbleweed to Development Tumbleweed and replaced by "…@64bit_cirrus" variants instead.

Out of the above two have failed:

• https://openqa.opensuse.org/tests/974526 in keymap_or_locale and force_scheduled_tasks but that happened in before in this scenario already, reported and handled as #53828
• https://openqa.opensuse.org/tests/974530#step/user_gui_login/6 handled as #53831

The situation could be easier by now because I updated all o3 workers to openSUSE Leap 15.1 in the meantime with a more recent qemu version so maybe it's enough to simply adjust the video= boot parameter on the go with EXTRABOOTPARAMS_DELETE_CHARACTERS=3 EXTRABOOTPARAMS_BOOT_LOCAL=' ' EXTRABOOTPARAMS_DELETE_NEEDLE_TARGET=grub-prompt-video_16bit

Dominik to take a look eventually when feature tickets and feedback to bugs is done, which is already quite some work :)

So the EXTRABOOTPARAMS_DELETE_CHARACTERS=3 EXTRABOOTPARAMS_BOOT_LOCAL=' ' EXTRABOOTPARAMS_DELETE_NEEDLE_TARGET=grub-prompt-video_16bit line from okurz doesn't seem to help while QEMUVGA=cirrus does.

https://openqa.opensuse.org/tests/1219149#step/setup_zdup/110
https://openqa.opensuse.org/tests/1219122# <-- with cirrus

Added QEMUVGA=cirrus for the following testsuites:

• zdup-Leap-15.0-gnome
• zdup-Leap-42.1-gnome
• zdup-Leap-42.2-gnome
• zdup-Leap-42.3-gnome
• zdup-Leap-42.3-kde

Added QEMUVGA=cirrus for the following testsuites:

• zdup-Leap-42.1-kde
• zdup-Leap-42.2-kde
• zdup-13.2
• zdup-13.1-gnome
• zdup-13.1-kde

Defining QEMUVGA via YAML schedule as discussed in #65265 to prevent breaking aarch64 tests.

@@ -159,6 +159,8 @@
- suse_patches-systemd_testsuite
- zdup-Leap-15.0-gnome:
machine: 64bit

• settings:
• QEMUVGA: cirrus
  • desktopapps-other-wayland: machine: 64bit_virtio
  • desktopapps-documentation-wayland:

dheidler wrote:

So the EXTRABOOTPARAMS_DELETE_CHARACTERS=3 EXTRABOOTPARAMS_BOOT_LOCAL=' ' EXTRABOOTPARAMS_DELETE_NEEDLE_TARGET=grub-prompt-video_16bit line from okurz doesn't seem to help while QEMUVGA=cirrus does.

https://openqa.opensuse.org/tests/1219149#step/setup_zdup/110
https://openqa.opensuse.org/tests/1219122# <-- with cirrus

Just keep in mind that the desired end state should be to not run any tests on cirrus anymore.

okurz wrote:

Just keep in mind that the desired end state should be to not run any tests on cirrus anymore.

I don't see this happening. I tried your EXTRABOOTPARAMS but they don't seem to work and I don't think it is worth the time to figure out why.
Eventually all cirrus based images will be so old that we might not use them anymore so cirrus based tests can be faded out.

dheidler wrote:

[…] Eventually all cirrus based images will be so old that we might not use them anymore so cirrus based tests can be faded out.

Yes, I hope this is an approach that will work :)

For some reason using SETTINGS in yamkl schedule had no effect.

Maybe this will help:

@@ -159,8 +159,17 @@
     - suse_patches-systemd_testsuite
     - zdup-Leap-15.0-gnome:
         machine: 64bit
+        description: ''
         settings:
           QEMUVGA: cirrus
+          BOOTFROM: cdrom
+          DESKTOP: gnome
+          HDD_1: '%DISTRI%-15.0-%ARCH%-GM-gnome@%MACHINE%.qcow2'
+          MAX_JOB_TIME: '14400'
+          UEFI_PFLASH_VARS: '%DISTRI%-15.0-%ARCH%-GM-gnome@%MACHINE%-uefi-vars.qcow2'
+          WORKER_CLASS: heavyload
+          ZDUP: '1'
+        testsuite: null
     - desktopapps-other-wayland:
         machine: 64bit_virtio
     - desktopapps-documentation-wayland:

# TW DEVEL jobgroup
@@ -209,19 +209,6 @@
     - zdup-Leap-42.3-gnome:
         machine: 64bit
     - suse_patches-systemd_testsuite
-    - zdup-Leap-15.0-gnome:
-        machine: 64bit
-        description: ''
-        settings:
-          QEMUVGA: cirrus
-          BOOTFROM: cdrom
-          DESKTOP: gnome
-          HDD_1: '%DISTRI%-15.0-%ARCH%-GM-gnome@%MACHINE%.qcow2'
-          MAX_JOB_TIME: '14400'
-          UEFI_PFLASH_VARS: '%DISTRI%-15.0-%ARCH%-GM-gnome@%MACHINE%-uefi-vars.qcow2'
-          WORKER_CLASS: heavyload
-          ZDUP: '1'
-        testsuite: null
     - desktopapps-other-wayland:
         machine: 64bit_virtio
     - desktopapps-documentation-wayland:

... as someone already scheduled that test for TW jobgroup: https://openqa.opensuse.org/tests/1272508

Dimstar is fine with dropping.

dheidler wrote:

Maybe dimstar deleted that job - anyway it didn't get scheduled for the current TW so I guess we are done here.

Nope, wasn't me.

And checkin the Development/Tumbleweed group, the definition is still there:

opensuse-Tumbleweed-DVD-x86_64:
- zdup-Leap-42.1-gnome:
    machine: 64bit

And from what I can see https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20200721&groupid=38 lists the job as well as scheduled/running