action #114523
closedDeal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64 size:M
0%
Description
Observation¶
Same as on #111992 but now on aarch64 OSD workers, crosscheck o3 worker aarch64
Steps to reproduce¶
Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label, call openqa-query-for-job-label poo#114523
Suggestions¶
- DONE Ensure to have workaround on all aarch64 OSD workers for #111992, e.g.
sudo zypper -n in --oldpackage http://download.opensuse.org/ports/aarch64/distribution/leap/15.3/repo/oss/noarch/qemu-uefi-aarch64-202008-10.8.1.noarch.rpm
- Now try again to remove zypper lock and trigger according openQA tests to crosscheck both on osd+o3
Rollback steps¶
- Remove zypper lock for qemu-uefi-aarch64 on all workers
Updated by okurz over 2 years ago
- Copied from action #111992: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl size:M added
Updated by okurz over 2 years ago
- Subject changed from Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64 to Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64, auto_review:"(?s)aarch64.*uefi.*Stall detected.*no candidate needle.*bootloader-shim-import-prompt.*grub2.*inst-bootmenu"
Updated by okurz over 2 years ago
- Subject changed from Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64, auto_review:"(?s)aarch64.*uefi.*Stall detected.*no candidate needle.*bootloader-shim-import-prompt.*grub2.*inst-bootmenu" to Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64, auto_review:"(?s)aarch64.*uefi.*Stall detected.*no candidate needle.*bootloader-shim-import-prompt.*grub2.*inst-bootmenu":retry
Deployed old packages onto
sudo salt --no-color --state-output=changes -C 'G@roles:worker and G@osarch:aarch64' cmd.run 'sudo zypper -n in --oldpackage http://download.opensuse.org/ports/aarch64/distribution/leap/15.3/repo/oss/noarch/qemu-uefi-aarch64-202008-10.8.1.noarch.rpm'
I triggered a test https://openqa.suse.de/tests/9196630# and realized that people went crazy creating new needles already for the wrong resolution. I pointed that out to mgrifalconi in https://suse.slack.com/archives/C02CANHLANP/p1658478957406679?thread_ts=1658478131.778769&cid=C02CANHLANP
Now after applying that workaround with the old package install that seems to work
openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/9192234 _GROUP=0 BUILD= TEST=okurz_poo_111992_workaround_downgraded_qemu-uefi-aarch64 SCHEDULE=tests/installation/bootloader_start WORKER_CLASS=openqaworker-arm-2
->
Created job #9196670: sle-15-SP3-Server-DVD-Updates-aarch64-Build20220721-1-qam-gnome@aarch64-virtio -> https://openqa.suse.de/t9196670
Updated by okurz over 2 years ago
- Subject changed from Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64, auto_review:"(?s)aarch64.*uefi.*Stall detected.*no candidate needle.*bootloader-shim-import-prompt.*grub2.*inst-bootmenu":retry to Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64, auto_review:"(?s)aarch64.*uefi.*no candidate needle.*grub":retry
And called
for i in openqa.opensuse.org openqa.suse.de; do export host=$i; bash -ex ./openqa-monitor-investigation-candidates | bash -e ./openqa-label-known-issues; done
Updated by okurz over 2 years ago
- Related to action #113794: Use prepared OVMF image with expected settings size:M added
Updated by okurz over 2 years ago
- Related to action #114493: [qe-core][aarch64][installation]test fails in bootloader_start, needle mismatch on installer boot memu added
Updated by okurz over 2 years ago
- Related to action #114550: [qe-core] Ignored warnings about too many needles and detected stalls, in particular when checking grub2 added
Updated by okurz over 2 years ago
Richard Fan in https://suse.slack.com/archives/C02CANHLANP/p1658892342572309 brought up that more tests are again affected. I might have missed the lock or a machine wasn't reachable during the operation? I executed sudo salt --no-color --state-output=changes -C 'G@roles:worker and G@osarch:aarch64' cmd.run 'zypper -n in --oldpackage http://download.opensuse.org/ports/aarch64/distribution/leap/15.3/repo/oss/noarch/qemu-uefi-aarch64-202008-10.8.1.noarch.rpm && zypper al qemu-uefi-aarch64' but openqaworker-arm-3 just crashed again right now. It might come back online again with the wrong version.
Updated by okurz over 2 years ago
- Related to action #114769: Have jobs fail if screen checks take too long, e.g. if there are "two many needles" after warning about it added
Updated by okurz over 2 years ago
openqaworker-arm-3 was now reachable so I could rollback the package qemu-uefi-aarch64. I have applied workarounds to all OSD machines regarding OVMF usage as far as they are available. In case you still encounter failed jobs from the passed with the "wrong resolution" for the initial bootup screen retrigger them. And also I recommend to delete any needles you might have created recently in the wrong resolution to prevent the bootloader screen to become inefficient due to the amount of needles so that eventually jobs would miss the bootloader screen and fail with annoying random failures.
Updated by okurz over 2 years ago
- Status changed from Feedback to Blocked
- Priority changed from Urgent to High
Workaround should still be effective on all, waiting for #113794
Updated by openqa_review over 2 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: yast2_gui
https://openqa.opensuse.org/tests/2526840#step/yast2_control_center/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by tinita about 2 years ago
See #113794 for current progress and what to do
Updated by mkittler about 2 years ago
The regex in the ticket title isn't specific enough, see note on #116614.
Updated by tinita about 2 years ago
- Subject changed from Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64, auto_review:"(?s)aarch64.*uefi.*no candidate needle.*grub":retry to Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64, auto_review:"(?s)aarch64.*uefi.*no candidate needle[^\n]*grub":retry
I changed the title and assume that no candidate needle[^\n]*grub should be matched on the same line.
Worked for me as expected on an example logfile.
#116614
Updated by jlausuch about 2 years ago
We still have some issues with aarch64 jobs:
https://openqa.suse.de/tests/9537583
https://openqa.suse.de/tests/9538772
https://openqa.suse.de/tests/9537585
https://openqa.suse.de/tests/9537545
https://openqa.suse.de/tests/9537544
workers:
openqaworker-arm-1
openqaworker-arm-2
openqaworker-arm-3
Updated by tinita about 2 years ago
@jlausuch can you help me get the connection with the resolution issue? I see that in your linked jobs needles don't match, but the screenshots are totally different, so it doesn't seem to be a resolution issue. Or am I missing something?
Updated by openqa_review about 2 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: xfstests_xfs-dangrous-tests
https://openqa.suse.de/tests/9679280#step/run/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 32 days if nothing changes in this ticket.
Updated by livdywan about 2 years ago
To clarify:
Blocked by https://bugzilla.opensuse.org/show_bug.cgi?id=1204067
Updated by openqa_review about 2 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: install_only@aarch64-uefi_http_boot
https://openqa.opensuse.org/tests/2885807#step/bootloader_uefi/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 48 days if nothing changes in this ticket.
Updated by openqa_review almost 2 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test:
https://openqa.suse.de/tests/10357436#step/boot_without_secureboot/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by tinita almost 2 years ago
The regex in the ticket title exceeds PCRE's backtrack limit, I see on osd:
grep failed: cmd=>grep -qPzo '(?s)aarch64.*uefi.*no candidate needle[^\n]*grub' '/tmp/tmp.HKYSbhPnjQ'< output='grep: exceeded PCRE's backtracking limit'
Updated by favogt almost 2 years ago
- Subject changed from Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64, auto_review:"(?s)aarch64.*uefi.*no candidate needle[^\n]*grub":retry to Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64
Let's just drop the autoreview regex, there are too many false positives anyway
Updated by openqa_review almost 2 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: online_sles15sp2_ltss_pscc_basesys-srv-desk-dev-contm-lgm-py2-tsm-wsm-pcm_all_full
https://openqa.suse.de/tests/10437121#step/check_os_release/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by openqa_review over 1 year ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: jeos-containers-podman
https://openqa.suse.de/tests/10653531#step/bootloader_uefi/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 56 days if nothing changes in this ticket.
Updated by okurz over 1 year ago
- Tags changed from reactive work to reactive work, infra
- Status changed from Blocked to New
- Assignee deleted (
okurz)
#111992 was resolved, the updated packages can be tried on aarch64 for now as well.
Updated by okurz over 1 year ago
- Subject changed from Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64 to Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl, but on aarch64 size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by tjyrinki_suse over 1 year ago
- Blocks action #115919: [security] test fails in tpm2_measured_boot added
Updated by openqa_review over 1 year ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: jeos-filesystem
https://openqa.suse.de/tests/11050552#step/bootloader_uefi/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by okurz 2 months ago ยท Edited
- Category set to Regressions/Crashes
- Status changed from Workable to In Progress
- Assignee set to okurz
- Target version changed from future to Ready
this was brought up in https://suse.slack.com/archives/C02CANHLANP/p1726731665437149 by Jose Lausuch. Apparently we don't have the downgraded package on newer arm workers. I will just remove the package lock and upgrade the package on openqaworker-arm-1.
Did sudo salt --no-color -C 'G@roles:worker and G@osarch:aarch64' cmd.run 'zypper rl qemu-uefi-aarch64'
openqa-query-for-job-label poo#114523 does not return any matches anymore.
Updated by okurz 2 months ago
- Status changed from In Progress to Resolved
No more problematic jobs encountered. I also followed the Slack conversation and no more problems were mentioned there as well. https://bugzilla.opensuse.org/show_bug.cgi?id=1204067 was already VERIFIED FIXED in before.