openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842022-08-29T08:44:02ZopenSUSE Project Management Tool
Redmine openQA Infrastructure - action #115925 (New): aarch64: Random QEMU failures while retrieving host...https://progress.opensuse.org/issues/1159252022-08-29T08:44:02ZMDouchamartin.doucha@suse.com
<p>Since the worker upgrade to Leap 15.4, some aarch64 jobs have randomly failed with the following error: <code>qemu-system-aarch64: Failed to retrieve host CPU features</code><br>
Example: <a href="https://openqa.suse.de/tests/9401654" class="external">https://openqa.suse.de/tests/9401654</a></p>
openQA Project - action #114643 (New): Add support for virtio keyboard and mouse on aarch64 QEMUhttps://progress.opensuse.org/issues/1146432022-07-25T12:33:10ZMDouchamartin.doucha@suse.com
<p>QEMU aarch64 VMs are currently hardcoded to use USB keyboard in OpenQA. We now need to test SLE-15SP4 kernel-azure where this does not work because the whole USB subsystem is intentionally disabled and therefore the framebuffer console gets no keyboard input:<br>
<a href="https://openqa.suse.de/tests/9122772#step/update_kernel/95" class="external">https://openqa.suse.de/tests/9122772#step/update_kernel/95</a></p>
<p>I can get the tests to work by setting <code>QEMU_APPEND=device virtio-keyboard -device virtio-mouse</code>. Please implement proper support for virtio input devices in the QEMU backend.</p>
openQA Project - action #112337 (Workable): [ui/ux][easy] OpenQA admin UI: Link to last match of ...https://progress.opensuse.org/issues/1123372022-06-13T13:03:14ZMDouchamartin.doucha@suse.com
<p>[ui/ux][easy] OpenQA admin UI: Link to last match of a needle points to invalid URL size:M</p>
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Some "Last Match" links in <a href="https://openqa.suse.de/admin/needles" class="external">https://openqa.suse.de/admin/needles</a> (if the needle had a recent match) point to invalid URL: <a href="https://openqa.suse.de/admin/undefined" class="external">https://openqa.suse.de/admin/undefined</a></p>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<p>For example of the issue, on <a href="https://openqa.suse.de/admin/needles" class="external">https://openqa.suse.de/admin/needles</a> enter <code>license-insert-disc</code> into the search input box.</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>entrance level issue</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Link is fixed</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Extend tests to ensure we have that covered</li>
</ul>
openQA Infrastructure - action #108266 (New): grenache: script_run() commands randomly time out s...https://progress.opensuse.org/issues/1082662022-03-14T09:36:30ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Since the NBG server room was moved, I'm seeing a lot of random script_run() command timeouts on grenache. I suspect network issues.<br>
<a href="https://openqa.suse.de/tests/8320677#step/sighold02/12">https://openqa.suse.de/tests/8320677#step/sighold02/12</a><br>
<a href="https://openqa.suse.de/tests/8294410#step/fallocate06/8">https://openqa.suse.de/tests/8294410#step/fallocate06/8</a><br>
<a href="https://openqa.suse.de/tests/8294334#step/boot_ltp/42">https://openqa.suse.de/tests/8294334#step/boot_ltp/42</a></p>
<pre><code> Test died: command 'vmstat -w' timed out at /usr/lib/os-autoinst/testapi.pm line 1039.
# Test died: Timed out waiting for LTP test case which may still be running or the OS may have crashed! at sle/tests/kernel/run_ltp.pm line 337.
# Test died: command 'rpm -qi kernel-default > /tmp/kernel-pkg.txt 2>&1' timed out at /usr/lib/os-autoinst/testapi.pm line 1039.
main::init_backend() called at /usr/bin/isotovideo line 258
[2022-03-09T16:12:24.052826+01:00] [info] ::: consoles::serial_screen::read_until: Matched output from SUT in 1 loops & 0.00229895696975291 seconds: Use of uninitialized value $regexp in concatenation (.) or string at /usr/lib/os-autoinst/testapi.pm line 927.
testapi::wait_serial(undef, undef, 0, "no_regex", 1) called at sle/tests/kernel/run_ltp.pm line 317
run_ltp::run(run_ltp=HASH(0x1001999aee8), LTP::TestInfo=HASH(0x1001b24d630)) called at /usr/lib/os-autoinst/basetest.pm line 356
cf. last good
[2022-03-12T07:06:13.797172+01:00] [info] ::: consoles::serial_screen::read_until: Matched output from SUT in 1 loops & 0.00224426796194166 seconds:
Use of uninitialized value $regexp in concatenation (.) or string at /usr/lib/os-autoinst/testapi.pm line 927.
testapi::wait_serial(undef, undef, 0, "no_regex", 1) called at sle/tests/kernel/run_ltp.pm line 317
run_ltp::run(run_ltp=HASH(0x1003570fb08), LTP::TestInfo=HASH(0x1003547afa8)) called at /usr/lib/os-autoinst/basetest.pm line 356
eval {...} called at /usr/lib/os-autoinst/basetest.pm line 354
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Users no longer file complaints about script_run timing out</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Find a reproducer or database query to identify recent cases e.g. ask Martin. EDIT: mdoucha responded that there is no special query available. Next suggestion: Just pick any recent job where the problem happened, trigger 1k jobs for investigation, e.g. according priority or over weekend, etc.</li>
<li>Look into warnings in logs</li>
<li>"Use of uninitialized value $regexp in concatenation (.) or string" is already fixed</li>
<li>last good: <a href="https://openqa.suse.de/tests/8315985">https://openqa.suse.de/tests/8315985</a></li>
<li>[debug] Current version is 4.6.1647014989.7540333c [interface v25]
<ul>
<li>Do <code>git log --no-merges 7540333c..$first_bad</code></li>
</ul></li>
<li><p>Investigate the timeout handling c.f. recent improvements to VNC connection code and handling former blocking code paths</p>
<ul>
<li>We don't have a screenshot to compare the serial output to</li>
<li>Maybe we can check the serial logs for comparison?</li>
</ul></li>
</ul>
<p>All these occurences are on the same machine, which is s390x-kvm-sle12</p>
<p>One problem I see is that in <a href="https://openqa.suse.de/tests/8505116#step/shutdown_ltp/6">https://openqa.suse.de/tests/8505116#step/shutdown_ltp/6</a> we have a serial terminal. If there would be VNC we would be able to see if the command was executed or not. I also don't see the commands in <a href="https://openqa.suse.de/tests/8505116/logfile?filename=serial_terminal.txt">https://openqa.suse.de/tests/8505116/logfile?filename=serial_terminal.txt</a> nor serial0.txt .</p>
<p>We should try to resolve the ambiguity if commands just never write to the serial terminal as they time out or if actual data is going missing from SUT to worker.</p>
<p>What would you say, what is the best way to reproduce the issue? If we have a reproducer we can try to make it as small as possible and then fix it, maybe just increase the timeout. Maybe ensure that we cath any console related processes in the background if they are still responsive.</p>
<a name="Further-suggestions-from-SUSE-QE-Tools-unblock-2022-05-11"></a>
<h3 >Further suggestions from SUSE QE Tools unblock 2022-05-11<a href="#Further-suggestions-from-SUSE-QE-Tools-unblock-2022-05-11" class="wiki-anchor">¶</a></h3>
<ul>
<li>As suggested in <a class="issue tracker-4 status-1 priority-4 priority-default child" title="action: grenache: script_run() commands randomly time out since server room move (New)" href="https://progress.opensuse.org/issues/108266#note-22">#108266#note-22</a>, similar as we do for openQA worker hosts there should be monitoring to critical components (out of scope for SUSE QE Tools, delegate to SUSE QE Core)</li>
<li>within the code called by script_run using ssh
<ul>
<li>retry</li>
<li>check if the ssh connection is still there at all</li>
<li>provide more details when failing</li>
</ul></li>
<li>Add in the message on timeout how long we waited</li>
</ul>
openQA Infrastructure - action #105867 (Resolved): OpenQA bot schedules jobs with incomplete INCI...https://progress.opensuse.org/issues/1058672022-02-03T10:23:54ZMDouchamartin.doucha@suse.com
<p>This week, the OpenQA bot has been scheduling kernel tests without adding the Basesystem/LTSS repository to INCIDENT_REPO. Only the livepatching repository was added. This happened on <a href="https://openqa.suse.de/tests/8085238#settings" class="external">SLE-12SP4</a>, <a href="https://openqa.suse.de/tests/8082278#settings" class="external">SLE-15SP2</a> (<a href="https://openqa.suse.de/tests/8081179#settings" class="external">twice</a>) and <a href="https://openqa.suse.de/tests/8087134#settings" class="external">SLE-15SP1</a>:</p>
<pre><code>INCIDENT_REPO=http://download.suse.de/ibs/SUSE:/Maintenance:/22660/SUSE_Updates_SLE-Module-Live-Patching_15-SP1_x86_64
</code></pre>
<p>Some of these tests have already been rescheduled with the correct settings but SLE-15SP1 is still affected. Current S:M:22660 incident data in QEM dashboard API:</p>
<pre><code>{"approved":false,"channels":["SUSE:SLE-15-SP1:Update","SUSE:Updates:SLE-Product-HA:15-SP1:x86_64","SUSE:Updates:SLE-Product-HA:15-SP1:s390x","SUSE:Updates:SLE-Product-HA:15-SP1:ppc64le","SUSE:Updates:SLE-Product-HA:15-SP1:aarch64","SUSE:Updates:Storage:6:aarch64","SUSE:Updates:Storage:6:x86_64","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:x86_64","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:s390x","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:ppc64le","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:aarch64","SUSE:Updates:SLE-Module-Live-Patching:15-SP1:x86_64","SUSE:Updates:SLE-Module-Live-Patching:15-SP1:ppc64le","SUSE:Updates:SUSE-CAASP:4.0:x86_64","SUSE:Updates:SLE-Product-SLES:15-SP1-BCL:x86_64","SUSE:Updates:SLE-Product-HPC:15-SP1-ESPOS:aarch64","SUSE:Updates:SLE-Product-HPC:15-SP1-ESPOS:x86_64","SUSE:Updates:SLE-Product-SLES_SAP:15-SP1:ppc64le","SUSE:Updates:SLE-Product-SLES_SAP:15-SP1:x86_64","SUSE:Updates:SLE-Product-SLES:15-SP1-LTSS:x86_64","SUSE:Updates:SLE-Product-SLES:15-SP1-LTSS:s390x","SUSE:Updates:SLE-Product-SLES:15-SP1-LTSS:ppc64le","SUSE:Updates:SLE-Product-SLES:15-SP1-LTSS:aarch64","SUSE:Updates:SLE-Product-HPC:15-SP1-LTSS:x86_64","SUSE:Updates:SLE-Product-HPC:15-SP1-LTSS:aarch64","SUSE:Updates:openSUSE-SLE:15.3","SUSE:Updates:openSUSE-SLE:15.4","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:aarch64","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:ppc64le","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:s390x","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:x86_64"],"emu":false,"inReview":false,"inReviewQAM":false,"isActive":true,"number":22660,"packages":["dtb-aarch64","kernel-debug","kernel-default","kernel-docs","kernel-kvmsmall","kernel-livepatch-SLE15-SP1_Update_28","kernel-obs-build","kernel-obs-qa","kernel-source","kernel-syms","kernel-vanilla","kernel-zfcpdump"],"project":"SUSE:Maintenance:22660","rr_number":null}
</code></pre> openQA Project - action #96507 (New): Job terminated prematurely during needle check auto_review:...https://progress.opensuse.org/issues/965072021-08-03T11:18:44ZMDouchamartin.doucha@suse.com
<p>Two LTP jobs have failed recently in a similar way while waiting for needle match.</p>
<p><a href="https://openqa.suse.de/tests/6638157" class="external">https://openqa.suse.de/tests/6638157</a><br>
<a href="https://openqa.suse.de/tests/6625293" class="external">https://openqa.suse.de/tests/6625293</a></p>
<p>The first job has some interesting output in <a href="https://openqa.suse.de/tests/6638157/logfile?filename=autoinst-log.txt" class="external">autoinst-log.txt</a>:</p>
<pre><code>[2021-08-03T07:24:58.610 CEST] [debug] no change: 0.5s
[2021-08-03T07:25:03.632 CEST] [debug] WARNING: check_asserted_screen took 4.02 seconds for 35 candidate needles - make your needles more specific
[2021-08-03T07:25:03.632 CEST] [debug] no match: -0.5s, best candidate: linux-login-20181005 (0.29)
*** Error in `/usr/bin/isotovideo: backen': free(): invalid pointer: 0x00005560f6795e00 ***
[2021-08-03T07:25:05.336 CEST] [debug] backend process exited: 0
[2021-08-03T07:25:05.336 CEST] [debug] stopping command server 30808 because test execution ended
</code></pre>
<p>The other job does not show any obvious error in <a href="https://openqa.suse.de/tests/6625293/logfile?filename=autoinst-log.txt" class="external">the log</a>:</p>
<pre><code>[2021-07-31T07:21:58.852 CEST] [debug] no change: 1785.4s
[2021-07-31T07:22:01.325 CEST] [debug] backend process exited: 0
[2021-07-31T07:22:01.326 CEST] [debug] stopping command server 21557 because test execution ended
</code></pre>
<a name="Suggestion"></a>
<h2 >Suggestion<a href="#Suggestion" class="wiki-anchor">¶</a></h2>
<ul>
<li>Reproduce locally</li>
<li>Extend <code>t/01-test_needle.t</code> to reproduce this case</li>
<li>Have the test create and destroy multiple needs in a loop</li>
<li>Try using an optimized build for tinycv (e.g. via cmake argument <code>-DCMAKE_BUILD_TYPE=RelWithDebInfo</code>, see <a href="https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html" class="external">https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html</a>) if the problem is not reproducible</li>
<li>Provoke more threading in opencv</li>
<li>Investigate memory-handling on the Perl side, passed to opencv, in baseclass (check os-autoinst readme)</li>
</ul>
openQA Project - action #94531 (New): OpenQA worker randomly skips uploading artefacts for whole ...https://progress.opensuse.org/issues/945312021-06-23T08:21:44ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Once in a while, some jobs randomly end up with one or more modules missing all screenshots and console output. The boxes do appear in OpenQA web UI but only show <code>Unable to read foo-123.txt.</code> or blank box instead of a screenshot:<br>
<a href="https://openqa.suse.de/tests/6308588#step/cn_pec_sh/1" class="external">https://openqa.suse.de/tests/6308588#step/cn_pec_sh/1</a></p>
<p>Worker-log.txt shows that the missing files were not uploaded at all. The worker uploaded all artefacts for <code>update_kernel</code> and <code>install_ltp</code> but there's not a single line with <code>Uploading artefact cn_pec_sh-*.txt</code> or <code>Uploading artefact shutdown_ltp-*.txt</code> (except for log assets uploaded by calling <code>upload_logs()</code> in the test itself):<br>
<a href="https://openqa.suse.de/tests/6308588/logfile?filename=worker-log.txt" class="external">https://openqa.suse.de/tests/6308588/logfile?filename=worker-log.txt</a></p>
<p>I've seen this happen randomly multiple times, usually it's just one module in the middle of a test run. Everything before it and after it gets uploaded correctly.</p>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<ul>
<li>Look for jobs with test modules steps that include <code>Unable to read</code></li>
</ul>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>another job in the same scenario as the original one shows correct expected data in steps, <a href="https://openqa.suse.de/tests/6312792#step/cn_pec_sh/1" class="external">https://openqa.suse.de/tests/6312792#step/cn_pec_sh/1</a> shows</p>
<pre><code># wait_serial expected: "# "
# Result:
#
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> test module steps are uploaded again for openqa.suse.de LTP cases in general</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Find fail ratio</li>
<li>Try to reproduce problem locally</li>
<li>Find out what files the openQA worker should try to upload</li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>If we find a problem in the custom LTP runner don't try to fix that on openQA side but have it be fixed within os-autoinst-distri-opensuse</li>
</ul>
openQA Tests - action #93112 (Resolved): [qe-core][s390x] bootloader_zkvm fails: Cannot allocate ...https://progress.opensuse.org/issues/931122021-05-25T15:32:22ZMDouchamartin.doucha@suse.com
<p>s390 jobs randomly fail in <code>bootloader_zkvm</code>. autoinst-log.txt shows the following error:</p>
<pre><code>[debug] [run_ssh_cmd(virsh start openQA-SUT-4 2> >(tee /tmp/os-autoinst-openQA-SUT-4-stderr.log >&2))] stderr:
error: Failed to start domain openQA-SUT-4
error: internal error: qemu unexpectedly closed the monitor: 2021-05-18T11:23:21.183643Z qemu-system-s390x: cannot set up guest memory 's390.ram': Cannot allocate memory
</code></pre>
<p><a href="https://openqa.suse.de/tests/6044126#step/bootloader_zkvm/28" class="external">https://openqa.suse.de/tests/6044126#step/bootloader_zkvm/28</a><br>
<a href="https://openqa.suse.de/tests/6044006#step/bootloader_zkvm/28" class="external">https://openqa.suse.de/tests/6044006#step/bootloader_zkvm/28</a></p>
<p>This appears to be the same problem as <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: [sle][functional][u][s390x[kvm] test fails in bootloader_zkvm - "Cannot allocate memory" when ins... (Resolved)" href="https://progress.opensuse.org/issues/45326">#45326</a> and <a class="issue tracker-4 status-6 priority-4 priority-default closed" title="action: [functional][u] test fails in bootloader_zkvm - qemu-system-s390x: cannot set up guest memory 's3... (Rejected)" href="https://progress.opensuse.org/issues/48404">#48404</a>.</p>
<p>Additional links: <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Server-DVD-Incidents-Kernel&machine=s390x-kvm-sle12&test=install_ltp%2Bsle%2BServer-DVD-Incidents-Kernel&version=15-SP2" class="external">latest job with bootloader_zkvm</a></p>
openQA Project - action #92533 (New): Module-centric test result overviewhttps://progress.opensuse.org/issues/925332021-05-11T15:21:55ZMDouchamartin.doucha@suse.com
<p>In response to the discussion about <a href="https://confluence.suse.com/display/~vpelcak/Draft+-+Change+in+openQA+Review" class="external">OpenQA review process changes</a>, I'd like to propose a new test result overview that is module-centric. All current overviews are job-centric which makes it difficult to compare the results of a single module under different configurations.</p>
<p>Requirements:</p>
<ul>
<li>Show all available results of a single test module on the same page</li>
<li>Renamed instances of the same Perl module will be treated as different modules but there will be quick navigation links between them</li>
<li>Filtering and easily configurable grouping by standard OpenQA job filters (distri, arch, flavor, etc.)</li>
<li>Filtering by module result (passed/failed/softfailed/skipped/none)</li>
<li>Result grouping by test version (package version or Git commit)</li>
<li>Option to show only the latest results (per job group/build) or everything</li>
<li>Quick link to parent OpenQA job from each module result</li>
</ul>
openQA Project - action #70615 (New): Calling select_serial_terminal() twice on s390x svirt backe...https://progress.opensuse.org/issues/706152020-08-27T15:23:01ZMDouchamartin.doucha@suse.com
<p>When the same job calls <code>select_serial_terminal()</code> twice on s390x svirt worker (e.g. once before and once after reboot), the test will crash with the following error:</p>
<pre><code># wait_serial expected: qr/login:\s*$/ui
# Result:
Script started, file is /tmp/serial_terminal.txt.DjErjAe114GKpV_a
Connected to domain openQA-SUT-3
Escape character is ^]
error: operation failed: Active console session exists for this domain
CONSOLE_EXIT_DjErjAe114GKpV_a: 1
Script done, file is /tmp/serial_terminal.txt.DjErjAe114GKpV_a
</code></pre>
<hr>
<pre><code># Test died: Failed to wait for login prompt at /var/lib/openqa/cache/openqa.suse.de/tests/sle/lib/serial_terminal.pm line 113.
</code></pre>
<p><a href="https://openqa.suse.de/tests/4600947#step/install_klp_product/32" class="external">https://openqa.suse.de/tests/4600947#step/install_klp_product/32</a></p>
<p>Calling <code>select_serial_terminal()</code> multiple times works fine on other archs.</p>
openQA Project - action #70612 (New): better error handling in testapi function script_output (wa...https://progress.opensuse.org/issues/706122020-08-27T14:16:53ZMDouchamartin.doucha@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>I've just spent 2 hours staring into code trying to figure out why <code>get_patches()</code> in lib/qam.pm rejected an update that has the right incident ID and is marker as needed.<br>
<a href="https://openqa.suse.de/tests/4596773#step/update_kernel/63" class="external">https://openqa.suse.de/tests/4596773#step/update_kernel/63</a></p>
<p>Then I've noticed that the leading output marker is malformed due to VNC typing issue so <code>script_output()</code> simply returned an empty string to <code>get_patches()</code>.</p>
<p>When <code>script_output()</code> fails to parse the output, it shouldn't silently return an empty string. Suggestion: It should throw an exception.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Parsing errors in script_output can be easily distinguished from a false boolean result from the internal called script command</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Review and potentially extend os-autoinst t/03-testapi.t for how script_output behaves on an error like lost characters leading to unparseable responses</li>
</ul>
openQA Project - action #70189 (Rejected): openQA-common package broken on Tumbleweedhttps://progress.opensuse.org/issues/701892020-08-18T11:40:34ZMDouchamartin.doucha@suse.com
<p>When I install the standard <code>openQA-common</code> package on Tumbleweed and try to access the local webUI, I get error 502 and <code>journalctl</code> shows Perl errors with traceback (see below). When I replace the <code>/usr/share/openqa</code> directory with a symlink to my local copy of OpenQA git repo and restart the OpenQA services, everything works fine.</p>
<pre><code>$ journalctl -u openqa-webuid -b0
srp 18 13:25:25 dhcp165.suse.cz systemd[1]: Started The openQA web UI.
srp 18 13:25:26 dhcp165.suse.cz openqa-webui-daemon[17382]: [2020-08-18 13:25:26.96072] [17382] [warn] Deprecated use of config key '[audit]: blacklist'. Use '[audit]: blocklist' instead
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://127.0.0.1:9526
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://[::1]:9526
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 14, near "include_branding 'docbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 17, near "include_branding 'sponsorbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Mojo::Reactor::Poll: I/O watcher failed: syntax error at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Context:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 30: } );
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 31: % end
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 32:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 33: <link rel="icon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 34: type="image/png" sizes="16x16"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 35: href="<%= icon_url 'logo-16.png' %>">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 36: <link rel="icon" href="<%= icon_url 'logo.svg'%>" sizes="any" type="image/svg+xml">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 37:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 38: </head>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 39: <body>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 40: <nav class="navbar navbar-static-top navbar-default">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Traceback (most recent call first):
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 163, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 173, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPLRenderer.pm", line 40, in "Mojolicious::Plugin::EPLRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPRenderer.pm", line 39, in "Mojolicious::Plugin::EPRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 221, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 110, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 152, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 164, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 123, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 110, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 50, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 70, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 203, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 15, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 18, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 141, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server.pm", line 66, in "Mojo::Server"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 103, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Transaction/HTTP.pm", line 60, in "Mojo::Transaction::HTTP"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 218, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 199, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 109, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 57, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 60, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 103, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop.pm", line 133, in "Mojo::IOLoop"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 152, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 93, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 78, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Command/prefork.pm", line 31, in "Mojolicious::Command::prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Commands.pm", line 57, in "Mojolicious::Commands"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 186, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/../lib/OpenQA/WebAPI.pm", line 493, in "OpenQA::WebAPI"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/openqa", line 35, in "main"
</code></pre> openQA Project - action #66619 (Rejected): OpenQA jobs roll back to the wrong snapshot on hard te...https://progress.opensuse.org/issues/666192020-05-07T11:54:05ZMDouchamartin.doucha@suse.com
<p>When a job includes multiple modules that create a snapshot, VM rollback appears to always use the very first snapshot instead of the last one.</p>
<p>Example: <a href="https://openqa.suse.de/tests/4203253#step/AD044/6" class="external">https://openqa.suse.de/tests/4203253#step/AD044/6</a><br>
Module AD043 failed and triggered VM rollback. The remaining modules then fail with the following error:</p>
<pre><code>/tmp/aiodio/junkfile: No such file or directory
</code></pre>
<p>This means that the VM was rolled back all the way to <code>boot_ltp</code>. But it was supposed to use the snapshot created by <code>create_junkfile_ltp</code>.</p>
<p>This does not appear to be a new issue. The same error appears in all LTP aiodio jobs which failed since VM rollback was enabled for them by <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9264" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9264</a><br>
Oldest known example: <a href="https://openqa.suse.de/tests/3987350#step/AD037/6" class="external">https://openqa.suse.de/tests/3987350#step/AD037/6</a></p>
openQA Tests - action #64285 (New): [qe-core][qem] Aggregate tests with GM base imagehttps://progress.opensuse.org/issues/642852020-03-06T16:39:37ZMDouchamartin.doucha@suse.com
<p>This is a test scenario designed to detect weak dependency breakage which caused certificate issues on SLE-12. <a href="https://bugzilla.suse.com/show_bug.cgi?id=1165915" class="external">https://bugzilla.suse.com/show_bug.cgi?id=1165915</a></p>
<p>Scenario:</p>
<ol>
<li>Start with GM base image of target SLE (only packages from GM pool)</li>
<li>Collect package names from incident repos</li>
<li>Install corresponding packages from GM pool repos</li>
<li>Enable both update repos <strong>AND</strong> incident repos</li>
<li>Do full system update</li>
<li>Run package-specific tests</li>
</ol>
<p>If you don't install old packages from GM pool first, zypper will order packages correctly through transitive dependencies. We're specifically trying to break transitive dependencies here.</p>
<p>If you separate system update from incident installation (splitting step 4), you may accidentally force correct ordering of transitive dependencies through release timing. In that case, dependency bugs will show up only if the packages with broken weak dependency both end up in testing queue at the same time (not guaranteed), of after both have been released (oh sh*t).</p>
openQA Tests - action #57131 (Resolved): install_ltp job fails in update_kernel (12SP4@ppc64le)https://progress.opensuse.org/issues/571312019-09-20T09:00:32ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-12-SP4-Server-DVD-Incidents-Kernel-ppc64le-install_ltp+sle+Server-DVD-Incidents-Kernel@ppc64le-virtio consistently fails in <a href="https://openqa.suse.de/tests/3384280/modules/update_kernel/steps/32" class="external">update_kernel</a> due to DNS error. Zypper almost always fails to resolve IP address of update repository host. The failure happens at different points in the test job (sometimes in module update_kernel, sometimes in module install_ltp) but it's always a DNS resolution error.</p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>install ltp with maintenance kernel/kgraft update</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/3342870" class="external">4.12.14-358.1.g6790685</a><br>
Oldest known failure of this type and build branch: <a href="https://openqa.suse.de/tests/3127191" class="external">4.12.14-322.1.g0619c2b</a><br>
Oldest known failure of this type in other 12SP4@ppc64le branches: <a href="https://openqa.suse.de/tests/3064111" class="external">:11846:kernel-ec2</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/3330947" class="external">4.12.14-356.1.gff88a5c</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=ppc64le&distri=sle&flavor=Server-DVD-Incidents-Kernel&machine=ppc64le-virtio&test=install_ltp%2Bsle%2BServer-DVD-Incidents-Kernel&version=12-SP4" class="external">latest</a></p>