openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-11-15T15:06:53ZopenSUSE Project Management Tool
Redmine openQA Project - action #150917 (Resolved): Restarting a job together with failed children will b...https://progress.opensuse.org/issues/1509172023-11-15T15:06:53ZMDouchamartin.doucha@suse.com
<p>Restarting a job together with failed children will break dependencies of the new job</p>
<p>Restarting a job with parent/child jobs using the <code>Skip restarting "OK" (passed/softfailed) children</code> drop-down link will break dependencies of the new job. In addition to the restarted children, it'll also link to all parents and children of the original job:<br>
<a href="https://openqa.suse.de/tests/12817616#dependencies" class="external">https://openqa.suse.de/tests/12817616#dependencies</a></p>
<p>The broken dependencies then prevent further restarts.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: After restarting a chained parent (using "Skip restarting OK" but actually regardless of the option) none of the chained children have two parents</li>
<li><strong>AC2</strong>: The same counts for other dependency types.</li>
</ul>
<a name="Suggestion"></a>
<h2 >Suggestion<a href="#Suggestion" class="wiki-anchor">¶</a></h2>
<ul>
<li>Find out why we have not seen this in before. Does nobody else use that or care about it?</li>
<li>Replicate the scenario by extending/adjusting the unit tests</li>
<li>Clone the scenario locally to test this for real (no reason to run any jobs, just fake results via DB updates)</li>
</ul>
openQA Infrastructure - action #125798 (Resolved): Visual differences in GRUB menu on different x...https://progress.opensuse.org/issues/1257982023-03-10T17:09:54ZMDouchamartin.doucha@suse.com
<p>Here are 5 different LTP jobs booting the exact same UEFI/SecureBoot QCOW image on different workers:<br>
<a href="https://openqa.suse.de/tests/10651590" class="external">https://openqa.suse.de/tests/10651590</a> openqaworker16:14, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10658203" class="external">https://openqa.suse.de/tests/10658203</a> openqaworker16:18, pass<br>
<a href="https://openqa.suse.de/tests/10659306" class="external">https://openqa.suse.de/tests/10659306</a> openqaworker16:7, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659346" class="external">https://openqa.suse.de/tests/10659346</a> openqaworker17:12, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659359" class="external">https://openqa.suse.de/tests/10659359</a> worker9:11, pass</p>
<p>It appears that GRUB menu size changes depending on not just the worker but also specific worker slot.</p>
<p>Possibly related to <a href="https://progress.opensuse.org/issues/114523" class="external">poo#114523</a> but this time it's happening on x86_64.</p>
openQA Infrastructure - action #123960 (Resolved): s390x tests fail to log into VNC console on wo...https://progress.opensuse.org/issues/1239602023-02-06T09:49:51ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>s390x tests started randomly failing last week when they try to log into the freshly booted test system. There are multiple instances across all SLE releases both before and after applying incident patches so this is most likely an infra issue.<br>
<a href="https://openqa.suse.de/tests/10427719#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10427719#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10427825#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10427825#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10432751#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10432751#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10424294#step/boot_ltp/21" class="external">https://openqa.suse.de/tests/10424294#step/boot_ltp/21</a></p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/10424537" class="external">:27598:kgraft-patch-SLE12-SP5_Update_35</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/10424525" class="external">:27599:kgraft-patch-SLE12-SP5_Update_36</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Server-DVD-Incidents-Kernel&machine=s390x-kvm-sle12&test=install_ltp%2Bsle%2BServer-DVD-Incidents-Kernel&version=12-SP5" class="external">latest</a></p>
openQA Project - action #123664 (New): os-autoinst does not flush serial console buffer on snapsh...https://progress.opensuse.org/issues/1236642023-01-25T15:19:39ZMDouchamartin.doucha@suse.com
<p>When tests fail with kernel backtraces, the stale backtrace will be reported again after snapshot reload and trigger bogus failure even when the next test is successful.<br>
Example: <a href="https://openqa.suse.de/tests/10371180#step/cve-2017-1000111/13" class="external">https://openqa.suse.de/tests/10371180#step/cve-2017-1000111/13</a><br>
dmesg log: <a href="https://openqa.suse.de/tests/10371180/logfile?filename=serial0.txt" class="external">https://openqa.suse.de/tests/10371180/logfile?filename=serial0.txt</a></p>
<p>Here, the LTP test <code>cve-2017-18075</code> triggered kernel warning and failed. But the same kernel warning gets reported again at the end of test <code>cve-2017-1000111</code> which was successful and the dmesg log does not show any additional backtraces from it.</p>
<p>This appears to be os-autoinst regression because IIRC kernel backtrace detection used to work fine and only reported errors once.</p>
openQA Infrastructure - action #120339 (Resolved): QEMU DNS fails to resolve openqa.suse.de via I...https://progress.opensuse.org/issues/1203392022-11-11T12:48:43ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h1 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h1>
<p>LTP test <code>host</code> <a href="https://openqa.suse.de/tests/9927713#step/host/8" class="external">started failing today</a>. The QEMU DNS service running at 10.0.2.3 correctly resolves hostnames to IP addresses but reverse lookup fails. <a href="https://openqa.suse.de/tests/9915478#step/host/8" class="external">Old tests</a> which passed up until yesterday are now <a href="https://openqa.suse.de/tests/9930979#step/host/8" class="external">also failing upon restart</a> so this appears to be a QEMU configuration issue. The physical worker machine can resolve IP addresses without issue.</p>
<p>This issue is confirmed on worker3, worker5, worker8 and worker13. Other workers may be affected as well. PPC64LE QEMU workers do not seem to be affected, though.</p>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Revert removal of faulty DNS</li>
</ul>
<pre><code>sudo salt --no-color --state-output=changes -C 'G@roles:worker' cmd.run 'sudo sed -i "s/\(NETCONFIG_DNS_POLICY=\)\"\"/\1\"auto\"/;s/\(NETCONFIG_DNS_STATIC_SERVERS=\)\"10.160.0.1 10.100.2.10\"/\1\"\"/" /etc/sysconfig/network/config && sudo netconfig update -f'
</code></pre> openQA Project - action #119461 (Resolved): Serial failure autodetection overrides test result wh...https://progress.opensuse.org/issues/1194612022-10-26T16:09:53ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>The serial failure detection code in <a href="https://github.com/os-autoinst/os-autoinst/blob/2156ecc810939cf3bce8ca705cf39423dd6c194d/basetest.pm#L632" class="external">basetest::parse_serial_output_qemu()</a> unconditionally changes <code>$self->{result}</code> which can result in test failure being turned into softfail or even pass. Result should be changed only when the new value is a "higher level" of failure than the current value (e.g. <code>softfail</code> -> <code>fail</code>, but never <code>fail</code> -> <code>softfail</code>).<br>
Example: <a href="https://openqa.suse.de/tests/9481132#step/nfs41_01/12" class="external">https://openqa.suse.de/tests/9481132#step/nfs41_01/12</a></p>
<p>The nfs41_01 test in the example job failed with exit code 137 (SIGKILL) but the CPU soft lockup box at the end (serial failure with type <code>info</code>) changed the result to <code>pass</code>. This continued in all subsequent test modules until the job timed out.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Test module results are never reset to a lower level</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Extend tests in t/17-basetest.t with the updated expectation (The main bunch of tests was added in 2018 by riafarov who is not active anymore)</li>
<li>Investigate the code path in <code>basetest::parse_serial_output_qemu</code></li>
<li>Update the implementation</li>
</ul>
openQA Project - action #109929 (New): Snapshot rollback after SUT reboot breaks console switchinghttps://progress.opensuse.org/issues/1099292022-04-13T16:08:03ZMDouchamartin.doucha@suse.com
<p>If the SUT gets rebooted between snapshot creation and rollback, console state will not be properly restored and <code>select_console()</code> will fail in some cases because it'll try to log in when the user is already logged in since before the snapshot:<br>
<a href="https://openqa.suse.de/tests/8545133#step/select_console#1/2" class="external">https://openqa.suse.de/tests/8545133#step/select_console#1/2</a></p>
<p>Steps to reproduce:</p>
<ol>
<li>Activate 2 or more consoles</li>
<li>Create snapshot</li>
<li>Reboot SUT and call <code>wait_boot()</code> (<code>reset_consoles()</code> will be called by <code>wait_boot()</code> here)</li>
<li>Activate the same consoles again (all of them will be added to <code>$autotest::last_milestone->{activated_consoles}</code> due to <code>reset_consoles()</code> above)</li>
<li>Trigger snapshot rollback</li>
<li>Select any console activated in step 1 except the one that was selected during snapshot creation</li>
</ol>
<p>If the console selected in the last step expects login prompt on first activation and then keeps the session open even while not selected, it'll fail to activate after snapshot rollback. The console session was left open in the snapshot but the test backend will wrongly believe that it's closed due to steps 3 and 4 causing another console reset during snapshot rollback.</p>
openQA Project - action #106898 (Resolved): Protection against asset clobberinghttps://progress.opensuse.org/issues/1068982022-02-16T10:33:47ZMDouchamartin.doucha@suse.com
<p>QCOW images in OpenQA occasionally get corrupted because multiple jobs try to publish the same file at the same time, either due to <code>PUBLISH_*</code> setting misconfiguration or duplicate install jobs scheduled in parallel. For example, this job failed to start:<br>
<a href="https://openqa.suse.de/tests/8162749" class="external">https://openqa.suse.de/tests/8162749</a></p>
<p>because these three install jobs finished 20 minutes apart and tried to upload the same QCOW image:<br>
<a href="https://openqa.suse.de/tests/8162347" class="external">https://openqa.suse.de/tests/8162347</a><br>
<a href="https://openqa.suse.de/tests/8161501" class="external">https://openqa.suse.de/tests/8161501</a><br>
<a href="https://openqa.suse.de/tests/8160547" class="external">https://openqa.suse.de/tests/8160547</a></p>
<p>Please add some sort of protection against asset clobbering via <code>PUBLISH_*</code> variables:</p>
<ul>
<li>two jobs must not publish the same file in parallel</li>
<li>jobs must not publish a file while another job may be downloading the previous version</li>
<li><code>PUBLISH_*</code> misconfiguration (e.g. copy-paste mistakes among multiple testsuites) should be detected and reported in the WebUI, for example as the reason why install job was terminated</li>
</ul>
openQA Project - action #94850 (Resolved): QEMU 6.0 fails to start if job has QEMU_NUMA=1https://progress.opensuse.org/issues/948502021-06-29T10:05:52ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Originally reported by pvorel relating to <a href="http://quasar.suse.cz/tests/6860#">http://quasar.suse.cz/tests/6860#</a></p>
<p>It appears that QEMU 6.0 requires explicit memory assignment to NUMA nodes using <code>-numa node,mem=...</code> command line arguments. OpenQA jobs with <code>QEMU_NUMA=1</code> currently fail to start on Tumbleweed because os-autoinst omits the <code>mem</code> option, which leads to the following error:</p>
<pre><code>[2021-06-28T13:26:00.308 CEST] [debug] starting: /usr/bin/qemu-system-x86_64 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 2048 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot order=c -device usb-ehci -device usb-tablet -smp 2 -numa node,nodeid=0 -numa node,nodeid=1 -enable-kvm -no-shutdown -vnc :91,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-overlay0-file,filename=/var/lib/openqa/pool/1/raid/hd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0-overlay0,file=hd0-overlay0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0-overlay0,bootindex=0,serial=hd0
[2021-06-28T13:26:00.312 CEST] [debug] Waiting for 0 attempts
[2021-06-28T13:26:00.423 CEST] [debug] Waiting for 1 attempts
[2021-06-28T13:26:00.424 CEST] [info] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 453.
[2021-06-28T13:26:00.424 CEST] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2021-06-28T13:26:00.424 CEST] [debug] Passing remaining frames to the video encoder
[2021-06-28T13:26:00.426 CEST] [debug] Waiting for video encoder to finalize the video
[2021-06-28T13:26:00.426 CEST] [debug] The built-in video encoder (pid 22047) terminated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: QEMU emulator version 6.0.0 (openSUSE Tumbleweed)
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on: warning: short-form boolean option 'server' deprecated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Please use server=on instead
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on: warning: short-form boolean option 'nowait' deprecated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Please use wait=off instead
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: total memory for NUMA nodes (0x0) should equal RAM size (0x80000000)
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> os-autoinst continues to start for qemu >= 6.0 and QEMU_NUMA=1</li>
<li><strong>AC2:</strong> os-autoinst jobs still work fine on qemu < 6.0 regardless of QEMU_NUMA setting</li>
</ul>
openQA Project - action #94531 (New): OpenQA worker randomly skips uploading artefacts for whole ...https://progress.opensuse.org/issues/945312021-06-23T08:21:44ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Once in a while, some jobs randomly end up with one or more modules missing all screenshots and console output. The boxes do appear in OpenQA web UI but only show <code>Unable to read foo-123.txt.</code> or blank box instead of a screenshot:<br>
<a href="https://openqa.suse.de/tests/6308588#step/cn_pec_sh/1" class="external">https://openqa.suse.de/tests/6308588#step/cn_pec_sh/1</a></p>
<p>Worker-log.txt shows that the missing files were not uploaded at all. The worker uploaded all artefacts for <code>update_kernel</code> and <code>install_ltp</code> but there's not a single line with <code>Uploading artefact cn_pec_sh-*.txt</code> or <code>Uploading artefact shutdown_ltp-*.txt</code> (except for log assets uploaded by calling <code>upload_logs()</code> in the test itself):<br>
<a href="https://openqa.suse.de/tests/6308588/logfile?filename=worker-log.txt" class="external">https://openqa.suse.de/tests/6308588/logfile?filename=worker-log.txt</a></p>
<p>I've seen this happen randomly multiple times, usually it's just one module in the middle of a test run. Everything before it and after it gets uploaded correctly.</p>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<ul>
<li>Look for jobs with test modules steps that include <code>Unable to read</code></li>
</ul>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>another job in the same scenario as the original one shows correct expected data in steps, <a href="https://openqa.suse.de/tests/6312792#step/cn_pec_sh/1" class="external">https://openqa.suse.de/tests/6312792#step/cn_pec_sh/1</a> shows</p>
<pre><code># wait_serial expected: "# "
# Result:
#
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> test module steps are uploaded again for openqa.suse.de LTP cases in general</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Find fail ratio</li>
<li>Try to reproduce problem locally</li>
<li>Find out what files the openQA worker should try to upload</li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>If we find a problem in the custom LTP runner don't try to fix that on openQA side but have it be fixed within os-autoinst-distri-opensuse</li>
</ul>
openQA Project - action #81142 (New): VNC console corruptionhttps://progress.opensuse.org/issues/811422020-12-17T11:07:10ZMDouchamartin.doucha@suse.com
<p>It appears that issue <a class="issue tracker-4 status-3 priority-3 priority-lowest closed behind-schedule" title="action: VNC console corruption on aarch64 (Resolved)" href="https://progress.opensuse.org/issues/61994">#61994</a> is back, this time on PPC64LE. LTP tests fail to boot because switching from boot animation to login screen corrupts VNC frame buffer and needles fail to match.<br>
<a href="https://openqa.suse.de/tests/5187741#step/boot_ltp/9" class="external">https://openqa.suse.de/tests/5187741#step/boot_ltp/9</a><br>
<a href="https://openqa.suse.de/tests/5187758#step/boot_ltp/8" class="external">https://openqa.suse.de/tests/5187758#step/boot_ltp/8</a></p>
openQA Project - action #70189 (Rejected): openQA-common package broken on Tumbleweedhttps://progress.opensuse.org/issues/701892020-08-18T11:40:34ZMDouchamartin.doucha@suse.com
<p>When I install the standard <code>openQA-common</code> package on Tumbleweed and try to access the local webUI, I get error 502 and <code>journalctl</code> shows Perl errors with traceback (see below). When I replace the <code>/usr/share/openqa</code> directory with a symlink to my local copy of OpenQA git repo and restart the OpenQA services, everything works fine.</p>
<pre><code>$ journalctl -u openqa-webuid -b0
srp 18 13:25:25 dhcp165.suse.cz systemd[1]: Started The openQA web UI.
srp 18 13:25:26 dhcp165.suse.cz openqa-webui-daemon[17382]: [2020-08-18 13:25:26.96072] [17382] [warn] Deprecated use of config key '[audit]: blacklist'. Use '[audit]: blocklist' instead
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://127.0.0.1:9526
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://[::1]:9526
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 14, near "include_branding 'docbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 17, near "include_branding 'sponsorbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Mojo::Reactor::Poll: I/O watcher failed: syntax error at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Context:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 30: } );
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 31: % end
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 32:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 33: <link rel="icon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 34: type="image/png" sizes="16x16"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 35: href="<%= icon_url 'logo-16.png' %>">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 36: <link rel="icon" href="<%= icon_url 'logo.svg'%>" sizes="any" type="image/svg+xml">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 37:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 38: </head>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 39: <body>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 40: <nav class="navbar navbar-static-top navbar-default">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Traceback (most recent call first):
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 163, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 173, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPLRenderer.pm", line 40, in "Mojolicious::Plugin::EPLRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPRenderer.pm", line 39, in "Mojolicious::Plugin::EPRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 221, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 110, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 152, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 164, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 123, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 110, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 50, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 70, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 203, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 15, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 18, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 141, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server.pm", line 66, in "Mojo::Server"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 103, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Transaction/HTTP.pm", line 60, in "Mojo::Transaction::HTTP"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 218, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 199, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 109, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 57, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 60, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 103, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop.pm", line 133, in "Mojo::IOLoop"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 152, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 93, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 78, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Command/prefork.pm", line 31, in "Mojolicious::Command::prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Commands.pm", line 57, in "Mojolicious::Commands"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 186, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/../lib/OpenQA/WebAPI.pm", line 493, in "OpenQA::WebAPI"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/openqa", line 35, in "main"
</code></pre> openQA Project - action #69700 (New): Predefined QEMU hardware profiles in os-autoinsthttps://progress.opensuse.org/issues/697002020-08-07T10:15:47ZMDouchamartin.doucha@suse.com
<p>We've recently had a <a href="https://bugzilla.suse.com/show_bug.cgi?id=1174887" class="external">regression</a> that made our kernels unbootable in QEMU VMs created in virt-manager. The regression was missed by OpenQA tests because os-autoinst uses VMs with minimal hardware configuration which didn't trigger the bug.</p>
<p>We should define multiple QEMU hardware profiles (named sets of extra device options for QEMU) which can then be selected through job settings. The hardware profiles don't need to cover every possible combination of devices, it'll be enough if each device model appears in them at least once. One of the profiles should be as close to virt-manager defaults as possible. Then it'll be sufficient to boot the existing LTP jobs on different hardware profiles. We don't need any extra tests beyond checking that the kernel is bootable.</p>
<p>Example profile that would trigger the regression:</p>
<pre><code>-machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1
-device pcie-pci-bridge,id=pci.9,bus=pci.2,addr=0x0
</code></pre> openQA Project - action #66619 (Rejected): OpenQA jobs roll back to the wrong snapshot on hard te...https://progress.opensuse.org/issues/666192020-05-07T11:54:05ZMDouchamartin.doucha@suse.com
<p>When a job includes multiple modules that create a snapshot, VM rollback appears to always use the very first snapshot instead of the last one.</p>
<p>Example: <a href="https://openqa.suse.de/tests/4203253#step/AD044/6" class="external">https://openqa.suse.de/tests/4203253#step/AD044/6</a><br>
Module AD043 failed and triggered VM rollback. The remaining modules then fail with the following error:</p>
<pre><code>/tmp/aiodio/junkfile: No such file or directory
</code></pre>
<p>This means that the VM was rolled back all the way to <code>boot_ltp</code>. But it was supposed to use the snapshot created by <code>create_junkfile_ltp</code>.</p>
<p>This does not appear to be a new issue. The same error appears in all LTP aiodio jobs which failed since VM rollback was enabled for them by <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9264" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9264</a><br>
Oldest known example: <a href="https://openqa.suse.de/tests/3987350#step/AD037/6" class="external">https://openqa.suse.de/tests/3987350#step/AD037/6</a></p>
openQA Infrastructure - action #58945 (Resolved): OpenQA worker service not restarted after OpenQ...https://progress.opensuse.org/issues/589452019-10-31T13:12:21ZMDouchamartin.doucha@suse.com
<p>The openqa-worker service on some openqa.suse.de workers doesn't get restarted after update. This may cause version mismatch between os-autoinst and openQA-common packages.</p>
<p>One example of this mismatch are these three verification runs for <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8329" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8329</a> below:<br>
openqaworker2: <a href="https://openqa.suse.de/tests/3541705" class="external">https://openqa.suse.de/tests/3541705</a> (openqa-worker service last restarted on 2019-10-30)<br>
openqaworker6: <a href="https://openqa.suse.de/tests/3541697" class="external">https://openqa.suse.de/tests/3541697</a> (openqa-worker service last restarted on 2019-09-18)<br>
openqaworker9: <a href="https://openqa.suse.de/tests/3544337" class="external">https://openqa.suse.de/tests/3544337</a> (openqa-worker service last restarted on 2019-09-18)</p>
<p>All three jobs ran the same test modules (see autoinst log) but all tests after intall_ltp were scheduled at runtime. Updating test schedule at runtime requires patches merged into OpenQA on 2019-09-27 so openqaworker6 and openqaworker9 didn't update test schedule due to still running openQA-common from mid-September, before the patches were merged.</p>