openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-11-15T15:06:53ZopenSUSE Project Management Tool
Redmine openQA Project - action #150917 (Resolved): Restarting a job together with failed children will b...https://progress.opensuse.org/issues/1509172023-11-15T15:06:53ZMDouchamartin.doucha@suse.com
<p>Restarting a job together with failed children will break dependencies of the new job</p>
<p>Restarting a job with parent/child jobs using the <code>Skip restarting "OK" (passed/softfailed) children</code> drop-down link will break dependencies of the new job. In addition to the restarted children, it'll also link to all parents and children of the original job:<br>
<a href="https://openqa.suse.de/tests/12817616#dependencies" class="external">https://openqa.suse.de/tests/12817616#dependencies</a></p>
<p>The broken dependencies then prevent further restarts.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: After restarting a chained parent (using "Skip restarting OK" but actually regardless of the option) none of the chained children have two parents</li>
<li><strong>AC2</strong>: The same counts for other dependency types.</li>
</ul>
<a name="Suggestion"></a>
<h2 >Suggestion<a href="#Suggestion" class="wiki-anchor">¶</a></h2>
<ul>
<li>Find out why we have not seen this in before. Does nobody else use that or care about it?</li>
<li>Replicate the scenario by extending/adjusting the unit tests</li>
<li>Clone the scenario locally to test this for real (no reason to run any jobs, just fake results via DB updates)</li>
</ul>
openQA Infrastructure - action #125798 (Resolved): Visual differences in GRUB menu on different x...https://progress.opensuse.org/issues/1257982023-03-10T17:09:54ZMDouchamartin.doucha@suse.com
<p>Here are 5 different LTP jobs booting the exact same UEFI/SecureBoot QCOW image on different workers:<br>
<a href="https://openqa.suse.de/tests/10651590" class="external">https://openqa.suse.de/tests/10651590</a> openqaworker16:14, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10658203" class="external">https://openqa.suse.de/tests/10658203</a> openqaworker16:18, pass<br>
<a href="https://openqa.suse.de/tests/10659306" class="external">https://openqa.suse.de/tests/10659306</a> openqaworker16:7, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659346" class="external">https://openqa.suse.de/tests/10659346</a> openqaworker17:12, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659359" class="external">https://openqa.suse.de/tests/10659359</a> worker9:11, pass</p>
<p>It appears that GRUB menu size changes depending on not just the worker but also specific worker slot.</p>
<p>Possibly related to <a href="https://progress.opensuse.org/issues/114523" class="external">poo#114523</a> but this time it's happening on x86_64.</p>
openQA Infrastructure - action #123960 (Resolved): s390x tests fail to log into VNC console on wo...https://progress.opensuse.org/issues/1239602023-02-06T09:49:51ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>s390x tests started randomly failing last week when they try to log into the freshly booted test system. There are multiple instances across all SLE releases both before and after applying incident patches so this is most likely an infra issue.<br>
<a href="https://openqa.suse.de/tests/10427719#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10427719#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10427825#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10427825#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10432751#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10432751#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10424294#step/boot_ltp/21" class="external">https://openqa.suse.de/tests/10424294#step/boot_ltp/21</a></p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/10424537" class="external">:27598:kgraft-patch-SLE12-SP5_Update_35</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/10424525" class="external">:27599:kgraft-patch-SLE12-SP5_Update_36</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Server-DVD-Incidents-Kernel&machine=s390x-kvm-sle12&test=install_ltp%2Bsle%2BServer-DVD-Incidents-Kernel&version=12-SP5" class="external">latest</a></p>
openQA Project - action #123664 (New): os-autoinst does not flush serial console buffer on snapsh...https://progress.opensuse.org/issues/1236642023-01-25T15:19:39ZMDouchamartin.doucha@suse.com
<p>When tests fail with kernel backtraces, the stale backtrace will be reported again after snapshot reload and trigger bogus failure even when the next test is successful.<br>
Example: <a href="https://openqa.suse.de/tests/10371180#step/cve-2017-1000111/13" class="external">https://openqa.suse.de/tests/10371180#step/cve-2017-1000111/13</a><br>
dmesg log: <a href="https://openqa.suse.de/tests/10371180/logfile?filename=serial0.txt" class="external">https://openqa.suse.de/tests/10371180/logfile?filename=serial0.txt</a></p>
<p>Here, the LTP test <code>cve-2017-18075</code> triggered kernel warning and failed. But the same kernel warning gets reported again at the end of test <code>cve-2017-1000111</code> which was successful and the dmesg log does not show any additional backtraces from it.</p>
<p>This appears to be os-autoinst regression because IIRC kernel backtrace detection used to work fine and only reported errors once.</p>
openQA Infrastructure - action #120339 (Resolved): QEMU DNS fails to resolve openqa.suse.de via I...https://progress.opensuse.org/issues/1203392022-11-11T12:48:43ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h1 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h1>
<p>LTP test <code>host</code> <a href="https://openqa.suse.de/tests/9927713#step/host/8" class="external">started failing today</a>. The QEMU DNS service running at 10.0.2.3 correctly resolves hostnames to IP addresses but reverse lookup fails. <a href="https://openqa.suse.de/tests/9915478#step/host/8" class="external">Old tests</a> which passed up until yesterday are now <a href="https://openqa.suse.de/tests/9930979#step/host/8" class="external">also failing upon restart</a> so this appears to be a QEMU configuration issue. The physical worker machine can resolve IP addresses without issue.</p>
<p>This issue is confirmed on worker3, worker5, worker8 and worker13. Other workers may be affected as well. PPC64LE QEMU workers do not seem to be affected, though.</p>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Revert removal of faulty DNS</li>
</ul>
<pre><code>sudo salt --no-color --state-output=changes -C 'G@roles:worker' cmd.run 'sudo sed -i "s/\(NETCONFIG_DNS_POLICY=\)\"\"/\1\"auto\"/;s/\(NETCONFIG_DNS_STATIC_SERVERS=\)\"10.160.0.1 10.100.2.10\"/\1\"\"/" /etc/sysconfig/network/config && sudo netconfig update -f'
</code></pre> openQA Project - action #119461 (Resolved): Serial failure autodetection overrides test result wh...https://progress.opensuse.org/issues/1194612022-10-26T16:09:53ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>The serial failure detection code in <a href="https://github.com/os-autoinst/os-autoinst/blob/2156ecc810939cf3bce8ca705cf39423dd6c194d/basetest.pm#L632" class="external">basetest::parse_serial_output_qemu()</a> unconditionally changes <code>$self->{result}</code> which can result in test failure being turned into softfail or even pass. Result should be changed only when the new value is a "higher level" of failure than the current value (e.g. <code>softfail</code> -> <code>fail</code>, but never <code>fail</code> -> <code>softfail</code>).<br>
Example: <a href="https://openqa.suse.de/tests/9481132#step/nfs41_01/12" class="external">https://openqa.suse.de/tests/9481132#step/nfs41_01/12</a></p>
<p>The nfs41_01 test in the example job failed with exit code 137 (SIGKILL) but the CPU soft lockup box at the end (serial failure with type <code>info</code>) changed the result to <code>pass</code>. This continued in all subsequent test modules until the job timed out.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Test module results are never reset to a lower level</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Extend tests in t/17-basetest.t with the updated expectation (The main bunch of tests was added in 2018 by riafarov who is not active anymore)</li>
<li>Investigate the code path in <code>basetest::parse_serial_output_qemu</code></li>
<li>Update the implementation</li>
</ul>
openQA Project - action #109929 (New): Snapshot rollback after SUT reboot breaks console switchinghttps://progress.opensuse.org/issues/1099292022-04-13T16:08:03ZMDouchamartin.doucha@suse.com
<p>If the SUT gets rebooted between snapshot creation and rollback, console state will not be properly restored and <code>select_console()</code> will fail in some cases because it'll try to log in when the user is already logged in since before the snapshot:<br>
<a href="https://openqa.suse.de/tests/8545133#step/select_console#1/2" class="external">https://openqa.suse.de/tests/8545133#step/select_console#1/2</a></p>
<p>Steps to reproduce:</p>
<ol>
<li>Activate 2 or more consoles</li>
<li>Create snapshot</li>
<li>Reboot SUT and call <code>wait_boot()</code> (<code>reset_consoles()</code> will be called by <code>wait_boot()</code> here)</li>
<li>Activate the same consoles again (all of them will be added to <code>$autotest::last_milestone->{activated_consoles}</code> due to <code>reset_consoles()</code> above)</li>
<li>Trigger snapshot rollback</li>
<li>Select any console activated in step 1 except the one that was selected during snapshot creation</li>
</ol>
<p>If the console selected in the last step expects login prompt on first activation and then keeps the session open even while not selected, it'll fail to activate after snapshot rollback. The console session was left open in the snapshot but the test backend will wrongly believe that it's closed due to steps 3 and 4 causing another console reset during snapshot rollback.</p>
openQA Project - action #106898 (Resolved): Protection against asset clobberinghttps://progress.opensuse.org/issues/1068982022-02-16T10:33:47ZMDouchamartin.doucha@suse.com
<p>QCOW images in OpenQA occasionally get corrupted because multiple jobs try to publish the same file at the same time, either due to <code>PUBLISH_*</code> setting misconfiguration or duplicate install jobs scheduled in parallel. For example, this job failed to start:<br>
<a href="https://openqa.suse.de/tests/8162749" class="external">https://openqa.suse.de/tests/8162749</a></p>
<p>because these three install jobs finished 20 minutes apart and tried to upload the same QCOW image:<br>
<a href="https://openqa.suse.de/tests/8162347" class="external">https://openqa.suse.de/tests/8162347</a><br>
<a href="https://openqa.suse.de/tests/8161501" class="external">https://openqa.suse.de/tests/8161501</a><br>
<a href="https://openqa.suse.de/tests/8160547" class="external">https://openqa.suse.de/tests/8160547</a></p>
<p>Please add some sort of protection against asset clobbering via <code>PUBLISH_*</code> variables:</p>
<ul>
<li>two jobs must not publish the same file in parallel</li>
<li>jobs must not publish a file while another job may be downloading the previous version</li>
<li><code>PUBLISH_*</code> misconfiguration (e.g. copy-paste mistakes among multiple testsuites) should be detected and reported in the WebUI, for example as the reason why install job was terminated</li>
</ul>
openQA Project - action #99246 (Resolved): Published QCOW images appear to be uncompressedhttps://progress.opensuse.org/issues/992462021-09-24T11:41:40ZMDouchamartin.doucha@suse.com
<p>QCOW images generated by OpenQA install jobs appear to be uncompressed now.</p>
<p>ppc64le: <a href="https://openqa.suse.de/tests/7211627#downloads" class="external">https://openqa.suse.de/tests/7211627#downloads</a><br>
HDD_1: 885MB<br>
PUBLISH_HDD_1: 7.1GB</p>
<p>x86_64: <a href="https://openqa.suse.de/tests/7211569#downloads" class="external">https://openqa.suse.de/tests/7211569#downloads</a><br>
HDD_1: 976MB<br>
PUBLISH_HDD_1: 4.2GB</p>
<p>The size difference should be only a few hundred megabytes at most.</p>
openQA Project - action #98841 (Resolved): qemu randomly fails to start on QA-Power8-5-kvm auto_r...https://progress.opensuse.org/issues/988412021-09-17T15:49:43ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>A few LTP jobs have failed to start today due to qemu error on QA-Power8-5-kvm worker:<br>
<a href="https://openqa.suse.de/tests/7138972">https://openqa.suse.de/tests/7138972</a><br>
<a href="https://openqa.suse.de/tests/7149857">https://openqa.suse.de/tests/7149857</a><br>
<a href="https://openqa.suse.de/tests/7153989">https://openqa.suse.de/tests/7153989</a></p>
<p>All of them have the following output in autoinst-log.txt:</p>
<pre><code>[2021-09-17T16:24:29.803 CEST] [info] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
QEMU terminated before QMP connection could be established. Check for errors below
[2021-09-17T16:24:29.804 CEST] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2021-09-17T16:24:29.805 CEST] [debug] Passing remaining frames to the video encoder
[2021-09-17T16:24:29.805 CEST] [debug] Waiting for video encoder to finalize the video
[2021-09-17T16:24:29.805 CEST] [debug] The built-in video encoder (pid 110385) terminated
[2021-09-17T16:24:29.807 CEST] [debug] QEMU: QEMU emulator version 4.2.1 (openSUSE Leap 15.2)
[2021-09-17T16:24:29.807 CEST] [debug] QEMU: Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
[2021-09-17T16:24:29.807 CEST] [warn] !!! : qemu-system-ppc64: Failed to allocate KVM HPT of order 25 (try smaller maxmem?): Cannot allocate memory
</code></pre>
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>QA-Power8-5-kvm has 256GB RAM. <a href="https://monitor.qa.suse.de/d/WDQA-Power8-5-kvm/worker-dashboard-qa-power8-5-kvm?viewPanel=12054&orgId=1&from=1631765162464&to=1632085860553">https://monitor.qa.suse.de/d/WDQA-Power8-5-kvm/worker-dashboard-qa-power8-5-kvm?viewPanel=12054&orgId=1&from=1631765162464&to=1632085860553</a> shows that some memory was used during the period when the test failed but nothing that should explain the inability to allocate the memory for the qemu VM. In the system journal there is</p>
<pre><code>Sep 17 16:24:28 QA-Power8-5-kvm worker[88148]: [debug] [pid:88148] REST-API call: POST http://openqa.suse.de/api/v1/jobs/7125263/status
Sep 17 16:24:29 QA-Power8-5-kvm worker[104911]: [info] [pid:108741] sle-15-SP4-ppc64le-Build36.1-HA-BV.qcow2: Processing chunk 501/5812, avg. speed ~976.562 KiB/s
Sep 17 16:24:29 QA-Power8-5-kvm worker[101413]: [debug] [pid:102598] Uploading artefact mq_timedreceive_15-1-2.txt
Sep 17 16:24:29 QA-Power8-5-kvm worker[96737]: [debug] [pid:96737] REST-API call: POST http://openqa.suse.de/api/v1/jobs/7125265/status
Sep 17 16:24:29 QA-Power8-5-kvm worker[109458]: [debug] [pid:110336] Uploading artefact bootloader_start-15.txt
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: 23 callbacks suppressed
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf00, 3cb100) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf04, 3cb104) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf08, 3cb108) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf0c, 3cb10c) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf10, 3cb110) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf14, 3cb114) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf18, 3cb118) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf1c, 3cb11c) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf20, 3cb120) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf24, 3cb124) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: cma: cma_alloc: alloc failed, req-size: 512 pages, ret: -16
Sep 17 16:24:30 QA-Power8-5-kvm worker[88148]: [debug] [pid:88148] Upload concluded (at wait_children)
Sep 17 16:24:30 QA-Power8-5-kvm worker[109557]: [info] [pid:109557] Isotovideo exit status: 1
Sep 17 16:24:30 QA-Power8-5-kvm worker[109557]: [debug] [pid:109557] Stopping job 7153989 from openqa.suse.de: 07153989-sle-15-SP3-Server-DVD-Incidents-Kernel-KOTD-ppc64le-Build5.3.18-302.1.g316993b-ltp_crashme@ppc64le-virtio - reason: died
Sep 17 16:24:30 QA-Power8-5-kvm worker[109557]: [debug] [pid:109557] REST-API call: POST http://openqa.suse.de/api/v1/jobs/7153989/status
Sep 17 16:24:30 QA-Power8-5-kvm worker[101413]: [debug] [pid:102598] Uploading artefact mq_timedreceive_7-1-2.txt
</code></pre>
<p>in particular the messages</p>
<pre><code>Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf20, 3cb120) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf24, 3cb124) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: cma: cma_alloc: alloc failed, req-size: 512 pages, ret: -16
</code></pre>
<p>so an allocation failure. We could report a bug about this but because KVM on SUSE with Power8 is unsupported so I don't expect any success.</p>
<p>We likely need to accept such issues and trigger a restart automatically by openQA.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> qemu ppc64le allocation errors cause automatic job retriggers by openQA</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Catch the error and make it "Incomplete"</li>
<li>Restart the incomplete job</li>
<li>Make openQA automatically detect the issue and trigger restart, e.g. based on <a href="https://github.com/os-autoinst/openQA/blob/master/etc/openqa/openqa.ini#L76">https://github.com/os-autoinst/openQA/blob/master/etc/openqa/openqa.ini#L76</a></li>
</ul>
openQA Project - action #94850 (Resolved): QEMU 6.0 fails to start if job has QEMU_NUMA=1https://progress.opensuse.org/issues/948502021-06-29T10:05:52ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Originally reported by pvorel relating to <a href="http://quasar.suse.cz/tests/6860#">http://quasar.suse.cz/tests/6860#</a></p>
<p>It appears that QEMU 6.0 requires explicit memory assignment to NUMA nodes using <code>-numa node,mem=...</code> command line arguments. OpenQA jobs with <code>QEMU_NUMA=1</code> currently fail to start on Tumbleweed because os-autoinst omits the <code>mem</code> option, which leads to the following error:</p>
<pre><code>[2021-06-28T13:26:00.308 CEST] [debug] starting: /usr/bin/qemu-system-x86_64 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 2048 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot order=c -device usb-ehci -device usb-tablet -smp 2 -numa node,nodeid=0 -numa node,nodeid=1 -enable-kvm -no-shutdown -vnc :91,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-overlay0-file,filename=/var/lib/openqa/pool/1/raid/hd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0-overlay0,file=hd0-overlay0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0-overlay0,bootindex=0,serial=hd0
[2021-06-28T13:26:00.312 CEST] [debug] Waiting for 0 attempts
[2021-06-28T13:26:00.423 CEST] [debug] Waiting for 1 attempts
[2021-06-28T13:26:00.424 CEST] [info] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 453.
[2021-06-28T13:26:00.424 CEST] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2021-06-28T13:26:00.424 CEST] [debug] Passing remaining frames to the video encoder
[2021-06-28T13:26:00.426 CEST] [debug] Waiting for video encoder to finalize the video
[2021-06-28T13:26:00.426 CEST] [debug] The built-in video encoder (pid 22047) terminated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: QEMU emulator version 6.0.0 (openSUSE Tumbleweed)
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on: warning: short-form boolean option 'server' deprecated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Please use server=on instead
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on: warning: short-form boolean option 'nowait' deprecated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Please use wait=off instead
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: total memory for NUMA nodes (0x0) should equal RAM size (0x80000000)
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> os-autoinst continues to start for qemu >= 6.0 and QEMU_NUMA=1</li>
<li><strong>AC2:</strong> os-autoinst jobs still work fine on qemu < 6.0 regardless of QEMU_NUMA setting</li>
</ul>
openQA Project - action #81142 (New): VNC console corruptionhttps://progress.opensuse.org/issues/811422020-12-17T11:07:10ZMDouchamartin.doucha@suse.com
<p>It appears that issue <a class="issue tracker-4 status-3 priority-3 priority-lowest closed behind-schedule" title="action: VNC console corruption on aarch64 (Resolved)" href="https://progress.opensuse.org/issues/61994">#61994</a> is back, this time on PPC64LE. LTP tests fail to boot because switching from boot animation to login screen corrupts VNC frame buffer and needles fail to match.<br>
<a href="https://openqa.suse.de/tests/5187741#step/boot_ltp/9" class="external">https://openqa.suse.de/tests/5187741#step/boot_ltp/9</a><br>
<a href="https://openqa.suse.de/tests/5187758#step/boot_ltp/8" class="external">https://openqa.suse.de/tests/5187758#step/boot_ltp/8</a></p>
openQA Project - action #70189 (Rejected): openQA-common package broken on Tumbleweedhttps://progress.opensuse.org/issues/701892020-08-18T11:40:34ZMDouchamartin.doucha@suse.com
<p>When I install the standard <code>openQA-common</code> package on Tumbleweed and try to access the local webUI, I get error 502 and <code>journalctl</code> shows Perl errors with traceback (see below). When I replace the <code>/usr/share/openqa</code> directory with a symlink to my local copy of OpenQA git repo and restart the OpenQA services, everything works fine.</p>
<pre><code>$ journalctl -u openqa-webuid -b0
srp 18 13:25:25 dhcp165.suse.cz systemd[1]: Started The openQA web UI.
srp 18 13:25:26 dhcp165.suse.cz openqa-webui-daemon[17382]: [2020-08-18 13:25:26.96072] [17382] [warn] Deprecated use of config key '[audit]: blacklist'. Use '[audit]: blocklist' instead
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://127.0.0.1:9526
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://[::1]:9526
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 14, near "include_branding 'docbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 17, near "include_branding 'sponsorbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Mojo::Reactor::Poll: I/O watcher failed: syntax error at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Context:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 30: } );
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 31: % end
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 32:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 33: <link rel="icon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 34: type="image/png" sizes="16x16"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 35: href="<%= icon_url 'logo-16.png' %>">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 36: <link rel="icon" href="<%= icon_url 'logo.svg'%>" sizes="any" type="image/svg+xml">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 37:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 38: </head>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 39: <body>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 40: <nav class="navbar navbar-static-top navbar-default">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Traceback (most recent call first):
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 163, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 173, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPLRenderer.pm", line 40, in "Mojolicious::Plugin::EPLRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPRenderer.pm", line 39, in "Mojolicious::Plugin::EPRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 221, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 110, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 152, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 164, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 123, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 110, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 50, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 70, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 203, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 15, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 18, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 141, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server.pm", line 66, in "Mojo::Server"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 103, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Transaction/HTTP.pm", line 60, in "Mojo::Transaction::HTTP"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 218, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 199, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 109, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 57, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 60, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 103, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop.pm", line 133, in "Mojo::IOLoop"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 152, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 93, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 78, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Command/prefork.pm", line 31, in "Mojolicious::Command::prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Commands.pm", line 57, in "Mojolicious::Commands"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 186, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/../lib/OpenQA/WebAPI.pm", line 493, in "OpenQA::WebAPI"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/openqa", line 35, in "main"
</code></pre> openQA Project - action #69700 (New): Predefined QEMU hardware profiles in os-autoinsthttps://progress.opensuse.org/issues/697002020-08-07T10:15:47ZMDouchamartin.doucha@suse.com
<p>We've recently had a <a href="https://bugzilla.suse.com/show_bug.cgi?id=1174887" class="external">regression</a> that made our kernels unbootable in QEMU VMs created in virt-manager. The regression was missed by OpenQA tests because os-autoinst uses VMs with minimal hardware configuration which didn't trigger the bug.</p>
<p>We should define multiple QEMU hardware profiles (named sets of extra device options for QEMU) which can then be selected through job settings. The hardware profiles don't need to cover every possible combination of devices, it'll be enough if each device model appears in them at least once. One of the profiles should be as close to virt-manager defaults as possible. Then it'll be sufficient to boot the existing LTP jobs on different hardware profiles. We don't need any extra tests beyond checking that the kernel is bootable.</p>
<p>Example profile that would trigger the regression:</p>
<pre><code>-machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1
-device pcie-pci-bridge,id=pci.9,bus=pci.2,addr=0x0
</code></pre> openQA Infrastructure - action #61994 (Resolved): VNC console corruption on aarch64https://progress.opensuse.org/issues/619942020-01-10T09:46:39ZMDouchamartin.doucha@suse.com
<p>A random problem sometimes appears on aarch64 test machines where the VM screen isn't properly cleared after boot and console output gets drawn over remnants of boot splash screen. Then the job fails because needles don't match. The problem appears less than once a week and job restart usually fixes it but it might be worth investigating further.<br>
<a href="https://openqa.suse.de/tests/3773959#step/update_kernel/6" class="external">https://openqa.suse.de/tests/3773959#step/update_kernel/6</a></p>