openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-15T10:13:41ZopenSUSE Project Management Tool
Redmine openQA Project - action #157333 (Closed): Log all job setting changes in autoinst-log.txthttps://progress.opensuse.org/issues/1573332024-03-15T10:13:41ZMDouchamartin.doucha@suse.com
<p>All job settings should be logged in autoinst-log.txt with source of the value (e.g. the place where <code>set_var()</code> was called or whether they were added from product/medium/worker etc.)</p>
openQA Project - action #150917 (Resolved): Restarting a job together with failed children will b...https://progress.opensuse.org/issues/1509172023-11-15T15:06:53ZMDouchamartin.doucha@suse.com
<p>Restarting a job together with failed children will break dependencies of the new job</p>
<p>Restarting a job with parent/child jobs using the <code>Skip restarting "OK" (passed/softfailed) children</code> drop-down link will break dependencies of the new job. In addition to the restarted children, it'll also link to all parents and children of the original job:<br>
<a href="https://openqa.suse.de/tests/12817616#dependencies" class="external">https://openqa.suse.de/tests/12817616#dependencies</a></p>
<p>The broken dependencies then prevent further restarts.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: After restarting a chained parent (using "Skip restarting OK" but actually regardless of the option) none of the chained children have two parents</li>
<li><strong>AC2</strong>: The same counts for other dependency types.</li>
</ul>
<a name="Suggestion"></a>
<h2 >Suggestion<a href="#Suggestion" class="wiki-anchor">¶</a></h2>
<ul>
<li>Find out why we have not seen this in before. Does nobody else use that or care about it?</li>
<li>Replicate the scenario by extending/adjusting the unit tests</li>
<li>Clone the scenario locally to test this for real (no reason to run any jobs, just fake results via DB updates)</li>
</ul>
openQA Infrastructure - action #125798 (Resolved): Visual differences in GRUB menu on different x...https://progress.opensuse.org/issues/1257982023-03-10T17:09:54ZMDouchamartin.doucha@suse.com
<p>Here are 5 different LTP jobs booting the exact same UEFI/SecureBoot QCOW image on different workers:<br>
<a href="https://openqa.suse.de/tests/10651590" class="external">https://openqa.suse.de/tests/10651590</a> openqaworker16:14, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10658203" class="external">https://openqa.suse.de/tests/10658203</a> openqaworker16:18, pass<br>
<a href="https://openqa.suse.de/tests/10659306" class="external">https://openqa.suse.de/tests/10659306</a> openqaworker16:7, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659346" class="external">https://openqa.suse.de/tests/10659346</a> openqaworker17:12, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659359" class="external">https://openqa.suse.de/tests/10659359</a> worker9:11, pass</p>
<p>It appears that GRUB menu size changes depending on not just the worker but also specific worker slot.</p>
<p>Possibly related to <a href="https://progress.opensuse.org/issues/114523" class="external">poo#114523</a> but this time it's happening on x86_64.</p>
openQA Project - action #124493 (Resolved): openqa-clone-job --skip-deps behavior contradicts doc...https://progress.opensuse.org/issues/1244932023-02-14T14:49:46ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Both <a href="http://open.qa/docs/#_handling_of_dependencies_when_cloning_jobs" class="external">OpenQA documentation</a> and <code>openqa-clone-job --help</code> say that <code>--skip-deps</code> and <code>--skip-chained-deps</code> should only prevent cloning of <strong>parent</strong> jobs. In reality, however, both options will prevent cloning of all (chained) dependencies regardless of parent/child relationship (even when you specify <code>--clone-children</code>). This means there is currently no way to clone a dependency subtree without parents using <code>openqa-clone-job</code>. The subtree can only be restarted in webUI which does not support modifying settings of the restarted jobs.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> There is a way to clone a dependency subtree without parents using <code>openqa-clone-job</code> (in accordance with the documentation).</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>It probably worked in the past, maybe a regression?</li>
<li>Create a set of dependent jobs locally (e.g. by setting dependencies manually within the database or by cloning a set of jobs from production) and run <code>openqa-clone-job</code> locally with parameters mention in description</li>
<li>Extend unit tests</li>
</ul>
openQA Project - action #124469 (Resolved): Allow partial product retrigger size:Mhttps://progress.opensuse.org/issues/1244692023-02-14T10:14:42ZMDouchamartin.doucha@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Fixing job failures sometimes requires editing medium and testsuite settings. It'd be useful to have a job restart option that'll behave like partial <code>isos post</code> but only for the target job and its descendants, without restarting any parent jobs or parallel job dependency branches. The restarted jobs would be created from scratch using the original <code>isos post</code> settings and the current testsuite/medium/job group configuration. Unlike normal restart, job settings of the original failed/cancelled jobs would be ignored.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: It is clear how the partial product re-trigger is supposed to work (how the "part" is specified)</li>
<li><strong>AC2</strong>: A solution exists to re-trigger a subset of tests re-evaluating scheduling settings (and not just re-triggering with the same settings)</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Follow comments in the ticket</li>
</ul>
openQA Infrastructure - action #120339 (Resolved): QEMU DNS fails to resolve openqa.suse.de via I...https://progress.opensuse.org/issues/1203392022-11-11T12:48:43ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h1 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h1>
<p>LTP test <code>host</code> <a href="https://openqa.suse.de/tests/9927713#step/host/8" class="external">started failing today</a>. The QEMU DNS service running at 10.0.2.3 correctly resolves hostnames to IP addresses but reverse lookup fails. <a href="https://openqa.suse.de/tests/9915478#step/host/8" class="external">Old tests</a> which passed up until yesterday are now <a href="https://openqa.suse.de/tests/9930979#step/host/8" class="external">also failing upon restart</a> so this appears to be a QEMU configuration issue. The physical worker machine can resolve IP addresses without issue.</p>
<p>This issue is confirmed on worker3, worker5, worker8 and worker13. Other workers may be affected as well. PPC64LE QEMU workers do not seem to be affected, though.</p>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Revert removal of faulty DNS</li>
</ul>
<pre><code>sudo salt --no-color --state-output=changes -C 'G@roles:worker' cmd.run 'sudo sed -i "s/\(NETCONFIG_DNS_POLICY=\)\"\"/\1\"auto\"/;s/\(NETCONFIG_DNS_STATIC_SERVERS=\)\"10.160.0.1 10.100.2.10\"/\1\"\"/" /etc/sysconfig/network/config && sudo netconfig update -f'
</code></pre> openQA Project - action #119461 (Resolved): Serial failure autodetection overrides test result wh...https://progress.opensuse.org/issues/1194612022-10-26T16:09:53ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>The serial failure detection code in <a href="https://github.com/os-autoinst/os-autoinst/blob/2156ecc810939cf3bce8ca705cf39423dd6c194d/basetest.pm#L632" class="external">basetest::parse_serial_output_qemu()</a> unconditionally changes <code>$self->{result}</code> which can result in test failure being turned into softfail or even pass. Result should be changed only when the new value is a "higher level" of failure than the current value (e.g. <code>softfail</code> -> <code>fail</code>, but never <code>fail</code> -> <code>softfail</code>).<br>
Example: <a href="https://openqa.suse.de/tests/9481132#step/nfs41_01/12" class="external">https://openqa.suse.de/tests/9481132#step/nfs41_01/12</a></p>
<p>The nfs41_01 test in the example job failed with exit code 137 (SIGKILL) but the CPU soft lockup box at the end (serial failure with type <code>info</code>) changed the result to <code>pass</code>. This continued in all subsequent test modules until the job timed out.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Test module results are never reset to a lower level</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Extend tests in t/17-basetest.t with the updated expectation (The main bunch of tests was added in 2018 by riafarov who is not active anymore)</li>
<li>Investigate the code path in <code>basetest::parse_serial_output_qemu</code></li>
<li>Update the implementation</li>
</ul>
openQA Infrastructure - action #107989 (Resolved): CPU-specific worker classeshttps://progress.opensuse.org/issues/1079892022-03-08T11:28:38ZMDouchamartin.doucha@suse.com
<p>We have a few tests which require specific CPU types, for example the CPU vulnerability mitigation tests. It'd be useful to have worker classes like <code>x86_64_amd</code> and <code>x86_64_intel</code> so that we can schedule tests on workers which have the required features or vulnerabilities.</p>
openQA Project - action #106898 (Resolved): Protection against asset clobberinghttps://progress.opensuse.org/issues/1068982022-02-16T10:33:47ZMDouchamartin.doucha@suse.com
<p>QCOW images in OpenQA occasionally get corrupted because multiple jobs try to publish the same file at the same time, either due to <code>PUBLISH_*</code> setting misconfiguration or duplicate install jobs scheduled in parallel. For example, this job failed to start:<br>
<a href="https://openqa.suse.de/tests/8162749" class="external">https://openqa.suse.de/tests/8162749</a></p>
<p>because these three install jobs finished 20 minutes apart and tried to upload the same QCOW image:<br>
<a href="https://openqa.suse.de/tests/8162347" class="external">https://openqa.suse.de/tests/8162347</a><br>
<a href="https://openqa.suse.de/tests/8161501" class="external">https://openqa.suse.de/tests/8161501</a><br>
<a href="https://openqa.suse.de/tests/8160547" class="external">https://openqa.suse.de/tests/8160547</a></p>
<p>Please add some sort of protection against asset clobbering via <code>PUBLISH_*</code> variables:</p>
<ul>
<li>two jobs must not publish the same file in parallel</li>
<li>jobs must not publish a file while another job may be downloading the previous version</li>
<li><code>PUBLISH_*</code> misconfiguration (e.g. copy-paste mistakes among multiple testsuites) should be detected and reported in the WebUI, for example as the reason why install job was terminated</li>
</ul>
openQA Project - action #99246 (Resolved): Published QCOW images appear to be uncompressedhttps://progress.opensuse.org/issues/992462021-09-24T11:41:40ZMDouchamartin.doucha@suse.com
<p>QCOW images generated by OpenQA install jobs appear to be uncompressed now.</p>
<p>ppc64le: <a href="https://openqa.suse.de/tests/7211627#downloads" class="external">https://openqa.suse.de/tests/7211627#downloads</a><br>
HDD_1: 885MB<br>
PUBLISH_HDD_1: 7.1GB</p>
<p>x86_64: <a href="https://openqa.suse.de/tests/7211569#downloads" class="external">https://openqa.suse.de/tests/7211569#downloads</a><br>
HDD_1: 976MB<br>
PUBLISH_HDD_1: 4.2GB</p>
<p>The size difference should be only a few hundred megabytes at most.</p>
openQA Project - action #98841 (Resolved): qemu randomly fails to start on QA-Power8-5-kvm auto_r...https://progress.opensuse.org/issues/988412021-09-17T15:49:43ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>A few LTP jobs have failed to start today due to qemu error on QA-Power8-5-kvm worker:<br>
<a href="https://openqa.suse.de/tests/7138972">https://openqa.suse.de/tests/7138972</a><br>
<a href="https://openqa.suse.de/tests/7149857">https://openqa.suse.de/tests/7149857</a><br>
<a href="https://openqa.suse.de/tests/7153989">https://openqa.suse.de/tests/7153989</a></p>
<p>All of them have the following output in autoinst-log.txt:</p>
<pre><code>[2021-09-17T16:24:29.803 CEST] [info] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
QEMU terminated before QMP connection could be established. Check for errors below
[2021-09-17T16:24:29.804 CEST] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2021-09-17T16:24:29.805 CEST] [debug] Passing remaining frames to the video encoder
[2021-09-17T16:24:29.805 CEST] [debug] Waiting for video encoder to finalize the video
[2021-09-17T16:24:29.805 CEST] [debug] The built-in video encoder (pid 110385) terminated
[2021-09-17T16:24:29.807 CEST] [debug] QEMU: QEMU emulator version 4.2.1 (openSUSE Leap 15.2)
[2021-09-17T16:24:29.807 CEST] [debug] QEMU: Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
[2021-09-17T16:24:29.807 CEST] [warn] !!! : qemu-system-ppc64: Failed to allocate KVM HPT of order 25 (try smaller maxmem?): Cannot allocate memory
</code></pre>
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>QA-Power8-5-kvm has 256GB RAM. <a href="https://monitor.qa.suse.de/d/WDQA-Power8-5-kvm/worker-dashboard-qa-power8-5-kvm?viewPanel=12054&orgId=1&from=1631765162464&to=1632085860553">https://monitor.qa.suse.de/d/WDQA-Power8-5-kvm/worker-dashboard-qa-power8-5-kvm?viewPanel=12054&orgId=1&from=1631765162464&to=1632085860553</a> shows that some memory was used during the period when the test failed but nothing that should explain the inability to allocate the memory for the qemu VM. In the system journal there is</p>
<pre><code>Sep 17 16:24:28 QA-Power8-5-kvm worker[88148]: [debug] [pid:88148] REST-API call: POST http://openqa.suse.de/api/v1/jobs/7125263/status
Sep 17 16:24:29 QA-Power8-5-kvm worker[104911]: [info] [pid:108741] sle-15-SP4-ppc64le-Build36.1-HA-BV.qcow2: Processing chunk 501/5812, avg. speed ~976.562 KiB/s
Sep 17 16:24:29 QA-Power8-5-kvm worker[101413]: [debug] [pid:102598] Uploading artefact mq_timedreceive_15-1-2.txt
Sep 17 16:24:29 QA-Power8-5-kvm worker[96737]: [debug] [pid:96737] REST-API call: POST http://openqa.suse.de/api/v1/jobs/7125265/status
Sep 17 16:24:29 QA-Power8-5-kvm worker[109458]: [debug] [pid:110336] Uploading artefact bootloader_start-15.txt
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: 23 callbacks suppressed
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf00, 3cb100) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf04, 3cb104) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf08, 3cb108) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf0c, 3cb10c) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf10, 3cb110) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf14, 3cb114) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf18, 3cb118) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf1c, 3cb11c) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf20, 3cb120) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf24, 3cb124) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: cma: cma_alloc: alloc failed, req-size: 512 pages, ret: -16
Sep 17 16:24:30 QA-Power8-5-kvm worker[88148]: [debug] [pid:88148] Upload concluded (at wait_children)
Sep 17 16:24:30 QA-Power8-5-kvm worker[109557]: [info] [pid:109557] Isotovideo exit status: 1
Sep 17 16:24:30 QA-Power8-5-kvm worker[109557]: [debug] [pid:109557] Stopping job 7153989 from openqa.suse.de: 07153989-sle-15-SP3-Server-DVD-Incidents-Kernel-KOTD-ppc64le-Build5.3.18-302.1.g316993b-ltp_crashme@ppc64le-virtio - reason: died
Sep 17 16:24:30 QA-Power8-5-kvm worker[109557]: [debug] [pid:109557] REST-API call: POST http://openqa.suse.de/api/v1/jobs/7153989/status
Sep 17 16:24:30 QA-Power8-5-kvm worker[101413]: [debug] [pid:102598] Uploading artefact mq_timedreceive_7-1-2.txt
</code></pre>
<p>in particular the messages</p>
<pre><code>Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf20, 3cb120) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: alloc_contig_range: [3caf24, 3cb124) PFNs busy
Sep 17 16:24:29 QA-Power8-5-kvm kernel: cma: cma_alloc: alloc failed, req-size: 512 pages, ret: -16
</code></pre>
<p>so an allocation failure. We could report a bug about this but because KVM on SUSE with Power8 is unsupported so I don't expect any success.</p>
<p>We likely need to accept such issues and trigger a restart automatically by openQA.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> qemu ppc64le allocation errors cause automatic job retriggers by openQA</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Catch the error and make it "Incomplete"</li>
<li>Restart the incomplete job</li>
<li>Make openQA automatically detect the issue and trigger restart, e.g. based on <a href="https://github.com/os-autoinst/openQA/blob/master/etc/openqa/openqa.ini#L76">https://github.com/os-autoinst/openQA/blob/master/etc/openqa/openqa.ini#L76</a></li>
</ul>
openQA Project - action #96007 (Resolved): OpenQA jobs randomly time out during setup phasehttps://progress.opensuse.org/issues/960072021-07-26T10:23:59ZMDouchamartin.doucha@suse.com
<p>OpenQA jobs have been incompleting more than usual in the past few weeks. The incompletes I've seen just today all show the following sequence of messages in worker.log:</p>
<pre><code>[2021-07-24T16:28:45.444 CEST] [debug] started mgmt loop with pid 59928
[2021-07-24T16:28:45.510 CEST] [debug] qemu version detected: 4.2.1
[2021-07-24T16:28:45.512 CEST] [debug] running /usr/bin/chattr -f +C /var/lib/openqa/pool/9/raid
[2021-07-24T18:28:42.557 CEST] [debug] isotovideo received signal TERM
[2021-07-24T18:28:42.558 CEST] [debug] backend got TERM
</code></pre>
<p><a href="https://openqa.suse.de/tests/6552459" class="external">https://openqa.suse.de/tests/6552459</a><br>
<a href="https://openqa.suse.de/tests/6555414" class="external">https://openqa.suse.de/tests/6555414</a><br>
<a href="https://openqa.suse.de/tests/6543695" class="external">https://openqa.suse.de/tests/6543695</a></p>
<p>I'll update this ticket if I find any similar jobs where the last operation before timeout isn't <code>chattr -f +C</code>.</p>
openQA Project - action #94850 (Resolved): QEMU 6.0 fails to start if job has QEMU_NUMA=1https://progress.opensuse.org/issues/948502021-06-29T10:05:52ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Originally reported by pvorel relating to <a href="http://quasar.suse.cz/tests/6860#">http://quasar.suse.cz/tests/6860#</a></p>
<p>It appears that QEMU 6.0 requires explicit memory assignment to NUMA nodes using <code>-numa node,mem=...</code> command line arguments. OpenQA jobs with <code>QEMU_NUMA=1</code> currently fail to start on Tumbleweed because os-autoinst omits the <code>mem</code> option, which leads to the following error:</p>
<pre><code>[2021-06-28T13:26:00.308 CEST] [debug] starting: /usr/bin/qemu-system-x86_64 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 2048 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot order=c -device usb-ehci -device usb-tablet -smp 2 -numa node,nodeid=0 -numa node,nodeid=1 -enable-kvm -no-shutdown -vnc :91,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-overlay0-file,filename=/var/lib/openqa/pool/1/raid/hd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0-overlay0,file=hd0-overlay0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0-overlay0,bootindex=0,serial=hd0
[2021-06-28T13:26:00.312 CEST] [debug] Waiting for 0 attempts
[2021-06-28T13:26:00.423 CEST] [debug] Waiting for 1 attempts
[2021-06-28T13:26:00.424 CEST] [info] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 453.
[2021-06-28T13:26:00.424 CEST] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2021-06-28T13:26:00.424 CEST] [debug] Passing remaining frames to the video encoder
[2021-06-28T13:26:00.426 CEST] [debug] Waiting for video encoder to finalize the video
[2021-06-28T13:26:00.426 CEST] [debug] The built-in video encoder (pid 22047) terminated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: QEMU emulator version 6.0.0 (openSUSE Tumbleweed)
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on: warning: short-form boolean option 'server' deprecated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Please use server=on instead
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on: warning: short-form boolean option 'nowait' deprecated
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: Please use wait=off instead
[2021-06-28T13:26:00.427 CEST] [debug] QEMU: qemu-system-x86_64: total memory for NUMA nodes (0x0) should equal RAM size (0x80000000)
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> os-autoinst continues to start for qemu >= 6.0 and QEMU_NUMA=1</li>
<li><strong>AC2:</strong> os-autoinst jobs still work fine on qemu < 6.0 regardless of QEMU_NUMA setting</li>
</ul>
openQA Project - action #70189 (Rejected): openQA-common package broken on Tumbleweedhttps://progress.opensuse.org/issues/701892020-08-18T11:40:34ZMDouchamartin.doucha@suse.com
<p>When I install the standard <code>openQA-common</code> package on Tumbleweed and try to access the local webUI, I get error 502 and <code>journalctl</code> shows Perl errors with traceback (see below). When I replace the <code>/usr/share/openqa</code> directory with a symlink to my local copy of OpenQA git repo and restart the OpenQA services, everything works fine.</p>
<pre><code>$ journalctl -u openqa-webuid -b0
srp 18 13:25:25 dhcp165.suse.cz systemd[1]: Started The openQA web UI.
srp 18 13:25:26 dhcp165.suse.cz openqa-webui-daemon[17382]: [2020-08-18 13:25:26.96072] [17382] [warn] Deprecated use of config key '[audit]: blacklist'. Use '[audit]: blocklist' instead
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://127.0.0.1:9526
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://[::1]:9526
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 14, near "include_branding 'docbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 17, near "include_branding 'sponsorbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Mojo::Reactor::Poll: I/O watcher failed: syntax error at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Context:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 30: } );
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 31: % end
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 32:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 33: <link rel="icon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 34: type="image/png" sizes="16x16"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 35: href="<%= icon_url 'logo-16.png' %>">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 36: <link rel="icon" href="<%= icon_url 'logo.svg'%>" sizes="any" type="image/svg+xml">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 37:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 38: </head>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 39: <body>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 40: <nav class="navbar navbar-static-top navbar-default">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Traceback (most recent call first):
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 163, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 173, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPLRenderer.pm", line 40, in "Mojolicious::Plugin::EPLRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPRenderer.pm", line 39, in "Mojolicious::Plugin::EPRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 221, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 110, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 152, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 164, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 123, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 110, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 50, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 70, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 203, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 15, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 18, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 141, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server.pm", line 66, in "Mojo::Server"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 103, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Transaction/HTTP.pm", line 60, in "Mojo::Transaction::HTTP"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 218, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 199, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 109, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 57, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 60, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 103, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop.pm", line 133, in "Mojo::IOLoop"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 152, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 93, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 78, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Command/prefork.pm", line 31, in "Mojolicious::Command::prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Commands.pm", line 57, in "Mojolicious::Commands"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 186, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/../lib/OpenQA/WebAPI.pm", line 493, in "OpenQA::WebAPI"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/openqa", line 35, in "main"
</code></pre> openQA Infrastructure - action #61994 (Resolved): VNC console corruption on aarch64https://progress.opensuse.org/issues/619942020-01-10T09:46:39ZMDouchamartin.doucha@suse.com
<p>A random problem sometimes appears on aarch64 test machines where the VM screen isn't properly cleared after boot and console output gets drawn over remnants of boot splash screen. Then the job fails because needles don't match. The problem appears less than once a week and job restart usually fixes it but it might be worth investigating further.<br>
<a href="https://openqa.suse.de/tests/3773959#step/update_kernel/6" class="external">https://openqa.suse.de/tests/3773959#step/update_kernel/6</a></p>