openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-15T10:13:41ZopenSUSE Project Management Tool
Redmine openQA Project - action #157333 (Closed): Log all job setting changes in autoinst-log.txthttps://progress.opensuse.org/issues/1573332024-03-15T10:13:41ZMDouchamartin.doucha@suse.com
<p>All job settings should be logged in autoinst-log.txt with source of the value (e.g. the place where <code>set_var()</code> was called or whether they were added from product/medium/worker etc.)</p>
openQA Project - action #150917 (Resolved): Restarting a job together with failed children will b...https://progress.opensuse.org/issues/1509172023-11-15T15:06:53ZMDouchamartin.doucha@suse.com
<p>Restarting a job together with failed children will break dependencies of the new job</p>
<p>Restarting a job with parent/child jobs using the <code>Skip restarting "OK" (passed/softfailed) children</code> drop-down link will break dependencies of the new job. In addition to the restarted children, it'll also link to all parents and children of the original job:<br>
<a href="https://openqa.suse.de/tests/12817616#dependencies" class="external">https://openqa.suse.de/tests/12817616#dependencies</a></p>
<p>The broken dependencies then prevent further restarts.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: After restarting a chained parent (using "Skip restarting OK" but actually regardless of the option) none of the chained children have two parents</li>
<li><strong>AC2</strong>: The same counts for other dependency types.</li>
</ul>
<a name="Suggestion"></a>
<h2 >Suggestion<a href="#Suggestion" class="wiki-anchor">¶</a></h2>
<ul>
<li>Find out why we have not seen this in before. Does nobody else use that or care about it?</li>
<li>Replicate the scenario by extending/adjusting the unit tests</li>
<li>Clone the scenario locally to test this for real (no reason to run any jobs, just fake results via DB updates)</li>
</ul>
openQA Infrastructure - action #138746 (Resolved): [tools] s390x VM randomly fails to open QCOW d...https://progress.opensuse.org/issues/1387462023-10-30T12:45:10ZMDouchamartin.doucha@suse.com
<p>s390x tests randomly fail to boot because the VM does not have permission to open the disk image. Multiple workers have the same issue. Restarting the job usually fixes the issue. Examples:</p>
<p><a href="https://openqa.suse.de/tests/12711015#step/bootloader_zkvm/31" class="external">https://openqa.suse.de/tests/12711015#step/bootloader_zkvm/31</a><br>
<a href="https://openqa.suse.de/tests/12711015/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/12711015/logfile?filename=autoinst-log.txt</a></p>
<p><a href="https://openqa.suse.de/tests/12716015#step/bootloader_zkvm/31" class="external">https://openqa.suse.de/tests/12716015#step/bootloader_zkvm/31</a><br>
<a href="https://openqa.suse.de/tests/12716015/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/12716015/logfile?filename=autoinst-log.txt</a></p>
<p><a href="https://openqa.suse.de/tests/12708886#step/bootloader_start/34" class="external">https://openqa.suse.de/tests/12708886#step/bootloader_start/34</a><br>
<a href="https://openqa.suse.de/tests/12708886/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/12708886/logfile?filename=autoinst-log.txt</a></p>
<pre><code>[2023-10-28T00:17:57.550325+02:00] [debug] [pid:56810] [run_ssh_cmd(virsh start openQA-SUT-6 2> >(tee /tmp/os-autoinst-openQA-SUT-6-stderr.log >&2))] stderr:
error: Failed to start domain 'openQA-SUT-6'
error: internal error: process exited while connecting to monitor: 2023-10-27T22:17:57.331249Z qemu-system-s390x: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images//SLES-15-SP4-s390x-mru-install-minimal-with-addons-Build20231027-1-Server-DVD-Updates-s390x-kvm.qcow2","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images//SLES-15-SP4-s390x-mru-install-minimal-with-addons-Build20231027-1-Server-DVD-Updates-s390x-kvm.qcow2': Permission denied
</code></pre> openQA Infrastructure - action #125798 (Resolved): Visual differences in GRUB menu on different x...https://progress.opensuse.org/issues/1257982023-03-10T17:09:54ZMDouchamartin.doucha@suse.com
<p>Here are 5 different LTP jobs booting the exact same UEFI/SecureBoot QCOW image on different workers:<br>
<a href="https://openqa.suse.de/tests/10651590" class="external">https://openqa.suse.de/tests/10651590</a> openqaworker16:14, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10658203" class="external">https://openqa.suse.de/tests/10658203</a> openqaworker16:18, pass<br>
<a href="https://openqa.suse.de/tests/10659306" class="external">https://openqa.suse.de/tests/10659306</a> openqaworker16:7, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659346" class="external">https://openqa.suse.de/tests/10659346</a> openqaworker17:12, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659359" class="external">https://openqa.suse.de/tests/10659359</a> worker9:11, pass</p>
<p>It appears that GRUB menu size changes depending on not just the worker but also specific worker slot.</p>
<p>Possibly related to <a href="https://progress.opensuse.org/issues/114523" class="external">poo#114523</a> but this time it's happening on x86_64.</p>
openQA Project - action #124493 (Resolved): openqa-clone-job --skip-deps behavior contradicts doc...https://progress.opensuse.org/issues/1244932023-02-14T14:49:46ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Both <a href="http://open.qa/docs/#_handling_of_dependencies_when_cloning_jobs" class="external">OpenQA documentation</a> and <code>openqa-clone-job --help</code> say that <code>--skip-deps</code> and <code>--skip-chained-deps</code> should only prevent cloning of <strong>parent</strong> jobs. In reality, however, both options will prevent cloning of all (chained) dependencies regardless of parent/child relationship (even when you specify <code>--clone-children</code>). This means there is currently no way to clone a dependency subtree without parents using <code>openqa-clone-job</code>. The subtree can only be restarted in webUI which does not support modifying settings of the restarted jobs.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> There is a way to clone a dependency subtree without parents using <code>openqa-clone-job</code> (in accordance with the documentation).</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>It probably worked in the past, maybe a regression?</li>
<li>Create a set of dependent jobs locally (e.g. by setting dependencies manually within the database or by cloning a set of jobs from production) and run <code>openqa-clone-job</code> locally with parameters mention in description</li>
<li>Extend unit tests</li>
</ul>
openQA Project - action #124469 (Resolved): Allow partial product retrigger size:Mhttps://progress.opensuse.org/issues/1244692023-02-14T10:14:42ZMDouchamartin.doucha@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Fixing job failures sometimes requires editing medium and testsuite settings. It'd be useful to have a job restart option that'll behave like partial <code>isos post</code> but only for the target job and its descendants, without restarting any parent jobs or parallel job dependency branches. The restarted jobs would be created from scratch using the original <code>isos post</code> settings and the current testsuite/medium/job group configuration. Unlike normal restart, job settings of the original failed/cancelled jobs would be ignored.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: It is clear how the partial product re-trigger is supposed to work (how the "part" is specified)</li>
<li><strong>AC2</strong>: A solution exists to re-trigger a subset of tests re-evaluating scheduling settings (and not just re-triggering with the same settings)</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Follow comments in the ticket</li>
</ul>
openQA Infrastructure - action #123960 (Resolved): s390x tests fail to log into VNC console on wo...https://progress.opensuse.org/issues/1239602023-02-06T09:49:51ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>s390x tests started randomly failing last week when they try to log into the freshly booted test system. There are multiple instances across all SLE releases both before and after applying incident patches so this is most likely an infra issue.<br>
<a href="https://openqa.suse.de/tests/10427719#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10427719#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10427825#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10427825#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10432751#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10432751#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10424294#step/boot_ltp/21" class="external">https://openqa.suse.de/tests/10424294#step/boot_ltp/21</a></p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/10424537" class="external">:27598:kgraft-patch-SLE12-SP5_Update_35</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/10424525" class="external">:27599:kgraft-patch-SLE12-SP5_Update_36</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Server-DVD-Incidents-Kernel&machine=s390x-kvm-sle12&test=install_ltp%2Bsle%2BServer-DVD-Incidents-Kernel&version=12-SP5" class="external">latest</a></p>
QA - action #123748 (Resolved): [tools] Add support for excluding packages from test flavor in bo...https://progress.opensuse.org/issues/1237482023-01-27T12:53:19ZMDouchamartin.doucha@suse.com
<p>SLE-15SP4 livepatching channel will include packages for userspace livepatching which need standard single incident and aggregate tests. Incident scheduling logic in bot config therefore needs support for package exclusion so that the livepatching channel can be enabled for single incidents without flooding the job groups with kernel livepatch tests. Example:</p>
<pre><code>Server-DVD-Incidents:
archs:
- x86_64
issues:
...
exclude_packages:
- kernel-livepatch
</code></pre>
<p>Any incident that contains package with the given name (or name prefix) will be skipped for the parent flavor regardless of what else it contains.</p>
openQA Project - action #123664 (New): os-autoinst does not flush serial console buffer on snapsh...https://progress.opensuse.org/issues/1236642023-01-25T15:19:39ZMDouchamartin.doucha@suse.com
<p>When tests fail with kernel backtraces, the stale backtrace will be reported again after snapshot reload and trigger bogus failure even when the next test is successful.<br>
Example: <a href="https://openqa.suse.de/tests/10371180#step/cve-2017-1000111/13" class="external">https://openqa.suse.de/tests/10371180#step/cve-2017-1000111/13</a><br>
dmesg log: <a href="https://openqa.suse.de/tests/10371180/logfile?filename=serial0.txt" class="external">https://openqa.suse.de/tests/10371180/logfile?filename=serial0.txt</a></p>
<p>Here, the LTP test <code>cve-2017-18075</code> triggered kernel warning and failed. But the same kernel warning gets reported again at the end of test <code>cve-2017-1000111</code> which was successful and the dmesg log does not show any additional backtraces from it.</p>
<p>This appears to be os-autoinst regression because IIRC kernel backtrace detection used to work fine and only reported errors once.</p>
openQA Project - action #121774 (In Progress): LTP cgroup test appears to crash OpenQA worker ins...https://progress.opensuse.org/issues/1217742022-12-09T13:36:47ZMDouchamartin.doucha@suse.com
<p>LTP test cgroup_fj_stress_blkio_4_4_each on latest SLE-15SP1 KOTD kernel appears to crash the OpenQA worker instance it's running on. The test itself will succeed but the OpenQA job will stay stuck in <code>wait_serial()</code> for several hours (despite 90 second timeout) until the whole job fails on MAX_JOB_TIME. There are 3 examples so far:<br>
<a href="https://openqa.suse.de/tests/10089424#step/cgroup_fj_stress_blkio_4_4_each/7" class="external">https://openqa.suse.de/tests/10089424#step/cgroup_fj_stress_blkio_4_4_each/7</a><br>
<a href="https://openqa.suse.de/tests/10111009#step/cgroup_fj_stress_blkio_4_4_each/7" class="external">https://openqa.suse.de/tests/10111009#step/cgroup_fj_stress_blkio_4_4_each/7</a><br>
<a href="https://openqa.suse.de/tests/10113099#step/cgroup_fj_stress_blkio_4_4_each/7" class="external">https://openqa.suse.de/tests/10113099#step/cgroup_fj_stress_blkio_4_4_each/7</a></p>
<p>I've seen this issue only on SLE-15SP1 KOTD builds 156 and 157. I have not seen any cases on other SLE versions.</p>
<p>Typical autoinst-log.txt entries related to the timeout:</p>
<pre><code>[2022-12-06T08:52:27.432374+01:00] [debug] <<< testapi::script_run(cmd="vmstat -w", output="", quiet=undef, timeout=30, die_on_timeout=1)
[2022-12-06T08:52:27.432549+01:00] [debug] tests/kernel/run_ltp.pm:334 called testapi::script_run
[2022-12-06T08:52:27.432710+01:00] [debug] <<< testapi::wait_serial(record_output=undef, regexp="# ", quiet=undef, no_regex=1, buffer_size=undef, expect_not_found=0, timeout=90)
[2022-12-06T10:39:58.278597+01:00] [debug] autotest received signal TERM, saving results of current test before exiting
[2022-12-06T10:39:58.278622+01:00] [debug] isotovideo received signal TERM
[2022-12-06T10:39:58.278748+01:00] [debug] backend got TERM
</code></pre>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/10091628" class="external">4.12.14-150100.156.1.gb6c27ee</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Incidents-Kernel-KOTD&machine=64bit&test=ltp_controllers&version=15-SP1" class="external">latest</a></p>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce:<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<ol>
<li>Run <code>ltp_controllers</code> testsuite on SLE-15SP1 KOTD</li>
<li>Wait.</li>
</ol>
openQA Infrastructure - action #120339 (Resolved): QEMU DNS fails to resolve openqa.suse.de via I...https://progress.opensuse.org/issues/1203392022-11-11T12:48:43ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h1 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h1>
<p>LTP test <code>host</code> <a href="https://openqa.suse.de/tests/9927713#step/host/8" class="external">started failing today</a>. The QEMU DNS service running at 10.0.2.3 correctly resolves hostnames to IP addresses but reverse lookup fails. <a href="https://openqa.suse.de/tests/9915478#step/host/8" class="external">Old tests</a> which passed up until yesterday are now <a href="https://openqa.suse.de/tests/9930979#step/host/8" class="external">also failing upon restart</a> so this appears to be a QEMU configuration issue. The physical worker machine can resolve IP addresses without issue.</p>
<p>This issue is confirmed on worker3, worker5, worker8 and worker13. Other workers may be affected as well. PPC64LE QEMU workers do not seem to be affected, though.</p>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Revert removal of faulty DNS</li>
</ul>
<pre><code>sudo salt --no-color --state-output=changes -C 'G@roles:worker' cmd.run 'sudo sed -i "s/\(NETCONFIG_DNS_POLICY=\)\"\"/\1\"auto\"/;s/\(NETCONFIG_DNS_STATIC_SERVERS=\)\"10.160.0.1 10.100.2.10\"/\1\"\"/" /etc/sysconfig/network/config && sudo netconfig update -f'
</code></pre> openQA Project - action #119461 (Resolved): Serial failure autodetection overrides test result wh...https://progress.opensuse.org/issues/1194612022-10-26T16:09:53ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>The serial failure detection code in <a href="https://github.com/os-autoinst/os-autoinst/blob/2156ecc810939cf3bce8ca705cf39423dd6c194d/basetest.pm#L632" class="external">basetest::parse_serial_output_qemu()</a> unconditionally changes <code>$self->{result}</code> which can result in test failure being turned into softfail or even pass. Result should be changed only when the new value is a "higher level" of failure than the current value (e.g. <code>softfail</code> -> <code>fail</code>, but never <code>fail</code> -> <code>softfail</code>).<br>
Example: <a href="https://openqa.suse.de/tests/9481132#step/nfs41_01/12" class="external">https://openqa.suse.de/tests/9481132#step/nfs41_01/12</a></p>
<p>The nfs41_01 test in the example job failed with exit code 137 (SIGKILL) but the CPU soft lockup box at the end (serial failure with type <code>info</code>) changed the result to <code>pass</code>. This continued in all subsequent test modules until the job timed out.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Test module results are never reset to a lower level</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Extend tests in t/17-basetest.t with the updated expectation (The main bunch of tests was added in 2018 by riafarov who is not active anymore)</li>
<li>Investigate the code path in <code>basetest::parse_serial_output_qemu</code></li>
<li>Update the implementation</li>
</ul>
openQA Tests - action #116287 (Rejected): [qe-core][s390x] SSH serial terminal connection issues ...https://progress.opensuse.org/issues/1162872022-09-06T13:54:08ZMDouchamartin.doucha@suse.com
<p>s390x livepatch tests had a lot of installation failures this month due to SSH serial terminal connection failures. Interestingly enough, the connection failures seem to happen around the same module step. serial_terminal.txt output appears to be out of sync with the terminal because part of the commands and output is missing even though it's listed in the update_kernel module details. The dmesg output in serial0.txt often (but not always) shows some key exchange SSH error followed by output from a completely different job:</p>
<pre><code>Welcome to SUSE Linux Enterprise Server 15 SP2 (s390x) - Kernel 5.3.18-24.83-default (ttysclp0).
eth0: 10.161.145.86 fe80::5054:ff:fe84:f877
susetest login: root
Password:
Last login: Mon Sep 5 10:18:10 from 10.160.0.147
susetest:~ #�(B systemctl is-active network
active
susetest:~ #�(B systemctl is-active sshd
active
susetest:~ #�(B 2022-09-05T10:25:03.604370-04:00 susetest sshd[4272]: error: kex_exchange_identification: Connection closed by remote host
2022-09-05T10:25:04.844743-04:00 susetest sshd[4273]: error: kex_exchange_identification: Connection closed by remote host
[ 107.444474] LTP: starting DI000 (dirty)
[ 107.445525] LTP: starting DS000 (dio_sparse)
[ 107.466125] LTP: starting abort01
[ 107.758318] LTP: starting accept01
</code></pre>
<p>12-SP4: <a href="https://openqa.suse.de/tests/9438804#step/update_kernel/337" class="external">https://openqa.suse.de/tests/9438804#step/update_kernel/337</a><br>
15-SP2: <a href="https://openqa.suse.de/tests/9457752#step/update_kernel/337" class="external">https://openqa.suse.de/tests/9457752#step/update_kernel/337</a><br>
15-SP3: <a href="https://openqa.suse.de/tests/9458645#step/update_kernel/337" class="external">https://openqa.suse.de/tests/9458645#step/update_kernel/337</a><br>
15-SP4: <a href="https://openqa.suse.de/tests/9455666#step/update_kernel/199" class="external">https://openqa.suse.de/tests/9455666#step/update_kernel/199</a></p>
<p>I could not find any such connection failure on SLE-12SP5. Other SLE releases don't support s390x livepatches and KOTD tests don't show this kind of issue. This looks like a kernel bug but I'd like an s390x expert to look at this before I create a Bugzilla ticket. And of course this has exposed logging issues in OpenQA.</p>
openQA Infrastructure - action #115925 (New): aarch64: Random QEMU failures while retrieving host...https://progress.opensuse.org/issues/1159252022-08-29T08:44:02ZMDouchamartin.doucha@suse.com
<p>Since the worker upgrade to Leap 15.4, some aarch64 jobs have randomly failed with the following error: <code>qemu-system-aarch64: Failed to retrieve host CPU features</code><br>
Example: <a href="https://openqa.suse.de/tests/9401654" class="external">https://openqa.suse.de/tests/9401654</a></p>
openQA Project - action #114643 (New): Add support for virtio keyboard and mouse on aarch64 QEMUhttps://progress.opensuse.org/issues/1146432022-07-25T12:33:10ZMDouchamartin.doucha@suse.com
<p>QEMU aarch64 VMs are currently hardcoded to use USB keyboard in OpenQA. We now need to test SLE-15SP4 kernel-azure where this does not work because the whole USB subsystem is intentionally disabled and therefore the framebuffer console gets no keyboard input:<br>
<a href="https://openqa.suse.de/tests/9122772#step/update_kernel/95" class="external">https://openqa.suse.de/tests/9122772#step/update_kernel/95</a></p>
<p>I can get the tests to work by setting <code>QEMU_APPEND=device virtio-keyboard -device virtio-mouse</code>. Please implement proper support for virtio input devices in the QEMU backend.</p>