openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-09-22T06:11:38ZopenSUSE Project Management Tool
Redmine openQA Tests - action #136298 (Resolved): Leap Micro test fails in transactional_updatehttps://progress.opensuse.org/issues/1362982023-09-22T06:11:38Zjlausuchjalausuch@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario leap-micro-5.5-MicroOS-Image-x86_64-microos_image_default@64bit fails in<br>
<a href="https://openqa.opensuse.org/tests/3591732/modules/transactional_update/steps/96" class="external">transactional_update</a></p>
<p>As the job says, there should be an expected change when <a href="https://openqa.opensuse.org/tests/3591732#step/transactional_update/23" class="external">installing the local rpm</a><br>
<code>transactional-update -n ptf install update-test-trivial/update-test-security-5.1-1.20.x86_64.rpm</code><br>
and <a href="https://openqa.opensuse.org/tests/3591732#step/transactional_update/73" class="external">adding the utt repo</a> along with <code>transactional-update -n cleanup up</code>. </p>
<p>The local package is <code>/data/microos/utt-opensuse-x86_64.tgz</code> and the repo is <code>/data/microos/utt-leap.repo</code>, both in the data directory:<br>
<a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/data/microos" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/data/microos</a></p>
<p>This was working on Leap Micro 5.4 (no existing jobs), so for some reason now the packages seem to be the same or the local rpm is already a higher version than the one in the repo, and zypper says <code>zypper: nothing to update</code>.</p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>MicroOS image boot with ignition disk and default tests. Default tests are transactional-update, rebootmgr, health_check, cockpit service and some other checks specific to MicroOS.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since the beginning.</p>
openQA Tests - action #129601 (Feedback): bootloader_uefi fails too often in aarch64https://progress.opensuse.org/issues/1296012023-05-19T05:49:51Zjlausuchjalausuch@suse.com
<p><a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/installation/bootloader_uefi.pm" class="external">installation/bootloader_uefi.pm</a> times out quite often because openQA does not have time to enter uefi grub2 configuration screen before the 10s grub timeout.</p>
<p>e.g. <br>
<a href="https://openqa.suse.de/tests/11157624#step/bootloader_uefi/11">https://openqa.suse.de/tests/11157624#step/bootloader_uefi/11</a><br>
<a href="https://openqa.suse.de/tests/11157620#step/bootloader_uefi/11">https://openqa.suse.de/tests/11157620#step/bootloader_uefi/11</a><br>
<a href="https://openqa.suse.de/tests/11157622#step/bootloader_uefi/11">https://openqa.suse.de/tests/11157622#step/bootloader_uefi/11</a></p>
<p>The culprit is that the first <code>bootloader-grub2</code> takes too long to match.</p>
<p>This is a successful run:</p>
<pre><code>[2023-05-17T22:29:48.135632+02:00] [debug] [pid:78975] <<< testapi::assert_screen(mustmatch=[
"bootloader-shim-import-prompt",
"bootloader-grub2"
], timeout=90)
[2023-05-17T22:29:49.928958+02:00] [warn] [pid:79369] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.82 seconds for 14 candidate needles - make your needles more specific
[2023-05-17T22:29:49.929428+02:00] [debug] [pid:79369] no match: 89.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:29:50.104691+02:00] [debug] [pid:79369] no change: 88.0s
[2023-05-17T22:29:50.588948+02:00] [debug] [pid:79369] no match: 88.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:29:51.105278+02:00] [debug] [pid:79369] no change: 87.0s
[2023-05-17T22:29:51.589146+02:00] [debug] [pid:79369] no match: 87.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:29:52.105972+02:00] [debug] [pid:79369] no change: 86.0s
[2023-05-17T22:29:52.589983+02:00] [debug] [pid:79369] no match: 86.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:00.232487+02:00] [warn] [pid:79369] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 7.13 seconds for 14 candidate needles - make your needles more specific
[2023-05-17T22:30:00.232953+02:00] [debug] [pid:79369] no match: 85.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:00.248237+02:00] [debug] [pid:79369] no change: 77.8s
[2023-05-17T22:30:00.733227+02:00] [debug] [pid:79369] no match: 77.8s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:01.236512+02:00] [debug] [pid:79369] no change: 76.9s
[2023-05-17T22:30:01.743323+02:00] [warn] [pid:79369] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.51 seconds for 14 candidate needles - make your needles more specific
[2023-05-17T22:30:01.743772+02:00] [debug] [pid:79369] no match: 76.9s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:09.594737+02:00] [warn] [pid:79369] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 7.36 seconds for 14 candidate needles - make your needles more specific
[2023-05-17T22:30:09.595227+02:00] [debug] [pid:79369] no match: 75.9s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:10.332028+02:00] [warn] [pid:79369] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.71 seconds for 14 candidate needles - make your needles more specific
[2023-05-17T22:30:10.332520+02:00] [debug] [pid:79369] no match: 68.5s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:11.328004+02:00] [warn] [pid:79369] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.72 seconds for 14 candidate needles - make your needles more specific
[2023-05-17T22:30:11.328472+02:00] [debug] [pid:79369] no match: 67.5s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:12.205373+02:00] [warn] [pid:79369] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.59 seconds for 14 candidate needles - make your needles more specific
[2023-05-17T22:30:12.205867+02:00] [debug] [pid:79369] no match: 66.5s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:19.704649+02:00] [warn] [pid:79369] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 7.09 seconds for 14 candidate needles - make your needles more specific
[2023-05-17T22:30:19.705116+02:00] [debug] [pid:79369] no match: 65.5s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-17T22:30:20.417002+02:00] [debug] [pid:78975] >>> testapi::_handle_found_needle: found bootmenu-jeos-20201204, similarity 1.00 @ 101/208
[2023-05-17T22:30:20.419694+02:00] [debug] [pid:78975] tests/installation/bootloader_uefi.pm:124 called bootloader_setup::uefi_bootmenu_params -> lib/bootloader_setup.pm:383 called testapi::send_key_until_needlematch
[2023-05-17T22:30:20.420160+02:00] [debug] [pid:78975] <<< testapi::check_screen(mustmatch="grub2-enter-edit-mode", timeout=0)
</code></pre>
<p>And this is a failed run:</p>
<pre><code>[2023-05-18T22:31:23.112731+02:00] [debug] [pid:20579] <<< testapi::assert_screen(mustmatch=[
"bootloader-shim-import-prompt",
"bootloader-grub2"
], timeout=90)
[2023-05-18T22:31:24.904170+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.82 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:24.904637+02:00] [debug] [pid:21382] no match: 89.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:25.081871+02:00] [debug] [pid:21382] no change: 88.0s
[2023-05-18T22:31:25.580143+02:00] [debug] [pid:21382] no match: 88.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:26.094924+02:00] [debug] [pid:21382] no change: 87.0s
[2023-05-18T22:31:26.569793+02:00] [debug] [pid:21382] no match: 87.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:27.083393+02:00] [debug] [pid:21382] no change: 86.0s
[2023-05-18T22:31:27.553981+02:00] [debug] [pid:21382] no match: 86.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:35.276260+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 7.19 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:35.276731+02:00] [debug] [pid:21382] no match: 85.0s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:36.083159+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.78 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:36.083646+02:00] [debug] [pid:21382] no match: 77.8s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:36.279755+02:00] [debug] [pid:21382] no change: 76.8s
[2023-05-18T22:31:37.020384+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.74 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:37.020854+02:00] [debug] [pid:21382] no match: 76.8s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:44.628366+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 7.35 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:44.628834+02:00] [debug] [pid:21382] no match: 75.8s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:45.401250+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.74 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:45.401731+02:00] [debug] [pid:21382] no match: 68.4s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:46.386500+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.74 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:46.386994+02:00] [debug] [pid:21382] no match: 67.4s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:47.387270+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.74 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:47.387869+02:00] [debug] [pid:21382] no match: 66.4s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:55.130059+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 7.48 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:55.130529+02:00] [debug] [pid:21382] no match: 65.4s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:55.715449+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.56 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:55.715931+02:00] [debug] [pid:21382] no match: 57.9s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:31:56.143795+02:00] [debug] [pid:21382] no change: 56.9s
[2023-05-18T22:31:56.669176+02:00] [warn] [pid:21382] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.52 seconds for 14 candidate needles - make your needles more specific
[2023-05-18T22:31:56.669652+02:00] [debug] [pid:21382] no match: 56.9s, best candidate: bootloader_uefi-20210917 (0.00)
[2023-05-18T22:32:04.606465+02:00] [debug] [pid:20579] >>> testapi::_handle_found_needle: found bootmenu-jeos-20201204, similarity 1.00 @ 101/208
[2023-05-18T22:32:04.610142+02:00] [debug] [pid:20579] tests/installation/bootloader_uefi.pm:124 called bootloader_setup::uefi_bootmenu_params -> lib/bootloader_setup.pm:383 called testapi::send_key_until_needlematch
[2023-05-18T22:32:04.610684+02:00] [debug] [pid:20579] <<< testapi::check_screen(mustmatch="grub2-enter-edit-mode", timeout=0)
</code></pre>
<p>So, in the first case, the needle takes around 25s to be matched after the VM is started, but in the second case it takes ~45s, which makes the grub to pick the default option and boot the system without a chance to go to the <code>grub2-enter-edit-mode</code> screen.</p>
openQA Tests - action #111010 (Resolved): type_password does not hide the string in autoinst-log.txthttps://progress.opensuse.org/issues/1110102022-05-12T11:30:11Zjlausuchjalausuch@suse.com
<p>I am doing some tests using <code>type_password</code> command to put some sensitive information into a file in a running system.<br>
According to the documentation [1]: <code>A convenience wrapper around type_string, which doesn’t log the string.</code></p>
<p>However, if I do:<br>
<code>type_password("echo 'yada yada' > /root/influxdb_conf\n");</code></p>
<p>This info is shown in autoinst-log.txt:</p>
<pre><code>[2022-05-12T13:05:09.897059+02:00] [debug] tests/jeos/image_info.pm:130 called testapi::type_password
[2022-05-12T13:05:09.897329+02:00] [debug] <<< testapi::type_string(string="echo 'yada yada' > /root/influxdb_conf\n", max_interval=100, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
</code></pre>
<p>This is using root-console. If I use serial_terminal, I get it in the <code>serial_terminal.txt</code> log:</p>
<pre><code>...
SCRIPT_FINISHEDGItcm-0-
# echo 'yada yada' > /root/influxdb_conf
</code></pre>
<p>Also, openQA tests are showing this </p>
<p>2 months ago (before <a href="https://github.com/os-autoinst/os-autoinst/pull/2002" class="external">https://github.com/os-autoinst/os-autoinst/pull/2002</a>)<br>
<a href="https://openqa.suse.de/tests/8405529/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/8405529/logfile?filename=autoinst-log.txt</a></p>
<pre><code>[2022-03-26T12:33:40.095557+01:00] [debug] tests/jeos/firstrun.pm:116 called testapi::type_password
[2022-03-26T12:33:40.095850+01:00] [debug] <<< testapi::type_string(string="SECRET STRING", max_interval=100, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
</code></pre>
<p>Now:<br>
<a href="https://openqa.suse.de/tests/8738443/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/8738443/logfile?filename=autoinst-log.txt</a></p>
<pre><code>[2022-05-12T06:09:58.666172+02:00] [debug] tests/jeos/firstrun.pm:116 called testapi::type_password
[2022-05-12T06:09:58.666561+02:00] [debug] <<< testapi::type_string(string="nots3cr3t", max_interval=100, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
</code></pre>
<p>[1] <a href="http://open.qa/api/testapi/#_type_password" class="external">http://open.qa/api/testapi/#_type_password</a></p>
openQA Tests - action #109882 (Resolved): Leap Micro 5.2 autoyast installation misses the repo-ma...https://progress.opensuse.org/issues/1098822022-04-12T22:14:27Zjlausuchjalausuch@suse.com
<p>After doing an <a href="https://openqa.opensuse.org/tests/2293631" class="external">autoyast installation</a>, the system doesn't have the repository </p>
<pre><code>install:~ # zypper lr -u
Repository priorities are without effect. All enabled repositories share the same priority.
# | Alias | Name | Enabled | GPG Check | Refresh | URI
--+---------------------------+---------------------------+---------+-----------+---------+--------------------------------------------------------
1 | openSUSE-Leap-Micro-5.2-1 | openSUSE-Leap-Micro-5.2-1 | No | ---- | ---- | cd:/?devices=/dev/disk/by-id/scsi-0QEMU_QEMU_CD-ROM_cd0
</code></pre>
<p>And because of this, the installation of some packages fail (e.g. <a href="https://openqa.opensuse.org/tests/2293631#step/cockpit_service/8" class="external">cockpit</a>)</p>
<p>This is how it should look like from the regular <a href="https://openqa.opensuse.org/tests/2294237" class="external">DVD installation</a>:</p>
<pre><code>localhost:~ # zypper lr -u
Repository priorities are without effect. All enabled repositories share the same priority.
# | Alias | Name | Enabled | GPG Check | Refresh | URI
--+---------------------------+----------------------------+---------+-----------+---------+----------------------------------------------------------------------------------------------------
1 | openSUSE-Leap-Micro-5.2-1 | openSUSE-Leap-Micro-5.2-1 | No | ---- | ---- | cd:/?devices=/dev/disk/by-id/scsi-0QEMU_QEMU_CD-ROM_cd0
2 | repo-main | Leap Micro Main Repository | Yes | (r ) Yes | Yes | https://download.opensuse.org/distribution/leap-micro/5.2/product/repo/Leap-Micro-5.2-x86_64-Media/
</code></pre>
<p>The used autoyast profile is <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/data/autoyast_opensuse/autoyast_leap-micro.xml.ep" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/data/autoyast_opensuse/autoyast_leap-micro.xml.ep</a></p>
openQA Tests - action #109046 (Resolved): [tools] auto_review:"Unable to find image SLES15-SP3-Je...https://progress.opensuse.org/issues/1090462022-03-28T06:23:08Zjlausuchjalausuch@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-15-SP4-Migration-from-SLE12-SPx-s390x-offline_sles12sp3_ltss_pscc_asmm-lgm_all_full@s390x-kvm-sle15 fails in<br>
<a href="https://openqa.suse.de/tests/8415599/modules/bootloader_zkvm/steps/18" class="external">bootloader_zkvm</a><br>
But the qcow exists:</p>
<pre><code>$ /var/lib/openqa/share/factory/hdd/fixed> ll SLES-12-SP3-s390x-GM-gnome-allpatterns.qcow2
-rw-r--r-- 1 geekotest nogroup 14485946368 Jan 14 08:45 SLES-12-SP3-s390x-GM-gnome-allpatterns.qcow2
</code></pre>
<p>And the asset is actually there:<br>
<img src="https://progress.opensuse.org/attachments/download/13001/hdd.png" alt="" loading="lazy" /></p>
<p>It happens in all jobs with <code>svirt</code> backend that have their HDD in fixed directory. So, all s390x jobs and some others that use that backend too:<br>
Examples:<br>
<a href="https://openqa.suse.de/tests/8414273" class="external">https://openqa.suse.de/tests/8414273</a><br>
<a href="https://openqa.suse.de/tests/8414268" class="external">https://openqa.suse.de/tests/8414268</a><br>
<a href="https://openqa.suse.de/tests/8414272" class="external">https://openqa.suse.de/tests/8414272</a><br>
<a href="https://openqa.suse.de/tests/8414267" class="external">https://openqa.suse.de/tests/8414267</a><br>
<a href="https://openqa.suse.de/tests/8414271" class="external">https://openqa.suse.de/tests/8414271</a><br>
<a href="https://openqa.suse.de/tests/8414266" class="external">https://openqa.suse.de/tests/8414266</a><br>
<a href="https://openqa.suse.de/tests/8414270" class="external">https://openqa.suse.de/tests/8414270</a><br>
<a href="https://openqa.suse.de/tests/8414265" class="external">https://openqa.suse.de/tests/8414265</a><br>
<a href="https://openqa.suse.de/tests/8414269" class="external">https://openqa.suse.de/tests/8414269</a><br>
<a href="https://openqa.suse.de/tests/8414264" class="external">https://openqa.suse.de/tests/8414264</a></p>
<p><a href="https://openqa.suse.de/tests/8414323" class="external">https://openqa.suse.de/tests/8414323</a><br>
<a href="https://openqa.suse.de/tests/8414322" class="external">https://openqa.suse.de/tests/8414322</a><br>
<a href="https://openqa.suse.de/tests/8414321" class="external">https://openqa.suse.de/tests/8414321</a><br>
<a href="https://openqa.suse.de/tests/8414320" class="external">https://openqa.suse.de/tests/8414320</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>The base test suite is used for job templates defined in YAML documents. It has no settings of its own.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/8345312" class="external">113.1</a></p>
<p>Find jobs referencing this ticket with the help of<br>
<a href="https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label" class="external">https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label</a> ,<br>
call <code>openqa-query-for-job-label poo#109046</code></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/8292036" class="external">108.1</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Migration-from-SLE12-SPx&machine=s390x-kvm-sle15&test=offline_sles12sp3_ltss_pscc_asmm-lgm_all_full&version=15-SP4" class="external">latest</a></p>
openQA Tests - action #108953 (Resolved): [tools] Performance issues in some s390 workershttps://progress.opensuse.org/issues/1089532022-03-25T07:13:32Zjlausuchjalausuch@suse.com
<p>This ticket is to collect examples of jobs that are failing due to some performance degradation, specially s390x workers.</p>
<p>Installation jobs:<br>
(some of these issues seem to be a slow key press so it doesn't reach the target needle on time):</p>
<ul>
<li><a href="https://openqa.suse.de/tests/8390044#step/select_patterns/5" class="external">https://openqa.suse.de/tests/8390044#step/select_patterns/5</a> - grenache-1:47</li>
<li><a href="https://openqa.suse.de/tests/8387127#step/disable_grub_timeout/18" class="external">https://openqa.suse.de/tests/8387127#step/disable_grub_timeout/18</a> - grenache-1:32</li>
<li><a href="https://openqa.suse.de/tests/8380005#step/add_update_test_repo/38" class="external">https://openqa.suse.de/tests/8380005#step/add_update_test_repo/38</a> - grenache-1:49
<a href="https://openqa.suse.de/tests/8380706#step/add_update_test_repo/10" class="external">https://openqa.suse.de/tests/8380706#step/add_update_test_repo/10</a> - grenache-1:49 </li>
<li><a href="https://openqa.suse.de/tests/8380553#step/cleanup_before_shutdown/4" class="external">https://openqa.suse.de/tests/8380553#step/cleanup_before_shutdown/4</a> (command 'rm -f /etc/udev/rules.d/70-persistent-net.rules' timed out ) - grenache-1:37</li>
</ul>
<p>Other jobs:</p>
<ul>
<li><a href="https://openqa.suse.de/tests/8387144#step/install_updates/14" class="external">https://openqa.suse.de/tests/8387144#step/install_updates/14</a> (command 'rm /tmp/journal_before' timed out) - grenache-1:44</li>
<li><a href="https://openqa.suse.de/tests/8387142#step/image_checks/13" class="external">https://openqa.suse.de/tests/8387142#step/image_checks/13</a> (script timeout: cat /usr/share/combustion-welcome) - grenache-1:33 </li>
</ul>
<p>Boot failures:</p>
<ul>
<li><a href="https://openqa.suse.de/tests/8380156#step/bootloader_zkvm/23" class="external">https://openqa.suse.de/tests/8380156#step/bootloader_zkvm/23</a></li>
<li><a href="https://openqa.suse.de/tests/8380915#step/bootloader_zkvm/22" class="external">https://openqa.suse.de/tests/8380915#step/bootloader_zkvm/22</a></li>
</ul>
qe-yam - action #107959 (Closed): 12-SP5-Server-DVD-Updates-s390x installation job fails too oftenhttps://progress.opensuse.org/issues/1079592022-03-08T08:16:49Zjlausuchjalausuch@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-12-SP5-Server-DVD-Updates-s390x-mru-install-minimal-with-addons@s390x-kvm-sle12 fails in<br>
<a href="https://openqa.suse.de/tests/8286121/modules/await_install/steps/2" class="external">await_install</a></p>
<p>This job is a parent job which many child jobs depend on and are blocked if this fails,<br>
which delays the maintenance review some hours.</p>
<p>It fails 40-50% of the times with <strong>timeout_exceeded</strong> or <strong>incomplete</strong>.</p>
<p>Examples:<br>
<a href="https://openqa.suse.de/tests/8280085" class="external">https://openqa.suse.de/tests/8280085</a><br>
<a href="https://openqa.suse.de/tests/8277975" class="external">https://openqa.suse.de/tests/8277975</a><br>
<a href="https://openqa.suse.de/tests/8277320" class="external">https://openqa.suse.de/tests/8277320</a><br>
<a href="https://openqa.suse.de/tests/8261669" class="external">https://openqa.suse.de/tests/8261669</a><br>
<a href="https://openqa.suse.de/tests/8247241" class="external">https://openqa.suse.de/tests/8247241</a><br>
<a href="https://openqa.suse.de/tests/8239743" class="external">https://openqa.suse.de/tests/8239743</a><br>
<a href="https://openqa.suse.de/tests/8227540" class="external">https://openqa.suse.de/tests/8227540</a><br>
<a href="https://openqa.suse.de/tests/8226549" class="external">https://openqa.suse.de/tests/8226549</a><br>
<a href="https://openqa.suse.de/tests/8194333" class="external">https://openqa.suse.de/tests/8194333</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Testsuite maintained at <a href="https://gitlab.suse.de/qa-maintenance/qam-openqa-yml" class="external">https://gitlab.suse.de/qa-maintenance/qam-openqa-yml</a>.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/8286121" class="external">20220308-1</a> (current job)</p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/8281074" class="external">20220307-1</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Server-DVD-Updates&machine=s390x-kvm-sle12&test=mru-install-minimal-with-addons&version=12-SP5" class="external">latest</a></p>
openQA Tests - action #107623 (Resolved): Investigate which update causes rootless_podman regress...https://progress.opensuse.org/issues/1076232022-02-25T08:13:10Zjlausuchjalausuch@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Since 24 Feb, one of the updates in <code>BASE_TEST_ISSUES</code> is causing a regression in rootless podman.<br>
<img src="https://progress.opensuse.org/attachments/download/12904/rootless%20podman.png" alt="" loading="lazy" /></p>
<p>It only happens in 15-SP3 Update jobs. <br>
<code>BASE_TEST_ISSUES=21897,22201,22540,22559,22637,22670,22688,22700,22721,22726,22748,22752,22756,22834,22837,22848,22857,22863,22881,22884,22909,22914,22921,22926,22933,22948,22951,22955,22961,22965,22970,22971,22973,22977,22990</code></p>
<p>podman version: 2.1.1</p>
<p>I have started doing some small investigation and neither <a href="https://smelt.suse.de/incident/22201/" class="external">https://smelt.suse.de/incident/22201/</a> or <a href="https://smelt.suse.de/incident/22843/" class="external">https://smelt.suse.de/incident/22843/</a> (buildah update) cause the issue.</p>
<p>openQA test in scenario sle-15-SP3-JeOS-for-kvm-and-xen-Updates-x86_64-jeos-containers@64bit-virtio-vga fails in<br>
<a href="https://openqa.suse.de/tests/8225835/modules/rootless_podman/steps/102" class="external">rootless_podman</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/8219750" class="external">20220224-1</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/8214690" class="external">20220223-1</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=JeOS-for-kvm-and-xen-Updates&machine=64bit-virtio-vga&test=jeos-containers&version=15-SP3" class="external">latest</a></p>
openQA Tests - action #107062 (Feedback): Multiple failures due to network issueshttps://progress.opensuse.org/issues/1070622022-02-18T08:27:41Zjlausuchjalausuch@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>I will use this ticket to collect the different errors I observe in our tests (at least for QE-C squad) that fail due to network issues.<br>
Normally a restart helps to get the job green again (in case we need it green), but this is not the ideal solution.</p>
<p>The idea of this ticket is to collect more potential issues caught by reviewers and propose solutions for some of them, in the code (retry same command several times might help) or in the infra side. </p>
<p>There is an example for each error I found, but from my experience reviewing jobs every day, these failures happen multiple times a day and randomly (difficult to predict).</p>
<p>1) <strong>SUSEConnect timeouts</strong> -> <a href="https://openqa.suse.de/tests/8189768#step/docker/34">https://openqa.suse.de/tests/8189768#step/docker/34</a><br>
<code>Test died: command 'SUSEConnect -p sle-module-containers/${VERSION_ID}/${CPU} ' timed out at /usr/lib/os-autoinst/testapi.pm line 1039.</code></p>
<p>Or <a href="https://openqa.suse.de/tests/8193554#step/suseconnect_scc/8">https://openqa.suse.de/tests/8193554#step/suseconnect_scc/8</a><br>
<code>Test died: command 'SUSEConnect -r $regcode' timed out at /usr/lib/os-autoinst/testapi.pm line 950.</code></p>
<p>2) <strong><a href="updates.suse.com" class="external">updates.suse.com</a> not reachable</strong> -> <a href="https://openqa.suse.de/tests/8189697#step/image_docker/1110">https://openqa.suse.de/tests/8189697#step/image_docker/1110</a></p>
<pre><code>Retrieving: kmod-25-6.10.1.aarch64.rpm [.........error]
Abort, retry, ignore? [a/r/i/...? shows all options] (a): a
Download (curl) error for 'https://updates.suse.com/SUSE/Updates/SLE-Module-Basesystem/15-SP2/aarch64/update/aarch64/kmod-25-6.10.1.aarch64.rpm?nE0jiYdfiOdLYjH0o-llNN2xIDXncon0vYw8z1aBPGx00H9S1eN413vUsfSJnzFrVz-CoZoGtSdsPKIDRAOQy3Xw2Tac3Yx5_1i8TPomSNiqhDJ0Ayxro23n46NHHB-XHq669RlHs17wiUFSJiSMCSh-YzdGdFw':
Error code: Connection failed
Error message: Could not resolve host: updates.suse.com
Problem occurred during or after installation or removal of packages:
Installation has been aborted as directed.
Please see the above error message for a hint.
</code></pre>
<p>3) <strong>SCC timeouts</strong> -> <a href="https://openqa.suse.de/tests/8189613#step/image_docker/316">https://openqa.suse.de/tests/8189613#step/image_docker/316</a></p>
<pre><code>docker run --entrypoint /usr/lib/zypp/plugins/services/container-suseconnect-zypp -i zypper_docker_derived lp
...
2022/02/18 07:16:19 Installed product: SLES-12.3-x86_64
2022/02/18 07:16:19 Registration server set to https://scc.suse.com
2022/02/18 07:16:30 Get https://scc.suse.com/connect/subscriptions/products?arch=x86_64&identifier=SLES&version=12.3: dial tcp: lookup scc.suse.com on 10.0.2.3:53: read udp 172.17.0.2:37151->10.0.2.3:53: i/o timeout
</code></pre>
<p>4) <strong>zypper ref timeout or error</strong> -> <a href="https://openqa.opensuse.org/tests/2193730#step/image_podman/124">https://openqa.opensuse.org/tests/2193730#step/image_podman/124</a></p>
<pre><code>podman run -i --name 'refreshed' --entrypoint '' registry.opensuse.org/opensuse/leap/15.3/images/totest/containers/opensuse/leap:15.3 zypper -nv ref
...
Retrieving: cb71cb070e8aac79327e6f1b6edc5317122ca1f72970299c3cb2cf505e18b27f-deltainfo.xml.gz [........................done (82.3 KiB/s)]
Retrieving: 832729371fe20bc1a4d27e59d76c10ffe2c0b5a1ff71c4e934e7a11baa24a74b-primary.xml.gz [............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................error (87.0 KiB/s)]
WJdDM-124-
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> All existing subtasks are resolved, no additional work needed on top</li>
</ul>
openQA Tests - action #93339 (Resolved): test fails in validate_btrfshttps://progress.opensuse.org/issues/933392021-06-01T09:16:42Zjlausuchjalausuch@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-15-SP3-Server-DVD-Updates-x86_64-sle_image_on_sle_host@64bit fails in<br>
<a href="https://openqa.suse.de/tests/6149331/modules/validate_btrfs/steps/39" class="external">validate_btrfs</a><br>
Other failures:<br>
All Container jobs, e.g. <a href="https://openqa.suse.de/tests/6151606" class="external">https://openqa.suse.de/tests/6151606</a><br>
All Public Cloud jobs, e.g. <a href="https://openqa.suse.de/tests/6149926#step/validate_btrfs/39" class="external">https://openqa.suse.de/tests/6149926#step/validate_btrfs/39</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>The base test suite is used for job templates defined in YAML documents. It has no settings of its own.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/6149331" class="external">20210601-1</a> (current job)</p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: (unknown) (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=sle_image_on_sle_host&version=15-SP3" class="external">latest</a></p>
openQA Tests - action #89287 (Resolved): test fails in yast2_lanhttps://progress.opensuse.org/issues/892872021-03-01T14:48:17Zjlausuchjalausuch@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-15-SP3-JeOS-for-MS-HyperV-x86_64-jeos-main@svirt-hyperv fails in<br>
<a href="https://openqa.suse.de/tests/5552124/modules/yast2_lan/steps/21" class="external">yast2_lan</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/5462492" class="external">23.32</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: (unknown) (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=JeOS-for-MS-HyperV&machine=svirt-hyperv&test=jeos-main&version=15-SP3" class="external">latest</a></p>
openQA Tests - action #65115 (Resolved): [qac][public cloud] Storage perf test failing in ssh con...https://progress.opensuse.org/issues/651152020-04-01T06:24:17Zjlausuchjalausuch@suse.com
<p>PC tools helper VM fails to connect to the VM</p>
<p><a href="https://openqa.suse.de/tests/4070444#" class="external">https://openqa.suse.de/tests/4070444#</a><br>
<a href="https://openqa.suse.de/tests/4067508#" class="external">https://openqa.suse.de/tests/4067508#</a></p>
<pre><code># nc -vz -w 1 34.76.215.65 22; echo Hsf3L-$?-
Connection to 34.76.215.65 22 port [tcp/ssh] succeeded!
Hsf3L-0-
# cat > /tmp/scripthW6mh.sh << 'EOT_hW6mh'; echo hW6mh-$?-
> ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR -i '/root/.ssh/id_rsa' "susetest@34.76.215.65" -- 'uname -r'
> EOT_hW6mh
hW6mh-0-
# echo hW6mh; bash -oe pipefail /tmp/scripthW6mh.sh ; echo SCRIPT_FINISHEDhW6mh-$?-
hW6mh
susetest@34.76.215.65: Permission denied (publickey).
</code></pre> openQA Tests - action #64724 (Resolved): [kernel][xfstests] Parted fails to create a partitionhttps://progress.opensuse.org/issues/647242020-03-22T18:31:09Zjlausuchjalausuch@suse.com
<p><a href="https://openqa.suse.de/tests/4017525#step/partition/13" class="external">https://openqa.suse.de/tests/4017525#step/partition/13</a></p>
<pre><code>parted /dev/vdb --script -- mklabel gpt
Error: partition(s) on /dev/vdb are being used.
</code></pre> openQA Tests - action #58220 (Resolved): [kernel] fadump LVM test failshttps://progress.opensuse.org/issues/582202019-10-15T20:25:44Zjlausuchjalausuch@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Failure when going into grub screen<br>
<a href="https://openqa.suse.de/tests/3479900#step/kdump_and_crash/64" class="external">grub screen</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Maintainer: Petr Cervinka <a href="mailto:pcervinka@suse.com">pcervinka@suse.com</a></p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/3479900" class="external">0358</a><br>
But it also failed in some <a href="https://openqa.suse.de/tests/3395111" class="external">older run</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>This is an example of <a href="https://openqa.suse.de/tests/3470330#step/kdump_and_crash/64" class="external">successful run</a> in the previous build.</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
openQA Tests - action #58148 (Rejected): [kernel][multipath] Timeout in qa_kernel_multipathhttps://progress.opensuse.org/issues/581482019-10-14T15:19:53Zjlausuchjalausuch@suse.com
<p>The test times out after 2 hours. Need to investigate why.</p>
<p><a href="https://openqa.suse.de/tests/3470142" class="external">https://openqa.suse.de/tests/3470142</a><br>
<a href="https://openqa.suse.de/tests/3470021" class="external">https://openqa.suse.de/tests/3470021</a></p>