openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-06-01T08:11:53ZopenSUSE Project Management Tool
Redmine openQA Infrastructure - action #130201 (Resolved): openqa_bootstrap: fetchneedles not called duri...https://progress.opensuse.org/issues/1302012023-06-01T08:11:53Zdimstardimstar@opensuse.org
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>The test tries to confirm the minimalx testsuite is running by querying the openQA server a few times, prepping for 'running'</p>
<p>The 'running' part is no longer found in the grep output (test_running does not fail though?) which results in the final needle match to not find the minimalx (has moved to the finished tests at the bottom, but the screen is too small to show it)</p>
<p>test_running used to be like 30s in execution time, which resulted in us catching minimalx in test_restults to be in 'running state'; now, as test_running takes almost 3 minutes, test_result no longer sees the same</p>
<p>openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-openqa_bootstrap@64bit fails in<br>
<a href="https://openqa.opensuse.org/tests/3328313/modules/test_running/steps/1" class="external">test_running</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Maintainer: dheidler. Install openQA using openqa-bootstrap script.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.opensuse.org/tests/3327355" class="external">20230531</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.opensuse.org/tests/3325858" class="external">20230530</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=openqa_bootstrap&version=Tumbleweed" class="external">latest</a></p>
openQA Infrastructure - action #129121 (New): performance: various tests fail receiving data at u...https://progress.opensuse.org/issues/1291212023-05-11T09:10:02Zdimstardimstar@opensuse.org
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>This is seen in multiple tests, where data from sites like github, or flathub, are being transferred at too low rate to finish the test successfully</p>
<p>e.g.</p>
<p><a href="https://openqa.opensuse.org/tests/3282120#step/tvm/11" class="external">https://openqa.opensuse.org/tests/3282120#step/tvm/11</a><br>
downloading data fro mgithubusercontent; transfer rates of ~ 110k</p>
<p><a href="https://openqa.opensuse.org/tests/3282069#step/helm_K3S/20" class="external">https://openqa.opensuse.org/tests/3282069#step/helm_K3S/20</a><br>
downloading from github.com (according to later output in the tests, download succeeded after the timeout was reached)</p>
<p><a href="https://openqa.opensuse.org/tests/3282034#step/podman_pods/138" class="external">https://openqa.opensuse.org/tests/3282034#step/podman_pods/138</a><br>
downloading from github.com</p>
<p><a href="https://openqa.opensuse.org/tests/3282068#step/flatpak/20" class="external">https://openqa.opensuse.org/tests/3282068#step/flatpak/20</a><br>
receiving data from flatub; if the grub call would just not match anything, we'd see a return value != 0 on the console, but the command simply does not return</p>
<p>The combination of issues seen does not indicate this being an actual product bug - but rather performance of o3 -> internet being quite poor</p>
openQA Infrastructure - action #122776 (Resolved): containers_tw_image_on_ubuntu_host: error 403 ...https://progress.opensuse.org/issues/1227762023-01-06T08:20:23Zdimstardimstar@opensuse.org
<p>Seen in <a href="https://openqa.opensuse.org/tests/3017400" class="external">https://openqa.opensuse.org/tests/3017400</a> (multiple reruns; the day before this worked)</p>
<p>[2023-01-06T08:14:24.856791+01:00] [debug] Found HDD_1, caching ubuntu-22.04.qcow2<br>
[2023-01-06T08:14:24.860952+01:00] [info] Downloading ubuntu-22.04.qcow2, request #490426 sent to Cache Service<br>
[2023-01-06T08:14:35.043522+01:00] [info] Download of ubuntu-22.04.qcow2 processed:<br>
[info] [#490426] Cache size of "/var/lib/openqa/cache" is 398 GiB, with limit 400 GiB<br>
[info] [#490426] Downloading "ubuntu-22.04.qcow2" from "<a href="http://openqa1-opensuse/tests/3017400/asset/hdd/ubuntu-22.04.qcow2" class="external">http://openqa1-opensuse/tests/3017400/asset/hdd/ubuntu-22.04.qcow2</a>"<br>
[info] [#490426] Download of "/var/lib/openqa/cache/openqa1-opensuse/ubuntu-22.04.qcow2" failed: 403 Forbidden</p>
openQA Infrastructure - action #121789 (Resolved): MultiMachine tests lose ability to communicatehttps://progress.opensuse.org/issues/1217892022-12-10T06:16:47Zdimstardimstar@opensuse.org
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>This was already an issue yesterday, appeared again today, so this needs a structural fix to not sure up anymore.</p>
<p>Yesterday, Fabian rebooted ow20, and things worked. Today it stopped working again</p>
<p>openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-ovs-server@64bit fails in<br>
<a href="https://openqa.opensuse.org/tests/2944685/modules/ovs_server/steps/15" class="external">ovs_server</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Regression test for openvswitch-ipsec. Maintainer: Anna Minou</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.opensuse.org/tests/2944279" class="external">20221209</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.opensuse.org/tests/2943222" class="external">20221208</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=ovs-server&version=Tumbleweed" class="external">latest</a></p>
openQA Infrastructure - action #103524 (Resolved): OW1: performance loss size:Mhttps://progress.opensuse.org/issues/1035242021-12-06T08:33:05Zdimstardimstar@opensuse.org
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>A lot of tests are failing since Friday, even the rather 'simple' raid install tests. The failure happens very often in 'await_install', after having spent 1 hour being busy installing RPMs. Those tests generally pass in 20 minutes through the installer. </p>
<p>This started happening across the product to be tested as well as also in stagings, at the same time, which makes 'product-related issues' less probable than ow1 related issues.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> openqaworker1 is back in production with a stable set of worker instances</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Reduce the number of openQA worker instances until we don't commonly have failing or incompleting jobs</li>
<li>If there are recurring I/O errors showing up in logs consider replacing hardware again or the other NVMe if that is faulty</li>
</ul>
<a name="Rollback-actions"></a>
<h2 >Rollback actions<a href="#Rollback-actions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Use a proper RAID again of all NVMe devices for /var/lib/openqa (fvogdt has used an unusual direct use of /dev/nvme0n1 for /var/lib/openqa for now)</li>
<li>Enable additional worker instances again after hardware replacement: <code>systemctl unmask --now openqa-worker-auto-restart@{16..20}</code></li>
<li>Increase WORKERCACHE in openqaworker1:/etc/openqa/workers.ini to a higher value, e.g. 400GB again after replacement of NVMe an increase of space</li>
<li>Increase number of worker instances again, i.e. enable worker instance [7..16]</li>
</ul>
openQA Infrastructure - action #96272 (Resolved): Test failed incomplete: "Reason: backend died: ...https://progress.opensuse.org/issues/962722021-07-29T11:53:44Zdimstardimstar@opensuse.org
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>This error is seen every now and then, a test seems to be running all right, and suddenly goes 'incomplete' with an error like</p>
<pre><code>Reason: backend died: QEMU exited unexpectedly, see log for details
</code></pre>
<p>Example: <a href="https://openqa.opensuse.org/tests/1860795" class="external">https://openqa.opensuse.org/tests/1860795</a></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Reason should be sth like "QEMU startup failed: samba not installed" or similar</li>
</ul>
openQA Infrastructure - action #75259 (Resolved): 100% of powerpc tests incomplete auto_review:"(...https://progress.opensuse.org/issues/752592020-10-25T08:59:38Zdimstardimstar@opensuse.org
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Currently, all powerpc tests end with result incomplete</p>
<p>E.g <a href="https://openqa.opensuse.org/tests/1448500">https://openqa.opensuse.org/tests/1448500</a></p>
<p>From the log:</p>
<p>[2020-10-25T08:55:26.0807 UTC] [info] +++ setup notes +++<br>
[2020-10-25T08:55:26.0807 UTC] [info] Running on power8:6 (Linux 5.3.18-lp152.47-default #1 SMP Thu Oct 15 16:05:25 UTC 2020 (41f7396) ppc64le)<br>
[2020-10-25T08:55:26.0810 UTC] [debug] Found ASSET_256, caching openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256<br>
[2020-10-25T08:55:26.0812 UTC] [info] Downloading openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256, request #37952 sent to Cache Service<br>
[2020-10-25T08:55:31.0853 UTC] [info] Download of openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256 processed:<br>
[info] [#37952] Cache size of "/var/lib/openqa/cache" is 600GiB, with limit 600GiB<br>
[info] [#37952] Downloading "openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256" from "<a href="http://openqa1-opensuse/tests/1448500/asset/other/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256">http://openqa1-opensuse/tests/1448500/asset/other/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256</a>"<br>
[info] [#37952] Content of "/var/lib/openqa/cache/openqa1-opensuse/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256" has not changed, updating last use</p>
<p>[2020-10-25T08:55:31.0856 UTC] [debug] Found ISO, caching openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso<br>
[2020-10-25T08:55:31.0858 UTC] [info] Downloading openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso, request #37957 sent to Cache Service<br>
[2020-10-25T08:55:36.0899 UTC] [info] Download of openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso processed:<br>
[info] [#37957] Cache size of "/var/lib/openqa/cache" is 600GiB, with limit 600GiB<br>
[info] [#37957] Downloading "openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso" from "<a href="http://openqa1-opensuse/tests/1448500/asset/iso/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso">http://openqa1-opensuse/tests/1448500/asset/iso/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso</a>"<br>
[info] [#37957] Content of "/var/lib/openqa/cache/openqa1-opensuse/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso" has not changed, updating last use</p>
<p>[2020-10-25T08:55:36.0902 UTC] [info] Rsync from 'rsync://openqa1-opensuse/tests' to '/var/lib/openqa/cache/openqa1-opensuse', request #37961 sent to Cache Service<br>
[2020-10-25T08:55:41.0954 UTC] [info] Output of rsync:<br>
[info] [#37961] Calling: rsync -avHP rsync://openqa1-opensuse/tests/ --delete /var/lib/openqa/cache/openqa1-opensuse/tests/<br>
receiving incremental file list</p>
<p>sent 1,713 bytes received 1,099,636 bytes 734,232.67 bytes/sec<br>
total size is 4,532,146,856 speedup is 4,115.09</p>
<p>[2020-10-25T08:55:41.0954 UTC] [info] Finished to rsync tests<br>
[2020-10-25T08:55:41.0958 UTC] [debug] +++ worker notes +++<br>
�[37m[2020-10-25T08:55:42.377 UTC] [debug] Current version is 4.6.1603469837.f54bdeab [interface v20]<br>
�[0m�[37m[2020-10-25T08:55:42.384 UTC] [debug] git hash in /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse: c12d1da2e511d1cb87aeb302b752bc035a2c8c60<br>
�[0m�[37m[2020-10-25T08:55:42.560 UTC] [debug] usingenv DESKTOP=kde<br>
�[0m�[37m[2020-10-25T08:55:42.560 UTC] [debug] usingenv DISTRI=opensuse<br>
�[0m�[37m[2020-10-25T08:55:42.560 UTC] [debug] usingenv QEMUCPU=host<br>
�[0m�[37m[2020-10-25T08:55:42.560 UTC] [debug] usingenv QEMUCPUS=4<br>
�[0m�[37m[2020-10-25T08:55:42.560 UTC] [debug] usingenv INSTLANG=en_US<br>
�[0m�[37m[2020-10-25T08:55:42.560 UTC] [debug] usingenv DVD=1<br>
�[0m�[37m[2020-10-25T08:55:42.560 UTC] [debug] usingenv KDE=1<br>
�[0m�[37m[2020-10-25T08:55:42.560 UTC] [debug] usingenv ISO=/var/lib/openqa/pool/6/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso<br>
�[0m�[37m[2020-10-25T08:55:42.561 UTC] [debug] usingenv ISO_MAXSIZE=4700372992<br>
�[0m�[37m[2020-10-25T08:55:42.561 UTC] [debug] usingenv NOIMAGES=1<br>
�[0m�[37m[2020-10-25T08:55:42.561 UTC] [debug] scheduling mediacheck tests/installation/mediacheck.pm<br>
�[0m[2020-10-25T08:55:42.567 UTC] [info] cmdsrv: daemon reachable under http://*:20063/qL0cvoaoKR5zOTFr/<br>
�[37m[2020-10-25T08:55:42.568 UTC] [debug] git hash in /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles: 5482b32dfe22ba8e0376db50c26e07cc01ee56fb<br>
�[0m�[37m[2020-10-25T08:55:42.568 UTC] [debug] init needles from /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles<br>
�[0m[2020-10-25T08:55:42.569 UTC] [info] Listening at "http://[::]:20063"<br>
Web application available at http://[::]:20063<br>
�[37m[2020-10-25T08:55:42.884 UTC] [debug] inst-console-20200224 contains inst-console twice<br>
�[0m�[37m[2020-10-25T08:55:43.715 UTC] [debug] loaded 8201 needles<br>
�[0m�[37m[2020-10-25T08:55:43.862 UTC] [debug] Blocking SIGTERM<br>
�[0m�[37m[2020-10-25T08:55:43.884 UTC] [debug] Unblocking SIGTERM<br>
�[0m�[37m[2020-10-25T08:55:43.981 UTC] [debug] Blocking SIGTERM<br>
�[0m19934: channel_out 15, channel_in 14<br>
�[37m[2020-10-25T08:55:44.000 UTC] [debug] Unblocking SIGTERM<br>
�[0m19982: cmdpipe 13, rsppipe 16<br>
�[37m[2020-10-25T08:55:44.001 UTC] [debug] started mgmt loop with pid 19982<br>
�[0mremove_tree qemuscreenshot<br>
�[37m[2020-10-25T08:55:44.025 UTC] [debug] qemu version detected: 4.2.1<br>
�[0m�[37m[2020-10-25T08:55:44.026 UTC] [debug] running /usr/bin/chattr -f +C /var/lib/openqa/pool/6/raid<br>
�[0m�[37m[2020-10-25T08:55:44.032 UTC] [debug] running /usr/bin/qemu-img info --output=json /var/lib/openqa/pool/6/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso<br>
�[0m�[37m[2020-10-25T08:55:44.039 UTC] [debug] running /usr/bin/qemu-img create -f qcow2 /var/lib/openqa/pool/6/raid/hd0 20G<br>
�[0m�[37m[2020-10-25T08:55:44.076 UTC] [debug] Formatting '/var/lib/openqa/pool/6/raid/hd0', fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16<br>
�[0m�[37m[2020-10-25T08:55:44.076 UTC] [debug] running /usr/bin/qemu-img create -f qcow2 -b /var/lib/openqa/pool/6/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso /var/lib/openqa/pool/6/raid/cd0-overlay0 3745804288<br>
�[0m�[37m[2020-10-25T08:55:44.119 UTC] [debug] Formatting '/var/lib/openqa/pool/6/raid/cd0-overlay0', fmt=qcow2 size=3745804288 backing_file=/var/lib/openqa/pool/6/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso cluster_size=65536 lazy_refcounts=off refcount_bits=16<br>
�[0m�[37m[2020-10-25T08:55:44.120 UTC] [debug] starting: /usr/bin/qemu-system-ppc64 -g 1024x768 -vga std -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.driveA= -m 4096 -machine usb=off -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d -device nec-usb-xhci -device usb-tablet -device usb-kbd -smp 4 -enable-kvm -no-shutdown -vnc :96,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-file,filename=/var/lib/openqa/pool/6/raid/hd0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0,file=hd0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0,serial=hd0 -blockdev driver=file,node-name=cd0-overlay0-file,filename=/var/lib/openqa/pool/6/raid/cd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=cd0-overlay0,file=cd0-overlay0-file,cache.no-flush=on -device scsi-cd,id=cd0-device,drive=cd0-overlay0,serial=cd0<br>
�[0m�[37m[2020-10-25T08:55:44.123 UTC] [debug] Waiting for 0 attempts<br>
�[0m�[37m[2020-10-25T08:55:44.185 UTC] [debug] Waiting for 1 attempts<br>
�[0m�[37m[2020-10-25T08:55:44.185 UTC] [debug] Backend process died, backend errors are reported below in the following lines:<br>
QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 444.</p>
<p>�[0m�[33m[2020-10-25T08:55:44.185 UTC] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json<br>
�[0m�[37m[2020-10-25T08:55:44.186 UTC] [debug] Passing remaining frames to the video encoder<br>
�[0m�[37m[2020-10-25T08:55:44.187 UTC] [debug] Waiting for video encoder to finalize the video<br>
�[0m�[37m[2020-10-25T08:55:44.187 UTC] [debug] The built-in video encoder (pid 20002) terminated<br>
�[0m�[37m[2020-10-25T08:55:44.188 UTC] [debug] QEMU: QEMU emulator version 4.2.1 (openSUSE Leap 15.2)<br>
�[0m�[37m[2020-10-25T08:55:44.188 UTC] [debug] QEMU: Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers<br>
�[0m�[37m[2020-10-25T08:55:44.188 UTC] [debug] QEMU: Unknown host!<br>
�[0m�[37m[2020-10-25T08:55:44.188 UTC] [debug] QEMU: Unknown host!<br>
�[0m�[37m[2020-10-25T08:55:44.188 UTC] [debug] QEMU: Unknown host!<br>
�[0m�[37m[2020-10-25T08:55:44.188 UTC] [debug] QEMU: qemu-system-ppc64: Requested safe cache capability level not supported by kvm, try appending -machine cap-cfpc=broken</p>
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>This very likely happened due to the upgrade of the worker host to openSUSE Leap 15.2 in <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Upgrade o3 workers to openSUSE Leap 15.2 after openqa-aarch64 already done (Resolved)" href="https://progress.opensuse.org/issues/73189">#73189</a></p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>First consider rolling back the installation or maybe just the qemu version</li>
<li>Try the suggestion in the log file: "try appending -machine cap-cfpc=broken", to the qemu command line in jobs, then add that maybe in the openQA "machine" configuration?</li>
<li>Consider including the necessary setting in os-autoinst itself</li>
</ul>
openQA Infrastructure - action #73606 (Resolved): o3: multimachine tests all incomplete earlyhttps://progress.opensuse.org/issues/736062020-10-20T08:11:01Zdimstardimstar@opensuse.org
<p>Since Oct 19/20, all multimachine tests end directly as incomplete.</p>
<p>The log files shows errors like</p>
<p>�[0m�[37m[2020-10-20T10:07:52.256 CEST] [debug] QEMU: qemu-system-x86_64: could not configure /dev/net/tun (tap18): Operation not permitted</p>
<p>Having all multi-machine tests failing/unavailable is blocking Tumbleweed snapshots</p>
openQA Infrastructure - action #69475 (Resolved): [tools] openQA child task fails to download ass...https://progress.opensuse.org/issues/694752020-07-31T08:21:47Zdimstardimstar@opensuse.org
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-autoyast_reinstall_gnome@64bit fails in<br>
<a href="https://openqa.opensuse.org/tests/1348359/modules/installation/steps/32" class="external">installation</a></p>
<p>The job clone_system uploads an autoyast profile, which should then be consumed by the autoyast_reinstall_gnome test. The file seems to be there as part of the assets, but the worker fails to download it</p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.opensuse.org/tests/1348029" class="external">20200730</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.opensuse.org/tests/1283179" class="external">20200528</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=autoyast_reinstall_gnome&version=Tumbleweed" class="external">latest</a></p>
openQA Infrastructure - action #61085 (Resolved): New Tumbleweed Snapshot (20191216) does not sho...https://progress.opensuse.org/issues/610852019-12-17T10:31:48Zdimstardimstar@opensuse.org
<p>Snapshot 1216 has been tagged for QA at 12/17 07:45 CET. Until now (11L30 CET), the snapshot does not show up on openQA</p>
<p>openqa_rsync.log contains a bunch of errors though:</p>
<pre><code>error scheduling openSUSE-MicroOS-DVD-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: VERSION"error scheduling openSUSE-Kubic-DVD-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: VERSION"error scheduling openSUSE-Tumbleweed-NET-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-DVD-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-GNOME-Live-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-Rescue-CD-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-XFCE-Live-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"rsync of tumbleweed done at Tue Dec 17 08:27:58 UTC 2019
</code></pre> openQA Project - action #18730 (Rejected): openqaworker1 is causing weird test failureshttps://progress.opensuse.org/issues/187302017-04-24T08:33:35Zdimstardimstar@opensuse.org
<p>We had seen this already last week (coolo did some magic and it got better):</p>
<p>A bunch of tests fail early on when starting from the DVD, already when setting up the internal system (failed to hard link files in var/adm/....)</p>
<p>Sample test: <a href="https://openqa.opensuse.org/tests/391696#step/installation_mode/3" class="external">https://openqa.opensuse.org/tests/391696#step/installation_mode/3</a></p>
<p>Last week it was observed that all runs on ow4 were fine, ow1 failed (the currently linked one is a i586 test, where ow4 does not advertise the capability to run it)</p>
openQA Project - action #10208 (Resolved): Tests sometimes fail with 'timeout reading hmp socket'https://progress.opensuse.org/issues/102082016-01-13T15:05:06Zdimstardimstar@opensuse.org
<p>from autoinst-log.txt (<a href="https://openqa.opensuse.org/tests/113600/file/autoinst-log.txt">https://openqa.opensuse.org/tests/113600/file/autoinst-log.txt</a>)</p>
<pre><code>DIE ERROR: timeout reading hmp socket
at /usr/lib/os-autoinst/backend/baseclass.pm line 58 thread 1.
backend::baseclass::die_handler("ERROR: timeout reading hmp socket\x{a}") called at /usr/lib/os-autoinst/backend/qemu.pm line 723 thread 1
backend::qemu::_read_hmp(backend::qemu=HASH(0x57e1d98)) called at /usr/lib/os-autoinst/backend/qemu.pm line 790 thread 1
backend::qemu::_send_hmp(backend::qemu=HASH(0x57e1d98), "savevm lastgood") called at /usr/lib/os-autoinst/backend/qemu.pm line 138 thread 1
backend::qemu::do_savevm(backend::qemu=HASH(0x57e1d98), HASH(0x7f7b781f65d0)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 53 thread 1
backend::baseclass::handle_command(backend::qemu=HASH(0x57e1d98), HASH(0x7f7b781f28a0)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 374 thread 1
backend::baseclass::check_socket(backend::qemu=HASH(0x57e1d98), IO::Handle=GLOB(0x45d9820)) called at /usr/lib/os-autoinst/backend/qemu.pm line 815 thread 1
backend::qemu::check_socket(backend::qemu=HASH(0x57e1d98), IO::Handle=GLOB(0x45d9820)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 176 thread 1
eval {...} called at /usr/lib/os-autoinst/backend/baseclass.pm line 143 thread 1
backend::baseclass::run_capture_loop(backend::qemu=HASH(0x57e1d98), IO::Select=ARRAY(0x5433ed0)) called at /usr/lib/os-autoinst/backend/baseclass.pm line 98 thread 1
backend::baseclass::run(backend::qemu=HASH(0x57e1d98), 14, 17) called at /usr/lib/os-autoinst/backend/driver.pm line 66 thread 1
backend::driver::_run(backend::qemu=HASH(0x57e1d98), 14, 17) called at /usr/lib/os-autoinst/backend/driver.pm line 53 thread 1
eval {...} called at /usr/lib/os-autoinst/backend/driver.pm line 53 thread 1
waitpid for 28972 returned 28972
QEMU: qemu-system-x86_64: /home/abuild/rpmbuild/BUILD/qemu-2.1.3/hw/i386/kvm/clock.c:69: kvmclock_current_nsec: Assertion `time.tsc_timestamp <= migration_tsc' failed.
sending magic and exit
received magic close
</code></pre>
<p>openQA is nice and auto-reschedules those tests, but it would be even better to find why those fail</p>
openQA Project - action #7442 (Resolved): Cloning a job can lose job settingshttps://progress.opensuse.org/issues/74422015-05-09T08:33:18Zdimstardimstar@opensuse.org
<p>The job <a href="https://openqa.opensuse.org/tests/61594" class="external">https://openqa.opensuse.org/tests/61594</a> does not have the variable LIVETEST set to 1, which results in the test failing</p>
<p>This job was cloned from <a href="https://openqa.opensuse.org/tests/61517" class="external">https://openqa.opensuse.org/tests/61517</a> which had the variable set. So the cloning of a job seems to be able to lose variables on the way.</p>
openQA Project - action #6784 (Resolved): Test marked 'failed' without apparent reasonhttps://progress.opensuse.org/issues/67842015-03-19T13:04:23Zdimstardimstar@opensuse.org
<p>This is something I have seen more often lately, but surprisingly only on the rescuesystem tests (maybe because it's so short-living?)</p>
<p>A sample run:<br>
<a href="https://openqa.opensuse.org/tests/52724" class="external">https://openqa.opensuse.org/tests/52724</a></p>
<p>The step "rescuesystem_validate_131" has no return value; in autoinst-log.txt, it is visible that the awaited serial string has been seen (can also be confirmed in serial.txt)</p>
<p>The end of the log:<br>
`Debug: /var/lib/openqa/share/tests/opensuse/tests/installation/rescuesystem_validate_131.pm:10 called testapi::type_string<br>
<<< type_string(string=$VAR1 = '\'cat /mnt/etc/SUSE-brand > /dev/ttyS0<br>
\'';<br>
, max_interval=$VAR1 = '\'250\'';<br>
)<br>
Debug: /var/lib/openqa/share/tests/opensuse/tests/installation/rescuesystem_validate_131.pm:11 called testapi::wait_serial<br>
<<< wait_serial(regex=$VAR1 = 'VERSION = 13.1';<br>
, timeout=$VAR1 = 2;<br>
)</p>
<blockquote>
<blockquote>
<blockquote>
<p>wait_serial: VERSION = 13.1: ok<br>
||| finished rescuesystem_validate_131 installation at 2015-03-18 23:12:39 (6 s)<br>
isotovideo done<br>
waitpid for 15800 returned 0<br>
sending TERM to qemu pid: 15800<br>
waitpid for 15800 returned 15800<br>
QEMU: qemu: terminating on signal 15 from pid 15757<br>
sending magic and exit<br>
received magic close<br>
killing commands thread<br>
done joining commands thread<br>
+++ worker notes +++<br>
end time: 2015-03-18 23:12:52<br>
result: done`</p>
</blockquote>
</blockquote>
</blockquote>
openQA Project - action #5420 (Resolved): openQA considers previously 'failed' tests now as 'inco...https://progress.opensuse.org/issues/54202014-12-09T13:05:36Zdimstardimstar@opensuse.org
<p><a href="https://openqa.opensuse.org/tests/38422" class="external">https://openqa.opensuse.org/tests/38422</a> is a sample test - it appears to be 'failed' to the user (and was consdiered failed until the re-deployment of openQA on Dec 9).</p>
<p>Since this upgrade, it is now considered 'incomplete'</p>