openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-10-02T08:58:55ZopenSUSE Project Management Tool
Redmine openQA Project - action #137300 (Resolved): [FIRING:1] (Incomplete jobs (not restarted) of last 2...https://progress.opensuse.org/issues/1373002023-10-02T08:58:55Ztinitatina.mueller+trick-redmine@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<pre><code>Firing [stats.openqa-monitor.qa.suse.de]
Incomplete jobs (not restarted) of last 24h alert
View alert [stats.openqa-monitor.qa.suse.de]
Values
B0=314
Labels
alertname
Incomplete jobs (not restarted) of last 24h alert
</code></pre>
<p><a href="http://stats.openqa-monitor.qa.suse.de/alerting/grafana/cXo2cmBVk/view" class="external">http://stats.openqa-monitor.qa.suse.de/alerting/grafana/cXo2cmBVk/view</a><br>
<a href="http://stats.openqa-monitor.qa.suse.de/alerting/grafana/cXo2cmBVk/view" class="external">http://stats.openqa-monitor.qa.suse.de/alerting/grafana/cXo2cmBVk/view</a></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Alert is not triggered anymore</li>
<li><strong>AC2:</strong> It is known what triggered the alert originally</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate what happened on September 28th</li>
<li>run <code>select id,test,reason from jobs where result='incomplete' and t_created >= '2023-09-27' and t_created <= '2023-09-29' limit 30;</code> to find anything obvious by reason. I think we can group by shortened reason</li>
</ul>
openQA Infrastructure - action #136238 (Resolved): test incompletes with auto_review:"qemu-system...https://progress.opensuse.org/issues/1362382023-09-21T09:08:22Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://openqa.suse.de/tests/12221977" class="external">https://openqa.suse.de/tests/12221977</a></p>
<p>[2023-09-21T10:34:53.639587+02:00] [warn] [pid:52210] !!! : qemu-system-x86_64: -vnc :111,share=force-shared: Failed to find an available port: Address already in use</p>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<p>Find jobs referencing this ticket with the help of<br>
<a href="https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label" class="external">https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label</a> ,<br>
call <code>openqa-query-for-job-label poo#136238</code></p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Try and add a diagnostic message i.e. whatever is currently occupied on that port</li>
<li>find other occurences of this happening if any, e.g. waiting for openqa-label-known-issues to label more and use the above query or manual SQL or whatever. If it's a one-off then just reject and don't waste time here :)</li>
<li>Check other logs from the affected system at the recorded times, maybe some references in worker log or system journal</li>
<li>Find patterns of commonality of affected jobs, maybe just some specific worker machines or something</li>
<li>Fix it for good</li>
</ul>
openQA Infrastructure - action #124146 (Resolved): [alert] Incomplete jobs (not restarted) of las...https://progress.opensuse.org/issues/1241462023-02-08T16:07:15Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>See<br>
<a href="https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&viewPanel=17" class="external">https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&viewPanel=17</a><br>
since 2023-02-08 0800</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Incomplete jobs alerts are not seen on Grafana</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>The incompletes on broken downloads are likely what's expected by design</li>
<li>Alerts were triggered because we got too many incompletes</li>
<li>The asset never existed in the first place?</li>
<li>Maybe a clean-up deleted an asset too fast (unlikely), this should be possible to find out from the asset table</li>
<li>Can we detect missing assets when scheduling a job?
<ul>
<li>Leave it in scheduled? Show a message "Asset Xyz missing"</li>
</ul></li>
<li>Reach out to whoever is scheduling jobs without valid assets</li>
<li>Research if the missing link to <a href="https://openqa.suse.de/tests/10447238" class="external">the parent jopb</a> is actually an openQA bug that's making the private asset inaccessible</li>
</ul>
openQA Project - action #103584 (Resolved): job incompletes with exception in OpenCV code "Assert...https://progress.opensuse.org/issues/1035842021-12-07T07:38:24Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://openqa.opensuse.org/tests/2073991/logfile?filename=autoinst-log.txt" class="external">https://openqa.opensuse.org/tests/2073991/logfile?filename=autoinst-log.txt</a> shows</p>
<pre><code>[2021-12-07T08:35:39.375289+01:00] [debug] activate_console, console: root-ssh, type: ssh
[2021-12-07T08:35:39.375987+01:00] [debug] tests/jeos/prepare_firstboot.pm:32 called testapi::select_console -> lib/susedistribution.pm:807 called susedistribution::handle_password_prompt -> lib/susedistribution.pm:48 called testapi::assert_screen
[2021-12-07T08:35:39.376345+01:00] [debug] <<< testapi::assert_screen(mustmatch="password-prompt", timeout=60)
terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(4.5.4) /home/abuild/rpmbuild/BUILD/opencv-4.5.4/modules/imgproc/src/smooth.dispatch.cpp:293: error: (-215:Assertion failed) ksize.width > 0 && ksize.width % 2 == 1 && ksize.height > 0 && ksize.height % 2 == 1 in function 'createGaussianKernels'
Unexpected end of data 0
X connection to :38581 broken (explicit kill or server shutdown).
[2021-12-07T08:35:39.418186+01:00] [debug] backend process exited: 0
</code></pre>
<p>os-autoinst version is 4.6.1638289529.0a3f5b98 [interface v24]. This is running on siodtw01:4 (Linux 5.15.5-1-default #1 SMP Thu Nov 25 09:36:40 UTC 2021 (83fc974) aarch64)</p>
openQA Project - action #80548 (New): job incomplete with auto_review:"isotovideo died: Can.t mkd...https://progress.opensuse.org/issues/805482020-11-27T21:26:58Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://openqa.suse.de/tests/5084011" class="external">https://openqa.suse.de/tests/5084011</a> shows</p>
<pre><code>[2020-11-27T20:08:34.0611 CET] [info] [pid:6784] +++ setup notes +++
[2020-11-27T20:08:34.0611 CET] [info] [pid:6784] Running on QA-Power8-5-kvm:1 (Linux 4.12.14-lp151.28.79-default #1 SMP Wed Nov 11 08:17:16 UTC 2020 (472d149) ppc64le)
…
[2020-11-27T20:08:49.0756 CET] [debug] [pid:81266] +++ worker notes +++
�[37m[2020-11-27T20:08:50.151 CET] [debug] Current version is 4.6.1605852905.39d54910 [interface v20]
�[0mCan't mkdir('testresults'): Structure needs cleaning at /usr/lib/os-autoinst/bmwqemu.pm line 140
81266: EXIT 1
[2020-11-27T20:08:50.0180 CET] [info] [pid:6784] Isotovideo exit status: 1
[2020-11-27T20:08:50.0212 CET] [info] [pid:6784] +++ worker notes +++
[2020-11-27T20:08:50.0212 CET] [info] [pid:6784] End time: 2020-11-27 19:08:50
[2020-11-27T20:08:50.0213 CET] [info] [pid:6784] Result: died
</code></pre>
<a name="Workaround"></a>
<h2 >Workaround<a href="#Workaround" class="wiki-anchor">¶</a></h2>
<p>Retrigger</p>
openQA Project - action #80356 (Workable): incompletes with auto_review:"Cache service.*error: Co...https://progress.opensuse.org/issues/803562020-11-25T11:08:03Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://openqa.suse.de/tests/5068376" class="external">https://openqa.suse.de/tests/5068376</a><br>
shows<br>
Reason: setup failure: Cache service info error: Connection refused</p>
<p>with worker-log.txt containing:</p>
<pre><code>[2020-11-25T10:28:01.0913 CET] [error] [pid:40152] Unable to setup job 5068376: Cache service info error: Connection refused
</code></pre>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<p>Find jobs referencing this ticket with the help of<br>
<a href="https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label" class="external">https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label</a> ,<br>
to look for this ticket call <code>openqa-query-for-job-label poo#80356</code></p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>If connection refused, retry</li>
</ul>
<a name="Workaround"></a>
<h2 >Workaround<a href="#Workaround" class="wiki-anchor">¶</a></h2>
<p>Retrigger</p>
openQA Project - action #80226 (New): job incomplete with autoinst-log.txt ending just in the middlehttps://progress.opensuse.org/issues/802262020-11-23T19:09:11Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://openqa.suse.de/tests/5057715/file/autoinst-log.txt">https://openqa.suse.de/tests/5057715/file/autoinst-log.txt</a> shows at the end of the log file</p>
<pre><code>�[0m[2020-11-23T13:50:54.432 CET] [debug] tests/installation/await_install.pm:129 called testapi::check_screen
[2020-11-23T13:50:54.432 CET] [debug] <<< testapi::check_screen(mustmatch=[
"rebootnow",
"yast2_wrong_digest",
"yast2_package_retry",
"yast_error",
"ERROR-removing-package",
"DIALOG-packages-notifications"
], timeout=30)
�[37m[2020-11-23T13:50:54.523 CET] [debug] no match: 29.9s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:50:55.524 CET] [debug] no match: 28.9s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:50:56.525 CET] [debug] no match: 27.9s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:50:57.526 CET] [debug] no match: 26.9s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:50:59.609 CET] [debug] WARNING: check_asserted_screen took 1.17 seconds for 18 candidate needles - make your needles more specific
�[0m�[37m[2020-11-23T13:50:59.610 CET] [debug] no match: 25.9s, best candidate: rebootnow-20160504 (0.51)
�[0m�[37m[2020-11-23T13:50:59.710 CET] [debug] no match: 24.8s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:00.703 CET] [debug] no match: 23.8s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:01.703 CET] [debug] no match: 22.8s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:02.704 CET] [debug] no match: 21.8s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:04.787 CET] [debug] WARNING: check_asserted_screen took 1.17 seconds for 18 candidate needles - make your needles more specific
�[0m�[37m[2020-11-23T13:51:04.787 CET] [debug] no match: 20.8s, best candidate: DIALOG-packages-notifications-20191220 (0.39)
�[0m�[37m[2020-11-23T13:51:04.889 CET] [debug] no match: 19.6s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:05.884 CET] [debug] no match: 18.6s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:06.882 CET] [debug] no match: 17.6s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:07.883 CET] [debug] no match: 16.6s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:09.965 CET] [debug] WARNING: check_asserted_screen took 1.17 seconds for 18 candidate needles - make your needles more specific
�[0m�[37m[2020-11-23T13:51:09.965 CET] [debug] no match: 15.6s, best candidate: DIALOG-packages-notifications-20191220 (0.39)
�[0m�[37m[2020-11-23T13:51:10.061 CET] [debug] no match: 14.4s, best candidate: DIALOG-packages-notifications-20191220 (0.18)
�[0m�[37m[2020-11-23T13:51:10.971 CET] [debug] no change: 13.4s
�[0m�[37m[2020-11-23T13:51:11.972 CET] [debug] no change: 12.4s
�[0m�[37m[2020-11-23T13:51:12.973 CET] [debug] no change: 11.4
</code></pre>
<p>and that's it. Nothing more.</p>
<p><a href="https://openqa.suse.de/tests/5057715/file/worker-log.txt">https://openqa.suse.de/tests/5057715/file/worker-log.txt</a> looks pretty complete:</p>
<pre><code>[2020-11-23T13:51:08.0820 CET] [debug] [pid:9516] REST-API call: POST http://openqa.suse.de/api/v1/jobs/5057715/status
[2020-11-23T13:51:13.0051 CET] [debug] [pid:9516] Stopping job 5057715 from openqa.suse.de: 05057715-sle-15-SP3-Migration-from-SLE12-SPx-ppc64le-Build88.1-offline_sles12sp4_ltss_pscc_lp_def_full@ppc64le - reason: died
[2020-11-23T13:51:13.0052 CET] [debug] [pid:9516] REST-API call: POST http://openqa.suse.de/api/v1/jobs/5057715/status
[2020-11-23T13:51:13.0053 CET] [debug] [pid:9516] Registered process:63427
[2020-11-23T13:51:15.0053 CET] [debug] [pid:9516] Registered process:63482
[2020-11-23T13:51:17.0085 CET] [info] [pid:68579] Uploading patch_sle-rpm-qa.txt
[2020-11-23T13:51:17.0121 CET] [info] [pid:68579] Uploading record_disk_info-btrfs-filesystem-df.txt
[2020-11-23T13:51:17.0148 CET] [info] [pid:68579] Uploading record_disk_info-btrfs-filesystem-usage.txt
[2020-11-23T13:51:17.0177 CET] [info] [pid:68579] Uploading record_disk_info-df.txt
[2020-11-23T13:51:17.0215 CET] [info] [pid:68579] Uploading record_disk_info-snapper-list.txt
[2020-11-23T13:51:17.0286 CET] [info] [pid:68579] Uploading vars.json
[2020-11-23T13:51:17.0323 CET] [info] [pid:68579] Uploading autoinst-log.txt
[2020-11-23T13:51:17.0367 CET] [info] [pid:68579] Uploading worker-log.txt
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> If the autoinst-log.txt is incomplete and not even showing that the job finished, the job should automatically retrigger</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Research how we currently automatically retrigger incompletes within the worker code</li>
<li>Try to reproduce this error in tests by simulating premature isotovideo endings where isotovideo does not even have a proper chance to run till the end</li>
<li>Extend to retrigger if there was no proper exit code or anything from isotovideo</li>
</ul>
openQA Project - action #80108 (Resolved): HDD images not available for aarch64 Tumbleweed (clean...https://progress.opensuse.org/issues/801082020-11-20T12:11:38Zggardet_armguillaume.gardet@arm.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>We have some incompletes due to missing qcow2 images:</p>
<ul>
<li><a href="https://openqa.opensuse.org/tests/1479248" class="external">https://openqa.opensuse.org/tests/1479248</a></li>
<li><a href="https://openqa.opensuse.org/tests/1479255" class="external">https://openqa.opensuse.org/tests/1479255</a></li>
<li><a href="https://openqa.opensuse.org/tests/1479268" class="external">https://openqa.opensuse.org/tests/1479268</a></li>
<li><a href="https://openqa.opensuse.org/tests/1479189" class="external">https://openqa.opensuse.org/tests/1479189</a></li>
</ul>
<p>Checking <a href="https://openqa.opensuse.org/admin/assets" class="external">https://openqa.opensuse.org/admin/assets</a> I can find some HDD images from previous snapshots, such as <code>hdd/opensuse-Tumbleweed-aarch64-20201114-textmode@aarch64.qcow2</code> whereas the same image for 20201119 is missing.</p>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<p>TBC</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> assets are only deleted if the corresponding assets from previous builds (or "older" assets) of comparable size have been deleted first</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Get mentioned logs from aarch64.o.o</li>
<li>Look into logs, crosscheck with assets, e.g. in o3</li>
</ul>
<a name="Workaround"></a>
<h2 >Workaround<a href="#Workaround" class="wiki-anchor">¶</a></h2>
<p>Retrigger image creation jobs</p>
openQA Project - action #69691 (Workable): Improve incomplete output for qemu related problems, e...https://progress.opensuse.org/issues/696912020-08-07T09:06:05Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://openqa.opensuse.org/tests/1355023">https://openqa.opensuse.org/tests/1355023</a> shows reason "backend died: can't open qmp at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 448." so one needs to take a look into the logfile which shows:</p>
<pre><code>�[0m�[37m[2020-08-06T12:11:40.169 UTC] [debug] starting: /usr/bin/qemu-system-ppc64 -g 1024x768 -vga std -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -soundhw hda -global isa-fdc.driveA= -m 4096 -machine usb=off -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot order=c -device nec-usb-xhci -device usb-tablet -device usb-kbd -smp 4 -enable-kvm -no-shutdown -vnc :93,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-overlay2-file,filename=/var/lib/openqa/pool/3/raid/hd0-overlay2,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0-overlay2,file=hd0-overlay2-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0-overlay2,bootindex=0,serial=hd0 -blockdev driver=file,node-name=cd0-overlay2-file,filename=/var/lib/openqa/pool/3/raid/cd0-overlay2,cache.no-flush=on -blockdev driver=qcow2,node-name=cd0-overlay2,file=cd0-overlay2-file,cache.no-flush=on -device scsi-cd,id=cd0-device,drive=cd0-overlay2,serial=cd0 -incoming defer
�[0m�[37m[2020-08-06T12:11:40.174 UTC] [debug] Waiting for 0 attempts
…
�[0m�[37m[2020-08-06T12:11:58.443 UTC] [debug] Waiting for 19 attempts
�[0m�[37m[2020-08-06T12:11:59.444 UTC] [debug] Backend process died, backend errors are reported below in the following lines:
can't open qmp at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 448.
�[0m�[33m[2020-08-06T12:11:59.444 UTC] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
�[0m�[37m[2020-08-06T12:11:59.445 UTC] [debug] flushing frames
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: QEMU emulator version 3.1.1.1 (openSUSE Leap 15.1)
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: Unknown host!
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: Unknown host!
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: Unknown host!
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: Unknown host!
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: Unknown host!
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: Unknown host!
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] QEMU: qemu-system-ppc64: Failed to allocate KVM HPT of order 25 (try smaller maxmem?): Cannot allocate memory
�[0m�[37m[2020-08-06T12:11:59.447 UTC] [debug] sending magic and exit
�[0m�[37m[2020-08-06T12:11:59.448 UTC] [debug] received magic close
�[0m�[37m[2020-08-06T12:11:59.449 UTC] [debug] THERE IS NOTHING TO READ 15 4 3
</code></pre>
<p>where the latter is only visible in the log file.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> The reason includes the content from the last qemu output</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>For qemu related problems try to parse the (last) line that starts with "QEMU: " and put that into the reason instead of "can't open qmp"</li>
</ul>
openQA Project - action #69553 (Resolved): job incompletes with "Failed to rsync tests: exit code...https://progress.opensuse.org/issues/695532020-08-04T07:26:45Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://openqa.opensuse.org/tests/1352135" class="external">https://openqa.opensuse.org/tests/1352135</a> shows reason "setup failure: Failed to rsync tests: exit code 10", autoinst-log.txt shows:</p>
<pre><code>[2020-08-03T19:34:17.0982 UTC] [info] Rsync from 'rsync://openqa1-opensuse/tests' to '/var/lib/openqa/cache/openqa1-opensuse', request #5593 sent to Cache Service
[2020-08-03T19:34:44.0427 UTC] [info] Output of rsync:
[info] [#5593] Calling: rsync -avHP rsync://openqa1-opensuse/tests/ --delete /var/lib/openqa/cache/openqa1-opensuse/tests/
[2020-08-03T19:34:44.0655 UTC] [info] +++ worker notes +++
[2020-08-03T19:34:44.0655 UTC] [info] End time: 2020-08-03 19:34:44
[2020-08-03T19:34:44.0656 UTC] [info] Result: setup failure
</code></pre>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<p>Unclear how this can be reproduced but as long as auto_review is finding related issues we can find these jobs with <a href="https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label:" class="external">https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label:</a></p>
<pre><code>openqa-query-for-job-label poo#69553
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><del>Find out what "exit code 10" means</del> "exit code 10" means "Error in socket I/O" (see man page). Improve feedback to users, e.g. what are possible reasons for the problem and what to do to fix or workaround</li>
</ul>
<a name="Workaround"></a>
<h2 >Workaround<a href="#Workaround" class="wiki-anchor">¶</a></h2>
<p>Retriggering the job should work</p>