openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842019-08-23T08:20:01ZopenSUSE Project Management Tool
Redmine openQA Infrastructure - action #55868 (Resolved): USB device is removed from gonzo.qa.suse.de.https://progress.opensuse.org/issues/558682019-08-23T08:20:01Zxlaixlai@suse.com
<p>We equipped on some servers some usb device to test virtualization pvusb feature. However today from a failure test <a href="https://openqa.suse.de/tests/3280807">https://openqa.suse.de/tests/3280807</a>, we noticed that the logitech mouse usb device on it was accidentally removed by someone and resulted in test failure.</p>
<p>SUT_IP: gonzo.qa.suse.de<br>
Missing USB device: <br>
Bus 003 Device 002: ID 046d:c001 Logitech, Inc. N48/M-BB48/M-UK96A [FirstMouse Plus]</p>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/24624">@nicksinger</a> Would you please help to find out who moved it and ask him/her to recover it? I know it is not easy since so many people have access to lab. If you can not find it, would you please help to remind people to not move our usb devices? Really appreciate if you can help paste some warning msg on these machines.</p>
<p>SUTs with usb device:<br>
21:<br>
WORKER_CLASS: virt-mm-64bit-ipmi,virt-pvusb-64bit-ipmi,64bit-ipmi<br>
IPMI_HOSTNAME: sp.kermit.qa.suse.de<br>
IPMI_PASSWORD: ADMIN<br>
IPMI_USER: ADMIN<br>
MAX_JOB_TIME: 32000<br>
SUT_IP: kermit.qa.suse.de<br>
PVUSB_DEVICE: Keyboard<br>
22: //missing now<br>
WORKER_CLASS: virt-mm-64bit-ipmi,virt-pvusb-64bit-ipmi,64bit-ipmi<br>
IPMI_HOSTNAME: sp.gonzo.qa.suse.de<br>
IPMI_PASSWORD: ADMIN<br>
IPMI_USER: ADMIN<br>
MAX_JOB_TIME: 32000<br>
SUT_IP: gonzo.qa.suse.de<br>
PVUSB_DEVICE: Logitech<br>
23:<br>
WORKER_CLASS: virt-mm-64bit-ipmi,virt-pvusb-64bit-ipmi,64bit-ipmi<br>
IPMI_HOSTNAME: sp.fozzie.qa.suse.de<br>
IPMI_PASSWORD: ADMIN<br>
IPMI_USER: ADMIN<br>
MAX_JOB_TIME: 32000<br>
SUT_IP: fozzie-1.qa.suse.de<br>
PVUSB_DEVICE: Logitech<br>
24:<br>
WORKER_CLASS: virt-mm-64bit-ipmi,virt-pvusb-64bit-ipmi,64bit-ipmi<br>
IPMI_HOSTNAME: sp.scooter.qa.suse.de<br>
IPMI_PASSWORD: ADMIN<br>
IPMI_USER: ADMIN<br>
MAX_JOB_TIME: 32000<br>
SUT_IP: scooter-1.qa.suse.de</p>
<pre><code> 16:
WORKER_CLASS: virt-pvusb-64bit-ipmi,64bit-ipmi
IPMI_HOSTNAME: 10.162.28.200
IPMI_PASSWORD: qatesting
IPMI_USER: admin
MAX_JOB_TIME: 32000
SUT_IP: 10.162.2.75
PVUSB_DEVICE: Winbond
17: # new worker quinn.qa.suse.de
WORKER_CLASS: virt-pvusb-64bit-ipmi,64bit-ipmi
IPMI_HOSTNAME: 10.162.3.87
IPMI_PASSWORD: ADMIN
IPMI_USER: ADMIN
SUT_IP: 10.162.2.87
MAX_JOB_TIME: 32000
PVUSB_DEVICE: Verbatim
</code></pre> openQA Tests - action #55100 (Resolved): [hyperv] Need to delete ISO with issue when checksum doe...https://progress.opensuse.org/issues/551002019-08-05T07:00:31Zxlaixlai@suse.com
<p>All vmware&hyperv jobs in virtualization job group fail by similar error <a href="https://openqa.suse.de/tests/3204511#step/welcome/10" class="external">https://openqa.suse.de/tests/3204511#step/welcome/10</a>.</p>
<p>Need to find why checksum does not match and fix it.</p>
openQA Infrastructure - action #51710 (Resolved): [openqa infra] Need 15sp1 repo under repo/fixedhttps://progress.opensuse.org/issues/517102019-05-21T03:21:09Zxlaixlai@suse.com
<p>In 12sp5 test plan, virtualization needs to cover tests on sle15sp1 host. So we need 15sp1 gmc repo under /var/lib/openqa/share/factory/repo/fixed just like other released products SLE12SP3. But I do not have permission to do so. Can anyone help to do it?</p>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/24624">@nicksinger</a>, please help to reassign if you find other more suitable one. Thanks!</p>
<p>Log:<br>
xlai@openqa:/var/lib/openqa/share/factory/repo/fixed> ln -s ../SLE-15-SP1-Installer-DVD-x86_64-Build227.1-Media1 ./SLE-15-SP1-Installer-DVD-x86_64-GM-DVD1<br>
ln: failed to create symbolic link './SLE-15-SP1-Installer-DVD-x86_64-GM-DVD1': Permission denied<br>
xlai@openqa:/var/lib/openqa/share/factory/repo/fixed> </p>
openQA Infrastructure - action #48029 (Resolved): [network failure] Can not access ipmi worker gr...https://progress.opensuse.org/issues/480292019-02-18T06:35:35Zxlaixlai@suse.com
<p>All recent triggered jobs were incomplete on this worker because not able to establish ipmitool connection. Local ping to that ip also failed.</p>
<p>Worker config:<br>
10:<br>
WORKER_CLASS: 64bit-ipmi<br>
IPMI_HOSTNAME: openqaipmi5-sp.qa.suse.de<br>
IPMI_PASSWORD: qatesting<br>
IPMI_USER: admin<br>
MAX_JOB_TIME: 32000<br>
SUT_IP: openqaipmi5.qa.suse.de</p>
<p>Ping from beijing failure:</p>
<p>linux-gepp:~ # ping openqaipmi5-sp.qa.suse.de<br>
PING openqaipmi5-sp.qa.suse.de (10.162.28.160) 56(84) bytes of data.<br>
<sup>C</sup><br>
--- openqaipmi5-sp.qa.suse.de ping statistics ---<br>
92 packets transmitted, 0 received, 100% packet loss, time 91728ms</p>
<p>linux-gepp:~ # </p>
openQA Project - action #44978 (Rejected): [ipmi unstability] jobs got blue screen making openqa ...https://progress.opensuse.org/issues/449782018-12-11T03:25:09Zxlaixlai@suse.com
<p>The blue screen looks like <a href="https://openqa.suse.de/tests/2319257#step/boot_from_pxe/25" class="external">https://openqa.suse.de/tests/2319257#step/boot_from_pxe/25</a>, and when it happens, needle matching mechenism of openqa will not work(creating new needle based on the blue screen not work and with such needle when retrigger jobs, job will fail at any needle matching).</p>
openQA Infrastructure - action #44498 (Resolved): [ipmi][grenache-1] Incomplete job due to no spa...https://progress.opensuse.org/issues/444982018-11-29T07:43:06Zxlaixlai@suse.com
<p>Key log:<br>
[2018-11-28T11:43:34.336 CET] [debug] Backend process died, backend errors are reported below in the following lines Can't write to log: No space left on device at /usr/lib/os-autoinst/bmwqemu.pm line 200.</p>
<p>XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":57369"<br>
after 15738 requests (15581 known processed) with 0 events remaining.<br>
[2018-11-28T11:43:34.660 CET] [debug] backend process exited: 200<br>
xterm: fatal IO error 11 (Resource temporarily unavailable) or KillClient on X server ":57369"<br>
[2018-11-28T11:43:34.740 CET] [debug] Driver backend collected unknown process with pid 477868 and exit status: 1<br>
[2018-11-28T11:43:34.740 CET] [debug] Driver backend collected unknown process with pid 477844 and exit status: 0<br>
[2018-11-28T11:43:34.740 CET] [debug] sysread failed: <br>
[2018-11-28T11:43:34.746 CET] [debug] Driver backend collected unknown process with pid 477870 and exit status: 84<br>
[2018-11-28T11:43:34.746 CET] [debug] Driver backend collected unknown process with pid 477880 and exit status: 0<br>
[2018-11-28T11:43:34.751 CET] [debug] commands process exited: 0<br>
[2018-11-28T11:43:37.0493 CET] [info] +++ worker notes +++<br>
[2018-11-28T11:43:37.0494 CET] [info] end time: 2018-11-28 10:43:37<br>
[2018-11-28T11:43:37.0495 CET] [info] result: died<br>
[2018-11-28T11:43:37.0496 CET] [info] uploading video.ogv<br>
[2018-11-28T11:43:37.0583 CET] [info] uploading vars.json<br>
[2018-11-28T11:43:37.0612 CET] [info] uploading serial0.txt</p>
<p>Failure job:<br>
grenache-1:16<br>
<a href="https://openqa.suse.de/tests/2285123/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/2285123/file/autoinst-log.txt</a></p>
<p>grenache-1:10<br>
<a href="https://openqa.suse.de/tests/2285129" class="external">https://openqa.suse.de/tests/2285129</a></p>
<p>grenache-1:17<br>
<a href="https://openqa.suse.de/tests/2285036/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/2285036/file/autoinst-log.txt</a></p>
openQA Infrastructure - action #44351 (Resolved): [ipmi] Workers/jobs stuck https://progress.opensuse.org/issues/443512018-11-26T10:20:30Zxlaixlai@suse.com
<p>Currently at least 4 ipmi workers (openqaworker2:23/24/25/26) are stuck. They can not finish jobs or take new jobs for over a day and some even several days.</p>
<p>Despite the misleading message from the developer mode, this issue has nothing to do with the developer mode. (The misleading message is also already fixed on latest master.)</p>
<p>This issue is likely tied to the latest refactoring of the worker cache. At least the <code>autoinst-log.txt</code> leads to that conclusion:</p>
<pre><code>tail -f /var/lib/openqa/pool/26/autoinst-log.txt
[2018-11-26T11:07:10.0806 CET] [info] +++ setup notes +++
[2018-11-26T11:07:10.0806 CET] [info] start time: 2018-11-26 10:07:10
[2018-11-26T11:07:10.0806 CET] [info] running on openqaworker2:26 (Linux 4.7.5-2.g02c4d35-default #1 SMP PREEMPT Mon Sep 26 08:11:45 UTC 2016 (02c4d35) x86_64)
[2018-11-26T11:07:10.0829 CET] [debug] Downloading SLE-15-SP1-Installer-DVD-x86_64-Build100.4-Media1.iso - request sent to Cache Service.
</code></pre>
<p>Since spvm jobs on grenache show the same symptom, this is likely also not ipmi-specific.</p>
<p>Job links: <a href="https://openqa.suse.de/tests/2272494" class="external">https://openqa.suse.de/tests/2272494</a> (openqaworker2), <a href="https://openqa.suse.de/tests/2278365" class="external">https://openqa.suse.de/tests/2278365</a> (grenache-1)</p>
openQA Project - action #39074 (Closed): [OpenQA][API] upload_logs fails from time to time.https://progress.opensuse.org/issues/390742018-08-02T03:35:53Zxlaixlai@suse.com
<p>On ipmi backend, upload_logs fails from time to time in jobs. Fail reason is "curl: fail creating formpost data"</p>
<p>It happened since build 0312. </p>
<p>Failure job(build 0315 has several such failure jobs):<br>
<a href="http://openqa.suse.de/tests/1880375#step/update_package/24" class="external">http://openqa.suse.de/tests/1880375#step/update_package/24</a></p>
openQA Tests - action #33136 (Resolved): [virtualization][functional][y] XEN/KVM host text mode ...https://progress.opensuse.org/issues/331362018-03-13T10:51:53Zxlaixlai@suse.com
<p>On build 499.1, most virtualization jobs fail at host installation phase, because PR <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4565" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4565</a> does not add needed needles for such installations.</p>
<p>Virtualization tests really depend on and affected by common installation or special change for xen/kvm roles or special change for ipmi backend. Not only test process delayed, but also we need spend quite some effort to debug the root cause.</p>
<p>Suggestion for future openqa PR-- to deliver to openqa PR reviewers:<br>
when there is code change on common installation or special change for xen/kvm roles or special change for ipmi backend, please add verification run on <a href="https://openqa.suse.de/tests/1536711" class="external">https://openqa.suse.de/tests/1536711</a> (kvm) and <a href="https://openqa.suse.de/tests/1534947" class="external">https://openqa.suse.de/tests/1534947</a> (xen) and make sure the host installation part pass , before it gets merged.</p>
openQA Tests - action #23514 (Resolved): [labs][64bit-ipmi_debug worker] SLE15 shows interface se...https://progress.opensuse.org/issues/235142017-08-22T02:17:44Zxlaixlai@suse.comopenQA Project - action #16602 (Resolved): Comment with unicode characters do not work because of...https://progress.opensuse.org/issues/166022017-02-09T02:24:15Zxlaixlai@suse.com
<a name="observation"></a>
<h2 >observation<a href="#observation" class="wiki-anchor">¶</a></h2>
<p>An error message is reported when I updated result for a build on virtualization group on openqa.suse.de</p>
<p>What was typed on comment:</p>
<pre><code>**Date:** <2017-02-09>
**Build:** 0234
**Arch:** x86_64
**Status: <font color="red">Red</font>
**New Product bugs:**
* No
**Requirement:**
* one more ipmi machine: poo#15376
**Existing Product bugs:**
* sles12sp3 pv guest installation: bsc#1012005
* pvusb bug on sles12sp3 xen: bsc#1023948
**Existing openqa issues:**
* guest migration job fail/multi-machine: poo#15376
* ipmi ikvm unstable: poo#14338
* ipmi serial console unstable: poo#13914
* ipmi keyboard simulator unstable: poo#12982 poo#16088
</code></pre>
<a name="steps-to-reproduce"></a>
<h2 >steps to reproduce<a href="#steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<ul>
<li>on an openQA instance with enabled rabbitmq plugin</li>
<li>type character '/' in any comment</li>
<li>observe reported error</li>
</ul>
<a name="problem"></a>
<h2 >problem<a href="#problem" class="wiki-anchor">¶</a></h2>
<p>H1. Mojo::RabbitMQ::Client does not handle unicode input correctly<br>
<a href="https://github.com/inway/mojo-rabbitmq-client/commit/c043447d4fa53a1456515c4def1c4d1367cd87b0#commitcomment-20829251" class="external">https://github.com/inway/mojo-rabbitmq-client/commit/c043447d4fa53a1456515c4def1c4d1367cd87b0#commitcomment-20829251</a><br>
H2. <code>lib/OpenQA/WebAPI/Plugin/AMQP.pm</code> does not forward unicode correctly to Mojo::RabbitMQ::Client</p>
<a name="workaround"></a>
<h2 >workaround<a href="#workaround" class="wiki-anchor">¶</a></h2>
<p>only type ascii within comments.</p>
openQA Infrastructure - action #16088 (Rejected): [ipmi] Do not respond to send_key.https://progress.opensuse.org/issues/160882017-01-19T06:02:20Zxlaixlai@suse.com
<p>In job <a href="https://openqa.suse.de/tests/716647#step/reboot_and_wait_up_normal2/3" class="external">https://openqa.suse.de/tests/716647#step/reboot_and_wait_up_normal2/3</a>, we use send_key_until_needle_match api to select xen grub menuentry, however after catching a not matching screen, and a send_key is sent, screen does not change.</p>
openQA Tests - action #15376 (Resolved): Need one more stable ipmi physical machine for virtualiz...https://progress.opensuse.org/issues/153762016-12-08T08:44:37Zxlaixlai@suse.com
<p>Requirement: <br>
Please add one more ipmi machine with stable ipmi.</p>
<p>Background:<br>
Currently, we only get one stable ipmi machine which is pointed to by worker openqaw2:1. As we add more virtualization tests, we have multi machine jobs involving, including guest migration which was added into openqa already. Also we plan to add virt-v2v, post copy migration tests before sle12sp3 milestone starts. They all require 2 ipmi machines.</p>
openQA Tests - action #14338 (Resolved): [ipmi] ikvm does not get timely image backhttps://progress.opensuse.org/issues/143382016-10-20T09:24:20Zxlaixlai@suse.com
<p>Assert_screen does not get any screenshot back, although it fails, so ikvm does not respond any more</p>
<p>Failed test<br>
<a href="https://openqa.suse.de/tests/619762#" class="external">https://openqa.suse.de/tests/619762#</a></p>
openQA Tests - action #13916 (Rejected): [ipmi] What should be typed by type_string is not typed ...https://progress.opensuse.org/issues/139162016-09-27T01:38:14Zxlaixlai@suse.com
<p>In build <a href="https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2144&groupid=46" class="external">https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2144&groupid=46</a>, two tests failed because what should be typed by type_string is not actually typed at all to screen . There is no ipmi connection issue reported from either serial0.txt or autoinst log. But from screenshot, the string required is not typed at all.</p>
<p>Detailed failure:<br>
<a href="https://openqa.suse.de/tests/589661#comments" class="external">https://openqa.suse.de/tests/589661#comments</a><br>
<a href="https://openqa.suse.de/tests/589666#comments" class="external">https://openqa.suse.de/tests/589666#comments</a></p>