openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-09-01T03:20:18ZopenSUSE Project Management Tool
Redmine openQA Tests - action #134969 (New): [qe-core] Require latest released quarterly update images fo...https://progress.opensuse.org/issues/1349692023-09-01T03:20:18Zxlaixlai@suse.com
<p>We recently get a suggestion and agreed in discussions with Heiko Rommel, Jan Stehlik, and Antoine Ginies, that when testing released products, it is better to use <code>latest QU image + scc updates</code>.</p>
<p>For detailed requirements,</p>
<ul>
<li>all supported products' latest QU image are needed,
<ul>
<li>sle12sp3~sle12sp5 (12sp3 has teradata tests after LTSS ends, 12sp4 seems ending but not final confirmed yet)</li>
<li>sle15sp2-sle15sp5 (and sle15sp6/7 when they are released)</li>
</ul></li>
<li>available in three locations' mirror server and pxe server so that local openqa servers' jobs can access them-- DE NUE2 lab(OSD), PRG1 lab (<a href="http://openqa.qam.suse.cz/" class="external">http://openqa.qam.suse.cz/</a> resides in) and BeiJing lab (<a href="http://openqa.qa2.suse.asia/" class="external">http://openqa.qa2.suse.asia/</a> resides in)
<ul>
<li>the QU images can be accessible via http(s) in repository way (unpacked from iso), and are put with FIXED names like SLExxSPx-QU-LATEST </li>
<li>the latest QU for all products are added in pxe server</li>
</ul></li>
</ul>
<p>I am not sure if this ticket's scope can be fully covered by openqa infrastructure (some should). It will help a lot, if anyone familiar with it can help point out which parts should go to which team. We will then split the ticket properly. Thanks in advance!</p>
openQA Project - coordination #100688 (Resolved): [epic][virtualization][3rd party hypervisor] Ad...https://progress.opensuse.org/issues/1006882021-10-11T07:38:22Zxlaixlai@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>In vmware 7.0, the VNC server is completely removed. However the svirt backend that is used to do vmware virtualization tests heavily relies on VNC to interact with guests. So we have to rework the backend to make it compatible with vmware 7.0, while keeping the current way for vmware 6.5.<br>
In vSphere 7.0, the ESXi built-in VNC server has been removed. Users will no longer be able to connect to a virtual machine using a VNC client by setting the RemoteDisplay.vnc.enable configure to be true. <br>
Instead, users should use the VM Console via the vSphere Client, the ESXi Host Client, or the VMware Remote Console, to connect virtual machines. Customers desiring VNC access to a VM should use the VirtualMachine.AcquireTicket("webmks") API, which offers a VNC-over-websocket connection. The webmks ticket offers authenticated access to the virtual machine console. For more information, please refer to the VMware HTML Console SDK Documentation(<a href="http://www.vmware.com/support/developer/html-console/">http://www.vmware.com/support/developer/html-console/</a>).</p>
<a name="Impact-of-this-ticket"></a>
<h3 >Impact of this ticket<a href="#Impact-of-this-ticket" class="wiki-anchor">¶</a></h3>
<p>It blocks all VT test on vmware 7.0.<br>
According to latest info from Ralf, vmware cloud will potentially be used by SAP as a replacement of xen. So we should give high enough priority to vmware testing. And 7.0 is the current latest vmware version.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> There is support for Vmware7.0 in os-autoinst to get a graphical connection with guests comparable to existing openQA tests</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>DONE: Research task <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [virtualization][3rd party hypervisor][timeboxed:10h][research] Learn about VMWare VirtualMachine... (Resolved)" href="https://progress.opensuse.org/issues/106083">#106083</a> : Learn about VirtualMachine.AcquireTicket("webmks") API first and refine ticket to understand if we can use "VNC as-is" or need further tunneling, etc.
<ul>
<li>Some curl commands to get started with the API: <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [virtualization][3rd party hypervisor][timeboxed:10h][research] Learn about VMWare VirtualMachine... (Resolved)" href="https://progress.opensuse.org/issues/106083#note-11">#106083#note-11</a></li>
<li>Further details: <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [virtualization][3rd party hypervisor][timeboxed:10h][research] Learn about VMWare VirtualMachine... (Resolved)" href="https://progress.opensuse.org/issues/106083#note-10">#106083#note-10</a></li>
<li>Further links to the VMWare documentation: <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [virtualization][3rd party hypervisor][timeboxed:10h][research] Learn about VMWare VirtualMachine... (Resolved)" href="https://progress.opensuse.org/issues/106083">#106083</a>?#note-4</li>
<li>To test and investigate yourself: Just start a VM via the web UI (see <a class="issue tracker-6 status-3 priority-4 priority-default closed child parent" title="coordination: [epic][virtualization][3rd party hypervisor] Add svirt backend compatibility for vmware 7.0 (Resolved)" href="https://progress.opensuse.org/issues/100688#note-25">#100688#note-25</a> for URL and credentials), open the screen and monitor the traffic.</li>
<li>It should be possible to do all the requests and the web socket connection via Mojolicious.</li>
<li>Our VNC code likely needs to be decoupled from reading/writing on a network socket directly (so we can instead read/write data via binary web socket messages).</li>
<li>Hopefully the server will only use formats the client supports. Otherwise we might need to implement support for further formats in our VNC client.</li>
</ul></li>
<li>Download evaluation version of VMWare 7, install it locally (your notebook or workstation), try to get something running locally.</li>
<li>DONE: Ask virtualization team for servers which we can use for testing</li>
<li>Create pull request and ask domain experts to test in their near-production or production environment before going ahead</li>
<li>Improve existing unit tests for VNC module to increase its test coverage (before doing any actual changes) -> <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Improve existing unit tests for VNC module to increase its test coverage (before doing any actual... (Resolved)" href="https://progress.opensuse.org/issues/107026">#107026</a></li>
<li>Create integration test for the VNC module (using VNC-over-websockets) to test outside of a whole test run</li>
<li>Document how to test manually, e.g. just in the git commit</li>
<li>Consider alternatives to what customers would also use rather than our own custom VNC over websockets implementation. This allows to mitigate implementation risks and provides better, more realistic tests
<ul>
<li>Automate VMWare tooling as part of tests itself, e.g. the web interface</li>
<li>Start VM with just serial terminal and spawn VNC server within the SUT, compare to s390x z/VM test implementations </li>
</ul></li>
</ul>
openQA Project - action #90818 (Resolved): [openqa][tool] Not able to get group_overview json out...https://progress.opensuse.org/issues/908182021-04-08T06:15:56Zxlaixlai@suse.com
<p>I meet a new issue when querying osd group_overview json format output after osd deployed the latest update this Wednesday(April 7, 2021). It had been working for long time before it. Would you please help to have a look? It blocks our openqa job retrigger tool. Thanks a lot!</p>
<p><strong>Details:</strong></p>
<ul>
<li><p>Before update, works well, output is similar with below(did not paste original because output is very long):<br>
2021-04-05 01:00:01, general_utils.py, DEBUG:Command_args for subprocess to run is: ['/usr/share/openqa/script/client', '--json-output', '--host', '<a href="http://openqa.q">http://openqa.q</a><br>
a2.suse.asia', '--apikey', 'keyxxxx', '--apisecret', 'secretxxx', '--apibase', '/', 'group_overview/20', 'limit_builds=1']<br>
2021-04-05 01:00:02, general_utils.py, DEBUG:Command output is: b'{\n "comments" : [],\n "build_results" : [\n {\n "escaped_build" : "162_7",\n<br><br>
"failed" : 3,\n "build" : "162.7",\n "softfailed" : 0,\n "escaped_id" : "15_SP3-162_7",\n "passed" : 116,\n "version" : "15-SP<br>
3",\n "escaped_version" : "15_SP3",\n "oldest" : "2021-03-15T10:51:17",\n "unfinished" : 0,\n "distris" : {\n "sle" : 1\n<br><br>
},\n "key" : "15-SP3-162.7",\n "labeled" : 0,\n "skipped" : 1,\n "total" : 120,\n "reviewed" : "",\n "all_passed" : ""\n<br><br>
}\n ],\n "max_jobs" : 120,\n "pinned_comments" : [],\n "group" : {\n "id" : 20,\n "is_parent" : null,\n "name" : "SLE-15-SP3-Performance",\n<br><br>
"rendered_description" : null\n },\n "description" : null\n}\n'<br>
2021-04-05 01:00:02, general_utils.py, DEBUG:After json load, data is {'comments': [], 'build_results': [{'escaped_build': '162_7', 'failed': 3, 'build': '162.7',<br>
'softfailed': 0, 'escaped_id': '15_SP3-162_7', 'passed': 116, 'version': '15-SP3', 'escaped_version': '15_SP3', 'oldest': '2021-03-15T10:51:17', 'unfinished': 0, 'distris':<br>
{'sle': 1}, 'key': '15-SP3-162.7', 'labeled': 0, 'skipped': 1, 'total': 120, 'reviewed': '', 'all_passed': ''}], 'max_jobs': 120, 'pinned_comments': [], 'group': {'id': 20<br>
, 'is_parent': None, 'name': 'SLE-15-SP3-Performance', 'rendered_description': None}, 'description': None}<br>
2021-04-05 01:00:02, openqa_job_retrigger.py, INFO:Group 20's latest build is 162.7, will handle this build.</p></li>
<li><p>After update, return reports error:<br>
qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host <a href="http://openqa.suse.de">http://openqa.suse.de</a> --apikey --apisecret --apibase / group_overview/263 limit_builds=1<br><br>
hash- or arrayref expected (not a simple scalar, use allow_nonref to allow this) at /usr/share/openqa/script/client line 174.<br>
qa2-dhcp-53:~ #</p></li>
</ul>
openQA Tests - action #55100 (Resolved): [hyperv] Need to delete ISO with issue when checksum doe...https://progress.opensuse.org/issues/551002019-08-05T07:00:31Zxlaixlai@suse.com
<p>All vmware&hyperv jobs in virtualization job group fail by similar error <a href="https://openqa.suse.de/tests/3204511#step/welcome/10" class="external">https://openqa.suse.de/tests/3204511#step/welcome/10</a>.</p>
<p>Need to find why checksum does not match and fix it.</p>
openQA Project - action #53177 (Resolved): [openqa][worker] Can not start worker service after in...https://progress.opensuse.org/issues/531772019-06-17T09:25:40Zxlaixlai@suse.com
<p>I have a leap 42.3. On it, I install the latest openqa-* and os-autoinst. The webui can be started normally. But the workers can not be started.</p>
<a name="logs"></a>
<h3 >logs<a href="#logs" class="wiki-anchor">¶</a></h3>
<p>Worker setting:</p>
<pre><code>[global]
WORKER_HOSTNAME=10.67.18.220
[22]
WORKER_CLASS=multi-machine-job-dedicated-ipmi
BACKEND=ipmi
IPMI_HOSTNAME=10.67.135.2
IPMI_PASSWORD=susetesting
IPMI_USER=root
#WORKER_HOSTNAME=147.2.212.149
WORKER_HOSTNAME=10.67.18.220
TIMEOUT_SCALE=3
VNC_TYPING_LIMIT=10
SUT_IP=10.67.131.2
PVUSB_DEVICE=Seagate
</code></pre>
<p>Start worker failure output:</p>
<pre><code>linux-gepp:/usr/share/openqa/script # /usr/share/openqa/script/worker --instance 22
Bareword "DBD::SQLite::INSERT" not allowed while "strict subs" in use at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/SQLite.pm line 93.
Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Worker/Cache.pm line 28.
BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Worker/Cache.pm line 28.
Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Worker/Engines/isotovideo.pm line 31.
BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Worker/Engines/isotovideo.pm line 31.
Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Worker/Jobs.pm line 26.
BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Worker/Jobs.pm line 26.
Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Worker/Commands.pm line 24.
BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Worker/Commands.pm line 24.
Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Worker.pm line 31.
BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Worker.pm line 31.
Compilation failed in require at /usr/share/openqa/script/worker line 106.
BEGIN failed--compilation aborted at /usr/share/openqa/script/worker line 106.
</code></pre>
<p>All installed openqa tool set:</p>
<pre><code>linux-gepp:/usr/share/openqa/script # rpm -qf /usr/share/openqa/script/../lib/OpenQA/Worker/Cache.pm
openQA-common-4.6.1560549641.c09044ae-1444.1.noarch
linux-gepp:/usr/share/openqa/script # zypper se -s openQA-common
Loading repository data...
Reading installed packages...
S | Name | Type | Version | Arch | Repository
---+---------------+---------+--------------------------------+--------+----------------
i+ | openQA-common | package | 4.6.1560549641.c09044ae-1444.1 | noarch | openqa_42.3
v | openQA-common | package | 4.4.1497257618.89b04ed8-1.1 | noarch | dependency-repo
linux-gepp:/usr/share/openqa/script # zypper ref
Repository 'dependency-repo' is up to date.
Repository 'language-perl-42.3' is up to date.
Repository 'language_perl' is up to date.
Repository 'openqa_42.3' is up to date.
Repository 'opensuse-42.3-install-repo' is up to date.
All repositories have been refreshed.
linux-gepp:/usr/share/openqa/script # rpm -qa | grep -i openqa
openQA-common-4.6.1560549641.c09044ae-1444.1.noarch
openQA-4.6.1560549641.c09044ae-1444.1.noarch
openQA-worker-4.6.1560549641.c09044ae-1444.1.noarch
openQA-client-4.6.1560549641.c09044ae-1444.1.noarch
openQA-local-db-4.6.1560549641.c09044ae-1444.1.noarch
linux-gepp:/usr/share/openqa/script # rpm -qa | grep os-autoinst
os-autoinst-4.5.1560548250.151bd110-120.1.x86_64
linux-gepp:/usr/share/openqa/script # zypper se -s os-autoinst
Loading repository data...
Reading installed packages...
S | Name | Type | Version | Arch | Repository
---+----------------------------------+------------+-------------------------------+--------+----------------
i+ | os-autoinst | package | 4.5.1560548250.151bd110-120.1 | x86_64 | openqa_42.3
v | os-autoinst | package | 4.4.1493206707.df989545-1.1 | x86_64 | dependency-repo
| os-autoinst | srcpackage | 4.5.1560548250.151bd110-120.1 | noarch | openqa_42.3
| os-autoinst-debuginfo | package | 4.5.1560548250.151bd110-120.1 | x86_64 | openqa_42.3
| os-autoinst-debugsource | package | 4.5.1560548250.151bd110-120.1 | x86_64 | openqa_42.3
| os-autoinst-distri-opensuse-deps | package | 1.1560758996.2a5be2f7b-201.1 | noarch | openqa_42.3
| os-autoinst-distri-opensuse-deps | srcpackage | 1.1560758996.2a5be2f7b-201.1 | noarch | openqa_42.3
| os-autoinst-openvswitch | package | 4.5.1560548250.151bd110-120.1 | x86_64 | openqa_42.3
| os-autoinst-openvswitch | package | 4.4.1493206707.df989545-1.1 | x86_64 | dependency-repo
| os-autoinst-test | srcpackage | 4.5.1560548250.151bd110-120.1 | noarch | openqa_42.3
</code></pre>
<p>Repo:</p>
<pre><code>linux-gepp:/usr/share/openqa/script # zypper lr -u
Repository priorities are without effect. All enabled repositories share the same priority.
# | Alias | Name | Enabled | GPG Check | Refresh | URI
---+----------------------------+-----------------------------------------+---------+-----------+---------+----------------------------------------------------------------------------------------------------------------------
1 | dependency-repo | dependency-repo | Yes | (r ) Yes | No | http://download.opensuse.org/distribution/leap/42.3/repo/oss/suse/
2 | language-perl-42.3 | language-perl-42.3 | Yes | (r ) Yes | No | https://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_Leap_42.3/
3 | language_perl | language_perl | Yes | (r ) Yes | Yes | http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_Leap_42.3/
4 | latest-opensuse | latest-opensuse | No | ---- | ---- | https://download.opensuse.org/repositories/openSUSE:/Factory/images/repo/openSUSE-Factory-DVD-x86_64-Build0186-Media/
5 | openQA | openQA | No | ---- | ---- | http://download.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_42.2
6 | openQA-perl-modules | openQA-perl-modules | No | ---- | ---- | http://download.opensuse.org/repositories/devel:/openQA:/Leap:/42.2/openSUSE_Leap_42.2
7 | openSUSE-42.2-0 | openSUSE-42.2-0 | No | ---- | ---- | http://mirror.bej.suse.com/dist/install/SLP/openSUSE-Leap-42.2-GM/x86_64/dvd1/
8 | openqa_42.3 | openqa_42.3 | Yes | (r ) Yes | No | https://download.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_42.3/
9 | opensuse-42.3-install-repo | opensuse-42.3-install-repo | Yes | (r ) Yes | No | http://mirror.bej.suse.com/dist/install/SLP/openSUSE-Leap-42.3/x86_64/dvd1/
10 | repo-debug-non-oss | openSUSE-Leap-42.2-Debug-Non-Oss | No | ---- | ---- | http://download.opensuse.org/debug/distribution/leap/42.2/repo/non-oss/
11 | repo-debug-update | openSUSE-Leap-42.2-Update-Debug | No | ---- | ---- | http://download.opensuse.org/debug/update/leap/42.2/oss/
12 | repo-debug-update-non-oss | openSUSE-Leap-42.2-Update-Debug-Non-Oss | No | ---- | ---- | http://download.opensuse.org/debug/update/leap/42.2/non-oss/
13 | repo-non-oss | openSUSE-Leap-42.2-Non-Oss | No | ---- | ---- | http://download.opensuse.org/distribution/leap/42.2/repo/non-oss/
14 | repo-oss | openSUSE-Leap-42.2-Oss | No | ---- | ---- | http://download.opensuse.org/distribution/leap/42.2/repo/oss/
15 | repo-source | openSUSE-Leap-42.2-Source | No | ---- | ---- | http://download.opensuse.org/source/distribution/leap/42.2/repo/oss/
16 | repo-source-non-oss | openSUSE-Leap-42.2-Source-Non-Oss | No | ---- | ---- | http://download.opensuse.org/source/distribution/leap/42.2/repo/non-oss/
17 | repo-update | openSUSE-Leap-42.2-Update | No | ---- | ---- | http://download.opensuse.org/update/leap/42.2/oss/
18 | repo-update-non-oss | openSUSE-Leap-42.2-Update-Non-Oss | No | ---- | ---- | http://download.opensuse.org/update/leap/42.2/non-oss/
linux-gepp:/usr/share/openqa/script
</code></pre> openQA Infrastructure - action #51710 (Resolved): [openqa infra] Need 15sp1 repo under repo/fixedhttps://progress.opensuse.org/issues/517102019-05-21T03:21:09Zxlaixlai@suse.com
<p>In 12sp5 test plan, virtualization needs to cover tests on sle15sp1 host. So we need 15sp1 gmc repo under /var/lib/openqa/share/factory/repo/fixed just like other released products SLE12SP3. But I do not have permission to do so. Can anyone help to do it?</p>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/24624">@nicksinger</a>, please help to reassign if you find other more suitable one. Thanks!</p>
<p>Log:<br>
xlai@openqa:/var/lib/openqa/share/factory/repo/fixed> ln -s ../SLE-15-SP1-Installer-DVD-x86_64-Build227.1-Media1 ./SLE-15-SP1-Installer-DVD-x86_64-GM-DVD1<br>
ln: failed to create symbolic link './SLE-15-SP1-Installer-DVD-x86_64-GM-DVD1': Permission denied<br>
xlai@openqa:/var/lib/openqa/share/factory/repo/fixed> </p>
openQA Infrastructure - action #48029 (Resolved): [network failure] Can not access ipmi worker gr...https://progress.opensuse.org/issues/480292019-02-18T06:35:35Zxlaixlai@suse.com
<p>All recent triggered jobs were incomplete on this worker because not able to establish ipmitool connection. Local ping to that ip also failed.</p>
<p>Worker config:<br>
10:<br>
WORKER_CLASS: 64bit-ipmi<br>
IPMI_HOSTNAME: openqaipmi5-sp.qa.suse.de<br>
IPMI_PASSWORD: qatesting<br>
IPMI_USER: admin<br>
MAX_JOB_TIME: 32000<br>
SUT_IP: openqaipmi5.qa.suse.de</p>
<p>Ping from beijing failure:</p>
<p>linux-gepp:~ # ping openqaipmi5-sp.qa.suse.de<br>
PING openqaipmi5-sp.qa.suse.de (10.162.28.160) 56(84) bytes of data.<br>
<sup>C</sup><br>
--- openqaipmi5-sp.qa.suse.de ping statistics ---<br>
92 packets transmitted, 0 received, 100% packet loss, time 91728ms</p>
<p>linux-gepp:~ # </p>
openQA Infrastructure - action #44498 (Resolved): [ipmi][grenache-1] Incomplete job due to no spa...https://progress.opensuse.org/issues/444982018-11-29T07:43:06Zxlaixlai@suse.com
<p>Key log:<br>
[2018-11-28T11:43:34.336 CET] [debug] Backend process died, backend errors are reported below in the following lines Can't write to log: No space left on device at /usr/lib/os-autoinst/bmwqemu.pm line 200.</p>
<p>XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":57369"<br>
after 15738 requests (15581 known processed) with 0 events remaining.<br>
[2018-11-28T11:43:34.660 CET] [debug] backend process exited: 200<br>
xterm: fatal IO error 11 (Resource temporarily unavailable) or KillClient on X server ":57369"<br>
[2018-11-28T11:43:34.740 CET] [debug] Driver backend collected unknown process with pid 477868 and exit status: 1<br>
[2018-11-28T11:43:34.740 CET] [debug] Driver backend collected unknown process with pid 477844 and exit status: 0<br>
[2018-11-28T11:43:34.740 CET] [debug] sysread failed: <br>
[2018-11-28T11:43:34.746 CET] [debug] Driver backend collected unknown process with pid 477870 and exit status: 84<br>
[2018-11-28T11:43:34.746 CET] [debug] Driver backend collected unknown process with pid 477880 and exit status: 0<br>
[2018-11-28T11:43:34.751 CET] [debug] commands process exited: 0<br>
[2018-11-28T11:43:37.0493 CET] [info] +++ worker notes +++<br>
[2018-11-28T11:43:37.0494 CET] [info] end time: 2018-11-28 10:43:37<br>
[2018-11-28T11:43:37.0495 CET] [info] result: died<br>
[2018-11-28T11:43:37.0496 CET] [info] uploading video.ogv<br>
[2018-11-28T11:43:37.0583 CET] [info] uploading vars.json<br>
[2018-11-28T11:43:37.0612 CET] [info] uploading serial0.txt</p>
<p>Failure job:<br>
grenache-1:16<br>
<a href="https://openqa.suse.de/tests/2285123/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/2285123/file/autoinst-log.txt</a></p>
<p>grenache-1:10<br>
<a href="https://openqa.suse.de/tests/2285129" class="external">https://openqa.suse.de/tests/2285129</a></p>
<p>grenache-1:17<br>
<a href="https://openqa.suse.de/tests/2285036/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/2285036/file/autoinst-log.txt</a></p>
openQA Infrastructure - action #44351 (Resolved): [ipmi] Workers/jobs stuck https://progress.opensuse.org/issues/443512018-11-26T10:20:30Zxlaixlai@suse.com
<p>Currently at least 4 ipmi workers (openqaworker2:23/24/25/26) are stuck. They can not finish jobs or take new jobs for over a day and some even several days.</p>
<p>Despite the misleading message from the developer mode, this issue has nothing to do with the developer mode. (The misleading message is also already fixed on latest master.)</p>
<p>This issue is likely tied to the latest refactoring of the worker cache. At least the <code>autoinst-log.txt</code> leads to that conclusion:</p>
<pre><code>tail -f /var/lib/openqa/pool/26/autoinst-log.txt
[2018-11-26T11:07:10.0806 CET] [info] +++ setup notes +++
[2018-11-26T11:07:10.0806 CET] [info] start time: 2018-11-26 10:07:10
[2018-11-26T11:07:10.0806 CET] [info] running on openqaworker2:26 (Linux 4.7.5-2.g02c4d35-default #1 SMP PREEMPT Mon Sep 26 08:11:45 UTC 2016 (02c4d35) x86_64)
[2018-11-26T11:07:10.0829 CET] [debug] Downloading SLE-15-SP1-Installer-DVD-x86_64-Build100.4-Media1.iso - request sent to Cache Service.
</code></pre>
<p>Since spvm jobs on grenache show the same symptom, this is likely also not ipmi-specific.</p>
<p>Job links: <a href="https://openqa.suse.de/tests/2272494" class="external">https://openqa.suse.de/tests/2272494</a> (openqaworker2), <a href="https://openqa.suse.de/tests/2278365" class="external">https://openqa.suse.de/tests/2278365</a> (grenache-1)</p>
openQA Infrastructure - action #40544 (Resolved): [OpenQA][IPMI backend] IPMI worker can not surv...https://progress.opensuse.org/issues/405442018-09-04T07:29:51Zxlaixlai@suse.com
<p>We have two dell machines, vh003.qa2.suse.asia and vh004.qa2.suse.asia. When they are binded with ipmi worker, the jobs on those two machines can not survive reboot. For example, after host installation when it boots to the new os, the sol console can only get black screen, not reactive at all. So does any other simple reboot.</p>
<p>After debugging by john and jerry, it is found that the reset_console operation leads to this failure because the existing sol console connection is not properly cleaned up and result in failure in the new sol console setup.</p>
<p>John and jerry also have their 2 proposals as solutions which are open for discussions. I will let them describe in more details in later comments.</p>
openQA Project - action #40148 (Resolved): [OpenQA][64bit-ipmi worker] Three online 64bit-ipmi wo...https://progress.opensuse.org/issues/401482018-08-23T03:20:40Zxlaixlai@suse.com
<p>Currently there are 3 online 64bit-ipmi workers(openqaw1:2, openqaworker2:24, openqaworker2:25) which haven't take jobs for over 10 hours. However there are a lot queened jobs in virtualization job group in 12sp4 build 0351. Only 3 other workers are taking jobs.</p>
<p>Seems openqa scheduler has some problem? This delays tests a lot. Build 0351 has been running for about 2 days, but virtualization still has not finished yet. Generally it should finish within 1 day.</p>
openQA Project - action #39806 (Resolved): [OpenQA][workers] 4/6 64bit-ipmi workers are downhttps://progress.opensuse.org/issues/398062018-08-16T05:30:02Zxlaixlai@suse.com
<p>On openqa.suse.de, only 2 64bit-ipmi workers are online, and they are openqaw1:1 and openqaworker2:24. The other 4 64bit-ipmi are down, openqaw1:2 and openqaworker2:23, openqaworker2:25, openqaworker2:26.</p>
<p>Because 4 workers are offline, virtualization jobs are quite delayed, still 22 jobs(nearly half) are untested yet.</p>
<p>Can anyone help to recover those workers?</p>
openQA Tests - action #23514 (Resolved): [labs][64bit-ipmi_debug worker] SLE15 shows interface se...https://progress.opensuse.org/issues/235142017-08-22T02:17:44Zxlaixlai@suse.comopenQA Tests - action #14338 (Resolved): [ipmi] ikvm does not get timely image backhttps://progress.opensuse.org/issues/143382016-10-20T09:24:20Zxlaixlai@suse.com
<p>Assert_screen does not get any screenshot back, although it fails, so ikvm does not respond any more</p>
<p>Failed test<br>
<a href="https://openqa.suse.de/tests/619762#" class="external">https://openqa.suse.de/tests/619762#</a></p>
openQA Tests - action #13914 (New): [qe-core][functional][ipmi] wait_serial does not get expected...https://progress.opensuse.org/issues/139142016-09-27T01:36:22Zxlaixlai@suse.com
<p>Test failed due to wait_serial does not get output. From serial0.txt, the ipmi session was already closed due to "excess errors received"</p>
<p>Failure step:<br>
<a href="https://openqa.suse.de/tests/587781#step/install_package/4" class="external">https://openqa.suse.de/tests/587781#step/install_package/4</a></p>
<p>Build link:<br>
<a href="https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2141&groupid=46" class="external">https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2141&groupid=46</a></p>
<p>Serial output link:<br>
<a href="https://openqa.suse.de/tests/587781/file/serial0.txt" class="external">https://openqa.suse.de/tests/587781/file/serial0.txt</a></p>
<p>Key serial output errors:</p>
<pre><code>[�[0;32m OK �[0m] Started Serial Getty on ttyS1.
[�[0;32m OK �[0m] Started Serial Getty on hvc0.
Starting X Display Manager...
[�[0;32m OK �[0m] Started Getty on tty1.
[�[0;32m OK �[0m] Reached target Login Prompts.
[�[0;32m OK �[0m] Started /etc/init.d/after.local Compatibility.
[�[0;32m OK �[0m] Started Load dom0 backend drivers.
Starting The Xen xenstore...
[SOL established]
[error received]: excess errors received
[closing the connection]
</code></pre>