https://progress.opensuse.org/
https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?1582917784
2018-01-16T17:07:15Z
openSUSE Project Management Tool
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=83527
2018-01-16T17:07:15Z
dasantiago
dasantiago@suse.com
<ul></ul><p>Usually this is a sympton that a qemu instance is running when it shouldn't and the start of the new instance is failing. To fix it, the current running qemu process should be killed.</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84055
2018-01-17T16:25:36Z
pcervinka
pcervinka@suse.com
<ul></ul><p>openqaworker10:4 looks fine, but worker openqaworker10:2 started to fail recently:<br>
<a href="https://openqa.suse.de/admin/workers/772" class="external">https://openqa.suse.de/admin/workers/772</a></p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84130
2018-01-18T07:56:53Z
pcervinka
pcervinka@suse.com
<ul></ul><p>openqaworker10:7 fails too:<br>
<a href="https://openqa.suse.de/admin/workers/777" class="external">https://openqa.suse.de/admin/workers/777</a></p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84145
2018-01-18T08:22:57Z
okurz
okurz@suse.com
<ul><li><strong>Category</strong> set to <i>Infrastructure</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>checking right now on that machine I can see two qemu processes which do not seem to be linked to workers. They had been running since yesterday. I killed these two qemu processes now and will check the instances 7 and 2. Last output from instance 7:</p>
<p>sudo systemctl status openqa-worker@7</p>
<pre><code>Jan 18 09:05:57 openqaworker10 worker[2703]: child 16762 died with exit status 256
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] +++ worker notes +++
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] end time: 2018-01-18 08:05:57
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] result: died
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] uploading video.ogv
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] uploading vars.json
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] uploading autoinst-log.txt
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] uploading worker-log.txt
Jan 18 09:05:57 openqaworker10 worker[2703]: can't open /var/lib/openqa/pool/7/testresults/test_order.json: No such file or directory at /usr/s...ine 756.
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] cleaning up 01395623-sle-12-SP3-Server-DVD-Updates-x86_64-Build20180118-1-mau-qa_kernel_ltp_mm@64bit
Hint: Some lines were ellipsized, use -l to show in full.
</code></pre>
<p>For instance 2 it looks comparable. instance 7 picked up a job and is executing that just fine: <a href="https://openqa.suse.de/tests/1395637#live" class="external">https://openqa.suse.de/tests/1395637#live</a></p>
<p>I will monitor both instances and report back.</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84196
2018-01-18T10:21:13Z
okurz
okurz@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>instance 7 successfully finished a job, instance 2 so far did not pick up a job but is online.</p>
<p>I assume that this was a temporary problem which will not appear anymore. If it does please report specifically the symptoms and we should try to solve it for good.</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84793
2018-01-21T14:39:24Z
okurz
okurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>In Progress</i></li><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li></ul><p>Apparently happened again. openqaworker10:4 failed, see <a href="https://openqa.suse.de/tests/1404119/file/autoinst-log.txt">https://openqa.suse.de/tests/1404119/file/autoinst-log.txt</a></p>
<p>Checking on the worker I see a left-over qemu process:</p>
<pre><code>_openqa+ 12546 0.8 0.4 5516308 1163948 ? Sl Jan19 18:24 /usr/bin/qemu-system-x86_64 -serial file:serial0 -soundhw ac97 -vga cirrus -global isa-fdc.driveA= -m 1024 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -device virtio-scsi-pci,id=scsi0 -device virtio-blk,drive=hd1,serial=1 -drive file=raid/l1,cache=unsafe,if=none,id=hd1,format=qcow2 -drive media=cdrom,if=none,id=cd0,format=raw,file=/var/lib/openqa/pool/4/SLE-12-SP3-Server-DVD-x86_64-GM-DVD1.iso -device scsi-cd,drive=cd0,bus=scsi0.0 -boot once=d,menu=on,splash-time=5000 -device usb-ehci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :94,share=force-shared -device virtio-serial -chardev socket,path=virtio_console,server,nowait,id=virtio_console,logfile=virtio_console.log -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -qmp unix:qmp_socket,server,nowait -monitor unix:hmp_socket,server,nowait -S -monitor telnet:127.0.0.1:20042,server,nowait
</code></pre>
<p>So the process is running since 2018-01-19 18:24 at least. Taking a look into the journal of this worker instance I find the last job that successfully executed before that is <a href="https://openqa.suse.de/tests/1400519#">https://openqa.suse.de/tests/1400519#</a> . And then</p>
<p><a href="https://openqa.suse.de/tests/1401276/file/autoinst-log.txt">https://openqa.suse.de/tests/1401276/file/autoinst-log.txt</a> reports</p>
<pre><code>[2018-01-19T22:29:24.0513 CET] [debug] considering VNC stalled, no update for 4.24 seconds
[2018-01-19T22:29:24.0538 CET] [debug] QEMU:
[2018-01-19T22:29:24.0538 CET] [debug] QEMU: (process:12546): GLib-CRITICAL **: Source ID 32305 was not found when attempting to remove it
[2018-01-19T22:29:24.0763 CET] [debug] pointer type 1 0 1024 768 -257
[2018-01-19T22:29:24.0763 CET] [debug] led state 0 1 1 -261
[2018-01-19T22:29:25.0014 CET] [debug] no change: 192.3s
[2018-01-19T22:29:26.0015 CET] [debug] no change: 191.3s
[2018-01-19T22:29:27.0016 CET] [debug] no change: 190.3s
[2018-01-19T22:29:28.0016 CET] [debug] no change: 189.3s
[2018-01-19T22:29:29.0018 CET] [debug] no change: 188.3s
[2018-01-19T22:29:30.0037 CET] [debug] MATCH(autoyast-system-login-console-20150529:0.00)
[2018-01-19T22:29:30.0051 CET] [debug] MATCH(autoyast-system-login-console-20170724:0.00)
[2018-01-19T22:29:30.0057 CET] [debug] MATCH(autoyast-system-login-console-minimal-20160113:0.95)
[2018-01-19T22:29:30.0063 CET] [debug] MATCH(autoyast-system-login-console-sles11sp4-20170419:1.00)
[2018-01-19T22:29:30.0073 CET] [debug] MATCH(linux-login-20150120:0.00)
[2018-01-19T22:29:30.0074 CET] [debug] MATCH(linux-login-20150922:0.95)
[2018-01-19T22:29:30.0077 CET] [debug] MATCH(linux-login-20170814:0.00)
[2018-01-19T22:29:30.0091 CET] [debug] MATCH(linux-login-bsc1055103-20170822:0.00)
[2018-01-19T22:29:30.0113 CET] [debug] MATCH(system-login-console-ipmitool-20170623:0.00)
[2018-01-19T22:29:30.0128 CET] [debug] MATCH(system-login-console-ipmitool-20170624:0.00)
[2018-01-19T22:29:30.0140 CET] [debug] MATCH(installation-autoyast-linuxlogin-20171204:0.29)
[2018-01-19T22:29:30.0150 CET] [debug] MATCH(online_migration_setup-emergency-shell-20170201:0.00)
[2018-01-19T22:29:30.0162 CET] [debug] MATCH(online_migration_setup-emergency-shell-20170330:0.00)
[2018-01-19T22:29:30.0172 CET] [debug] MATCH(snapper-emergency-shell-20170620:0.00)
[2018-01-19T22:29:30.0185 CET] [debug] MATCH(emergency-mode-20171017:0.00)
[2018-01-19T22:29:30.0366 CET] [debug] >>> testapi::_handle_found_needle: found autoyast-system-login-console-sles11sp4-20170419, similarity 1.00 @ 88/76
[2018-01-19T22:29:30.0369 CET] [debug] ||| finished update_minimal minimal at 2018-01-19 21:29:30 (666 s)
[2018-01-19T22:29:30.0370 CET] [debug] ||| starting check_logs tests/qam-minimal/check_logs.pm
[2018-01-19T22:29:30.0370 CET] [debug] /var/lib/openqa/cache/tests/sle/tests/qam-minimal/check_logs.pm:27 called testapi::select_console
[2018-01-19T22:29:30.0371 CET] [debug] <<< testapi::select_console(testapi_console='root-console')
[2018-01-19T22:29:30.0587 CET] [debug] activate_console, console: root-console, type: console
[2018-01-19T22:29:30.0587 CET] [debug] /var/lib/openqa/cache/tests/sle/tests/qam-minimal/check_logs.pm:27 called testapi::select_console
[2018-01-19T22:29:30.0587 CET] [debug] <<< testapi::assert_screen(mustmatch=[
'tty2-selected',
'text-logged-in-root',
'text-login'
], timeout=60)
*** Error in `/usr/bin/isotovideo: backen': corrupted double-linked list: 0x0000000009413f70 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x721af)[0x7f3add6411af]
/lib64/libc.so.6(+0x77706)[0x7f3add646706]
/lib64/libc.so.6(+0x786ee)[0x7f3add6476ee]
/usr/lib64/libopencv_core.so.3.1(+0x20754c)[0x7f3ad7c8e54c]
/usr/lib64/libopencv_core.so.3.1(_ZN2cv3Mat10deallocateEv+0x24)[0x7f3ad7c8e904]
/usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so(_Z13image_destroyP5Image+0x131)[0x7f3ad87b9d21]
/usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so(+0x4229)[0x7f3ad87b3229]
/usr/bin/isotovideo: backen(Perl_pp_entersub+0x586)[0x4ab526]
/usr/bin/isotovideo: backen(Perl_call_sv+0x607)[0x4372c7]
/usr/bin/isotovideo: backen[0x4b4404]
/usr/bin/isotovideo: backen(Perl_sv_clear+0x480)[0x4b4b80]
/usr/bin/isotovideo: backen(Perl_sv_free2+0xdd)[0x4b520d]
/usr/bin/isotovideo: backen(Perl_free_tmps+0x68)[0x4d6988]
/usr/bin/isotovideo: backen(Perl_pp_unstack+0x90)[0x4a4c10]
/usr/bin/isotovideo: backen(Perl_runops_standard+0x16)[0x4a3cf6]
/usr/bin/isotovideo: backen(perl_run+0x382)[0x43e112]
/usr/bin/isotovideo: backen(main+0x123)[0x41e963]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3add5ef6e5]
/usr/bin/isotovideo: backen(_start+0x29)[0x41e9a9]
======= Memory map: ========
00400000-0057a000 r-xp 00000000 00:27 407348 /usr/bin/perl
00779000-0077a000 r--p 00179000 00:27 407348 /usr/bin/perl
0077a000-0077f000 rw-p 0017a000 00:27 407348 /usr/bin/perl
00a1a000-06b72000 rw-p 00000000 00:00 0 [heap]
06b72000-0ed57000 rw-p 00000000 00:00 0 [heap]
7f3aa4000000-7f3aa4021000 rw-p 00000000 00:00 0
7f3aa4021000-7f3aa8000000 ---p 00000000 00:00 0
7f3aac000000-7f3aac021000 rw-p 00000000 00:00 0
7f3aac021000-7f3ab0000000 ---p 00000000 00:00 0
7f3ab0000000-7f3ab0021000 rw-p 00000000 00:00 0
7f3ab0021000-7f3ab4000000 ---p 00000000 00:00 0
7f3ab4000000-7f3ab4021000 rw-p 00000000 00:00 0
7f3ab4021000-7f3ab8000000 ---p 00000000 00:00 0
7f3ab8000000-7f3ab8021000 rw-p 00000000 00:00 0
7f3ab8021000-7f3abc000000 ---p 00000000 00:00 0
7f3abc000000-7f3abc021000 rw-p 00000000 00:00 0
7f3abc021000-7f3ac0000000 ---p 00000000 00:00 0
7f3ac0000000-7f3ac0021000 rw-p 00000000 00:00 0
7f3ac0021000-7f3ac4000000 ---p 00000000 00:00 0
7f3ac4000000-7f3ac4021000 rw-p 00000000 00:00 0
7f3ac4021000-7f3ac8000000 ---p 00000000 00:00 0
7f3ac9980000-7f3ac9981000 ---p 00000000 00:00 0
7f3ac9981000-7f3aca181000 rw-p 00000000 00:00 0
7f3acb7ff000-7f3acb800000 ---p 00000000 00:00 0
7f3acb800000-7f3acc000000 rw-p 00000000 00:00 0
7f3acc000000-7f3acc021000 rw-p 00000000 00:00 0
7f3acc021000-7f3ad0000000 ---p 00000000 00:00 0
7f3ad017d000-7f3ad017e000 ---p 00000000 00:00 0
7f3ad017e000-7f3ad097e000 rw-p 00000000 00:00 0
7f3ad097e000-7f3ad097f000 ---p 00000000 00:00 0
7f3ad097f000-7f3ad117f000 rw-p 00000000 00:00 0
7f3ad117f000-7f3ad1180000 ---p 00000000 00:00 0
7f3ad1180000-7f3ad1980000 rw-p 00000000 00:00 0
7f3ad1980000-7f3ad1981000 ---p 00000000 00:00 0
7f3ad1981000-7f3ad2181000 rw-p 00000000 00:00 0
7f3ad2181000-7f3ad2182000 ---p 00000000 00:00 0
7f3ad2182000-7f3ad2982000 rw-p 00000000 00:00 0
7f3ad2982000-7f3ad2983000 ---p 00000000 00:00 0
7f3ad2983000-7f3ad33c4000 rw-p 00000000 00:00 0
7f3ad36e7000-7f3ad36e9000 r-xp 00000000 00:27 71350 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Crypt/DES/DES.so
7f3ad36e9000-7f3ad38e9000 ---p 00002000 00:27 71350 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Crypt/DES/DES.so
7f3ad38e9000-7f3ad38ea000 r--p 00002000 00:27 71350 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Crypt/DES/DES.so
7f3ad38ea000-7f3ad38ed000 rw-p 00003000 00:27 71350 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Crypt/DES/DES.so
7f3ad38ef000-7f3ad38f2000 r-xp 00000000 00:27 3575 /usr/lib64/libXau.so.6.0.0
7f3ad38f2000-7f3ad3af1000 ---p 00003000 00:27 3575 /usr/lib64/libXau.so.6.0.0
7f3ad3af1000-7f3ad3af2000 r--p 00002000 00:27 3575 /usr/lib64/libXau.so.6.0.0
7f3ad3af2000-7f3ad3af3000 rw-p 00003000 00:27 3575 /usr/lib64/libXau.so.6.0.0
7f3ad3af7000-7f3ad3b06000 r-xp 00000000 00:27 3347 /usr/lib64/libdrm.so.2.4.0
7f3ad3b06000-7f3ad3d06000 ---p 0000f000 00:27 3347 /usr/lib64/libdrm.so.2.4.0
7f3ad3d06000-7f3ad3d07000 r--p 0000f000 00:27 3347 /usr/lib64/libdrm.so.2.4.0
7f3ad3d07000-7f3ad3d08000 rw-p 00010000 00:27 3347 /usr/lib64/libdrm.so.2.4.0
7f3ad3d0f000-7f3ad3d14000 r-xp 00000000 00:27 9002 /usr/lib64/libXxf86vm.so.1.0.0
7f3ad3d14000-7f3ad3f13000 ---p 00005000 00:27 9002 /usr/lib64/libXxf86vm.so.1.0.0
7f3ad3f13000-7f3ad3f14000 r--p 00004000 00:27 9002 /usr/lib64/libXxf86vm.so.1.0.0
7f3ad3f14000-7f3ad3f15000 rw-p 00005000 00:27 9002 /usr/lib64/libXxf86vm.so.1.0.0
7f3ad3f17000-7f3ad3f36000 r-xp 00000000 00:27 603327 /usr/lib64/libxcb.so.1.1.0
7f3ad3f36000-7f3ad4135000 ---p 0001f000 00:27 603327 /usr/lib64/libxcb.so.1.1.0
7f3ad4135000-7f3ad4136000 r--p 0001e000 00:27 603327 /usr/lib64/libxcb.so.1.1.0
7f3ad4136000-7f3ad4137000 rw-p 0001f000 00:27 603327 /usr/lib64/libxcb.so.1.1.0
7f3ad4137000-7f3ad413a000 r-xp 00000000 00:27 604963 /usr/lib64/libxcb-dri2.so.0.0.0
7f3ad413a000-7f3ad433a000 ---p 00003000 00:27 604963 /usr/lib64/libxcb-dri2.so.0.0.0
7f3ad433a000-7f3ad433b000 r--p 00003000 00:27 604963 /usr/lib64/libxcb-dri2.so.0.0.0
7f3ad433b000-7f3ad433c000 rw-p 00004000 00:27 604963 /usr/lib64/libxcb-dri2.so.0.0.0
7f3ad433f000-7f3ad4354000 r-xp 00000000 00:27 604955 /usr/lib64/libxcb-glx.so.0.0.0
7f3ad4354000-7f3ad4554000 ---p 00015000 00:27 604955 /usr/lib64/libxcb-glx.so.0.0.0
7f3ad4554000-7f3ad4556000 r--p 00015000 00:27 604955 /usr/lib64/libxcb-glx.so.0.0.0
7f3ad4556000-7f3ad4557000 rw-p 00017000 00:27 604955 /usr/lib64/libxcb-glx.so.0.0.0
7f3ad4557000-7f3ad468f000 r-xp 00000000 00:27 4392 /usr/lib64/libX11.so.6.3.0
7f3ad468f000-7f3ad488f000 ---p 00138000 00:27 4392 /usr/lib64/libX11.so.6.3.0
7f3ad488f000-7f3ad4890000 r--p 00138000 00:27 4392 /usr/lib64/libX11.so.6.3.0
7f3ad4890000-7f3ad4895000 rw-p 00139000 00:27 4392 /usr/lib64/libX11.so.6.3.0
7f3ad4897000-7f3ad4898000 r-xp 00000000 00:27 3578 /usr/lib64/libX11-xcb.so.1.0.0
7f3ad4898000-7f3ad4a97000 ---p 00001000 00:27 3578 /usr/lib64/libX11-xcb.so.1.0.0
7f3ad4a97000-7f3ad4a98000 r--p 00000000 00:27 3578 /usr/lib64/libX11-xcb.so.1.0.0
7f3ad4a98000-7f3ad4a99000 rw-p 00001000 00:27 3578 /usr/lib64/libX11-xcb.so.1.0.0
7f3ad4a9f000-7f3ad4aa4000 r-xp 00000000 00:27 8081 /usr/lib64/libXfixes.so.3.1.0
7f3ad4aa4000-7f3ad4ca3000 ---p 00005000 00:27 8081 /usr/lib64/libXfixes.so.3.1.0
7f3ad4ca3000-7f3ad4ca4000 r--p 00004000 00:27 8081 /usr/lib64/libXfixes.so.3.1.0
7f3ad4ca4000-7f3ad4ca5000 rw-p 00005000 00:27 8081 /usr/lib64/libXfixes.so.3.1.0
7f3ad4ca7000-7f3ad4ca9000 r-xp 00000000 00:27 8087 /usr/lib64/libXdamage.so.1.1.0
7f3ad4ca9000-7f3ad4ea8000 ---p 00002000 00:27 8087 /usr/lib64/libXdamage.so.1.1.0
7f3ad4ea8000-7f3ad4ea9000 r--p 00001000 00:27 8087 /usr/lib64/libXdamage.so.1.1.0
7f3ad4ea9000-7f3ad4eaa000 rw-p 00002000 00:27 8087 /usr/lib64/libXdamage.so.1.1.0
7f3ad4eaf000-7f3ad4ec0000 r-xp 00000000 00:27 8084 /usr/lib64/libXext.so.6.4.0
7f3ad4ec0000-7f3ad50bf000 ---p 00011000 00:27 8084 /usr/lib64/libXext.so.6.4.0
7f3ad50bf000-7f3ad50c0000 r--p 00010000 00:27 8084 /usr/lib64/libXext.so.6.4.0
7f3ad50c0000-7f3ad50c1000 rw-p 00011000 00:27 8084 /usr/lib64/libXext.so.6.4.0
7f3ad50c7000-7f3ad50f1000 r-xp 00000000 00:27 54067 /usr/lib64/libglapi.so.0.0.0
7f3ad50f1000-7f3ad52f0000 ---p 0002a000 00:27 54067 /usr/lib64/libglapi.so.0.0.0
7f3ad52f0000-7f3ad52f4000 r--p 00029000 00:27 54067 /usr/lib64/libglapi.so.0.0.0
7f3ad52f4000-7f3ad52f5000 rw-p 0002d000 00:27 54067 /usr/lib64/libglapi.so.0.0.0
7f3ad52f5000-7f3ad52f6000 rw-p 00000000 00:00 0
7f3ad52f7000-7f3ad52f8000 r-xp 00000000 00:27 2890 /usr/lib64/libxshmfence.so.1.0.0
7f3ad52f8000-7f3ad54f8000 ---p 00001000 00:27 2890 /usr/lib64/libxshmfence.so.1.0.0
7f3ad54f8000-7f3ad54f9000 r--p 00001000 00:27 2890 /usr/lib64/libxshmfence.so.1.0.0
7f3ad54f9000-7f3ad54fa000 rw-p 00002000 00:27 2890 /usr/lib64/libxshmfence.so.1.0.0
7f3ad54ff000-7f3ad5504000 r-xp 00000000 00:27 604939 /usr/lib64/libxcb-sync.so.1.0.0
7f3ad5504000-7f3ad5703000 ---p 00005000 00:27 604939 /usr/lib64/libxcb-sync.so.1.0.0
7f3ad5703000-7f3ad5704000 r--p 00004000 00:27 604939 /usr/lib64/libxcb-sync.so.1.0.0
7f3ad5704000-7f3ad5705000 rw-p 00005000 00:27 604939 /usr/lib64/libxcb-sync.so.1.0.0
7f3ad5707000-7f3ad5709000 r-xp 00000000 00:27 604951 /usr/lib64/libxcb-present.so.0.0.0
7f3ad5709000-7f3ad5908000 ---p 00002000 00:27 604951 /usr/lib64/libxcb-present.so.0.0.0
7f3ad5908000-7f3ad5909000 r--p 00001000 00:27 604951 /usr/lib64/libxcb-present.so.0.0.0
7f3ad5909000-7f3ad590a000 rw-p 00002000 00:27 604951 /usr/lib64/libxcb-present.so.0.0.0
7f3ad590f000-7f3ad5911000 r-xp 00000000 00:27 604959 /usr/lib64/libxcb-dri3.so.0.0.0
7f3ad5911000-7f3ad5b10000 ---p 00002000 00:27 604959 /usr/lib64/libxcb-dri3.so.0.0.0
7f3ad5b10000-7f3ad5b11000 r--p 00001000 00:27 604959 /usr/lib64/libxcb-dri3.so.0.0.0
7f3ad5b11000-7f3ad5b12000 rw-p 00002000 00:27 604959 /usr/lib64/libxcb-dri3.so.0.0.0
7f3ad5b17000-7f3ad5b1d000 r-xp 00000000 00:27 76606 /usr/lib64/libIlmThread-2_1.so.11.0.0
7f3ad5b1d000-7f3ad5d1c000 ---p 00006000 00:27 76606 /usr/lib64/libIlmThread-2_1.so.11.0.0
7f3ad5d1c000-7f3ad5d1d000 r--p 00005000 00:27 76606 /usr/lib64/libIlmThread-2_1.so.11.0.0
7f3ad5d1d000-7f3ad5d1e000 rw-p 00006000 00:27 76606 /usr/lib64/libIlmThread-2_1.so.11.0.0
7f3ad5d1f000-7f3ad5d3c000 r-xp 00000000 00:27 71046 /usr/lib64/libIex-2_1.so.11.0.0
7f3ad5d3c000-7f3ad5f3b000 ---p 0001d000 00:27 71046 /usr/lib64/libIex-2_1.so.11.0.0
7f3ad5f3b000-7f3ad5f3f000 r--p 0001c000 00:27 71046 /usr/lib64/libIex-2_1.so.11.0.0
7f3ad5f3f000-7f3ad5f40000 rw-p 00020000 00:27 71046 /usr/lib64/libIex-2_1.so.11.0.0
7f3ad5f47000-7f3ad5f50000 r-xp 00000000 00:27 3191 /usr/lib64/libjbig.so.2.0
7f3ad5f50000-7f3ad614f000 ---p 00009000 00:27 3191 /usr/lib64/libjbig.so.2.0
7f3ad614f000-7f3ad6150000 r--p 00008000 00:27 3191 /usr/lib64/libjbig.so.2.0
7f3ad6150000-7f3ad6153000 rw-p 00009000 00:27 3191 /usr/lib64/libjbig.so.2.0
7f3ad6157000-7f3ad617e000 r-xp 00000000 00:27 3167 /usr/lib64/liblzma.so.5.2.2
7f3ad617e000-7f3ad637e000 ---p 00027000 00:27 3167 /usr/lib64/liblzma.so.5.2.2
7f3ad637e000-7f3ad637f000 r--p 00027000 00:27 3167 /usr/lib64/liblzma.so.5.2.2
7f3ad637f000-7f3ad6380000 rw-p 00028000 00:27 3167 /usr/lib64/liblzma.so.5.2.2
7f3ad6387000-7f3ad63ef000 r-xp 00000000 00:27 55055 /usr/lib64/libGL.so.1.2.0
7f3ad63ef000-7f3ad65ee000 ---p 00068000 00:27 55055 /usr/lib64/libGL.so.1.2.0
7f3ad65ee000-7f3ad65f1000 r--p 00067000 00:27 55055 /usr/lib64/libGL.so.1.2.0
7f3ad65f1000-7f3ad65f2000 rw-p 0006a000 00:27 55055 /usr/lib64/libGL.so.1.2.0
7f3ad65f2000-7f3ad65f3000 rw-p 00000000 00:00 0
7f3ad65f7000-7f3ad6639000 r-xp 00000000 00:27 71043 /usr/lib64/libHalf.so.11.0.0
7f3ad6639000-7f3ad6838000 ---p 00042000 00:27 71043 /usr/lib64/libHalf.so.11.0.0
7f3ad6838000-7f3ad6839000 r--p 00041000 00:27 71043 /usr/lib64/libHalf.so.11.0.0
7f3ad6839000-7f3ad683a000 rw-p 00042000 00:27 71043 /usr/lib64/libHalf.so.11.0.0
7f3ad683f000-7f3ad6951000 r-xp 00000000 00:27 77951 /usr/lib64/libIlmImf-Imf_2_1.so.21.0.0
7f3ad6951000-7f3ad6b51000 ---p 00112000 00:27 77951 /usr/lib64/libIlmImf-Imf_2_1.so.21.0.0
7f3ad6b51000-7f3ad6b54000 r--p 00112000 00:27 77951 /usr/lib64/libIlmImf-Imf_2_1.so.21.0.0
7f3ad6b54000-7f3ad6b56000 rw-p 00115000 00:27 77951 /usr/lib64/libIlmImf-Imf_2_1.so.21.0.0
7f3ad6b56000-7f3ad6b57000 rw-p 00000000 00:00 0
7f3ad6b57000-7f3ad6ba6000 r-xp 00000000 00:27 54105 /usr/lib64/libjasper.so.1.0.0
7f3ad6ba6000-7f3ad6da5000 ---p 0004f000 00:27 54105 /usr/lib64/libjasper.so.1.0.0
7f3ad6da5000-7f3ad6da6000 r--p 0004e000 00:27 54105 /usr/lib64/libjasper.so.1.0.0
7f3ad6da6000-7f3ad6daa000 rw-p 0004f000 00:27 54105 /usr/lib64/libjasper.so.1.0.0
7f3ad6daa000-7f3ad6db1000 rw-p 00000000 00:00 0
7f3ad6db7000-7f3ad6e2a000 r-xp 00000000 00:27 54142 /usr/lib64/libtiff.so.5.2.6
7f3ad6e2a000-7f3ad7029000 ---p 00073000 00:27 54142 /usr/lib64/libtiff.so.5.2.6
7f3ad7029000-7f3ad702d000 r--p 00072000 00:27 54142 /usr/lib64/libtiff.so.5.2.6
7f3ad702d000-7f3ad702e000 rw-p 00076000 00:27 54142 /usr/lib64/libtiff.so.5.2.6
7f3ad702f000-7f3ad706a000 r-xp 00000000 00:27 3642 /usr/lib64/libpng16.so.16.8.0
7f3ad706a000-7f3ad726a000 ---p 0003b000 00:27 3642 /usr/lib64/libpng16.so.16.8.0
7f3ad726a000-7f3ad726b000 r--p 0003b000 00:27 3642 /usr/lib64/libpng16.so.16.8.0
7f3ad726b000-7f3ad726c000 rw-p 0003c000 00:27 3642 /usr/lib64/libpng16.so.16.8.0
7f3ad726f000-7f3ad72d7000 r-xp 00000000 00:27 407172 /usr/lib64/libjpeg.so.8.1.2
7f3ad72d7000-7f3ad74d7000 ---p 00068000 00:27 407172 /usr/lib64/libjpeg.so.8.1.2
7f3ad74d7000-7f3ad74d8000 r--p 00068000 00:27 407172 /usr/lib64/libjpeg.so.8.1.2
7f3ad74d8000-7f3ad74d9000 rw-p 00069000 00:27 407172 /usr/lib64/libjpeg.so.8.1.2
7f3ad74df000-7f3ad74f5000 r-xp 00000000 00:27 407157 /lib64/libgcc_s.so.1
7f3ad74f5000-7f3ad76f4000 ---p 00016000 00:27 407157 /lib64/libgcc_s.so.1
7f3ad76f4000-7f3ad76f5000 r--p 00015000 00:27 407157 /lib64/libgcc_s.so.1
7f3ad76f5000-7f3ad76f6000 rw-p 00016000 00:27 407157 /lib64/libgcc_s.so.1
7f3ad76f7000-7f3ad7872000 r-xp 00000000 00:27 414447 /usr/lib64/libstdc++.so.6.0.24
7f3ad7872000-7f3ad7a71000 ---p 0017b000 00:27 414447 /usr/lib64/libstdc++.so.6.0.24
7f3ad7a71000-7f3ad7a7b000 r--p 0017a000 00:27 414447 /usr/lib64/libstdc++.so.6.0.24
7f3ad7a7b000-7f3ad7a7d000 rw-p 00184000 00:27 414447 /usr/lib64/libstdc++.so.6.0.24
7f3ad7a7d000-7f3ad7a80000 rw-p 00000000 00:00 0
7f3ad7a87000-7f3ad7d54000 r-xp 00000000 00:27 83825 /usr/lib64/libopencv_core.so.3.1.0
7f3ad7d54000-7f3ad7f54000 ---p 002cd000 00:27 83825 /usr/lib64/libopencv_core.so.3.1.0
7f3ad7f54000-7f3ad7f59000 r--p 002cd000 00:27 83825 /usr/lib64/libopencv_core.so.3.1.0
7f3ad7f59000-7f3ad7f5c000 rw-p 002d2000 00:27 83825 /usr/lib64/libopencv_core.so.3.1.0
7f3ad7f5c000-7f3ad7f5f000 rw-p 00000000 00:00 0
7f3ad7f5f000-7f3ad8260000 r-xp 00000000 00:27 83837 /usr/lib64/libopencv_imgproc.so.3.1.0
7f3ad8260000-7f3ad8460000 ---p 00301000 00:27 83837 /usr/lib64/libopencv_imgproc.so.3.1.0
7f3ad8460000-7f3ad846a000 r--p 00301000 00:27 83837 /usr/lib64/libopencv_imgproc.so.3.1.0
7f3ad846a000-7f3ad8472000 rw-p 0030b000 00:27 83837 /usr/lib64/libopencv_imgproc.so.3.1.0
7f3ad8472000-7f3ad8508000 rw-p 00000000 00:00 0
7f3ad850f000-7f3ad85a4000 r-xp 00000000 00:27 83835 /usr/lib64/libopencv_imgcodecs.so.3.1.0
7f3ad85a4000-7f3ad87a4000 ---p 00095000 00:27 83835 /usr/lib64/libopencv_imgcodecs.so.3.1.0
7f3ad87a4000-7f3ad87a6000 r--p 00095000 00:27 83835 /usr/lib64/libopencv_imgcodecs.so.3.1.0
7f3ad87a6000-7f3ad87a7000 rw-p 00097000 00:27 83835 /usr/lib64/libopencv_imgcodecs.so.3.1.0
7f3ad87a7000-7f3ad87ab000 rw-p 00000000 00:00 0
7f3ad87af000-7f3ad87c3000 r-xp 00000000 00:27 662734 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
7f3ad87c3000-7f3ad89c2000 ---p 00014000 00:27 662734 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
7f3ad89c2000-7f3ad89c3000 r--p 00013000 00:27 662734 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
7f3ad89c3000-7f3ad89c4000 rw-p 00014000 00:27 662734 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
7f3ad89c7000-7f3ad89ee000 r-xp 00000000 00:27 54083 /usr/lib64/libexpat.so.1.6.0
7f3ad89ee000-7f3ad8bee000 ---p 00027000 00:27 54083 /usr/lib64/libexpat.so.1.6.0
7f3ad8bee000-7f3ad8bf0000 r--p 00027000 00:27 54083 /usr/lib64/libexpat.so.1.6.0
7f3ad8bf0000-7f3ad8bf1000 rw-p 00029000 00:27 54083 /usr/lib64/libexpat.so.1.6.0
7f3ad8bf7000-7f3ad8c0b000 r-xp 00000000 00:27 8519 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
7f3ad8c0b000-7f3ad8e0a000 ---p 00014000 00:27 8519 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
7f3ad8e0a000-7f3ad8e0b000 r--p 00013000 00:27 8519 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
7f3ad8e0b000-7f3ad8e0c000 rw-p 00014000 00:27 8519 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
7f3ad8e0f000-7f3ad8e55000 r-xp 00000000 00:27 3353 /lib64/libdbus-1.so.3.8.14
7f3ad8e55000-7f3ad9054000 ---p 00046000 00:27 3353 /lib64/libdbus-1.so.3.8.14
7f3ad9054000-7f3ad9055000 r--p 00045000 00:27 3353 /lib64/libdbus-1.so.3.8.14
7f3ad9055000-7f3ad9056000 rw-p 00046000 00:27 3353 /lib64/libdbus-1.so.3.8.14
7f3ad9057000-7f3ad9076000 r-xp 00000000 00:27 13715 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so
7f3ad9076000-7f3ad9275000 ---p 0001f000 00:27 13715 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so
7f3ad9275000-7f3ad9276000 r--p 0001e000 00:27 13715 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so
7f3ad9276000-7f3ad9277000 rw-p 0001f000 00:27 13715 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so
7f3ad9277000-7f3ad928b000 r-xp 00000000 00:27 72652 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7f3ad928b000-7f3ad948a000 ---p 00014000 00:27 72652 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7f3ad948a000-7f3ad948b000 r--p 00013000 00:27 72652 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7f3ad948b000-7f3ad948c000 rw-p 00014000 00:27 72652 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7f3ad948f000-7f3ad94b9000 r-xp 00000000 00:27 3972 /usr/lib64/libssh2.so.1.0.1
7f3ad94b9000-7f3ad96b8000 ---p 0002a000 00:27 3972 /usr/lib64/libssh2.so.1.0.1
7f3ad96b8000-7f3ad96b9000 r--p 00029000 00:27 3972 /usr/lib64/libssh2.so.1.0.1
7f3ad96b9000-7f3ad96ba000 rw-p 0002a000 00:27 3972 /usr/lib64/libssh2.so.1.0.1
7f3ad96bf000-7f3ad96e0000 r-xp 00000000 00:27 79872 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so
7f3ad96e0000-7f3ad98e0000 ---p 00021000 00:27 79872 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so
7f3ad98e0000-7f3ad98e1000 r--p 00021000 00:27 79872 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so
7f3ad98e1000-7f3ad98e2000 rw-p 00022000 00:27 79872 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so
7f3ad98e7000-7f3ad98e9000 r-xp 00000000 00:27 71623 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
7f3ad98e9000-7f3ad9ae8000 ---p 00002000 00:27 71623 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
7f3ad9ae8000-7f3ad9ae9000 r--p 00001000 00:27 71623 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
7f3ad9ae9000-7f3ad9aea000 rw-p 00002000 00:27 71623 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
7f3ad9aef000-7f3ad9af1000 r-xp 00000000 00:27 416382 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so
7f3ad9af1000-7f3ad9cf0000 ---p 00002000 00:27 416382 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so
7f3ad9cf0000-7f3ad9cf1000 r--p 00001000 00:27 416382 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so
7f3ad9cf1000-7f3ad9cf2000 rw-p 00002000 00:27 416382 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so
7f3ad9cf7000-7f3ad9cf9000 r-xp 00000000 00:27 416350 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/I18N/Langinfo/Langinfo.so
7f3ad9cf9000-7f3ad9ef9000 ---p 00002000 00:27 416350 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/I18N/Langinfo/Langinfo.so
7f3ad9ef9000-7f3ad9efa000 r--p 00002000 00:27 416350 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/I18N/Langinfo/Langinfo.so
7f3ad9efa000-7f3ad9efb000 rw-p 00003000 00:27 416350 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/I18N/Langinfo/Langinfo.so
7f3ad9eff000-7f3ad9f14000 r-xp 00000000 00:27 416374 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Storable/Storable.so
7f3ad9f14000-7f3ada113000 ---p 00015000 00:27 416374 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Storable/Storable.so
7f3ada113000-7f3ada114000 r--p 00014000 00:27 416374 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Storable/Storable.so
7f3ada114000-7f3ada115000 rw-p 00015000 00:27 416374 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Storable/Storable.so
7f3ada117000-7f3ada343000 r-xp 00000000 00:27 603305 /lib64/libcrypto.so.1.0.0
7f3ada343000-7f3ada543000 ---p 0022c000 00:27 603305 /lib64/libcrypto.so.1.0.0
7f3ada543000-7f3ada55e000 r--p 0022c000 00:27 603305 /lib64/libcrypto.so.1.0.0
7f3ada55e000-7f3ada56c000 rw-p 00247000 00:27 603305 /lib64/libcrypto.so.1.0.0
7f3ada56c000-7f3ada570000 rw-p 00000000 00:00 0
7f3ada577000-7f3ada5d8000 r-xp 00000000 00:27 603306 /lib64/libssl.so.1.0.0
7f3ada5d8000-7f3ada7d7000 ---p 00061000 00:27 603306 /lib64/libssl.so.1.0.0
7f3ada7d7000-7f3ada7db000 r--p 00060000 00:27 603306 /lib64/libssl.so.1.0.0
7f3ada7db000-7f3ada7e2000 rw-p 00064000 00:27 603306 /lib64/libssl.so.1.0.0
7f3ada7e7000-7f3ada856000 r-xp 00000000 00:27 8612 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7f3ada856000-7f3adaa56000 ---p 0006f000 00:27 8612 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7f3adaa56000-7f3adaa57000 r--p 0006f000 00:27 8612 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7f3adaa57000-7f3adaa59000 rw-p 00070000 00:27 8612 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7f3adaa5f000-7f3adaa6a000 r-xp 00000000 00:27 2580 /lib64/libnss_files-2.22.so
7f3adaa6a000-7f3adac69000 ---p 0000b000 00:27 2580 /lib64/libnss_files-2.22.so
7f3adac69000-7f3adac6a000 r--p 0000a000 00:27 2580 /lib64/libnss_files-2.22.so
7f3adac6a000-7f3adac6b000 rw-p 0000b000 00:27 2580 /lib64/libnss_files-2.22.so
7f3adac6b000-7f3adac71000 rw-p 00000000 00:00 0
7f3adac77000-7f3adac86000 r-xp 00000000 00:27 416309 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/B/B.so
7f3adac86000-7f3adae86000 ---p 0000f000 00:27 416309 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/B/B.so
7f3adae86000-7f3adae88000 r--p 0000f000 00:27 416309 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/B/B.so
7f3adae88000-7f3adae89000 rw-p 00011000 00:27 416309 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/B/B.so
7f3adae8f000-7f3adaea4000 r-xp 00000000 00:27 2881 /lib64/libz.so.1.2.8
7f3adaea4000-7f3adb0a3000 ---p 00015000 00:27 2881 /lib64/libz.so.1.2.8
7f3adb0a3000-7f3adb0a4000 r--p 00014000 00:27 2881 /lib64/libz.so.1.2.8
7f3adb0a4000-7f3adb0a5000 rw-p 00015000 00:27 2881 /lib64/libz.so.1.2.8
7f3adb0a7000-7f3adb0b5000 r-xp 00000000 00:27 416314 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7f3adb0b5000-7f3adb2b4000 ---p 0000e000 00:27 416314 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7f3adb2b4000-7f3adb2b5000 r--p 0000d000 00:27 416314 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7f3adb2b5000-7f3adb2b6000 rw-p 0000e000 00:27 416314 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7f3adb2b7000-7f3adb2bd000 r-xp 00000000 00:27 407336 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7f3adb2bd000-7f3adb4bc000 ---p 00006000 00:27 407336 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7f3adb4bc000-7f3adb4bd000 r--p 00005000 00:27 407336 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7f3adb4bd000-7f3adb4be000 rw-p 00006000 00:27 407336 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7f3adb4bf000-7f3adb512000 r-xp 00000000 00:27 416398 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/re/re.so
7f3adb512000-7f3adb711000 ---p 00053000 00:27 416398 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/re/re.so
7f3adb711000-7f3adb712000 r--p 00052000 00:27 416398 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/re/re.so
7f3adb712000-7f3adb713000 rw-p 00053000 00:27 416398 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/re/re.so
7f3adb717000-7f3adb71a000 r-xp 00000000 00:27 416354 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7f3adb71a000-7f3adb919000 ---p 00003000 00:27 416354 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7f3adb919000-7f3adb91a000 r--p 00002000 00:27 416354 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7f3adb91a000-7f3adb91b000 rw-p 00003000 00:27 416354 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7f3adb91f000-7f3adb927000 r-xp 00000000 00:27 416330 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Encode/Encode.so
7f3adb927000-7f3adbb26000 ---p 00008000 00:27 416330 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Encode/Encode.so
7f3adbb26000-7f3adbb27000 r--p 00007000 00:27 416330 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Encode/Encode.so
7f3adbb27000-7f3adbb28000 rw-p 00008000 00:27 416330 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Encode/Encode.so
7f3adbb2f000-7f3adbb3a000 r-xp 00000000 00:27 407330 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/SHA/SHA.so
7f3adbb3a000-7f3adbd39000 ---p 0000b000 00:27 407330 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/SHA/SHA.so
7f3adbd39000-7f3adbd3a000 r--p 0000a000 00:27 407330 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/SHA/SHA.so
7f3adbd3a000-7f3adbd3b000 rw-p 0000b000 00:27 407330 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/SHA/SHA.so
7f3adbd3f000-7f3adbd43000 r-xp 00000000 00:27 407328 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
7f3adbd43000-7f3adbf42000 ---p 00004000 00:27 407328 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
7f3adbf42000-7f3adbf43000 r--p 00003000 00:27 407328 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
7f3adbf43000-7f3adbf44000 rw-p 00004000 00:27 407328 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
7f3adbf47000-7f3adbf52000 r-xp 00000000 00:27 71602 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/Util/Util.so
7f3adbf52000-7f3adc151000 ---p 0000b000 00:27 71602 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/Util/Util.so
7f3adc151000-7f3adc152000 r--p 0000a000 00:27 71602 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/Util/Util.so
7f3adc152000-7f3adc153000 rw-p 0000b000 00:27 71602 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/Util/Util.so
7f3adc157000-7f3adc15f000 r-xp 00000000 00:27 407326 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7f3adc15f000-7f3adc35e000 ---p 00008000 00:27 407326 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7f3adc35e000-7f3adc35f000 r--p 00007000 00:27 407326 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7f3adc35f000-7f3adc360000 rw-p 00008000 00:27 407326 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7f3adc367000-7f3adc369000 r-xp 00000000 00:27 407324 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7f3adc369000-7f3adc568000 ---p 00002000 00:27 407324 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7f3adc568000-7f3adc569000 r--p 00001000 00:27 407324 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7f3adc569000-7f3adc56a000 rw-p 00002000 00:27 407324 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7f3adc56f000-7f3adc571000 r-xp 00000000 00:27 416394 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/attributes/attributes.so
7f3adc571000-7f3adc770000 ---p 00002000 00:27 416394 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/attributes/attributes.so
7f3adc770000-7f3adc771000 r--p 00001000 00:27 416394 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/attributes/attributes.so
7f3adc771000-7f3adc772000 rw-p 00002000 00:27 416394 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/attributes/attributes.so
7f3adc777000-7f3adc786000 r-xp 00000000 00:27 78188 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/JSON/XS/XS.so
7f3adc786000-7f3adc985000 ---p 0000f000 00:27 78188 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/JSON/XS/XS.so
7f3adc985000-7f3adc986000 r--p 0000e000 00:27 78188 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/JSON/XS/XS.so
7f3adc986000-7f3adc987000 rw-p 0000f000 00:27 78188 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/JSON/XS/XS.so
7f3adc987000-7f3adc999000 r-xp 00000000 00:27 407342 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7f3adc999000-7f3adcb98000 ---p 00012000 00:27 407342 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7f3adcb98000-7f3adcb9b000 r--p 00011000 00:27 407342 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7f3adcb9b000-7f3adcb9c000 rw-p 00014000 00:27 407342 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7f3adcb9f000-7f3adcba2000 r-xp 00000000 00:27 407332 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7f3adcba2000-7f3adcda2000 ---p 00003000 00:27 407332 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7f3adcda2000-7f3adcda3000 r--p 00003000 00:27 407332 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7f3adcda3000-7f3adcda4000 rw-p 00004000 00:27 407332 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7f3adcda7000-7f3adcdaf000 r-xp 00000000 00:27 407344 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Socket/Socket.so
7f3adcdaf000-7f3adcfaf000 ---p 00008000 00:27 407344 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Socket/Socket.so
7f3adcfaf000-7f3adcfb1000 r--p 00008000 00:27 407344 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Socket/Socket.so
7f3adcfb1000-7f3adcfb2000 rw-p 0000a000 00:27 407344 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Socket/Socket.so
7f3adcfb7000-7f3adcfbb000 r-xp 00000000 00:27 407338 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/IO/IO.so
7f3adcfbb000-7f3add1ba000 ---p 00004000 00:27 407338 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/IO/IO.so
7f3add1ba000-7f3add1bb000 r--p 00003000 00:27 407338 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/IO/IO.so
7f3add1bb000-7f3add1bc000 rw-p 00004000 00:27 407338 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/IO/IO.so
7f3add1bf000-7f3add1c6000 r-xp 00000000 00:27 2594 /lib64/librt-2.22.so
7f3add1c6000-7f3add3c5000 ---p 00007000 00:27 2594 /lib64/librt-2.22.so
7f3add3c5000-7f3add3c6000 r--p 00006000 00:27 2594 /lib64/librt-2.22.so
7f3add3c6000-7f3add3c7000 rw-p 00007000 00:27 2594 /lib64/librt-2.22.so
7f3add3c7000-7f3add3cc000 r-xp 00000000 00:27 416384 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7f3add3cc000-7f3add5cb000 ---p 00005000 00:27 416384 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7f3add5cb000-7f3add5cc000 r--p 00004000 00:27 416384 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7f3add5cc000-7f3add5cd000 rw-p 00005000 00:27 416384 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7f3add5cf000-7f3add768000 r-xp 00000000 00:27 2560 /lib64/libc-2.22.so
7f3add768000-7f3add968000 ---p 00199000 00:27 2560 /lib64/libc-2.22.so
7f3add968000-7f3add96c000 r--p 00199000 00:27 2560 /lib64/libc-2.22.so
7f3add96c000-7f3add96e000 rw-p 0019d000 00:27 2560 /lib64/libc-2.22.so
7f3add96e000-7f3add972000 rw-p 00000000 00:00 0
7f3add977000-7f3add98f000 r-xp 00000000 00:27 2590 /lib64/libpthread-2.22.so
7f3add98f000-7f3addb8e000 ---p 00018000 00:27 2590 /lib64/libpthread-2.22.so
7f3addb8e000-7f3addb8f000 r--p 00017000 00:27 2590 /lib64/libpthread-2.22.so
7f3addb8f000-7f3addb90000 rw-p 00018000 00:27 2590 /lib64/libpthread-2.22.so
7f3addb90000-7f3addb94000 rw-p 00000000 00:00 0
7f3addb97000-7f3addba2000 r-xp 00000000 00:27 2564 /lib64/libcrypt-2.22.so
7f3addba2000-7f3addda2000 ---p 0000b000 00:27 2564 /lib64/libcrypt-2.22.so
7f3addda2000-7f3addda3000 r--p 0000b000 00:27 2564 /lib64/libcrypt-2.22.so
7f3addda3000-7f3addda4000 rw-p 0000c000 00:27 2564 /lib64/libcrypt-2.22.so
7f3addda4000-7f3adddd2000 rw-p 00000000 00:00 0
7f3adddd7000-7f3adddd9000 r-xp 00000000 00:27 2566 /lib64/libdl-2.22.so
7f3adddd9000-7f3addfd9000 ---p 00002000 00:27 2566 /lib64/libdl-2.22.so
7f3addfd9000-7f3addfda000 r--p 00002000 00:27 2566 /lib64/libdl-2.22.so
7f3addfda000-7f3addfdb000 rw-p 00003000 00:27 2566 /lib64/libdl-2.22.so
7f3addfdf000-7f3ade0db000 r-xp 00000000 00:27 2568 /lib64/libm-2.22.so
7f3ade0db000-7f3ade2da000 ---p 000fc000 00:27 2568 /lib64/libm-2.22.so
7f3ade2da000-7f3ade2db000 r--p 000fb000 00:27 2568 /lib64/libm-2.22.so
7f3ade2db000-7f3ade2dc000 rw-p 000fc000 00:27 2568 /lib64/libm-2.22.so
7f3ade2df000-7f3ade300000 r-xp 00000000 00:27 2552 /lib64/ld-2.22.so
7f3ade3cc000-7f3ade44d000 rw-p 00000000 00:00 0
7f3ade477000-7f3ade4ac000 r--s 00000000 00:14 26836 /run/nscd/dbA9ZYZ5 (deleted)
7f3ade4af000-7f3ade4b6000 r--s 00000000 00:27 22586 /usr/lib64/gconv/gconv-modules.cache
7f3ade4b7000-7f3ade4fb000 r--p 00000000 00:27 22173 /usr/lib/locale/en_US.utf8/LC_CTYPE
7f3ade4ff000-7f3ade500000 r--p 00020000 00:27 2552 /lib64/ld-2.22.so
7f3ade500000-7f3ade501000 rw-p 00021000 00:27 2552 /lib64/ld-2.22.so
7f3ade501000-7f3ade502000 rw-p 00000000 00:00 0
7f3ade503000-7f3ade504000 rw-p 00000000 00:00 0
7f3ade504000-7f3ade509000 rw-p 00000000 00:00 0
7ffcf75f9000-7ffcf761a000 rw-p 00000000 00:00 0 [stack]
7ffcf76ef000-7ffcf76f2000 r--p 00000000 00:00 0 [vvar]
7ffcf76f2000-7ffcf76f4000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Unexpected end of data 0
[2018-01-19T22:29:31.0441 CET] [debug] backend 12520 died
can_read received kill signal at /usr/lib/os-autoinst/myjsonrpc.pm line 89.
</code></pre>
<p>so</p>
<ul>
<li>looks like qemu bug</li>
<li>isotovideo does not foresee this and fails to terminate the qemu process on dying</li>
</ul>
<p>I manually terminated the two still running old qemu processes on openqaworker10:</p>
<pre><code>_openqa+ 9482 99.2 0.4 5623116 1210348 ? Sl Jan19 2562:27 /usr/bin/qemu-system-x86_64 -serial file:seria
_openqa+ 12546 0.7 0.4 5516308 1163968 ? Sl Jan19 19:10 /usr/bin/qemu-system-x86_64 -serial file:serial
</code></pre>
<p>and will retrigger the failed jobs I find but of course that won't be enough.</p>
<p>Who can take over here to care about either qemu or isotovideo?</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84871
2018-01-22T05:45:15Z
okurz
okurz@suse.com
<ul><li><strong>Project</strong> changed from <i>openQA Tests</i> to <i>openQA Project</i></li><li><strong>Subject</strong> changed from <i>[qam] openqaworker10:4 - worker fail all tests</i> to <i>[qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind</i></li><li><strong>Category</strong> changed from <i>Infrastructure</i> to <i>Regressions/Crashes</i></li></ul>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84880
2018-01-22T05:48:59Z
okurz
okurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-5 priority-3 priority-lowest closed" href="/issues/30595">action #30595</a>: [ppc64le] Deploy bcache on tmpfs workers</i> added</li></ul>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84889
2018-01-22T06:26:25Z
coolo
coolo@suse.com
<ul></ul><p>where do you see a QEMU bug? I only see isotovideo crashing</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84892
2018-01-22T06:33:21Z
coolo
coolo@suse.com
<ul></ul><p>and it's the same 42.3 stack as on all workers. I will shutdown the worker</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=84907
2018-01-22T07:50:01Z
okurz
okurz@suse.com
<ul></ul><p>coolo wrote:</p>
<blockquote>
<p>where do you see a QEMU bug? I only see isotovideo crashing</p>
</blockquote>
<p>here:</p>
<pre><code>[2018-01-19T22:29:24.0538 CET] [debug] QEMU:
[2018-01-19T22:29:24.0538 CET] [debug] QEMU: (process:12546): GLib-CRITICAL **: Source ID 32305 was not found when attempting to remove it
</code></pre>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=85099
2018-01-22T11:23:05Z
coolo
coolo@suse.com
<ul></ul><p>this sounds basically like the same as what perl died on: memory corruption. And I find it rare that two processes have memory corruptions at the same time. We may need to do a memory check on this machine</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=85102
2018-01-22T11:32:06Z
coolo
coolo@suse.com
<ul></ul><p>The qemu problem is more frequent and also happens on other machines: <a href="http://paste.suse.de/19743" class="external">http://paste.suse.de/19743</a> (27/2000 jobs) - it always (when I looked) comes with a VNC stall. I would consider an out of memory problem, if worker10 wouldn't have 256GB</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=85105
2018-01-22T11:46:18Z
coolo
coolo@suse.com
<ul></ul><p>that perl crash is only once in the last 3000 jobs. So I would guess this is an effect of the matrix</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=85108
2018-01-22T11:54:45Z
coolo
coolo@suse.com
<ul><li><strong>Target version</strong> set to <i>Ready</i></li></ul><p>1401051 crashed differently - also on worker10 and also leaving qemu behind to create incompletes. So while the crashes are rare enough that I doubt we can do much about it (especially as we still have no monitoring for the workers to understand what's going on when it crashes), we should be able to cope with the backend.crash file</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=85111
2018-01-22T12:01:16Z
coolo
coolo@suse.com
<ul></ul><p>And here is the problem: both crashes didn't leave a backend.crashed file</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=85981
2018-01-24T15:43:01Z
okurz
okurz@suse.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed behind-schedule" href="/issues/30700">action #30700</a>: [sle][functional][sporadic][medium][sle12 sp4] test fails in memtest</i> added</li></ul>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=86116
2018-01-25T07:16:47Z
okurz
okurz@suse.com
<ul></ul><p>I triggered 20 memtest jobs and found some incompletes which look like the same symptoms: <a href="https://openqa.suse.de/tests/overview?build=0164&version=12-SP4&distri=sle&groupid=139" class="external">https://openqa.suse.de/tests/overview?build=0164&version=12-SP4&distri=sle&groupid=139</a></p>
<p>-> the incompletes:</p>
<ul>
<li><a href="https://openqa.suse.de/tests/1417046/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1417046/file/autoinst-log.txt</a> on openqaworker5:14</li>
<li><a href="https://openqa.suse.de/tests/1417057/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1417057/file/autoinst-log.txt</a> on openqaworker7:6</li>
<li><a href="https://openqa.suse.de/tests/1417050/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1417050/file/autoinst-log.txt</a> on openqaworker6:11</li>
<li><a href="https://openqa.suse.de/tests/1417049/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1417049/file/autoinst-log.txt</a> on openqaworker6:2</li>
</ul>
<p>That is 4/20 incompletes, the rest passed</p>
<p>So I doubt it's a "memory check needed on all machines at the same time".</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=86119
2018-01-25T07:20:17Z
okurz
okurz@suse.com
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-4 status-3 priority-4 priority-default closed behind-schedule" href="/issues/30700">action #30700</a>: [sle][functional][sporadic][medium][sle12 sp4] test fails in memtest</i>)</li></ul>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=86128
2018-01-25T07:20:29Z
okurz
okurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed behind-schedule" href="/issues/30700">action #30700</a>: [sle][functional][sporadic][medium][sle12 sp4] test fails in memtest</i> added</li></ul>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=86632
2018-01-26T10:15:07Z
szarate
<ul><li><strong>Assignee</strong> set to <i>dasantiago</i></li><li><strong>Target version</strong> changed from <i>Ready</i> to <i>Current Sprint</i></li></ul>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=87079
2018-01-29T11:58:07Z
dasantiago
dasantiago@suse.com
<ul></ul><p>Causes:<br>
worker 10:4 -> memory corruption: <a href="https://openqa.suse.de/tests/1401276/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1401276/file/autoinst-log.txt</a><br>
worker 5:14 -> out of disk space <a href="https://openqa.suse.de/tests/1414086/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1414086/file/autoinst-log.txt</a><br>
worker 7:6 -> out of disk space <a href="https://openqa.suse.de/tests/1414635/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1414635/file/autoinst-log.txt</a><br>
worker 6:11 -> no clue... :-(<br>
worker 6:2 -> broken pipe: <a href="https://openqa.suse.de/tests/1415737/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1415737/file/autoinst-log.txt</a> (but it looks the same sympton as out of disk space - worker 5:14)</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=87730
2018-01-31T09:46:06Z
dasantiago
dasantiago@suse.com
<ul></ul><p>malbec:3 -> disk space: <a href="https://openqa.suse.de/tests/1432891/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1432891/file/autoinst-log.txt</a><br>
malbec:2 -> disk space: <a href="https://openqa.suse.de/tests/1431618/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1431618/file/autoinst-log.txt</a></p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=88237
2018-02-01T11:04:55Z
coolo
coolo@suse.com
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/31126">action #31126</a>: [sle][functional][sle15] sle-15-Installer-DVD-x86_64-Build429.1-select_disk@64bit died because of qmp cannot be found</i> added</li></ul>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=91537
2018-02-16T10:48:20Z
pcervinka
pcervinka@suse.com
<ul></ul><p>Looks similar to error on openqaworker10:1 <a href="https://openqa.suse.de/tests/1478755/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1478755/file/autoinst-log.txt</a></p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=91600
2018-02-16T13:06:33Z
coolo
coolo@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>indeed. And so we deployed - and caused tons of other problems for now :)</p>
<p>but openqaworker10 is troublesome, I disabled it completely for now</p>
openQA Project - action #30388: [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
https://progress.opensuse.org/issues/30388?journal_id=94351
2018-02-26T11:07:15Z
szarate
<ul><li><strong>Target version</strong> changed from <i>Current Sprint</i> to <i>Done</i></li></ul>