Project

General

Profile

Actions
Copy link

action #30388

closed

[qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind

Added by pcervinka almost 7 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
dasantiago
Category:
Regressions/Crashes
Target version:
Done
Start date:
2018-02-26
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

openqaworker10:4 fail with error:

.
.
Use of uninitialized value in pattern match (m//) at /var/lib/openqa/cache/tests/sle/products/sle/main.pm line 363.
.
.

DIE can't open qmp: No such file or directory at /usr/lib/os-autoinst/backend/qemu.pm line 809.

 at /usr/lib/os-autoinst/backend/baseclass.pm line 80.
    backend::baseclass::die_handler('can\'t open qmp: No such file or directory at /usr/lib/os-aut...') called at /usr/lib/os-autoinst/backend/qemu.pm line 809
    backend::qemu::start_qemu('backend::qemu=HASH(0x72d4610)') called at /usr/lib/os-autoinst/backend/qemu.pm line 111
    backend::qemu::do_start_vm('backend::qemu=HASH(0x72d4610)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 308
    backend::baseclass::start_vm('backend::qemu=HASH(0x72d4610)', undef) called at /usr/lib/os-autoinst/backend/baseclass.pm line 75
    backend::baseclass::handle_command('backend::qemu=HASH(0x72d4610)', 'HASH(0x74e8fd8)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 487
    backend::baseclass::check_socket('backend::qemu=HASH(0x72d4610)', 'IO::Handle=GLOB(0x7302c78)') called at /usr/lib/os-autoinst/backend/qemu.pm line 928
    backend::qemu::check_socket('backend::qemu=HASH(0x72d4610)', 'IO::Handle=GLOB(0x7302c78)', 0) called at /usr/lib/os-autoinst/backend/baseclass.pm line 243
    eval {...} called at /usr/lib/os-autoinst/backend/baseclass.pm line 156
    backend::baseclass::run_capture_loop('backend::qemu=HASH(0x72d4610)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 129
    backend::baseclass::run('backend::qemu=HASH(0x72d4610)', 5, 8) called at /usr/lib/os-autoinst/backend/driver.pm line 85
    backend::driver::start('backend::driver=HASH(0x5553160)') called at /usr/lib/os-autoinst/backend/driver.pm line 48
    backend::driver::new('backend::driver', 'qemu') called at /usr/bin/isotovideo line 211
    main::init_backend() called at /usr/bin/isotovideo line 280

Looks like not properly updated test code, maybe hanged salt updater. It started 15.1.2018.
You can latest status:
https://openqa.suse.de/admin/workers/925

Error logs:
https://openqa.suse.de/tests/1389513/file/autoinst-log.txt
https://openqa.suse.de/tests/1389504/file/autoinst-log.txt
https://openqa.suse.de/tests/1389417/file/autoinst-log.txt
https://openqa.suse.de/tests/1389513/file/autoinst-log.txt

Subtasks 2 (0 open — 2 closed)

action #32263: [tools] Implement sub-reaper support in workers when starting isotovideoResolvedEDiGiacinto2018-02-26

Actions
action #32293: [tools] Implement sub-reaper support in isotovideoResolvedokurz2018-02-26

Actions

Related issues 3 (0 open — 3 closed)

Related to openQA Infrastructure (public) - action #30595: [ppc64le] Deploy bcache on tmpfs workersClosed2018-01-22

Actions
Related to openQA Tests (public) - action #30700: [sle][functional][sporadic][medium][sle12 sp4] test fails in memtestResolvedokurz2018-01-232018-02-27

Actions
Has duplicate openQA Project (public) - action #31126: [sle][functional][sle15] sle-15-Installer-DVD-x86_64-Build429.1-select_disk@64bit died because of qmp cannot be foundRejected2018-01-31

Actions
Actions
Copy link
 #1

Updated by dasantiago almost 7 years ago

Usually this is a sympton that a qemu instance is running when it shouldn't and the start of the new instance is failing. To fix it, the current running qemu process should be killed.

Actions
Copy link
 #2

Updated by pcervinka almost 7 years ago

openqaworker10:4 looks fine, but worker openqaworker10:2 started to fail recently:
https://openqa.suse.de/admin/workers/772

Actions
Copy link
 #3

Updated by pcervinka almost 7 years ago

openqaworker10:7 fails too:
https://openqa.suse.de/admin/workers/777

Actions
Copy link
 #4

Updated by okurz almost 7 years ago

  • Category set to Infrastructure
  • Status changed from New to In Progress
  • Assignee set to okurz

checking right now on that machine I can see two qemu processes which do not seem to be linked to workers. They had been running since yesterday. I killed these two qemu processes now and will check the instances 7 and 2. Last output from instance 7:

sudo systemctl status openqa-worker@7

Jan 18 09:05:57 openqaworker10 worker[2703]: child 16762 died with exit status 256
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] +++ worker notes +++
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] end time: 2018-01-18 08:05:57
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] result: died
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] uploading video.ogv
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] uploading vars.json
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] uploading autoinst-log.txt
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] uploading worker-log.txt
Jan 18 09:05:57 openqaworker10 worker[2703]: can't open /var/lib/openqa/pool/7/testresults/test_order.json: No such file or directory at /usr/s...ine 756.
Jan 18 09:05:57 openqaworker10 worker[2703]: [info] cleaning up 01395623-sle-12-SP3-Server-DVD-Updates-x86_64-Build20180118-1-mau-qa_kernel_ltp_mm@64bit
Hint: Some lines were ellipsized, use -l to show in full.

For instance 2 it looks comparable. instance 7 picked up a job and is executing that just fine: https://openqa.suse.de/tests/1395637#live

I will monitor both instances and report back.

Actions
Copy link
 #5

Updated by okurz almost 7 years ago

  • Status changed from In Progress to Resolved

instance 7 successfully finished a job, instance 2 so far did not pick up a job but is online.

I assume that this was a temporary problem which will not appear anymore. If it does please report specifically the symptoms and we should try to solve it for good.

Actions
Copy link
 #6

Updated by okurz almost 7 years ago

  • Status changed from Resolved to In Progress
  • Assignee deleted (okurz)

Apparently happened again. openqaworker10:4 failed, see https://openqa.suse.de/tests/1404119/file/autoinst-log.txt

Checking on the worker I see a left-over qemu process:

_openqa+ 12546  0.8  0.4 5516308 1163948 ?     Sl   Jan19  18:24 /usr/bin/qemu-system-x86_64 -serial file:serial0 -soundhw ac97 -vga cirrus -global isa-fdc.driveA= -m 1024 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -device virtio-scsi-pci,id=scsi0 -device virtio-blk,drive=hd1,serial=1 -drive file=raid/l1,cache=unsafe,if=none,id=hd1,format=qcow2 -drive media=cdrom,if=none,id=cd0,format=raw,file=/var/lib/openqa/pool/4/SLE-12-SP3-Server-DVD-x86_64-GM-DVD1.iso -device scsi-cd,drive=cd0,bus=scsi0.0 -boot once=d,menu=on,splash-time=5000 -device usb-ehci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :94,share=force-shared -device virtio-serial -chardev socket,path=virtio_console,server,nowait,id=virtio_console,logfile=virtio_console.log -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -qmp unix:qmp_socket,server,nowait -monitor unix:hmp_socket,server,nowait -S -monitor telnet:127.0.0.1:20042,server,nowait

So the process is running since 2018-01-19 18:24 at least. Taking a look into the journal of this worker instance I find the last job that successfully executed before that is https://openqa.suse.de/tests/1400519# . And then

https://openqa.suse.de/tests/1401276/file/autoinst-log.txt reports

[2018-01-19T22:29:24.0513 CET] [debug] considering VNC stalled, no update for 4.24 seconds
[2018-01-19T22:29:24.0538 CET] [debug] QEMU: 
[2018-01-19T22:29:24.0538 CET] [debug] QEMU: (process:12546): GLib-CRITICAL **: Source ID 32305 was not found when attempting to remove it
[2018-01-19T22:29:24.0763 CET] [debug] pointer type 1 0 1024 768 -257
[2018-01-19T22:29:24.0763 CET] [debug] led state 0 1 1 -261
[2018-01-19T22:29:25.0014 CET] [debug] no change: 192.3s
[2018-01-19T22:29:26.0015 CET] [debug] no change: 191.3s
[2018-01-19T22:29:27.0016 CET] [debug] no change: 190.3s
[2018-01-19T22:29:28.0016 CET] [debug] no change: 189.3s
[2018-01-19T22:29:29.0018 CET] [debug] no change: 188.3s
[2018-01-19T22:29:30.0037 CET] [debug] MATCH(autoyast-system-login-console-20150529:0.00)
[2018-01-19T22:29:30.0051 CET] [debug] MATCH(autoyast-system-login-console-20170724:0.00)
[2018-01-19T22:29:30.0057 CET] [debug] MATCH(autoyast-system-login-console-minimal-20160113:0.95)
[2018-01-19T22:29:30.0063 CET] [debug] MATCH(autoyast-system-login-console-sles11sp4-20170419:1.00)
[2018-01-19T22:29:30.0073 CET] [debug] MATCH(linux-login-20150120:0.00)
[2018-01-19T22:29:30.0074 CET] [debug] MATCH(linux-login-20150922:0.95)
[2018-01-19T22:29:30.0077 CET] [debug] MATCH(linux-login-20170814:0.00)
[2018-01-19T22:29:30.0091 CET] [debug] MATCH(linux-login-bsc1055103-20170822:0.00)
[2018-01-19T22:29:30.0113 CET] [debug] MATCH(system-login-console-ipmitool-20170623:0.00)
[2018-01-19T22:29:30.0128 CET] [debug] MATCH(system-login-console-ipmitool-20170624:0.00)
[2018-01-19T22:29:30.0140 CET] [debug] MATCH(installation-autoyast-linuxlogin-20171204:0.29)
[2018-01-19T22:29:30.0150 CET] [debug] MATCH(online_migration_setup-emergency-shell-20170201:0.00)
[2018-01-19T22:29:30.0162 CET] [debug] MATCH(online_migration_setup-emergency-shell-20170330:0.00)
[2018-01-19T22:29:30.0172 CET] [debug] MATCH(snapper-emergency-shell-20170620:0.00)
[2018-01-19T22:29:30.0185 CET] [debug] MATCH(emergency-mode-20171017:0.00)
[2018-01-19T22:29:30.0366 CET] [debug] >>> testapi::_handle_found_needle: found autoyast-system-login-console-sles11sp4-20170419, similarity 1.00 @ 88/76
[2018-01-19T22:29:30.0369 CET] [debug] ||| finished update_minimal minimal at 2018-01-19 21:29:30 (666 s)
[2018-01-19T22:29:30.0370 CET] [debug] ||| starting check_logs tests/qam-minimal/check_logs.pm
[2018-01-19T22:29:30.0370 CET] [debug] /var/lib/openqa/cache/tests/sle/tests/qam-minimal/check_logs.pm:27 called testapi::select_console
[2018-01-19T22:29:30.0371 CET] [debug] <<< testapi::select_console(testapi_console='root-console')
[2018-01-19T22:29:30.0587 CET] [debug] activate_console, console: root-console, type: console
[2018-01-19T22:29:30.0587 CET] [debug] /var/lib/openqa/cache/tests/sle/tests/qam-minimal/check_logs.pm:27 called testapi::select_console
[2018-01-19T22:29:30.0587 CET] [debug] <<< testapi::assert_screen(mustmatch=[
  'tty2-selected',
  'text-logged-in-root',
  'text-login'
], timeout=60)
*** Error in `/usr/bin/isotovideo: backen': corrupted double-linked list: 0x0000000009413f70 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x721af)[0x7f3add6411af]
/lib64/libc.so.6(+0x77706)[0x7f3add646706]
/lib64/libc.so.6(+0x786ee)[0x7f3add6476ee]
/usr/lib64/libopencv_core.so.3.1(+0x20754c)[0x7f3ad7c8e54c]
/usr/lib64/libopencv_core.so.3.1(_ZN2cv3Mat10deallocateEv+0x24)[0x7f3ad7c8e904]
/usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so(_Z13image_destroyP5Image+0x131)[0x7f3ad87b9d21]
/usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so(+0x4229)[0x7f3ad87b3229]
/usr/bin/isotovideo: backen(Perl_pp_entersub+0x586)[0x4ab526]
/usr/bin/isotovideo: backen(Perl_call_sv+0x607)[0x4372c7]
/usr/bin/isotovideo: backen[0x4b4404]
/usr/bin/isotovideo: backen(Perl_sv_clear+0x480)[0x4b4b80]
/usr/bin/isotovideo: backen(Perl_sv_free2+0xdd)[0x4b520d]
/usr/bin/isotovideo: backen(Perl_free_tmps+0x68)[0x4d6988]
/usr/bin/isotovideo: backen(Perl_pp_unstack+0x90)[0x4a4c10]
/usr/bin/isotovideo: backen(Perl_runops_standard+0x16)[0x4a3cf6]
/usr/bin/isotovideo: backen(perl_run+0x382)[0x43e112]
/usr/bin/isotovideo: backen(main+0x123)[0x41e963]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3add5ef6e5]
/usr/bin/isotovideo: backen(_start+0x29)[0x41e9a9]
======= Memory map: ========
00400000-0057a000 r-xp 00000000 00:27 407348                             /usr/bin/perl
00779000-0077a000 r--p 00179000 00:27 407348                             /usr/bin/perl
0077a000-0077f000 rw-p 0017a000 00:27 407348                             /usr/bin/perl
00a1a000-06b72000 rw-p 00000000 00:00 0                                  [heap]
06b72000-0ed57000 rw-p 00000000 00:00 0                                  [heap]
7f3aa4000000-7f3aa4021000 rw-p 00000000 00:00 0 
7f3aa4021000-7f3aa8000000 ---p 00000000 00:00 0 
7f3aac000000-7f3aac021000 rw-p 00000000 00:00 0 
7f3aac021000-7f3ab0000000 ---p 00000000 00:00 0 
7f3ab0000000-7f3ab0021000 rw-p 00000000 00:00 0 
7f3ab0021000-7f3ab4000000 ---p 00000000 00:00 0 
7f3ab4000000-7f3ab4021000 rw-p 00000000 00:00 0 
7f3ab4021000-7f3ab8000000 ---p 00000000 00:00 0 
7f3ab8000000-7f3ab8021000 rw-p 00000000 00:00 0 
7f3ab8021000-7f3abc000000 ---p 00000000 00:00 0 
7f3abc000000-7f3abc021000 rw-p 00000000 00:00 0 
7f3abc021000-7f3ac0000000 ---p 00000000 00:00 0 
7f3ac0000000-7f3ac0021000 rw-p 00000000 00:00 0 
7f3ac0021000-7f3ac4000000 ---p 00000000 00:00 0 
7f3ac4000000-7f3ac4021000 rw-p 00000000 00:00 0 
7f3ac4021000-7f3ac8000000 ---p 00000000 00:00 0 
7f3ac9980000-7f3ac9981000 ---p 00000000 00:00 0 
7f3ac9981000-7f3aca181000 rw-p 00000000 00:00 0 
7f3acb7ff000-7f3acb800000 ---p 00000000 00:00 0 
7f3acb800000-7f3acc000000 rw-p 00000000 00:00 0 
7f3acc000000-7f3acc021000 rw-p 00000000 00:00 0 
7f3acc021000-7f3ad0000000 ---p 00000000 00:00 0 
7f3ad017d000-7f3ad017e000 ---p 00000000 00:00 0 
7f3ad017e000-7f3ad097e000 rw-p 00000000 00:00 0 
7f3ad097e000-7f3ad097f000 ---p 00000000 00:00 0 
7f3ad097f000-7f3ad117f000 rw-p 00000000 00:00 0 
7f3ad117f000-7f3ad1180000 ---p 00000000 00:00 0 
7f3ad1180000-7f3ad1980000 rw-p 00000000 00:00 0 
7f3ad1980000-7f3ad1981000 ---p 00000000 00:00 0 
7f3ad1981000-7f3ad2181000 rw-p 00000000 00:00 0 
7f3ad2181000-7f3ad2182000 ---p 00000000 00:00 0 
7f3ad2182000-7f3ad2982000 rw-p 00000000 00:00 0 
7f3ad2982000-7f3ad2983000 ---p 00000000 00:00 0 
7f3ad2983000-7f3ad33c4000 rw-p 00000000 00:00 0 
7f3ad36e7000-7f3ad36e9000 r-xp 00000000 00:27 71350                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Crypt/DES/DES.so
7f3ad36e9000-7f3ad38e9000 ---p 00002000 00:27 71350                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Crypt/DES/DES.so
7f3ad38e9000-7f3ad38ea000 r--p 00002000 00:27 71350                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Crypt/DES/DES.so
7f3ad38ea000-7f3ad38ed000 rw-p 00003000 00:27 71350                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Crypt/DES/DES.so
7f3ad38ef000-7f3ad38f2000 r-xp 00000000 00:27 3575                       /usr/lib64/libXau.so.6.0.0
7f3ad38f2000-7f3ad3af1000 ---p 00003000 00:27 3575                       /usr/lib64/libXau.so.6.0.0
7f3ad3af1000-7f3ad3af2000 r--p 00002000 00:27 3575                       /usr/lib64/libXau.so.6.0.0
7f3ad3af2000-7f3ad3af3000 rw-p 00003000 00:27 3575                       /usr/lib64/libXau.so.6.0.0
7f3ad3af7000-7f3ad3b06000 r-xp 00000000 00:27 3347                       /usr/lib64/libdrm.so.2.4.0
7f3ad3b06000-7f3ad3d06000 ---p 0000f000 00:27 3347                       /usr/lib64/libdrm.so.2.4.0
7f3ad3d06000-7f3ad3d07000 r--p 0000f000 00:27 3347                       /usr/lib64/libdrm.so.2.4.0
7f3ad3d07000-7f3ad3d08000 rw-p 00010000 00:27 3347                       /usr/lib64/libdrm.so.2.4.0
7f3ad3d0f000-7f3ad3d14000 r-xp 00000000 00:27 9002                       /usr/lib64/libXxf86vm.so.1.0.0
7f3ad3d14000-7f3ad3f13000 ---p 00005000 00:27 9002                       /usr/lib64/libXxf86vm.so.1.0.0
7f3ad3f13000-7f3ad3f14000 r--p 00004000 00:27 9002                       /usr/lib64/libXxf86vm.so.1.0.0
7f3ad3f14000-7f3ad3f15000 rw-p 00005000 00:27 9002                       /usr/lib64/libXxf86vm.so.1.0.0
7f3ad3f17000-7f3ad3f36000 r-xp 00000000 00:27 603327                     /usr/lib64/libxcb.so.1.1.0
7f3ad3f36000-7f3ad4135000 ---p 0001f000 00:27 603327                     /usr/lib64/libxcb.so.1.1.0
7f3ad4135000-7f3ad4136000 r--p 0001e000 00:27 603327                     /usr/lib64/libxcb.so.1.1.0
7f3ad4136000-7f3ad4137000 rw-p 0001f000 00:27 603327                     /usr/lib64/libxcb.so.1.1.0
7f3ad4137000-7f3ad413a000 r-xp 00000000 00:27 604963                     /usr/lib64/libxcb-dri2.so.0.0.0
7f3ad413a000-7f3ad433a000 ---p 00003000 00:27 604963                     /usr/lib64/libxcb-dri2.so.0.0.0
7f3ad433a000-7f3ad433b000 r--p 00003000 00:27 604963                     /usr/lib64/libxcb-dri2.so.0.0.0
7f3ad433b000-7f3ad433c000 rw-p 00004000 00:27 604963                     /usr/lib64/libxcb-dri2.so.0.0.0
7f3ad433f000-7f3ad4354000 r-xp 00000000 00:27 604955                     /usr/lib64/libxcb-glx.so.0.0.0
7f3ad4354000-7f3ad4554000 ---p 00015000 00:27 604955                     /usr/lib64/libxcb-glx.so.0.0.0
7f3ad4554000-7f3ad4556000 r--p 00015000 00:27 604955                     /usr/lib64/libxcb-glx.so.0.0.0
7f3ad4556000-7f3ad4557000 rw-p 00017000 00:27 604955                     /usr/lib64/libxcb-glx.so.0.0.0
7f3ad4557000-7f3ad468f000 r-xp 00000000 00:27 4392                       /usr/lib64/libX11.so.6.3.0
7f3ad468f000-7f3ad488f000 ---p 00138000 00:27 4392                       /usr/lib64/libX11.so.6.3.0
7f3ad488f000-7f3ad4890000 r--p 00138000 00:27 4392                       /usr/lib64/libX11.so.6.3.0
7f3ad4890000-7f3ad4895000 rw-p 00139000 00:27 4392                       /usr/lib64/libX11.so.6.3.0
7f3ad4897000-7f3ad4898000 r-xp 00000000 00:27 3578                       /usr/lib64/libX11-xcb.so.1.0.0
7f3ad4898000-7f3ad4a97000 ---p 00001000 00:27 3578                       /usr/lib64/libX11-xcb.so.1.0.0
7f3ad4a97000-7f3ad4a98000 r--p 00000000 00:27 3578                       /usr/lib64/libX11-xcb.so.1.0.0
7f3ad4a98000-7f3ad4a99000 rw-p 00001000 00:27 3578                       /usr/lib64/libX11-xcb.so.1.0.0
7f3ad4a9f000-7f3ad4aa4000 r-xp 00000000 00:27 8081                       /usr/lib64/libXfixes.so.3.1.0
7f3ad4aa4000-7f3ad4ca3000 ---p 00005000 00:27 8081                       /usr/lib64/libXfixes.so.3.1.0
7f3ad4ca3000-7f3ad4ca4000 r--p 00004000 00:27 8081                       /usr/lib64/libXfixes.so.3.1.0
7f3ad4ca4000-7f3ad4ca5000 rw-p 00005000 00:27 8081                       /usr/lib64/libXfixes.so.3.1.0
7f3ad4ca7000-7f3ad4ca9000 r-xp 00000000 00:27 8087                       /usr/lib64/libXdamage.so.1.1.0
7f3ad4ca9000-7f3ad4ea8000 ---p 00002000 00:27 8087                       /usr/lib64/libXdamage.so.1.1.0
7f3ad4ea8000-7f3ad4ea9000 r--p 00001000 00:27 8087                       /usr/lib64/libXdamage.so.1.1.0
7f3ad4ea9000-7f3ad4eaa000 rw-p 00002000 00:27 8087                       /usr/lib64/libXdamage.so.1.1.0
7f3ad4eaf000-7f3ad4ec0000 r-xp 00000000 00:27 8084                       /usr/lib64/libXext.so.6.4.0
7f3ad4ec0000-7f3ad50bf000 ---p 00011000 00:27 8084                       /usr/lib64/libXext.so.6.4.0
7f3ad50bf000-7f3ad50c0000 r--p 00010000 00:27 8084                       /usr/lib64/libXext.so.6.4.0
7f3ad50c0000-7f3ad50c1000 rw-p 00011000 00:27 8084                       /usr/lib64/libXext.so.6.4.0
7f3ad50c7000-7f3ad50f1000 r-xp 00000000 00:27 54067                      /usr/lib64/libglapi.so.0.0.0
7f3ad50f1000-7f3ad52f0000 ---p 0002a000 00:27 54067                      /usr/lib64/libglapi.so.0.0.0
7f3ad52f0000-7f3ad52f4000 r--p 00029000 00:27 54067                      /usr/lib64/libglapi.so.0.0.0
7f3ad52f4000-7f3ad52f5000 rw-p 0002d000 00:27 54067                      /usr/lib64/libglapi.so.0.0.0
7f3ad52f5000-7f3ad52f6000 rw-p 00000000 00:00 0 
7f3ad52f7000-7f3ad52f8000 r-xp 00000000 00:27 2890                       /usr/lib64/libxshmfence.so.1.0.0
7f3ad52f8000-7f3ad54f8000 ---p 00001000 00:27 2890                       /usr/lib64/libxshmfence.so.1.0.0
7f3ad54f8000-7f3ad54f9000 r--p 00001000 00:27 2890                       /usr/lib64/libxshmfence.so.1.0.0
7f3ad54f9000-7f3ad54fa000 rw-p 00002000 00:27 2890                       /usr/lib64/libxshmfence.so.1.0.0
7f3ad54ff000-7f3ad5504000 r-xp 00000000 00:27 604939                     /usr/lib64/libxcb-sync.so.1.0.0
7f3ad5504000-7f3ad5703000 ---p 00005000 00:27 604939                     /usr/lib64/libxcb-sync.so.1.0.0
7f3ad5703000-7f3ad5704000 r--p 00004000 00:27 604939                     /usr/lib64/libxcb-sync.so.1.0.0
7f3ad5704000-7f3ad5705000 rw-p 00005000 00:27 604939                     /usr/lib64/libxcb-sync.so.1.0.0
7f3ad5707000-7f3ad5709000 r-xp 00000000 00:27 604951                     /usr/lib64/libxcb-present.so.0.0.0
7f3ad5709000-7f3ad5908000 ---p 00002000 00:27 604951                     /usr/lib64/libxcb-present.so.0.0.0
7f3ad5908000-7f3ad5909000 r--p 00001000 00:27 604951                     /usr/lib64/libxcb-present.so.0.0.0
7f3ad5909000-7f3ad590a000 rw-p 00002000 00:27 604951                     /usr/lib64/libxcb-present.so.0.0.0
7f3ad590f000-7f3ad5911000 r-xp 00000000 00:27 604959                     /usr/lib64/libxcb-dri3.so.0.0.0
7f3ad5911000-7f3ad5b10000 ---p 00002000 00:27 604959                     /usr/lib64/libxcb-dri3.so.0.0.0
7f3ad5b10000-7f3ad5b11000 r--p 00001000 00:27 604959                     /usr/lib64/libxcb-dri3.so.0.0.0
7f3ad5b11000-7f3ad5b12000 rw-p 00002000 00:27 604959                     /usr/lib64/libxcb-dri3.so.0.0.0
7f3ad5b17000-7f3ad5b1d000 r-xp 00000000 00:27 76606                      /usr/lib64/libIlmThread-2_1.so.11.0.0
7f3ad5b1d000-7f3ad5d1c000 ---p 00006000 00:27 76606                      /usr/lib64/libIlmThread-2_1.so.11.0.0
7f3ad5d1c000-7f3ad5d1d000 r--p 00005000 00:27 76606                      /usr/lib64/libIlmThread-2_1.so.11.0.0
7f3ad5d1d000-7f3ad5d1e000 rw-p 00006000 00:27 76606                      /usr/lib64/libIlmThread-2_1.so.11.0.0
7f3ad5d1f000-7f3ad5d3c000 r-xp 00000000 00:27 71046                      /usr/lib64/libIex-2_1.so.11.0.0
7f3ad5d3c000-7f3ad5f3b000 ---p 0001d000 00:27 71046                      /usr/lib64/libIex-2_1.so.11.0.0
7f3ad5f3b000-7f3ad5f3f000 r--p 0001c000 00:27 71046                      /usr/lib64/libIex-2_1.so.11.0.0
7f3ad5f3f000-7f3ad5f40000 rw-p 00020000 00:27 71046                      /usr/lib64/libIex-2_1.so.11.0.0
7f3ad5f47000-7f3ad5f50000 r-xp 00000000 00:27 3191                       /usr/lib64/libjbig.so.2.0
7f3ad5f50000-7f3ad614f000 ---p 00009000 00:27 3191                       /usr/lib64/libjbig.so.2.0
7f3ad614f000-7f3ad6150000 r--p 00008000 00:27 3191                       /usr/lib64/libjbig.so.2.0
7f3ad6150000-7f3ad6153000 rw-p 00009000 00:27 3191                       /usr/lib64/libjbig.so.2.0
7f3ad6157000-7f3ad617e000 r-xp 00000000 00:27 3167                       /usr/lib64/liblzma.so.5.2.2
7f3ad617e000-7f3ad637e000 ---p 00027000 00:27 3167                       /usr/lib64/liblzma.so.5.2.2
7f3ad637e000-7f3ad637f000 r--p 00027000 00:27 3167                       /usr/lib64/liblzma.so.5.2.2
7f3ad637f000-7f3ad6380000 rw-p 00028000 00:27 3167                       /usr/lib64/liblzma.so.5.2.2
7f3ad6387000-7f3ad63ef000 r-xp 00000000 00:27 55055                      /usr/lib64/libGL.so.1.2.0
7f3ad63ef000-7f3ad65ee000 ---p 00068000 00:27 55055                      /usr/lib64/libGL.so.1.2.0
7f3ad65ee000-7f3ad65f1000 r--p 00067000 00:27 55055                      /usr/lib64/libGL.so.1.2.0
7f3ad65f1000-7f3ad65f2000 rw-p 0006a000 00:27 55055                      /usr/lib64/libGL.so.1.2.0
7f3ad65f2000-7f3ad65f3000 rw-p 00000000 00:00 0 
7f3ad65f7000-7f3ad6639000 r-xp 00000000 00:27 71043                      /usr/lib64/libHalf.so.11.0.0
7f3ad6639000-7f3ad6838000 ---p 00042000 00:27 71043                      /usr/lib64/libHalf.so.11.0.0
7f3ad6838000-7f3ad6839000 r--p 00041000 00:27 71043                      /usr/lib64/libHalf.so.11.0.0
7f3ad6839000-7f3ad683a000 rw-p 00042000 00:27 71043                      /usr/lib64/libHalf.so.11.0.0
7f3ad683f000-7f3ad6951000 r-xp 00000000 00:27 77951                      /usr/lib64/libIlmImf-Imf_2_1.so.21.0.0
7f3ad6951000-7f3ad6b51000 ---p 00112000 00:27 77951                      /usr/lib64/libIlmImf-Imf_2_1.so.21.0.0
7f3ad6b51000-7f3ad6b54000 r--p 00112000 00:27 77951                      /usr/lib64/libIlmImf-Imf_2_1.so.21.0.0
7f3ad6b54000-7f3ad6b56000 rw-p 00115000 00:27 77951                      /usr/lib64/libIlmImf-Imf_2_1.so.21.0.0
7f3ad6b56000-7f3ad6b57000 rw-p 00000000 00:00 0 
7f3ad6b57000-7f3ad6ba6000 r-xp 00000000 00:27 54105                      /usr/lib64/libjasper.so.1.0.0
7f3ad6ba6000-7f3ad6da5000 ---p 0004f000 00:27 54105                      /usr/lib64/libjasper.so.1.0.0
7f3ad6da5000-7f3ad6da6000 r--p 0004e000 00:27 54105                      /usr/lib64/libjasper.so.1.0.0
7f3ad6da6000-7f3ad6daa000 rw-p 0004f000 00:27 54105                      /usr/lib64/libjasper.so.1.0.0
7f3ad6daa000-7f3ad6db1000 rw-p 00000000 00:00 0 
7f3ad6db7000-7f3ad6e2a000 r-xp 00000000 00:27 54142                      /usr/lib64/libtiff.so.5.2.6
7f3ad6e2a000-7f3ad7029000 ---p 00073000 00:27 54142                      /usr/lib64/libtiff.so.5.2.6
7f3ad7029000-7f3ad702d000 r--p 00072000 00:27 54142                      /usr/lib64/libtiff.so.5.2.6
7f3ad702d000-7f3ad702e000 rw-p 00076000 00:27 54142                      /usr/lib64/libtiff.so.5.2.6
7f3ad702f000-7f3ad706a000 r-xp 00000000 00:27 3642                       /usr/lib64/libpng16.so.16.8.0
7f3ad706a000-7f3ad726a000 ---p 0003b000 00:27 3642                       /usr/lib64/libpng16.so.16.8.0
7f3ad726a000-7f3ad726b000 r--p 0003b000 00:27 3642                       /usr/lib64/libpng16.so.16.8.0
7f3ad726b000-7f3ad726c000 rw-p 0003c000 00:27 3642                       /usr/lib64/libpng16.so.16.8.0
7f3ad726f000-7f3ad72d7000 r-xp 00000000 00:27 407172                     /usr/lib64/libjpeg.so.8.1.2
7f3ad72d7000-7f3ad74d7000 ---p 00068000 00:27 407172                     /usr/lib64/libjpeg.so.8.1.2
7f3ad74d7000-7f3ad74d8000 r--p 00068000 00:27 407172                     /usr/lib64/libjpeg.so.8.1.2
7f3ad74d8000-7f3ad74d9000 rw-p 00069000 00:27 407172                     /usr/lib64/libjpeg.so.8.1.2
7f3ad74df000-7f3ad74f5000 r-xp 00000000 00:27 407157                     /lib64/libgcc_s.so.1
7f3ad74f5000-7f3ad76f4000 ---p 00016000 00:27 407157                     /lib64/libgcc_s.so.1
7f3ad76f4000-7f3ad76f5000 r--p 00015000 00:27 407157                     /lib64/libgcc_s.so.1
7f3ad76f5000-7f3ad76f6000 rw-p 00016000 00:27 407157                     /lib64/libgcc_s.so.1
7f3ad76f7000-7f3ad7872000 r-xp 00000000 00:27 414447                     /usr/lib64/libstdc++.so.6.0.24
7f3ad7872000-7f3ad7a71000 ---p 0017b000 00:27 414447                     /usr/lib64/libstdc++.so.6.0.24
7f3ad7a71000-7f3ad7a7b000 r--p 0017a000 00:27 414447                     /usr/lib64/libstdc++.so.6.0.24
7f3ad7a7b000-7f3ad7a7d000 rw-p 00184000 00:27 414447                     /usr/lib64/libstdc++.so.6.0.24
7f3ad7a7d000-7f3ad7a80000 rw-p 00000000 00:00 0 
7f3ad7a87000-7f3ad7d54000 r-xp 00000000 00:27 83825                      /usr/lib64/libopencv_core.so.3.1.0
7f3ad7d54000-7f3ad7f54000 ---p 002cd000 00:27 83825                      /usr/lib64/libopencv_core.so.3.1.0
7f3ad7f54000-7f3ad7f59000 r--p 002cd000 00:27 83825                      /usr/lib64/libopencv_core.so.3.1.0
7f3ad7f59000-7f3ad7f5c000 rw-p 002d2000 00:27 83825                      /usr/lib64/libopencv_core.so.3.1.0
7f3ad7f5c000-7f3ad7f5f000 rw-p 00000000 00:00 0 
7f3ad7f5f000-7f3ad8260000 r-xp 00000000 00:27 83837                      /usr/lib64/libopencv_imgproc.so.3.1.0
7f3ad8260000-7f3ad8460000 ---p 00301000 00:27 83837                      /usr/lib64/libopencv_imgproc.so.3.1.0
7f3ad8460000-7f3ad846a000 r--p 00301000 00:27 83837                      /usr/lib64/libopencv_imgproc.so.3.1.0
7f3ad846a000-7f3ad8472000 rw-p 0030b000 00:27 83837                      /usr/lib64/libopencv_imgproc.so.3.1.0
7f3ad8472000-7f3ad8508000 rw-p 00000000 00:00 0 
7f3ad850f000-7f3ad85a4000 r-xp 00000000 00:27 83835                      /usr/lib64/libopencv_imgcodecs.so.3.1.0
7f3ad85a4000-7f3ad87a4000 ---p 00095000 00:27 83835                      /usr/lib64/libopencv_imgcodecs.so.3.1.0
7f3ad87a4000-7f3ad87a6000 r--p 00095000 00:27 83835                      /usr/lib64/libopencv_imgcodecs.so.3.1.0
7f3ad87a6000-7f3ad87a7000 rw-p 00097000 00:27 83835                      /usr/lib64/libopencv_imgcodecs.so.3.1.0
7f3ad87a7000-7f3ad87ab000 rw-p 00000000 00:00 0 
7f3ad87af000-7f3ad87c3000 r-xp 00000000 00:27 662734                     /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
7f3ad87c3000-7f3ad89c2000 ---p 00014000 00:27 662734                     /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
7f3ad89c2000-7f3ad89c3000 r--p 00013000 00:27 662734                     /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
7f3ad89c3000-7f3ad89c4000 rw-p 00014000 00:27 662734                     /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
7f3ad89c7000-7f3ad89ee000 r-xp 00000000 00:27 54083                      /usr/lib64/libexpat.so.1.6.0
7f3ad89ee000-7f3ad8bee000 ---p 00027000 00:27 54083                      /usr/lib64/libexpat.so.1.6.0
7f3ad8bee000-7f3ad8bf0000 r--p 00027000 00:27 54083                      /usr/lib64/libexpat.so.1.6.0
7f3ad8bf0000-7f3ad8bf1000 rw-p 00029000 00:27 54083                      /usr/lib64/libexpat.so.1.6.0
7f3ad8bf7000-7f3ad8c0b000 r-xp 00000000 00:27 8519                       /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
7f3ad8c0b000-7f3ad8e0a000 ---p 00014000 00:27 8519                       /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
7f3ad8e0a000-7f3ad8e0b000 r--p 00013000 00:27 8519                       /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
7f3ad8e0b000-7f3ad8e0c000 rw-p 00014000 00:27 8519                       /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/XML/Parser/Expat/Expat.so
7f3ad8e0f000-7f3ad8e55000 r-xp 00000000 00:27 3353                       /lib64/libdbus-1.so.3.8.14
7f3ad8e55000-7f3ad9054000 ---p 00046000 00:27 3353                       /lib64/libdbus-1.so.3.8.14
7f3ad9054000-7f3ad9055000 r--p 00045000 00:27 3353                       /lib64/libdbus-1.so.3.8.14
7f3ad9055000-7f3ad9056000 rw-p 00046000 00:27 3353                       /lib64/libdbus-1.so.3.8.14
7f3ad9057000-7f3ad9076000 r-xp 00000000 00:27 13715                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so
7f3ad9076000-7f3ad9275000 ---p 0001f000 00:27 13715                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so
7f3ad9275000-7f3ad9276000 r--p 0001e000 00:27 13715                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so
7f3ad9276000-7f3ad9277000 rw-p 0001f000 00:27 13715                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so
7f3ad9277000-7f3ad928b000 r-xp 00000000 00:27 72652                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7f3ad928b000-7f3ad948a000 ---p 00014000 00:27 72652                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7f3ad948a000-7f3ad948b000 r--p 00013000 00:27 72652                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7f3ad948b000-7f3ad948c000 rw-p 00014000 00:27 72652                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7f3ad948f000-7f3ad94b9000 r-xp 00000000 00:27 3972                       /usr/lib64/libssh2.so.1.0.1
7f3ad94b9000-7f3ad96b8000 ---p 0002a000 00:27 3972                       /usr/lib64/libssh2.so.1.0.1
7f3ad96b8000-7f3ad96b9000 r--p 00029000 00:27 3972                       /usr/lib64/libssh2.so.1.0.1
7f3ad96b9000-7f3ad96ba000 rw-p 0002a000 00:27 3972                       /usr/lib64/libssh2.so.1.0.1
7f3ad96bf000-7f3ad96e0000 r-xp 00000000 00:27 79872                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so
7f3ad96e0000-7f3ad98e0000 ---p 00021000 00:27 79872                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so
7f3ad98e0000-7f3ad98e1000 r--p 00021000 00:27 79872                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so
7f3ad98e1000-7f3ad98e2000 rw-p 00022000 00:27 79872                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSH2/SSH2.so
7f3ad98e7000-7f3ad98e9000 r-xp 00000000 00:27 71623                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
7f3ad98e9000-7f3ad9ae8000 ---p 00002000 00:27 71623                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
7f3ad9ae8000-7f3ad9ae9000 r--p 00001000 00:27 71623                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
7f3ad9ae9000-7f3ad9aea000 rw-p 00002000 00:27 71623                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Sub/Name/Name.so
7f3ad9aef000-7f3ad9af1000 r-xp 00000000 00:27 416382                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so
7f3ad9af1000-7f3ad9cf0000 ---p 00002000 00:27 416382                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so
7f3ad9cf0000-7f3ad9cf1000 r--p 00001000 00:27 416382                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so
7f3ad9cf1000-7f3ad9cf2000 rw-p 00002000 00:27 416382                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so
7f3ad9cf7000-7f3ad9cf9000 r-xp 00000000 00:27 416350                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/I18N/Langinfo/Langinfo.so
7f3ad9cf9000-7f3ad9ef9000 ---p 00002000 00:27 416350                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/I18N/Langinfo/Langinfo.so
7f3ad9ef9000-7f3ad9efa000 r--p 00002000 00:27 416350                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/I18N/Langinfo/Langinfo.so
7f3ad9efa000-7f3ad9efb000 rw-p 00003000 00:27 416350                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/I18N/Langinfo/Langinfo.so
7f3ad9eff000-7f3ad9f14000 r-xp 00000000 00:27 416374                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Storable/Storable.so
7f3ad9f14000-7f3ada113000 ---p 00015000 00:27 416374                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Storable/Storable.so
7f3ada113000-7f3ada114000 r--p 00014000 00:27 416374                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Storable/Storable.so
7f3ada114000-7f3ada115000 rw-p 00015000 00:27 416374                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Storable/Storable.so
7f3ada117000-7f3ada343000 r-xp 00000000 00:27 603305                     /lib64/libcrypto.so.1.0.0
7f3ada343000-7f3ada543000 ---p 0022c000 00:27 603305                     /lib64/libcrypto.so.1.0.0
7f3ada543000-7f3ada55e000 r--p 0022c000 00:27 603305                     /lib64/libcrypto.so.1.0.0
7f3ada55e000-7f3ada56c000 rw-p 00247000 00:27 603305                     /lib64/libcrypto.so.1.0.0
7f3ada56c000-7f3ada570000 rw-p 00000000 00:00 0 
7f3ada577000-7f3ada5d8000 r-xp 00000000 00:27 603306                     /lib64/libssl.so.1.0.0
7f3ada5d8000-7f3ada7d7000 ---p 00061000 00:27 603306                     /lib64/libssl.so.1.0.0
7f3ada7d7000-7f3ada7db000 r--p 00060000 00:27 603306                     /lib64/libssl.so.1.0.0
7f3ada7db000-7f3ada7e2000 rw-p 00064000 00:27 603306                     /lib64/libssl.so.1.0.0
7f3ada7e7000-7f3ada856000 r-xp 00000000 00:27 8612                       /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7f3ada856000-7f3adaa56000 ---p 0006f000 00:27 8612                       /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7f3adaa56000-7f3adaa57000 r--p 0006f000 00:27 8612                       /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7f3adaa57000-7f3adaa59000 rw-p 00070000 00:27 8612                       /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7f3adaa5f000-7f3adaa6a000 r-xp 00000000 00:27 2580                       /lib64/libnss_files-2.22.so
7f3adaa6a000-7f3adac69000 ---p 0000b000 00:27 2580                       /lib64/libnss_files-2.22.so
7f3adac69000-7f3adac6a000 r--p 0000a000 00:27 2580                       /lib64/libnss_files-2.22.so
7f3adac6a000-7f3adac6b000 rw-p 0000b000 00:27 2580                       /lib64/libnss_files-2.22.so
7f3adac6b000-7f3adac71000 rw-p 00000000 00:00 0 
7f3adac77000-7f3adac86000 r-xp 00000000 00:27 416309                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/B/B.so
7f3adac86000-7f3adae86000 ---p 0000f000 00:27 416309                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/B/B.so
7f3adae86000-7f3adae88000 r--p 0000f000 00:27 416309                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/B/B.so
7f3adae88000-7f3adae89000 rw-p 00011000 00:27 416309                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/B/B.so
7f3adae8f000-7f3adaea4000 r-xp 00000000 00:27 2881                       /lib64/libz.so.1.2.8
7f3adaea4000-7f3adb0a3000 ---p 00015000 00:27 2881                       /lib64/libz.so.1.2.8
7f3adb0a3000-7f3adb0a4000 r--p 00014000 00:27 2881                       /lib64/libz.so.1.2.8
7f3adb0a4000-7f3adb0a5000 rw-p 00015000 00:27 2881                       /lib64/libz.so.1.2.8
7f3adb0a7000-7f3adb0b5000 r-xp 00000000 00:27 416314                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7f3adb0b5000-7f3adb2b4000 ---p 0000e000 00:27 416314                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7f3adb2b4000-7f3adb2b5000 r--p 0000d000 00:27 416314                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7f3adb2b5000-7f3adb2b6000 rw-p 0000e000 00:27 416314                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7f3adb2b7000-7f3adb2bd000 r-xp 00000000 00:27 407336                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7f3adb2bd000-7f3adb4bc000 ---p 00006000 00:27 407336                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7f3adb4bc000-7f3adb4bd000 r--p 00005000 00:27 407336                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7f3adb4bd000-7f3adb4be000 rw-p 00006000 00:27 407336                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7f3adb4bf000-7f3adb512000 r-xp 00000000 00:27 416398                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/re/re.so
7f3adb512000-7f3adb711000 ---p 00053000 00:27 416398                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/re/re.so
7f3adb711000-7f3adb712000 r--p 00052000 00:27 416398                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/re/re.so
7f3adb712000-7f3adb713000 rw-p 00053000 00:27 416398                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/re/re.so
7f3adb717000-7f3adb71a000 r-xp 00000000 00:27 416354                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7f3adb71a000-7f3adb919000 ---p 00003000 00:27 416354                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7f3adb919000-7f3adb91a000 r--p 00002000 00:27 416354                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7f3adb91a000-7f3adb91b000 rw-p 00003000 00:27 416354                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7f3adb91f000-7f3adb927000 r-xp 00000000 00:27 416330                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Encode/Encode.so
7f3adb927000-7f3adbb26000 ---p 00008000 00:27 416330                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Encode/Encode.so
7f3adbb26000-7f3adbb27000 r--p 00007000 00:27 416330                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Encode/Encode.so
7f3adbb27000-7f3adbb28000 rw-p 00008000 00:27 416330                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Encode/Encode.so
7f3adbb2f000-7f3adbb3a000 r-xp 00000000 00:27 407330                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/SHA/SHA.so
7f3adbb3a000-7f3adbd39000 ---p 0000b000 00:27 407330                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/SHA/SHA.so
7f3adbd39000-7f3adbd3a000 r--p 0000a000 00:27 407330                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/SHA/SHA.so
7f3adbd3a000-7f3adbd3b000 rw-p 0000b000 00:27 407330                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/SHA/SHA.so
7f3adbd3f000-7f3adbd43000 r-xp 00000000 00:27 407328                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
7f3adbd43000-7f3adbf42000 ---p 00004000 00:27 407328                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
7f3adbf42000-7f3adbf43000 r--p 00003000 00:27 407328                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
7f3adbf43000-7f3adbf44000 rw-p 00004000 00:27 407328                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
7f3adbf47000-7f3adbf52000 r-xp 00000000 00:27 71602                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/Util/Util.so
7f3adbf52000-7f3adc151000 ---p 0000b000 00:27 71602                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/Util/Util.so
7f3adc151000-7f3adc152000 r--p 0000a000 00:27 71602                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/Util/Util.so
7f3adc152000-7f3adc153000 rw-p 0000b000 00:27 71602                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/List/Util/Util.so
7f3adc157000-7f3adc15f000 r-xp 00000000 00:27 407326                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7f3adc15f000-7f3adc35e000 ---p 00008000 00:27 407326                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7f3adc35e000-7f3adc35f000 r--p 00007000 00:27 407326                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7f3adc35f000-7f3adc360000 rw-p 00008000 00:27 407326                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7f3adc367000-7f3adc369000 r-xp 00000000 00:27 407324                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7f3adc369000-7f3adc568000 ---p 00002000 00:27 407324                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7f3adc568000-7f3adc569000 r--p 00001000 00:27 407324                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7f3adc569000-7f3adc56a000 rw-p 00002000 00:27 407324                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7f3adc56f000-7f3adc571000 r-xp 00000000 00:27 416394                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/attributes/attributes.so
7f3adc571000-7f3adc770000 ---p 00002000 00:27 416394                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/attributes/attributes.so
7f3adc770000-7f3adc771000 r--p 00001000 00:27 416394                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/attributes/attributes.so
7f3adc771000-7f3adc772000 rw-p 00002000 00:27 416394                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/attributes/attributes.so
7f3adc777000-7f3adc786000 r-xp 00000000 00:27 78188                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/JSON/XS/XS.so
7f3adc786000-7f3adc985000 ---p 0000f000 00:27 78188                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/JSON/XS/XS.so
7f3adc985000-7f3adc986000 r--p 0000e000 00:27 78188                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/JSON/XS/XS.so
7f3adc986000-7f3adc987000 rw-p 0000f000 00:27 78188                      /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/JSON/XS/XS.so
7f3adc987000-7f3adc999000 r-xp 00000000 00:27 407342                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7f3adc999000-7f3adcb98000 ---p 00012000 00:27 407342                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7f3adcb98000-7f3adcb9b000 r--p 00011000 00:27 407342                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7f3adcb9b000-7f3adcb9c000 rw-p 00014000 00:27 407342                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7f3adcb9f000-7f3adcba2000 r-xp 00000000 00:27 407332                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7f3adcba2000-7f3adcda2000 ---p 00003000 00:27 407332                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7f3adcda2000-7f3adcda3000 r--p 00003000 00:27 407332                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7f3adcda3000-7f3adcda4000 rw-p 00004000 00:27 407332                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7f3adcda7000-7f3adcdaf000 r-xp 00000000 00:27 407344                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Socket/Socket.so
7f3adcdaf000-7f3adcfaf000 ---p 00008000 00:27 407344                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Socket/Socket.so
7f3adcfaf000-7f3adcfb1000 r--p 00008000 00:27 407344                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Socket/Socket.so
7f3adcfb1000-7f3adcfb2000 rw-p 0000a000 00:27 407344                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Socket/Socket.so
7f3adcfb7000-7f3adcfbb000 r-xp 00000000 00:27 407338                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/IO/IO.so
7f3adcfbb000-7f3add1ba000 ---p 00004000 00:27 407338                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/IO/IO.so
7f3add1ba000-7f3add1bb000 r--p 00003000 00:27 407338                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/IO/IO.so
7f3add1bb000-7f3add1bc000 rw-p 00004000 00:27 407338                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/IO/IO.so
7f3add1bf000-7f3add1c6000 r-xp 00000000 00:27 2594                       /lib64/librt-2.22.so
7f3add1c6000-7f3add3c5000 ---p 00007000 00:27 2594                       /lib64/librt-2.22.so
7f3add3c5000-7f3add3c6000 r--p 00006000 00:27 2594                       /lib64/librt-2.22.so
7f3add3c6000-7f3add3c7000 rw-p 00007000 00:27 2594                       /lib64/librt-2.22.so
7f3add3c7000-7f3add3cc000 r-xp 00000000 00:27 416384                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7f3add3cc000-7f3add5cb000 ---p 00005000 00:27 416384                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7f3add5cb000-7f3add5cc000 r--p 00004000 00:27 416384                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7f3add5cc000-7f3add5cd000 rw-p 00005000 00:27 416384                     /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7f3add5cf000-7f3add768000 r-xp 00000000 00:27 2560                       /lib64/libc-2.22.so
7f3add768000-7f3add968000 ---p 00199000 00:27 2560                       /lib64/libc-2.22.so
7f3add968000-7f3add96c000 r--p 00199000 00:27 2560                       /lib64/libc-2.22.so
7f3add96c000-7f3add96e000 rw-p 0019d000 00:27 2560                       /lib64/libc-2.22.so
7f3add96e000-7f3add972000 rw-p 00000000 00:00 0 
7f3add977000-7f3add98f000 r-xp 00000000 00:27 2590                       /lib64/libpthread-2.22.so
7f3add98f000-7f3addb8e000 ---p 00018000 00:27 2590                       /lib64/libpthread-2.22.so
7f3addb8e000-7f3addb8f000 r--p 00017000 00:27 2590                       /lib64/libpthread-2.22.so
7f3addb8f000-7f3addb90000 rw-p 00018000 00:27 2590                       /lib64/libpthread-2.22.so
7f3addb90000-7f3addb94000 rw-p 00000000 00:00 0 
7f3addb97000-7f3addba2000 r-xp 00000000 00:27 2564                       /lib64/libcrypt-2.22.so
7f3addba2000-7f3addda2000 ---p 0000b000 00:27 2564                       /lib64/libcrypt-2.22.so
7f3addda2000-7f3addda3000 r--p 0000b000 00:27 2564                       /lib64/libcrypt-2.22.so
7f3addda3000-7f3addda4000 rw-p 0000c000 00:27 2564                       /lib64/libcrypt-2.22.so
7f3addda4000-7f3adddd2000 rw-p 00000000 00:00 0 
7f3adddd7000-7f3adddd9000 r-xp 00000000 00:27 2566                       /lib64/libdl-2.22.so
7f3adddd9000-7f3addfd9000 ---p 00002000 00:27 2566                       /lib64/libdl-2.22.so
7f3addfd9000-7f3addfda000 r--p 00002000 00:27 2566                       /lib64/libdl-2.22.so
7f3addfda000-7f3addfdb000 rw-p 00003000 00:27 2566                       /lib64/libdl-2.22.so
7f3addfdf000-7f3ade0db000 r-xp 00000000 00:27 2568                       /lib64/libm-2.22.so
7f3ade0db000-7f3ade2da000 ---p 000fc000 00:27 2568                       /lib64/libm-2.22.so
7f3ade2da000-7f3ade2db000 r--p 000fb000 00:27 2568                       /lib64/libm-2.22.so
7f3ade2db000-7f3ade2dc000 rw-p 000fc000 00:27 2568                       /lib64/libm-2.22.so
7f3ade2df000-7f3ade300000 r-xp 00000000 00:27 2552                       /lib64/ld-2.22.so
7f3ade3cc000-7f3ade44d000 rw-p 00000000 00:00 0 
7f3ade477000-7f3ade4ac000 r--s 00000000 00:14 26836                      /run/nscd/dbA9ZYZ5 (deleted)
7f3ade4af000-7f3ade4b6000 r--s 00000000 00:27 22586                      /usr/lib64/gconv/gconv-modules.cache
7f3ade4b7000-7f3ade4fb000 r--p 00000000 00:27 22173                      /usr/lib/locale/en_US.utf8/LC_CTYPE
7f3ade4ff000-7f3ade500000 r--p 00020000 00:27 2552                       /lib64/ld-2.22.so
7f3ade500000-7f3ade501000 rw-p 00021000 00:27 2552                       /lib64/ld-2.22.so
7f3ade501000-7f3ade502000 rw-p 00000000 00:00 0 
7f3ade503000-7f3ade504000 rw-p 00000000 00:00 0 
7f3ade504000-7f3ade509000 rw-p 00000000 00:00 0 
7ffcf75f9000-7ffcf761a000 rw-p 00000000 00:00 0                          [stack]
7ffcf76ef000-7ffcf76f2000 r--p 00000000 00:00 0                          [vvar]
7ffcf76f2000-7ffcf76f4000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Unexpected end of data 0
[2018-01-19T22:29:31.0441 CET] [debug] backend 12520 died
can_read received kill signal at /usr/lib/os-autoinst/myjsonrpc.pm line 89.

so

  • looks like qemu bug
  • isotovideo does not foresee this and fails to terminate the qemu process on dying

I manually terminated the two still running old qemu processes on openqaworker10:

_openqa+  9482 99.2  0.4 5623116 1210348 ?     Sl   Jan19 2562:27 /usr/bin/qemu-system-x86_64 -serial file:seria
_openqa+ 12546  0.7  0.4 5516308 1163968 ?     Sl   Jan19  19:10 /usr/bin/qemu-system-x86_64 -serial file:serial

and will retrigger the failed jobs I find but of course that won't be enough.

Who can take over here to care about either qemu or isotovideo?

Actions
Copy link
 #7

Updated by okurz almost 7 years ago

  • Project changed from openQA Tests (public) to openQA Project (public)
  • Subject changed from [qam] openqaworker10:4 - worker fail all tests to [qam] openqaworker10:4 - worker fail all tests - qemu instances are left behind
  • Category changed from Infrastructure to Regressions/Crashes
Actions
Copy link
 #8

Updated by okurz almost 7 years ago

  • Related to action #30595: [ppc64le] Deploy bcache on tmpfs workers added
Actions
Copy link
 #9

Updated by coolo almost 7 years ago

where do you see a QEMU bug? I only see isotovideo crashing

Actions
Copy link
 #10

Updated by coolo almost 7 years ago

and it's the same 42.3 stack as on all workers. I will shutdown the worker

Actions
Copy link
 #11

Updated by okurz almost 7 years ago

coolo wrote:

where do you see a QEMU bug? I only see isotovideo crashing

here:

[2018-01-19T22:29:24.0538 CET] [debug] QEMU: 
[2018-01-19T22:29:24.0538 CET] [debug] QEMU: (process:12546): GLib-CRITICAL **: Source ID 32305 was not found when attempting to remove it
Actions
Copy link
 #12

Updated by coolo almost 7 years ago

this sounds basically like the same as what perl died on: memory corruption. And I find it rare that two processes have memory corruptions at the same time. We may need to do a memory check on this machine

Actions
Copy link
 #13

Updated by coolo almost 7 years ago

The qemu problem is more frequent and also happens on other machines: http://paste.suse.de/19743 (27/2000 jobs) - it always (when I looked) comes with a VNC stall. I would consider an out of memory problem, if worker10 wouldn't have 256GB

Actions
Copy link
 #14

Updated by coolo almost 7 years ago

that perl crash is only once in the last 3000 jobs. So I would guess this is an effect of the matrix

Actions
Copy link
 #15

Updated by coolo almost 7 years ago

  • Target version set to Ready

1401051 crashed differently - also on worker10 and also leaving qemu behind to create incompletes. So while the crashes are rare enough that I doubt we can do much about it (especially as we still have no monitoring for the workers to understand what's going on when it crashes), we should be able to cope with the backend.crash file

Actions
Copy link
 #16

Updated by coolo almost 7 years ago

And here is the problem: both crashes didn't leave a backend.crashed file

Actions
Copy link
 #17

Updated by okurz almost 7 years ago

  • Blocks action #30700: [sle][functional][sporadic][medium][sle12 sp4] test fails in memtest added
Actions
Copy link
 #18

Updated by okurz almost 7 years ago

I triggered 20 memtest jobs and found some incompletes which look like the same symptoms: https://openqa.suse.de/tests/overview?build=0164&version=12-SP4&distri=sle&groupid=139

-> the incompletes:

That is 4/20 incompletes, the rest passed

So I doubt it's a "memory check needed on all machines at the same time".

Actions
Copy link
 #19

Updated by okurz almost 7 years ago

  • Blocks deleted (action #30700: [sle][functional][sporadic][medium][sle12 sp4] test fails in memtest)
Actions
Copy link
 #20

Updated by okurz almost 7 years ago

  • Related to action #30700: [sle][functional][sporadic][medium][sle12 sp4] test fails in memtest added
Actions
Copy link
 #21

Updated by szarate almost 7 years ago

  • Assignee set to dasantiago
  • Target version changed from Ready to Current Sprint
Actions
Copy link
 #22

Updated by dasantiago almost 7 years ago

Causes:
worker 10:4 -> memory corruption: https://openqa.suse.de/tests/1401276/file/autoinst-log.txt
worker 5:14 -> out of disk space https://openqa.suse.de/tests/1414086/file/autoinst-log.txt
worker 7:6 -> out of disk space https://openqa.suse.de/tests/1414635/file/autoinst-log.txt
worker 6:11 -> no clue... :-(
worker 6:2 -> broken pipe: https://openqa.suse.de/tests/1415737/file/autoinst-log.txt (but it looks the same sympton as out of disk space - worker 5:14)

Actions
Copy link
 #24

Updated by coolo almost 7 years ago

  • Has duplicate action #31126: [sle][functional][sle15] sle-15-Installer-DVD-x86_64-Build429.1-select_disk@64bit died because of qmp cannot be found added
Actions
Copy link
 #25

Updated by pcervinka almost 7 years ago

Looks similar to error on openqaworker10:1 https://openqa.suse.de/tests/1478755/file/autoinst-log.txt

Actions
Copy link
 #26

Updated by coolo almost 7 years ago

  • Status changed from In Progress to Resolved

indeed. And so we deployed - and caused tons of other problems for now :)

but openqaworker10 is troublesome, I disabled it completely for now

Actions
Copy link
 #27

Updated by szarate almost 7 years ago

  • Target version changed from Current Sprint to Done
Actions
Copy link

Also available in: Atom PDF