action #96507
Job terminated prematurely during needle check auto_review:"isotovideo: backen.*: free.*: invalid pointer":retry
Start date:
2021-08-03
Due date:
% Done:
0%
Estimated time:
Difficulty:
Description
Two LTP jobs have failed recently in a similar way while waiting for needle match.
https://openqa.suse.de/tests/6638157
https://openqa.suse.de/tests/6625293
The first job has some interesting output in autoinst-log.txt:
[2021-08-03T07:24:58.610 CEST] [debug] no change: 0.5s [2021-08-03T07:25:03.632 CEST] [debug] WARNING: check_asserted_screen took 4.02 seconds for 35 candidate needles - make your needles more specific [2021-08-03T07:25:03.632 CEST] [debug] no match: -0.5s, best candidate: linux-login-20181005 (0.29) *** Error in `/usr/bin/isotovideo: backen': free(): invalid pointer: 0x00005560f6795e00 *** [2021-08-03T07:25:05.336 CEST] [debug] backend process exited: 0 [2021-08-03T07:25:05.336 CEST] [debug] stopping command server 30808 because test execution ended
The other job does not show any obvious error in the log:
[2021-07-31T07:21:58.852 CEST] [debug] no change: 1785.4s [2021-07-31T07:22:01.325 CEST] [debug] backend process exited: 0 [2021-07-31T07:22:01.326 CEST] [debug] stopping command server 21557 because test execution ended
Suggestion¶
- Reproduce locally
- Extend
t/01-test_needle.tto reproduce this case - Have the test create and destroy multiple needs in a loop
- Try using an optimized build for tinycv (e.g. via cmake argument
-DCMAKE_BUILD_TYPE=RelWithDebInfo, see https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html) if the problem is not reproducible - Provoke more threading in opencv
- Investigate memory-handling on the Perl side, passed to opencv, in baseclass (check os-autoinst readme)
History
#1
Updated by okurz almost 2 years ago
Updated by - Category set to Concrete Bugs
- Priority changed from Normal to High
- Target version set to Ready
as first step https://github.com/os-autoinst/scripts/blob/master/README.md#auto-review---automatically-detect-known-issues-in-openqa-jobs-label-openqa-jobs-with-ticket-references-and-optionally-retrigger should be used to detect the issue and automatically retrigger
#2
Updated by mkittler almost 2 years ago
Updated by - File stacktrace stacktrace added
- Subject changed from Job terminated prematurely during needle check to Job terminated prematurely during needle check auto_review:"isotovideo: backen.*: free.*: invalid pointer":retry
I've been adding the regex but it doesn't look like we're running auto_review currently/anymore.
Luckily the stacktrace is still available so I've attached it. That's the unmangled version of the relevant thread:
(gdb) bt #0 0x00007fa5d9349420 in raise () from /lib64/libc.so.6 #1 0x00007fa5d934aa01 in abort () from /lib64/libc.so.6 #2 0x00007fa5d938c877 in __libc_message () from /lib64/libc.so.6 #3 0x00007fa5d9393093 in malloc_printerr () from /lib64/libc.so.6 #4 0x00007fa5d9394999 in _int_free () from /lib64/libc.so.6 #5 0x00007fa5d1659589 in cv::MatAllocator::unmap(cv::UMatData*) const () from /usr/lib64/libopencv_core.so.3.3 #6 0x00007fa5d26968a1 in image_destroy(Image*) () from /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/auto/tinycv/tinycv.so #7 0x00007fa5d269d32e in ?? () from /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/auto/tinycv/tinycv.so #8 0x00005560ecf47ac1 in Perl_pp_entersub (my_perl=0x5560ee4cb260) at pp_hot.c:4231 #9 0x00005560eceb8ca8 in Perl_call_sv (my_perl=my_perl@entry=0x5560ee4cb260, sv=0x5560f319f950, flags=flags@entry=45) at perl.c:2848 #10 0x00005560ecf4ca4e in S_curse (my_perl=my_perl@entry=0x5560ee4cb260, sv=sv@entry=0x5560f3237ef0, check_refcnt=check_refcnt@entry=true) at sv.c:6972 #11 0x00005560ecf4d2f8 in Perl_sv_clear (my_perl=my_perl@entry=0x5560ee4cb260, orig_sv=orig_sv@entry=0x5560f31f6058) at sv.c:6576 #12 0x00005560ecf4d7b1 in Perl_sv_free2 (my_perl=0x5560ee4cb260, sv=0x5560f31f6058, rc=<optimized out>) at sv.c:7073 #13 0x00005560ecf7fa5c in Perl_leave_scope (my_perl=my_perl@entry=0x5560ee4cb260, base=<optimized out>) at scope.c:1178 #14 0x00005560ecf83a08 in Perl_dounwind (my_perl=my_perl@entry=0x5560ee4cb260, cxix=cxix@entry=20) at pp_ctl.c:1552 #15 0x00005560ecf8eef1 in Perl_dounwind (cxix=20, my_perl=0x5560ee4cb260) at pp_ctl.c:2471 #16 Perl_pp_return (my_perl=0x5560ee4cb260) at pp_ctl.c:2477 #17 0x00005560ecf3f8e6 in Perl_runops_standard (my_perl=0x5560ee4cb260) at run.c:41 #18 0x00005560ecec07d7 in S_run_body (oldscope=<optimized out>, my_perl=<optimized out>) at perl.c:2524 #19 perl_run (my_perl=0x5560ee4cb260) at perl.c:2447 #20 0x00005560ece97ef2 in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:123
Looks like Perl passed a bad pointer to the XS function for destroying the tinycv::Image object so its destructor passes a bad pointer to the destructor of the cv::Mat object.
#3
Updated by mkittler almost 2 years ago
- File stacktrace2 stacktrace2 added
- File stacktrace3 stacktrace3 added
- File stacktrace4 stacktrace4 added
There are 3 more stack traces, likely one of them corresponds to the 2nd job. So there are basically two crashes, one involving image_destroy and one involving image_search.
By the way, I haven't seen such stack traces before (e.g. when looking at backend stack traces for different crashes).
#4
Updated by mkittler almost 2 years ago
It also happened on openqaworker13 (backtrace with image_search) so it is not specific openqaworker10.
#5
Updated by cdywan almost 2 years ago
Updated by - Subject changed from Job terminated prematurely during needle check auto_review:"isotovideo: backen.*: free.*: invalid pointer":retry to Job terminated prematurely during needle check auto_review:"isotovideo: backen.*: free.*: invalid pointer":retry size:M
- Description updated (diff)
- Status changed from New to Workable
#6
Updated by mkittler almost 2 years ago
- Description updated (diff)
- Assignee set to mkittler
#7
Updated by mkittler almost 2 years ago
This test should run the relevant functions many times over and over again: https://github.com/Martchus/os-autoinst/pull/new/debug-segfault-2
However, so far I failed to reproduce the problem.
Since compiler warnings are often useful to find bugs I've been creating https://github.com/os-autoinst/os-autoinst/pull/1740. I couldn't find any clear problems, though.
#8
Updated by MDoucha almost 2 years ago
Updated by Here's another similar job failure with a slightly different error in the log:
https://openqa.suse.de/tests/6795622
[2021-08-10T20:26:11.866 CEST] [debug] led state 0 1 1 -261 [2021-08-10T20:26:12.353 CEST] [debug] no change: 1785.1s *** Error in `/usr/bin/isotovideo: backen': corrupted size vs. prev_size: 0x000055acfb6e0670 *** Unexpected end of data 0 [2021-08-10T20:26:13.726 CEST] [debug] backend process exited: 0 [2021-08-10T20:26:13.727 CEST] [debug] stopping command server 30489 because test execution ended
#9
Updated by mkittler almost 2 years ago
Strangely there's no coredump from the day this job ran on openqaworker10 where the job ran according to the database. However, there are several other coredumps which I've attached. Many have actually no tinycv.so-frames within the stack trace; not sure whether they might even be bugs in Perl. One backtrace contains an image_scale-frame which is new (so far I've only seen image_search and image_destroy). The TBB threads all look the same and unproblematic (so the issue that signal handlers are invoked in these threads doesn't seem to be re-occurring).
#10
Updated by mkittler almost 2 years ago
I extended my test which runs image functionality by a few other functions but still cannot reproduce the issue locally. I've also tested against OpenCV 3. I also still couldn't find anything wrong with our code. It could be improved here and there according to certain guidelines and to avoid certain warnings but there's nothing which should cause UB or memory corruption.
I'm currently running out of ideas.
#11
Updated by mkittler almost 2 years ago
I've been running https://github.com/Martchus/os-autoinst/pull/new/debug-segfault-2 in a Leap 15.2 container but still couldn't reproduce the issue.
#12
Updated by mkittler almost 2 years ago
- Assignee deleted (
mkittler)
Not sure what the problem could be and how to reproduce it.
#13
Updated by okurz almost 2 years ago
- Subject changed from Job terminated prematurely during needle check auto_review:"isotovideo: backen.*: free.*: invalid pointer":retry size:M to Job terminated prematurely during needle check auto_review:"isotovideo: backen.*: free.*: invalid pointer":retry
- Status changed from Workable to New
#14
Updated by cdywan almost 2 years ago
- Description updated (diff)
#15
Updated by okurz almost 2 years ago
- Priority changed from High to Low
- Target version changed from Ready to future
During the daily 2021-08-31 we decided that we don't have a good idea how to fix the actual problem. I see the following alternatives:
- With the above information provided and assuming that "auto_review" handles automatic retriggering at least on o3+osd move the ticket out of the backlog and wait for further information. Please anyone who sees the issue again or has any additional information comment on the ticket
- Ensure that openQA has a proper reason in an incomplete job so that openQA can automatically retrigger, see https://github.com/os-autoinst/openQA/blob/master/etc/openqa/openqa.ini#L76
- Ask perl experts if they can read out something better from the stack trace
#16
Updated by MDoucha over 1 year ago
We have one more failed job due to this bug:
https://openqa.suse.de/tests/7095789
[2021-09-14T03:04:46.243 CEST] [debug] no change: 1386.7s *** Error in `/usr/bin/isotovideo: backen': corrupted size vs. prev_size: 0x000055ddc3398e70 *** [2021-09-14T03:04:50.611 CEST] [debug] backend process exited: 0 [2021-09-14T03:04:50.612 CEST] [debug] stopping command server 22002 because test execution ended
#17
Updated by MDoucha over 1 year ago
We have another one:
https://openqa.suse.de/tests/7345106
[2021-10-08T10:15:12.457542+02:00] [debug] WARNING: check_asserted_screen took 4.09 seconds for 36 candidate needles - make your needles more specific [2021-10-08T10:15:12.457694+02:00] [debug] no match: -0.1s, best candidate: linux-login-20181005 (0.29) *** Error in `/usr/bin/isotovideo: backen': realloc(): invalid pointer: 0x000056449d7fdab0 *** [2021-10-08T10:15:14.621326+02:00] [debug] backend process exited: 0 [2021-10-08T10:15:14.622132+02:00] [debug] stopping command server 2326 because test execution ended
#18
Updated by MDoucha over 1 year ago
And one more without any error message in the logs: https://openqa.suse.de/tests/7363250
#19
Updated by MDoucha over 1 year ago
We have more examples. The first one is expecially interesting due to new error message:
https://openqa.suse.de/tests/7668281
[2021-11-13T09:42:19.094394+01:00] [debug] led state 0 1 1 -261 not enough bytes for zrle [2021-11-13T09:42:21.140436+01:00] [debug] backend process exited: 0 [2021-11-13T09:42:21.141354+01:00] [debug] stopping command server 3912 because test execution ended
Looks like OpenCV crashed while trying to load a PNG file generated by isotovideo
https://openqa.suse.de/tests/7668232
*** Error in `/usr/bin/isotovideo: backen': realloc(): invalid pointer: 0x0000561b1e635070 ***
https://openqa.suse.de/tests/7668284
(no error message)
#20
Updated by MDoucha over 1 year ago
We have another example with a slightly different error message:
https://openqa.suse.de/tests/8190266
[2022-02-18T13:55:15.963440+01:00] [debug] no match: -1.8s, best candidate: linux-login-20181005 (0.29) realloc(): invalid pointer [2022-02-18T13:55:21.860752+01:00] [debug] backend process exited: 0 [2022-02-18T13:55:21.861512+01:00] [info] ::: backend::driver::__ANON__: Driver backend collected unknown process with pid 1908 and exit status: 0 [2022-02-18T13:55:21.862025+01:00] [debug] stopping command server 523 because test execution ended
#21
Updated by MDoucha over 1 year ago
One more example, this time with a completely different error message indicating a heap smash somewhere in needle processing:
https://openqa.suse.de/tests/8235310
[2022-02-26T03:35:28.028820+01:00] [debug] no match: 1781.5s, best candidate: linux-login-20181005 (0.29) malloc(): largebin double linked list corrupted (bk) [2022-02-26T03:35:33.426188+01:00] [debug] backend process exited: 0 [2022-02-26T03:35:33.427064+01:00] [debug] stopping command server 3839 because test execution ended
#22
Updated by MDoucha about 1 year ago
Bug is still present.
https://openqa.suse.de/tests/8887048
[2022-06-04T23:31:14.088669+02:00] [debug] no match: -4.0s, best candidate: linux-login-20181005 (0.29) free(): invalid pointer [2022-06-04T23:31:20.473338+02:00] [debug] backend process exited: 0 [2022-06-04T23:31:20.474203+02:00] [debug] stopping command server 1855 because test execution ended
#23
Updated by MDoucha 10 months ago
Another example appeared today:
https://openqa.suse.de/tests/9290554
[2022-08-09T05:28:11.257407+02:00] [debug] no match: 1786.0s, best candidate: linux-login-20181005 (0.29) corrupted size vs. prev_size [2022-08-09T05:28:17.352546+02:00] [debug] backend process exited: 0 [2022-08-09T05:28:17.353032+02:00] [debug] stopping command server 24638 because test execution ended
#24
Updated by okurz 10 months ago
Checking the latest job I found that the job ran on openqaworker13, so let's exercise coredumpctl:
Last login: Thu Jul 21 18:59:52 2022 from 10.163.25.150
Have a lot of fun...
okurz@openqaworker13:~> sudo coredumpctl list
TIME PID UID GID SIG COREFILE EXE SIZE
Wed 2022-08-03 12:13:41 CEST 23061 483 65534 SIGSEGV missing /usr/bin/perl n/a
Wed 2022-08-03 13:12:47 CEST 18276 483 65534 SIGSEGV missing /usr/bin/perl n/a
Wed 2022-08-03 13:12:48 CEST 18117 483 65534 SIGSEGV missing /usr/bin/perl n/a
Wed 2022-08-03 13:12:49 CEST 17936 483 65534 SIGSEGV missing /usr/bin/perl n/a
Wed 2022-08-03 13:12:52 CEST 22879 483 65534 SIGSEGV missing /usr/bin/perl n/a
Wed 2022-08-03 13:12:54 CEST 25036 483 65534 SIGSEGV missing /usr/bin/perl n/a
Wed 2022-08-03 14:12:55 CEST 14583 483 65534 SIGSEGV missing /usr/bin/perl n/a
Wed 2022-08-03 23:11:44 CEST 3602 483 65534 SIGSEGV missing /usr/bin/perl n/a
Thu 2022-08-04 12:12:02 CEST 11086 483 65534 SIGSEGV missing /usr/bin/perl n/a
Thu 2022-08-04 14:12:50 CEST 5693 483 65534 SIGSEGV missing /usr/bin/perl n/a
Thu 2022-08-04 15:13:35 CEST 30588 483 65534 SIGSEGV missing /usr/bin/perl n/a
Thu 2022-08-04 16:13:29 CEST 23778 483 65534 SIGSEGV missing /usr/bin/perl n/a
Thu 2022-08-04 16:13:33 CEST 23464 483 65534 SIGSEGV missing /usr/bin/perl n/a
Thu 2022-08-04 17:13:38 CEST 8868 483 65534 SIGSEGV missing /usr/bin/perl n/a
Thu 2022-08-04 17:13:40 CEST 8597 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 10:12:49 CEST 4877 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 10:12:52 CEST 4695 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 11:12:18 CEST 21429 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 11:12:18 CEST 20475 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 11:12:24 CEST 20294 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 12:12:03 CEST 7290 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 12:12:06 CEST 7103 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 12:12:07 CEST 7470 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 13:15:29 CEST 26573 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 14:13:31 CEST 10826 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 14:13:33 CEST 10580 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 14:13:37 CEST 10827 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 15:12:04 CEST 27707 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 15:12:04 CEST 27895 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 15:12:07 CEST 28396 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 16:12:20 CEST 15831 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 16:12:21 CEST 16015 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 17:12:57 CEST 4461 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 18:12:08 CEST 25740 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 18:12:13 CEST 24448 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 18:12:17 CEST 24763 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 19:12:09 CEST 11781 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 19:12:13 CEST 11708 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 19:12:15 CEST 11474 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 20:12:21 CEST 30801 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 20:12:21 CEST 31179 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 20:12:27 CEST 30999 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 21:12:18 CEST 13408 483 65534 SIGSEGV missing /usr/bin/perl n/a
Fri 2022-08-05 21:12:19 CEST 13679 483 65534 SIGSEGV missing /usr/bin/perl n/a
Tue 2022-08-09 05:11:39 CEST 21430 483 65534 SIGSEGV present /usr/bin/perl 42.6M
Tue 2022-08-09 05:28:17 CEST 24794 483 65534 SIGABRT present /usr/bin/perl 34.5M
Tue 2022-08-09 11:12:02 CEST 8623 483 65534 SIGSEGV present /usr/bin/perl 44.3M
Tue 2022-08-09 11:12:09 CEST 8443 483 65534 SIGSEGV present /usr/bin/perl 30.7M
Tue 2022-08-09 13:13:25 CEST 19555 483 65534 SIGSEGV present /usr/bin/perl 50.8M
Tue 2022-08-09 13:13:31 CEST 24990 483 65534 SIGSEGV present /usr/bin/perl 49.8M
Tue 2022-08-09 14:13:29 CEST 8027 483 65534 SIGSEGV present /usr/bin/perl 39.6M
Tue 2022-08-09 17:14:58 CEST 6773 483 65534 SIGSEGV present /usr/bin/perl 44.6M
Tue 2022-08-09 20:13:31 CEST 13193 483 65534 SIGSEGV present /usr/bin/perl 51.9M
Tue 2022-08-09 20:13:38 CEST 13372 483 65534 SIGSEGV present /usr/bin/perl 41.6M
Tue 2022-08-09 20:13:39 CEST 13005 483 65534 SIGSEGV present /usr/bin/perl 43.2M
Tue 2022-08-09 21:15:01 CEST 2115 483 65534 SIGSEGV present /usr/bin/perl 33.1M
Tue 2022-08-09 21:15:05 CEST 1888 483 65534 SIGSEGV present /usr/bin/perl 47.7M
Tue 2022-08-09 21:15:09 CEST 1328 483 65534 SIGSEGV present /usr/bin/perl 45.0M
Tue 2022-08-09 21:15:09 CEST 1067 483 65534 SIGSEGV present /usr/bin/perl 44.1M
Tue 2022-08-09 22:13:20 CEST 13473 483 65534 SIGSEGV present /usr/bin/perl 43.9M
Tue 2022-08-09 22:13:26 CEST 13660 483 65534 SIGSEGV present /usr/bin/perl 38.0M
Tue 2022-08-09 22:13:26 CEST 13839 483 65534 SIGSEGV present /usr/bin/perl 31.0M
Tue 2022-08-09 23:17:25 CEST 25368 483 65534 SIGSEGV present /usr/bin/perl 34.0M
Wed 2022-08-10 08:14:31 CEST 6738 483 65534 SIGSEGV present /usr/bin/perl 56.9M
Wed 2022-08-10 09:14:04 CEST 16219 483 65534 SIGSEGV present /usr/bin/perl 32.4M
Wed 2022-08-10 09:14:04 CEST 16399 483 65534 SIGSEGV present /usr/bin/perl 50.5M
Wed 2022-08-10 10:15:10 CEST 8711 483 65534 SIGSEGV present /usr/bin/perl 35.3M
Wed 2022-08-10 10:15:14 CEST 8476 483 65534 SIGSEGV present /usr/bin/perl 45.9M
Wed 2022-08-10 10:15:17 CEST 10829 483 65534 SIGSEGV present /usr/bin/perl 37.5M
…
okurz@openqaworker13:~> sudo coredumpctl dump
PID: 10829 (/usr/bin/isotov)
UID: 483 (_openqa-worker)
GID: 65534 (nobody)
Signal: 11 (SEGV)
Timestamp: Wed 2022-08-10 10:15:11 CEST (47min ago)
Command Line: $'/usr/bin/isotovideo: backen'
Executable: /usr/bin/perl
Control Group: /openqa.slice/openqa-worker.slice/openqa-worker-auto-restart@14.service
Unit: openqa-worker-auto-restart@14.service
Slice: openqa-worker.slice
Boot ID: 0c7a2997278b4887a01bfcfb5860e4c8
Machine ID: 625985c3f939414a1676d1d05a732110
Hostname: openqaworker13
Storage: /var/lib/systemd/coredump/core.\x2fusr\x2fbin\x2fisotov.483.0c7a2997278b4887a01bfcfb5860e4c8.10829.1660119311000000.zst (present)
Disk Size: 37.5M
Message: Process 10829 (/usr/bin/isotov) of user 483 dumped core.
Found module linux-vdso.so.1 with build-id: 7af0a0593d314958d54ccd203c5583a0559ae344
Found module DES.so with build-id: 4f1ace8e0f5612a7d6028ad3fa2450b62f346192
Found module libXau.so.6 with build-id: 7f7563c2e3af15370c200ce7ac1707fb085eb610
Found module libxcb.so.1 with build-id: 807011c756cf840f3b3fe4d99ef887b156e6b0b0
Found module libquadmath.so.0 with build-id: 2fc9e2b7504362902e670b97beaea72fdbe7cc8b
Found module libX11.so.6 with build-id: c8c57e64bcc5ab41d469a347b6df2c77f3db52e1
Found module libgfortran.so.5 with build-id: 153fc3d5cec8eb41b919bf9ba9247d048a0416ca
Found module libGLdispatch.so.0 with build-id: ce3cdddc8aa7e7893467c3fb0bae87ac90e369a8
Found module libGLX.so.0 with build-id: f9e775ea54c5e9fbacf1bd62d10eb0f7d2f39615
Found module libIlmThread-2_2.so.23 with build-id: 6a951b5f7047bde205063e1e8b0b3ced2869a781
Found module libIex-2_2.so.23 with build-id: e1f82a90100fcbbf6307d700cccb2fc653854695
Found module libHalf.so.23 with build-id: 722331adff14ba3c7a2247dc3af2f0124dbec0d1
Found module libjbig.so.2 with build-id: 6b85f4086bcd2e05e67fdb34521f5faeb16864de
Found module libtbb.so.2 with build-id: 08af210dc6a420306041f0de13750b2a120012f7
Found module libopenblas_pthreads.so.0 with build-id: 014cf542cb85a7629472440a64c8a8515b5d683c
Found module libGL.so.1 with build-id: 63979a9f6d9b1d72dcae0af4d0ef7ba28b748e5c
Found module libIlmImf-2_2.so.23 with build-id: 45cef729cf8422aeb450ddf2f6fae4fa151d6a76
Found module libopenjp2.so.7 with build-id: e0de6a6a23b5e36a0b29d293d2a4d0be07b3322f
Found module libtiff.so.5 with build-id: 8804cb29a6360449a4860382b5ecd58053bcdf9e
Found module libpng16.so.16 with build-id: 5f75035c3b165fede9a1e071c20259bf6d8d9389
Found module libwebp.so.7 with build-id: 9746ce683ab1853dfbc46d9d0ed121e9babb326e
Found module libjpeg.so.8 with build-id: 182687a3447abdb232af225929ad0dfdb408af97
Found module libgcc_s.so.1 with build-id: 901d2941694928694e67eccbc43f98d28e20780e
Found module libstdc++.so.6 with build-id: 129da2ab12b9d1617fedbf0d0d37c0572d5205d7
Found module libopencv_core.so.405 with build-id: 52644dfd235455622229eb6650381f74357843f3
Found module libopencv_imgproc.so.405 with build-id: df315ab280c93105e20c0d705cde58b81d7f9361
Found module libopencv_imgcodecs.so.405 with build-id: 17bb15909b983691b42a3d55b3cada680fc9ae98
Found module tinycv.so with build-id: 3a004086a2b53c2cf8f5b1f567efc8ea0b9eb2a8
Found module libexpat.so.1 with build-id: 3e8a0812306353cfcdfa9c0635886cdb9d8bffbe
Found module Expat.so with build-id: 1b4f675bda87b2119dca64ca8dfe22c3b9be721b
Found module libgpg-error.so.0 with build-id: 7703a2c38d4dc0cb40e4b1943f4d3dea9d7b1cae
Found module libgcrypt.so.20 with build-id: e348edacaceb362a6238b91448b26ea534608a42
Found module libcap.so.2 with build-id: 99405ab6633a66ccf29924793bf6ae8d8212ac9e
Found module liblz4.so.1 with build-id: ef9ad25a65c1623b33a365c00c3bc781492e7eaf
Found module libzstd.so.1 with build-id: f6d9025f2bc2a278acc6ac774636449e3d96aa1c
Found module liblzma.so.5 with build-id: 2d656c3bd393d5f9e95fdbbfe4fbbdc19e0cf0ba
Found module libsystemd.so.0 with build-id: 9305a7295aee5cabea6c3a474180eaabec3674c5
Found module libdbus-1.so.3 with build-id: 3481059afa8626b601fde370ff80f0bb80134416
Found module DBus.so with build-id: 66f8d05db06803b96a7f0f9b77da7859abbbac60
Found module XS.so with build-id: 6b2e6db66f7e3fd8236084c1fccaea6d6db26bac
Found module Langinfo.so with build-id: f108b3b9b0e21813b99b1d5a7b3716e13f7a0905
Found module DateTime.so with build-id: 864cfa3ac980b32ae2600babbc8da613592cd347
Found module Identify.so with build-id: 0b540d41fca6ff68ce2cf67be14690c66b522037
Found module XS.so with build-id: 995693dc4fb5ea9226a8a556ed9972b52d57d2a6
Found module Magic.so with build-id: 1dcc486ab55bde9599259592ff3cdf69e6838205
Found module XS.so with build-id: 127db2f9658dc6b84c533f42be0467921dbb8f14
Found module libssh2.so.1 with build-id: b945803d69b0ab0b923678c4cf8524de9b881d16
Found module SSH2.so with build-id: 8b6c074accba984424a8e22427994a788a213555
Found module NamedCapture.so with build-id: 2a5a9679e6f4b1ecca59f0d05b51905534788b5e
Found module FieldHash.so with build-id: 082fa4d9c8577c3cf004195785ffe368b1d2bbf8
Found module LibYAML.so with build-id: 9479b2e483c2fb6e4de61e128c8dc4ca314e151b
Found module Encoder.so with build-id: ee8fe16a86b1b2c46c663ca3764b4eb9e2d58169
Found module Punycode.so with build-id: b23d52cfbc0edcf03e4bfca447899af0c2ed839f
Found module Moment.so with build-id: fc82044fd3164fc1b3a0e150ebdb9cbf7a92a2a1
Found module SysV.so with build-id: f34cffe234151636d556bf80482b88ca99c8f046
Found module libcrypto.so.1.1 with build-id: cac3d636b7b29f085ea4ef6f7144ce89da7028ee
Found module libssl.so.1.1 with build-id: 51298ce9a79fab765af2998b83b18923ae4bd178
Found module SSLeay.so with build-id: ec04cadce2a1fe543c4451babb7dbd6ebb40d21b
Found module libnss_files.so.2 with build-id: 3bd874a11fbb12c60dd00591f37fc85750f59ff7
Found module attributes.so with build-id: 75acd55889f3d977db14cb3c33a62d0a1a295f38
Found module XS.so with build-id: 9a17fd009b089df406e8d2d2c26c6a43dd99260a
Found module B.so with build-id: b1c41b466ce1eaab0e632350b5db1e63a2c62cf3
Found module POSIX.so with build-id: 4069d8368e3edccd4e8027c85582a05c02e9f3bf
Found module re.so with build-id: d3c78e06301487f6d281ba2c76a8dc9793a397b6
Found module Normalize.so with build-id: 441648e204d7eb1d29124a2b57328ee1752e3e43
Found module librt.so.1 with build-id: 0675f40d0cd91db7a59f64729876f50c736e8805
Found module HiRes.so with build-id: f63bba8cfcf3103bb6d2f2e7e60f119b651c06be
Found module Socket.so with build-id: 0905a48cfb1629cf15ff663a3f3bd5a26847c7e5
Found module Cwd.so with build-id: 69c21ed9c509ba956c9a946a05b7aadc94608c46
Found module Base64.so with build-id: daf63f65a7a6e4f85aa9088b4b1a624b3e01055d
Found module libz.so.1 with build-id: 239bdc18697b1d0c289c5ba124f409ed0e09311d
Found module Zlib.so with build-id: 90eba948a97872d422293760dc90a642d891fa62
Found module IO.so with build-id: 8ebc2d92820c201b427b2e35bae2733e89d7b928
Found module Glob.so with build-id: c297f283393d147a8e3841f1196439350a88420c
Found module Storable.so with build-id: 1c68a92233496eb3b8b5362db2ddadda0d833450
Found module Encode.so with build-id: 381ed126801b24b12f039844aa70900d3028adbd
Found module SHA.so with build-id: 1d5de8603bb01ded19d0f44def1c3428aee1a089
Found module Fcntl.so with build-id: d5c36f117e08029425139b8102e063fb31672f03
Found module MD5.so with build-id: a49138f7da271609fe5b133bb1fb6e749a5dfb57
Found module Dumper.so with build-id: da3946eb6052fb15573731b8bb3c1c027999df5b
Found module Util.so with build-id: e3a9c891dbf63ad5197942d01d5c81cb3bd93072
Found module mro.so with build-id: fca906ed3d1617bb04863579403e1f00562fa277
Found module ld-linux-x86-64.so.2 with build-id: 731fe3b33b5e43b0a0630c430313ab2eb78f75db
Found module libc.so.6 with build-id: 28910b266cdd8f0c54c7830b758e4a1339f255c1
Found module libpthread.so.0 with build-id: a0ff2a2c46cd120e7406351499c8ed73f7c11098
Found module libcrypt.so.1 with build-id: 6c8a33f3e01abf18d4dfe9ac0cebd5abd8cab7de
Found module libdl.so.2 with build-id: e76bf98a5f8b1af5ba349496c5e56e85b0701d84
Found module libm.so.6 with build-id: 9e4d04062d1b620555a4ac050e62b3936bfc0c02
Found module perl with build-id: 751c22e4522022c351280da89517ead74180decd
Stack trace of thread 21735:
#0 0x0000560e34ec0a00 Perl_csighandler (perl + 0xc0a00)
#1 0x00007f06f3ece8c0 __restore_rt (libpthread.so.0 + 0x168c0)
#2 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#3 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#4 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#5 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#6 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10829:
#0 0x00007f06f3b9bce1 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xdbce1)
#1 0x00007f06f3ba19e3 __nanosleep (libc.so.6 + 0xe19e3)
#2 0x00007f06f3ba18fa sleep (libc.so.6 + 0xe18fa)
#3 0x00007f06f19f2e92 n/a (HiRes.so + 0x2e92)
#4 0x0000560e34edfae1 Perl_pp_entersub (perl + 0xdfae1)
#5 0x0000560e34ed7906 Perl_runops_standard (perl + 0xd7906)
#6 0x0000560e34e50cdf Perl_call_sv (perl + 0x50cdf)
#7 0x0000560e34ec51b5 Perl_sighandler (perl + 0xc51b5)
#8 0x0000560e34ec0bc7 Perl_despatch_signals (perl + 0xc0bc7)
#9 0x0000560e34ed84a6 Perl_pp_unstack (perl + 0xd84a6)
#10 0x0000560e34ed7906 Perl_runops_standard (perl + 0xd7906)
#11 0x0000560e34e587f7 perl_run (perl + 0x587f7)
#12 0x0000560e34e2ff12 main (perl + 0x2ff12)
#13 0x00007f06f3af52bd __libc_start_main (libc.so.6 + 0x352bd)
#14 0x0000560e34e2ff5a _start (perl + 0x2ff5a)
Stack trace of thread 10866:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10861:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10863:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10864:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10868:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10862:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10873:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10878:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10865:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10869:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10877:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10870:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10876:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10867:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10871:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10880:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10872:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10875:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10886:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10884:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10881:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10874:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10879:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10889:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10882:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10890:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10885:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10887:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10888:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
Stack trace of thread 10883:
#0 0x00007f06f3bd0ec9 syscall (libc.so.6 + 0x110ec9)
#1 0x00007f06e6f8c3f5 n/a (libtbb.so.2 + 0x1c3f5)
#2 0x00007f06e6f8c439 n/a (libtbb.so.2 + 0x1c439)
#3 0x00007f06f3ec26ea start_thread (libpthread.so.0 + 0xa6ea)
#4 0x00007f06f3bd7a8f __clone (libc.so.6 + 0x117a8f)
with gdb thread apply all bt showing the same. I don't think this brings us any further though. Unless anyone has a clue how to either more easily reproduce or what to add as debugging messages you don't need to point to more failures.