QA » openQA Project

as first step https://github.com/os-autoinst/scripts/blob/master/README.md#auto-review---automatically-detect-known-issues-in-openqa-jobs-label-openqa-jobs-with-ticket-references-and-optionally-retrigger should be used to detect the issue and automatically retrigger

I've been adding the regex but it doesn't look like we're running auto_review currently/anymore.

Luckily the stacktrace is still available so I've attached it. That's the unmangled version of the relevant thread:

(gdb) bt
#0  0x00007fa5d9349420 in raise () from /lib64/libc.so.6
#1  0x00007fa5d934aa01 in abort () from /lib64/libc.so.6
#2  0x00007fa5d938c877 in __libc_message () from /lib64/libc.so.6
#3  0x00007fa5d9393093 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fa5d9394999 in _int_free () from /lib64/libc.so.6
#5  0x00007fa5d1659589 in cv::MatAllocator::unmap(cv::UMatData*) const () from /usr/lib64/libopencv_core.so.3.3
#6  0x00007fa5d26968a1 in image_destroy(Image*) () from /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
#7  0x00007fa5d269d32e in ?? () from /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/auto/tinycv/tinycv.so
#8  0x00005560ecf47ac1 in Perl_pp_entersub (my_perl=0x5560ee4cb260) at pp_hot.c:4231
#9  0x00005560eceb8ca8 in Perl_call_sv (my_perl=my_perl@entry=0x5560ee4cb260, sv=0x5560f319f950, flags=flags@entry=45) at perl.c:2848
#10 0x00005560ecf4ca4e in S_curse (my_perl=my_perl@entry=0x5560ee4cb260, sv=sv@entry=0x5560f3237ef0, check_refcnt=check_refcnt@entry=true) at sv.c:6972
#11 0x00005560ecf4d2f8 in Perl_sv_clear (my_perl=my_perl@entry=0x5560ee4cb260, orig_sv=orig_sv@entry=0x5560f31f6058) at sv.c:6576
#12 0x00005560ecf4d7b1 in Perl_sv_free2 (my_perl=0x5560ee4cb260, sv=0x5560f31f6058, rc=<optimized out>) at sv.c:7073
#13 0x00005560ecf7fa5c in Perl_leave_scope (my_perl=my_perl@entry=0x5560ee4cb260, base=<optimized out>) at scope.c:1178
#14 0x00005560ecf83a08 in Perl_dounwind (my_perl=my_perl@entry=0x5560ee4cb260, cxix=cxix@entry=20) at pp_ctl.c:1552
#15 0x00005560ecf8eef1 in Perl_dounwind (cxix=20, my_perl=0x5560ee4cb260) at pp_ctl.c:2471
#16 Perl_pp_return (my_perl=0x5560ee4cb260) at pp_ctl.c:2477
#17 0x00005560ecf3f8e6 in Perl_runops_standard (my_perl=0x5560ee4cb260) at run.c:41
#18 0x00005560ecec07d7 in S_run_body (oldscope=<optimized out>, my_perl=<optimized out>) at perl.c:2524
#19 perl_run (my_perl=0x5560ee4cb260) at perl.c:2447
#20 0x00005560ece97ef2 in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:123

Looks like Perl passed a bad pointer to the XS function for destroying the tinycv::Image object so its destructor passes a bad pointer to the destructor of the cv::Mat object.

There are 3 more stack traces, likely one of them corresponds to the 2nd job. So there are basically two crashes, one involving image_destroy and one involving image_search.

By the way, I haven't seen such stack traces before (e.g. when looking at backend stack traces for different crashes).

Strangely there's no coredump from the day this job ran on openqaworker10 where the job ran according to the database. However, there are several other coredumps which I've attached. Many have actually no tinycv.so-frames within the stack trace; not sure whether they might even be bugs in Perl. One backtrace contains an image_scale-frame which is new (so far I've only seen image_search and image_destroy). The TBB threads all look the same and unproblematic (so the issue that signal handlers are invoked in these threads doesn't seem to be re-occurring).

Not sure what the problem could be and how to reproduce it.

During the daily 2021-08-31 we decided that we don't have a good idea how to fix the actual problem. I see the following alternatives:

1. With the above information provided and assuming that "auto_review" handles automatic retriggering at least on o3+osd move the ticket out of the backlog and wait for further information. Please anyone who sees the issue again or has any additional information comment on the ticket
2. Ensure that openQA has a proper reason in an incomplete job so that openQA can automatically retrigger, see https://github.com/os-autoinst/openQA/blob/master/etc/openqa/openqa.ini#L76
3. Ask perl experts if they can read out something better from the stack trace