QA » openQA Project » openQA Tests

I've seen many timeout failures with script_output. I think we just need to add a timeout. So, something like

my $ver_linux_out = script_output("cat $ver_linux_log", 300);

Had issues with my openQA instance with spvm in last 2+days and couldn't progress on the case, but my findings are now:

• happens for ipmi as well https://openqa.suse.de/tests/3554259#step/boot_ltp/23 (notified pvorel as well)
• simple solution with timeout increase is not correct one
• when it works it usually takes 2 seconds and when looking on logs, all was printed, file is quite small, there is no reason to take it more

Failing even with timeout change to 180:

[2019-11-13T13:35:56.395 CET] [debug] /var/lib/openqa/share/tests/sle/tests/kernel/boot_ltp.pm:78 called testapi::script_output
[2019-11-13T13:35:56.395 CET] [debug] <<< testapi::type_string(string='(echo ORQ2Z; bash -eox pipefail /tmp/scriptORQ2Z.sh ; echo SCRIPT_FINISHEDORQ2Z-$?-) | tee /dev/sshserial
', max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2019-11-13T13:35:59.007 CET] [debug] /var/lib/openqa/share/tests/sle/tests/kernel/boot_ltp.pm:78 called testapi::script_output
[2019-11-13T13:35:59.007 CET] [debug] <<< testapi::wait_serial(quiet=undef, no_regex=0, regexp='SCRIPT_FINISHEDORQ2Z-\\d+-', timeout=180, record_output=1, expect_not_found=0, buffer_size=undef)
ORQ2Z

pcervinka wrote:

I was successfully reproducing failure http://10.100.12.105/tests/3512 with os-autoinst version 4.6.1572372410.447dab86-lp151.214.1. When updated to 4.6.1573588068.2f8e74c9-lp151.234.1 it started work without failures. Did verification with multiple cat commands http://10.100.12.105/tests/3517. Jobs on osd seems to be fine now https://openqa.suse.de/tests/3588069.

Seems to be related to fix https://github.com/os-autoinst/os-autoinst/pull/1258.

Let's see more test runs.

Nice! I hope it also applies for ipmi jobs too :) I remember similar errors when I was using script_output in ipmi jobs.
Thanks for doing these checks.