Project

General

Profile

action #59226

[kernel]ltp][spvm][sporadic] test fails in boot_ltp - cat /tmp/ver_linux_before.txt timeout

Added by pcervinka almost 2 years ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
QE Kernel - QE Kernel Done
Start date:
2019-11-08
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-15-SP2-Installer-DVD-ppc64le-ltp_cve_spvm@ppc64le-spvm fails in
boot_ltp

Test suite description

Reproducible

Fails since (at least) Build 77.1

Expected result

Last good: 77.1 (or more recent)

Further details

Always latest result in this scenario: latest

Randomly happens (but often), timeout increase will be probably needed.


Related issues

Related to openQA Project - action #59190: Broken SSH serial consoleResolved2019-11-07

History

#1 Updated by jlausuch almost 2 years ago

I've seen many timeout failures with script_output. I think we just need to add a timeout. So, something like

my $ver_linux_out = script_output("cat $ver_linux_log", 300);

#2 Updated by pcervinka almost 2 years ago

  • Status changed from New to In Progress
  • Target version changed from 454 to 445

#3 Updated by pcervinka almost 2 years ago

Had issues with my openQA instance with spvm in last 2+days and couldn't progress on the case, but my findings are now:

  • happens for ipmi as well https://openqa.suse.de/tests/3554259#step/boot_ltp/23 (notified pvorel as well)
  • simple solution with timeout increase is not correct one
  • when it works it usually takes 2 seconds and when looking on logs, all was printed, file is quite small, there is no reason to take it more

#4 Updated by pcervinka almost 2 years ago

Failing even with timeout change to 180:

[2019-11-13T13:35:56.395 CET] [debug] /var/lib/openqa/share/tests/sle/tests/kernel/boot_ltp.pm:78 called testapi::script_output
[2019-11-13T13:35:56.395 CET] [debug] <<< testapi::type_string(string='(echo ORQ2Z; bash -eox pipefail /tmp/scriptORQ2Z.sh ; echo SCRIPT_FINISHEDORQ2Z-$?-) | tee /dev/sshserial
', max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2019-11-13T13:35:59.007 CET] [debug] /var/lib/openqa/share/tests/sle/tests/kernel/boot_ltp.pm:78 called testapi::script_output
[2019-11-13T13:35:59.007 CET] [debug] <<< testapi::wait_serial(quiet=undef, no_regex=0, regexp='SCRIPT_FINISHEDORQ2Z-\\d+-', timeout=180, record_output=1, expect_not_found=0, buffer_size=undef)
ORQ2Z

#5 Updated by pcervinka almost 2 years ago

  • Status changed from In Progress to Feedback

I was successfully reproducing failure http://10.100.12.105/tests/3512 with os-autoinst version 4.6.1572372410.447dab86-lp151.214.1. When updated to 4.6.1573588068.2f8e74c9-lp151.234.1 it started work without failures. Did verification with multiple cat commands http://10.100.12.105/tests/3517. Jobs on osd seems to be fine now https://openqa.suse.de/tests/3588069.

Seems to be related to fix https://github.com/os-autoinst/os-autoinst/pull/1258.

Let's see more test runs.

#6 Updated by pcervinka almost 2 years ago

#8 Updated by jlausuch almost 2 years ago

pcervinka wrote:

I was successfully reproducing failure http://10.100.12.105/tests/3512 with os-autoinst version 4.6.1572372410.447dab86-lp151.214.1. When updated to 4.6.1573588068.2f8e74c9-lp151.234.1 it started work without failures. Did verification with multiple cat commands http://10.100.12.105/tests/3517. Jobs on osd seems to be fine now https://openqa.suse.de/tests/3588069.

Seems to be related to fix https://github.com/os-autoinst/os-autoinst/pull/1258.

Let's see more test runs.

Nice! I hope it also applies for ipmi jobs too :) I remember similar errors when I was using script_output in ipmi jobs.
Thanks for doing these checks.

#9 Updated by pcervinka almost 2 years ago

  • Status changed from Feedback to Resolved

Works fine, there are no more issues on spvm.

#10 Updated by jlausuch over 1 year ago

  • Target version changed from 445 to 457

#11 Updated by pcervinka 11 months ago

  • Target version changed from 457 to QE Kernel Done

Also available in: Atom PDF