Project

General

Profile

Actions

action #135521

closed

os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" size:M

Added by okurz 6 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2023-09-11
Due date:
% Done:

0%

Estimated time:

Description

Observations

os-autoinst $ prove -v -I. t/14-isotovideo.t 
t/14-isotovideo.t .. 
# Subtest: get the version number
    ok 1 - version printed
    ok 2 - no state file was written
    1..2
ok 1 - get the version number
# Subtest: color output can be configured via the command-line
    # Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null --color=yes casedir=/home/okurz/local/os-autoinst/os-autoinst/t/data/tests schedule=module1,bar/module2 _exit_after_schedule=1
    ok 1 - isotovideo exit code
    ok 2 - logs use colors when requested
    # Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null --color=no casedir=/home/okurz/local/os-autoinst/os-autoinst/t/data/tests schedule=module1,bar/module2 _exit_after_schedule=1
    ok 3 - isotovideo exit code
    ok 4 - no colors in logs
    1..4
ok 2 - color output can be configured via the command-line
# Subtest: standalone isotovideo without any parameters
    # Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null
    ok 1 - isotovideo exit code
    ok 2 - initialization error printed to user
    1..2
ok 3 - standalone isotovideo without any parameters
# Subtest: standalone isotovideo without vars.json file and only command line parameters
    # Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null casedir=/home/okurz/local/os-autoinst/os-autoinst/t/data/tests schedule=module1,bar/module2 _exit_after_schedule=1
    ok 1 - isotovideo exit code
    ok 2 - requested module module1 scheduled
    ok 3 - requested module bar/module2 scheduled
    1..3
ok 4 - standalone isotovideo without vars.json file and only command line parameters
# Subtest: standard tests based on simple vars.json file
    # Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null
    ok 1 - isotovideo exit code
    ok 2 - shutdown scheduled
    1..2
ok 5 - standard tests based on simple vars.json file
# Subtest: isotovideo with custom git repo parameters specified
    ok 1 - initialized test repo
    # Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null casedir=file:///tmp/14-isotovideo.t-iG1X/pool/repo.git#foo needles_dir=/home/okurz/local/os-autoinst/os-autoinst/t/data _exit_after_schedule=1
Bailout called.  Further testing stopped:  test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds
Bail out!  test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds
FAILED--Further testing stopped: test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds

Acceptance criteria

  • AC1: t/14-isotvideo.t passes prove -I. t/14-isotovideo.t consistently without running into timeout locally

Suggestions

Actions #1

Updated by livdywan 6 months ago

  • Subject changed from os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" to os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #3

Updated by okurz 5 months ago

  • Target version changed from Ready to Tools - Next
Actions #4

Updated by okurz 3 months ago

  • Target version changed from Tools - Next to Ready
Actions #5

Updated by mkittler 3 months ago ยท Edited

  • Status changed from Workable to In Progress
  • Assignee set to mkittler

FAILED--Further testing stopped: test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds

It looks like the limit has already been bumped because it is actually 30 seconds (use OpenQA::Test::TimeLimit '30';). Therefore the test also doesn't fail for me locally where it takes 28 seconds to execute. That's of course not much headroom. So maybe it makes sense to bump the timeout further to at least 35 seconds.

The subtest isotovideo with custom git repo parameters specified takes quite a while so maybe it can be optimized. I'll also have a look into upload assets on demand even in failed jobs which is also taking a little bit longer than the other sub tests.

Actions #6

Updated by mkittler 3 months ago

  • Status changed from In Progress to Feedback

This PR will speed up the test again: https://github.com/os-autoinst/os-autoinst/pull/2410

It also restores the previous limit of 20 seconds which now seems appropriate again (enough headroom).

Actions #7

Updated by okurz 3 months ago

mkittler wrote in #note-5:

FAILED--Further testing stopped: test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds

It looks like the limit has already been bumped because it is actually 30 seconds

Yes, that was done as part of https://github.com/os-autoinst/os-autoinst/pull/2363, see the comment above. Now the goal is to significantly reduce the runtime. Looks like you are already on a good track

Actions #8

Updated by mkittler 3 months ago

  • Status changed from Feedback to Resolved

The PR has been merged. It restored the performance of the test before the regression.

Actions

Also available in: Atom PDF