action #135521
closedos-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" size:M
Description
Observations¶
os-autoinst $ prove -v -I. t/14-isotovideo.t
t/14-isotovideo.t ..
# Subtest: get the version number
ok 1 - version printed
ok 2 - no state file was written
1..2
ok 1 - get the version number
# Subtest: color output can be configured via the command-line
# Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null --color=yes casedir=/home/okurz/local/os-autoinst/os-autoinst/t/data/tests schedule=module1,bar/module2 _exit_after_schedule=1
ok 1 - isotovideo exit code
ok 2 - logs use colors when requested
# Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null --color=no casedir=/home/okurz/local/os-autoinst/os-autoinst/t/data/tests schedule=module1,bar/module2 _exit_after_schedule=1
ok 3 - isotovideo exit code
ok 4 - no colors in logs
1..4
ok 2 - color output can be configured via the command-line
# Subtest: standalone isotovideo without any parameters
# Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null
ok 1 - isotovideo exit code
ok 2 - initialization error printed to user
1..2
ok 3 - standalone isotovideo without any parameters
# Subtest: standalone isotovideo without vars.json file and only command line parameters
# Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null casedir=/home/okurz/local/os-autoinst/os-autoinst/t/data/tests schedule=module1,bar/module2 _exit_after_schedule=1
ok 1 - isotovideo exit code
ok 2 - requested module module1 scheduled
ok 3 - requested module bar/module2 scheduled
1..3
ok 4 - standalone isotovideo without vars.json file and only command line parameters
# Subtest: standard tests based on simple vars.json file
# Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null
ok 1 - isotovideo exit code
ok 2 - shutdown scheduled
1..2
ok 5 - standard tests based on simple vars.json file
# Subtest: isotovideo with custom git repo parameters specified
ok 1 - initialized test repo
# Starting isotovideo with: /usr/bin/perl /home/okurz/local/os-autoinst/os-autoinst/isotovideo --workdir /tmp/14-isotovideo.t-iG1X/pool -d backend=null casedir=file:///tmp/14-isotovideo.t-iG1X/pool/repo.git#foo needles_dir=/home/okurz/local/os-autoinst/os-autoinst/t/data _exit_after_schedule=1
Bailout called. Further testing stopped: test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds
Bail out! test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds
FAILED--Further testing stopped: test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds
Acceptance criteria¶
- AC1: t/14-isotvideo.t passes
prove -I. t/14-isotovideo.t
consistently without running into timeout locally
Suggestions¶
- Use https://github.com/okurz/scripts/blob/master/count_fail_ratio or
retry
to find out failure statistics - Bisect where a performance regression was introduced
- Fix the overly long runtime of the test
- As applicable refactor further to improve runtime
- Adjust timeout value as applicable
Updated by livdywan about 1 year ago
- Subject changed from os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" to os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by okurz about 1 year ago
Updated by okurz about 1 year ago
- Target version changed from Ready to Tools - Next
Updated by mkittler 11 months ago ยท Edited
- Status changed from Workable to In Progress
- Assignee set to mkittler
FAILED--Further testing stopped: test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds
It looks like the limit has already been bumped because it is actually 30 seconds (use OpenQA::Test::TimeLimit '30';
). Therefore the test also doesn't fail for me locally where it takes 28 seconds to execute. That's of course not much headroom. So maybe it makes sense to bump the timeout further to at least 35 seconds.
The subtest isotovideo with custom git repo parameters specified
takes quite a while so maybe it can be optimized. I'll also have a look into upload assets on demand even in failed jobs
which is also taking a little bit longer than the other sub tests.
Updated by mkittler 11 months ago
- Status changed from In Progress to Feedback
This PR will speed up the test again: https://github.com/os-autoinst/os-autoinst/pull/2410
It also restores the previous limit of 20 seconds which now seems appropriate again (enough headroom).
Updated by okurz 11 months ago
mkittler wrote in #note-5:
FAILED--Further testing stopped: test 't/14-isotovideo.t' exceeds runtime limit of '20' seconds
It looks like the limit has already been bumped because it is actually 30 seconds
Yes, that was done as part of https://github.com/os-autoinst/os-autoinst/pull/2363, see the comment above. Now the goal is to significantly reduce the runtime. Looks like you are already on a good track
Updated by okurz 7 months ago
- Related to action #157339: os-autoinst t/14-isotovideo.t is again taking too long (>20s on my setup) size:M added