action #157339
closedcoordination #154777: [saga][epic] Shareable os-autoinst and test distribution plugins
coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributions
os-autoinst t/14-isotovideo.t is again taking too long (>20s on my setup) size:M
0%
Description
Motivation¶
While reviewing https://github.com/os-autoinst/os-autoinst/pull/2470 and trying t/14-isotovideo.t out I found that t/14-isotovideo.t runs into timeout of 20s so taking too long. We had cases like that in the past and should try to identify the bottleneck of that test again and improve the runtime.
Acceptance criteria¶
- AC1: t/14-isotovideo.t consistently passes well below the timeout of 20s again locally
Suggestions¶
- Run
prove -I. -v t/14-isotovideo.tor other means to profile that test and improve the code parts taking the most time. Consider https://metacpan.org/pod/Devel::NYTProf for both the test and isotovideo itself, e.g.perl -d:NYTProf ./isotovideo --helporperl -d:NYTProf ./isotovideo . - Run
git bisectto find out where a regression might have come from, e.g. maybe 5eb5dfb8 "Report test module failures via exit codes" by Jose D. Gomez R - Consider trying to improve the runtime of isotovideo itself, not only making the test faster
- It might be necessary to accept the runtime and increase the limit accordingly after a useful amount of effort had been invested and limited to optimize
Updated by okurz 7 months ago
- Related to action #135521: os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" size:M added
Updated by okurz 7 months ago
- Related to action #127046: t: os-autoinst - qemu keeps running when test runs into timelimit size:M added
Updated by okurz 7 months ago
- Related to action #81899: Move code from isotovideo to a module size:M added
Updated by okurz 7 months ago
- Related to action #71110: Reduce waiting time in case of os-autoinst shutdown added
Updated by okurz 7 months ago
- Related to action #68938: Try to reduce waiting time in case of qemu (early-)exits auto_review:"QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 443":retry added
Updated by ybonatakis 7 months ago
- Status changed from Workable to In Progress
- Assignee set to ybonatakis
Updated by ybonatakis 7 months ago · Edited
https://github.com/os-autoinst/os-autoinst/pull/2474
No change needed in the test. At least in my laptop the timeout is satisfied
Updated by openqa_review 7 months ago
- Due date set to 2024-04-09
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan 7 months ago · Edited
After profiling isotovideo/ 14-isotovideo.t together we decided there would be a significant and doable win by replacing some or all of the isotovideo/--exit-status-from-test-results calls with tests of _exit_code_from_test_results (because the new process is very expensive).
As an aside it's currently not possible to enable profiling and coverage at the same time within invoke-tests, so I proposed a draft for it. Not sure how useful it is now, but it is there if we want it.
Another approach we tried was enabling profiling within the unit test by adding it to the isotovideo function in 14-isotovideo.t.
@ybonatakis Suggested approach: Move _exit_code_from_test_results into Runner.pm and call this function from 14-isotovideo.t. The "non-existent WHEELS_DIR" subtest uses a $runner_mock, so you could use a similar approach to call the function.
Updated by ybonatakis 6 months ago
_exit_code_from_test_results moved to Runner.pm.
We removed subtests from 14-isotovideo.t except one which stayed to check the real call.
Tests moved to t/19-isotovideo-command-processing.t which seemed the most suitable place.
wallclock drops from ~17 secs to 14 for 14-isotovideo.t
Updated by okurz 6 months ago
- Copied to action #158511: Try to optimize the startup time of isotovideo more added