action #157339
closed
coordination #154777: [saga][epic] Shareable os-autoinst and test distribution plugins
coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributions
os-autoinst t/14-isotovideo.t is again taking too long (>20s on my setup) size:M
Added by okurz 9 months ago.
Updated 9 months ago.
Category:
Regressions/Crashes
Description
Motivation¶
While reviewing https://github.com/os-autoinst/os-autoinst/pull/2470 and trying t/14-isotovideo.t out I found that t/14-isotovideo.t runs into timeout of 20s so taking too long. We had cases like that in the past and should try to identify the bottleneck of that test again and improve the runtime.
Acceptance criteria¶
- AC1: t/14-isotovideo.t consistently passes well below the timeout of 20s again locally
Suggestions¶
- Run
prove -I. -v t/14-isotovideo.t or other means to profile that test and improve the code parts taking the most time. Consider https://metacpan.org/pod/Devel::NYTProf for both the test and isotovideo itself, e.g. perl -d:NYTProf ./isotovideo --help or perl -d:NYTProf ./isotovideo .
- Run
git bisect to find out where a regression might have come from, e.g. maybe 5eb5dfb8 "Report test module failures via exit codes" by Jose D. Gomez R
- Consider trying to improve the runtime of isotovideo itself, not only making the test faster
- It might be necessary to accept the runtime and increase the limit accordingly after a useful amount of effort had been invested and limited to optimize
- Related to action #135521: os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" size:M added
- Related to action #127046: t: os-autoinst - qemu keeps running when test runs into timelimit size:M added
- Parent task set to #108527
- Related to action #81899: Move code from isotovideo to a module size:M added
- Related to action #71110: Reduce waiting time in case of os-autoinst shutdown added
- Related to action #68938: Try to reduce waiting time in case of qemu (early-)exits auto_review:"QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 443":retry added
- Subject changed from os-autoinst t/14-isotovideo.t is again taking too long (>20s on my setup) to os-autoinst t/14-isotovideo.t is again taking too long (>20s on my setup) size:M
- Description updated (diff)
- Status changed from New to Workable
- Status changed from Workable to In Progress
- Assignee set to ybonatakis
- Due date set to 2024-04-09
Setting due date based on mean cycle time of SUSE QE Tools
After profiling isotovideo/ 14-isotovideo.t together we decided there would be a significant and doable win by replacing some or all of the isotovideo/--exit-status-from-test-results calls with tests of _exit_code_from_test_results (because the new process is very expensive).
As an aside it's currently not possible to enable profiling and coverage at the same time within invoke-tests, so I proposed a draft for it. Not sure how useful it is now, but it is there if we want it.
Another approach we tried was enabling profiling within the unit test by adding it to the isotovideo function in 14-isotovideo.t.
@ybonatakis Suggested approach: Move _exit_code_from_test_results into Runner.pm and call this function from 14-isotovideo.t. The "non-existent WHEELS_DIR" subtest uses a $runner_mock, so you could use a similar approach to call the function.
_exit_code_from_test_results moved to Runner.pm.
We removed subtests from 14-isotovideo.t except one which stayed to check the real call.
Tests moved to t/19-isotovideo-command-processing.t which seemed the most suitable place.
wallclock drops from ~17 secs to 14 for 14-isotovideo.t
- Status changed from In Progress to Feedback
- Status changed from Feedback to Resolved
- Due date deleted (
2024-04-09)
- Copied to action #158511: Try to optimize the startup time of isotovideo more added
Also available in: Atom
PDF