Project

General

Profile

Actions

action #157339

closed

coordination #154777: [saga][epic] Shareable os-autoinst and test distribution plugins

coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributions

os-autoinst t/14-isotovideo.t is again taking too long (>20s on my setup) size:M

Added by okurz 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2024-03-15
Due date:
% Done:

0%

Estimated time:

Description

Motivation

While reviewing https://github.com/os-autoinst/os-autoinst/pull/2470 and trying t/14-isotovideo.t out I found that t/14-isotovideo.t runs into timeout of 20s so taking too long. We had cases like that in the past and should try to identify the bottleneck of that test again and improve the runtime.

Acceptance criteria

  • AC1: t/14-isotovideo.t consistently passes well below the timeout of 20s again locally

Suggestions

  • Run prove -I. -v t/14-isotovideo.t or other means to profile that test and improve the code parts taking the most time. Consider https://metacpan.org/pod/Devel::NYTProf for both the test and isotovideo itself, e.g. perl -d:NYTProf ./isotovideo --help or perl -d:NYTProf ./isotovideo .
  • Run git bisect to find out where a regression might have come from, e.g. maybe 5eb5dfb8 "Report test module failures via exit codes" by Jose D. Gomez R
  • Consider trying to improve the runtime of isotovideo itself, not only making the test faster
  • It might be necessary to accept the runtime and increase the limit accordingly after a useful amount of effort had been invested and limited to optimize

Related issues 6 (2 open4 closed)

Related to openQA Project - action #135521: os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" size:MResolvedmkittler2023-09-11

Actions
Related to openQA Project - action #127046: t: os-autoinst - qemu keeps running when test runs into timelimit size:MWorkable2023-04-01

Actions
Related to openQA Project - action #81899: Move code from isotovideo to a module size:MResolvedlivdywan2021-01-08

Actions
Related to openQA Project - action #71110: Reduce waiting time in case of os-autoinst shutdownResolvedmkittler

Actions
Related to openQA Project - action #68938: Try to reduce waiting time in case of qemu (early-)exits auto_review:"QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 443":retryResolvedmkittler2020-07-14

Actions
Copied to openQA Project - action #158511: Try to optimize the startup time of isotovideo moreNewokurz

Actions
Actions #1

Updated by okurz 3 months ago

  • Related to action #135521: os-autoinst test t/14-isotovideo.t commonly runs into timeout executed locally, in subtest 5 "standard tests based on simple vars.json file" size:M added
Actions #2

Updated by okurz 3 months ago

  • Related to action #127046: t: os-autoinst - qemu keeps running when test runs into timelimit size:M added
Actions #3

Updated by okurz 3 months ago

  • Parent task set to #108527
Actions #4

Updated by okurz 3 months ago

  • Related to action #81899: Move code from isotovideo to a module size:M added
Actions #5

Updated by okurz 3 months ago

  • Related to action #71110: Reduce waiting time in case of os-autoinst shutdown added
Actions #6

Updated by okurz 3 months ago

  • Related to action #68938: Try to reduce waiting time in case of qemu (early-)exits auto_review:"QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 443":retry added
Actions #7

Updated by okurz 3 months ago

  • Subject changed from os-autoinst t/14-isotovideo.t is again taking too long (>20s on my setup) to os-autoinst t/14-isotovideo.t is again taking too long (>20s on my setup) size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #8

Updated by ybonatakis 3 months ago

  • Status changed from Workable to In Progress
  • Assignee set to ybonatakis
Actions #9

Updated by ybonatakis 3 months ago · Edited

https://github.com/os-autoinst/os-autoinst/pull/2474
No change needed in the test. At least in my laptop the timeout is satisfied

Actions #10

Updated by openqa_review 3 months ago

  • Due date set to 2024-04-09

Setting due date based on mean cycle time of SUSE QE Tools

Actions #11

Updated by livdywan 3 months ago · Edited

After profiling isotovideo/ 14-isotovideo.t together we decided there would be a significant and doable win by replacing some or all of the isotovideo/--exit-status-from-test-results calls with tests of _exit_code_from_test_results (because the new process is very expensive).

As an aside it's currently not possible to enable profiling and coverage at the same time within invoke-tests, so I proposed a draft for it. Not sure how useful it is now, but it is there if we want it.
Another approach we tried was enabling profiling within the unit test by adding it to the isotovideo function in 14-isotovideo.t.

@ybonatakis Suggested approach: Move _exit_code_from_test_results into Runner.pm and call this function from 14-isotovideo.t. The "non-existent WHEELS_DIR" subtest uses a $runner_mock, so you could use a similar approach to call the function.

Actions #12

Updated by ybonatakis 3 months ago

_exit_code_from_test_results moved to Runner.pm.
We removed subtests from 14-isotovideo.t except one which stayed to check the real call.
Tests moved to t/19-isotovideo-command-processing.t which seemed the most suitable place.

wallclock drops from ~17 secs to 14 for 14-isotovideo.t

Actions #13

Updated by ybonatakis 2 months ago

  • Status changed from In Progress to Feedback
Actions #14

Updated by ybonatakis 2 months ago

  • Status changed from Feedback to Resolved
Actions #15

Updated by okurz 2 months ago

  • Due date deleted (2024-04-09)
Actions #16

Updated by okurz 2 months ago

  • Copied to action #158511: Try to optimize the startup time of isotovideo more added
Actions

Also available in: Atom PDF