action #78019
Updated by okurz about 4 years ago
## Observation https://build.opensuse.org/package/live_build_log/devel:openQA:TestGithub:OPR-1567/os-autoinst/openSUSE_Factory/x86_64 shows ``` [ 191s] 3: ./12-bmwqemu.t ........................... ok [ 191s] 3: ./15-logging.t ........................... ok [ 191s] 3: ./16-send_with_fd.t ...................... ok [ 191s] 3: ./17-basetest.t .......................... ok [ 191s] 3: Bailout called. Further testing stopped: test exceeds runtime limit of '10' seconds [ 191s] 3: FAILED--Further testing stopped: test exceeds runtime limit of '10' seconds [ 191s] 3/3 Test #3: test-perl-testsuite ..............***Failed 109.48 sec [ 191s] ``` this likely means that the *next* test module which is not explicitly mentioned here times out. That would be according to the alphabetical order "t/18-backend-qemu.t". Locally I ran: have inconsistent results: With `count_fail_ratio prove -v -I . -I external/os-autoinst-common/lib -l --timer t/18-backend-qemu.t` t/18-qemu-options.t` using https://github.com/okurz/scripts/tree/master/count_fail_ratio I observe a very consistent runtime of 1s. 10s (and no timeout) but the commit caa97e71 says it should be below 5s. Running tests on caa97e71 checked out I can reproduce the same times. So if a regression it could be in dependencies. ## Acceptance criteria * **AC1:** Stable test locally, in travis CI and OBS * **AC2:** No significant increase in runtime ## Suggestions * If not feasible to fix fast please at least prevent the flaky test result, e.g. by bumping the timeout in the test module and reference this ticket * As old travis CI logs do not give us any indication for the runtime of individual test modules I suggest same as we have for openQA to introduce an environment variable with which we can add the prove option `--timer` and call that within CI but not by default locally * Crosscheck locally, compare to old results * If necessary bump up the timeout but ensure that we do not have a performance regression that we addressed in caa97e71 ## Workaround Retrigger test