action #78019
Updated by okurz about 4 years ago
## Observation
https://build.opensuse.org/package/live_build_log/devel:openQA:TestGithub:OPR-1567/os-autoinst/openSUSE_Factory/x86_64 shows
```
[ 191s] 3: ./12-bmwqemu.t ........................... ok
[ 191s] 3: ./15-logging.t ........................... ok
[ 191s] 3: ./16-send_with_fd.t ...................... ok
[ 191s] 3: ./17-basetest.t .......................... ok
[ 191s] 3: Bailout called. Further testing stopped: test exceeds runtime limit of '10' seconds
[ 191s] 3: FAILED--Further testing stopped: test exceeds runtime limit of '10' seconds
[ 191s] 3/3 Test #3: test-perl-testsuite ..............***Failed 109.48 sec
[ 191s]
```
this likely means that the *next* test module which is not explicitly mentioned here times out. That would be according to the alphabetical order "t/18-backend-qemu.t".
Locally I have inconsistent results: With `count_fail_ratio prove -v -I . -I external/os-autoinst-common/lib -l --timer t/18-qemu-options.t` using https://github.com/okurz/scripts/tree/master/count_fail_ratio I observe a very consistent runtime of 10s (and no timeout) but the commit caa97e71 says it should be below 5s.
Running tests on caa97e71 checked out I can reproduce the same times. So if a regression it could be in dependencies.
## Acceptance criteria
* **AC1:** Stable test locally, in travis CI and OBS
* **AC2:** No significant increase in runtime
## Suggestions
* If not feasible to fix fast please at least prevent the flaky test result, e.g. by bumping the timeout in the test module and reference this ticket
* As old travis CI logs do not give us any indication for the runtime of individual test modules I suggest same as we have for openQA to introduce an environment variable with which we can add the prove option `--timer` and call that within CI but not by default locally
* Crosscheck locally, compare to old results
* If necessary bump up the timeout but ensure that we do not have a performance regression that we addressed in caa97e71
## Workaround
Retrigger test