Project

General

Profile

action #78019

Updated by okurz over 3 years ago

## Observation 

 https://build.opensuse.org/package/live_build_log/devel:openQA:TestGithub:OPR-1567/os-autoinst/openSUSE_Factory/x86_64 shows 

 ``` 
 [    191s] 3: ./12-bmwqemu.t ........................... ok 
 [    191s] 3: ./15-logging.t ........................... ok 
 [    191s] 3: ./16-send_with_fd.t ...................... ok 
 [    191s] 3: ./17-basetest.t .......................... ok 
 [    191s] 3: Bailout called.    Further testing stopped:    test exceeds runtime limit of '10' seconds 
 [    191s] 3: FAILED--Further testing stopped: test exceeds runtime limit of '10' seconds 
 [    191s] 3/3 Test #3: test-perl-testsuite ..............***Failed    109.48 sec 
 [    191s]  
 ``` 

 this likely means that the *next* test module which is not explicitly mentioned here times out. That would be according to the alphabetical order "t/18-backend-qemu.t". 

 Locally I ran: have inconsistent results: With `count_fail_ratio prove -v -I . -I external/os-autoinst-common/lib    -l --timer t/18-backend-qemu.t` t/18-qemu-options.t` using https://github.com/okurz/scripts/tree/master/count_fail_ratio I observe a very consistent runtime of 1s. 10s (and no timeout) but the commit caa97e71 says it should be below 5s. 

 Running tests on caa97e71 checked out I can reproduce the same times. So if a regression it could be in dependencies. 

 ## Acceptance criteria 
 * **AC1:** Stable test locally, in travis CI and OBS 
 * **AC2:** No significant increase in runtime 

 ## Suggestions 

 * If not feasible to fix fast please at least prevent the flaky test result, e.g. by bumping the timeout in the test module and reference this ticket 
 * As old travis CI logs do not give us any indication for the runtime of individual test modules I suggest same as we have for openQA to introduce an environment variable with which we can add the prove option `--timer` and call that within CI but not by default locally 
 * Crosscheck locally, compare to old results 
 * If necessary bump up the timeout but ensure that we do not have a performance regression that we addressed in caa97e71 

 ## Workaround 

 Retrigger test

Back