QA » openQA Project

Observation¶

https://build.opensuse.org/package/live_build_log/devel:openQA:TestGithub:OPR-1567/os-autoinst/openSUSE_Factory/x86_64 shows

[  191s] 3: ./12-bmwqemu.t ........................... ok
[  191s] 3: ./15-logging.t ........................... ok
[  191s] 3: ./16-send_with_fd.t ...................... ok
[  191s] 3: ./17-basetest.t .......................... ok
[  191s] 3: Bailout called.  Further testing stopped:  test exceeds runtime limit of '10' seconds
[  191s] 3: FAILED--Further testing stopped: test exceeds runtime limit of '10' seconds
[  191s] 3/3 Test #3: test-perl-testsuite ..............***Failed  109.48 sec
[  191s] 

this likely means that the next test module which is not explicitly mentioned here times out. That would be according to the alphabetical order "t/18-backend-qemu.t".

Locally I ran: With count_fail_ratio prove -v -I . -I external/os-autoinst-common/lib -l --timer t/18-backend-qemu.t using https://github.com/okurz/scripts/tree/master/count_fail_ratio I observe a very consistent runtime of 1s.

Running tests on caa97e71 checked out I can reproduce the same times. So if a regression it could be in dependencies.

Acceptance criteria¶

• AC1: Stable test locally, in travis CI and OBS
• AC2: No significant increase in runtime

Suggestions¶

• If not feasible to fix fast please at least prevent the flaky test result, e.g. by bumping the timeout in the test module and reference this ticket
• As old travis CI logs do not give us any indication for the runtime of individual test modules I suggest same as we have for openQA to introduce an environment variable with which we can add the prove option --timer and call that within CI but not by default locally
• Crosscheck locally, compare to old results
• If necessary bump up the timeout but ensure that we do not have a performance regression that we addressed in caa97e71

Workaround¶

Retrigger test