Project

General

Profile

action #78019

[sporadic] os-autoinst t/18-backend-qemu.t timed out in OBS checks after 10s

Added by okurz 2 months ago. Updated 2 months ago.

Status:
Rejected
Priority:
High
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2020-11-16
Due date:
% Done:

0%

Estimated time:
Difficulty:
medium

Description

Observation

https://build.opensuse.org/package/live_build_log/devel:openQA:TestGithub:OPR-1567/os-autoinst/openSUSE_Factory/x86_64 shows

[  191s] 3: ./12-bmwqemu.t ........................... ok
[  191s] 3: ./15-logging.t ........................... ok
[  191s] 3: ./16-send_with_fd.t ...................... ok
[  191s] 3: ./17-basetest.t .......................... ok
[  191s] 3: Bailout called.  Further testing stopped:  test exceeds runtime limit of '10' seconds
[  191s] 3: FAILED--Further testing stopped: test exceeds runtime limit of '10' seconds
[  191s] 3/3 Test #3: test-perl-testsuite ..............***Failed  109.48 sec
[  191s] 

this likely means that the next test module which is not explicitly mentioned here times out. That would be according to the alphabetical order "t/18-backend-qemu.t".

Locally I ran: With count_fail_ratio prove -v -I . -I external/os-autoinst-common/lib -l --timer t/18-backend-qemu.t using https://github.com/okurz/scripts/tree/master/count_fail_ratio I observe a very consistent runtime of 1s.

Running tests on caa97e71 checked out I can reproduce the same times. So if a regression it could be in dependencies.

Acceptance criteria

  • AC1: Stable test locally, in travis CI and OBS
  • AC2: No significant increase in runtime

Suggestions

  • If not feasible to fix fast please at least prevent the flaky test result, e.g. by bumping the timeout in the test module and reference this ticket
  • As old travis CI logs do not give us any indication for the runtime of individual test modules I suggest same as we have for openQA to introduce an environment variable with which we can add the prove option --timer and call that within CI but not by default locally
  • Crosscheck locally, compare to old results
  • If necessary bump up the timeout but ensure that we do not have a performance regression that we addressed in caa97e71

Workaround

Retrigger test

History

#1 Updated by okurz 2 months ago

  • Description updated (diff)

#2 Updated by okurz 2 months ago

  • Subject changed from [sporadic] os-autoinst t/18-backend-qemu.t timed out in OBS checks but should be way faster to [sporadic] os-autoinst t/18-backend-qemu.t timed out in OBS checks after 10s
  • Description updated (diff)
  • Status changed from Workable to Rejected
  • Assignee set to okurz

I ran env runs=500 count_fail_ratio prove -I . -I external/os-autoinst-common/lib -l t/18-backend-qemu.t with no problems at all, consistent runtime of 1s or less. Not much we can do unless this reappears. Because I am not aware of previous, similar problems I reject this ticket with no further changes conducted.

Also available in: Atom PDF