t/25-cache-service.t fails exceeding 90s timeout consistently size:M
Running t/25-cache-service.t locally for me consistently fails with
ok 10 - Asset sle-12-SP3firstname.lastname@example.org downloaded correctly ok 11 - Asset sle-12-SP3email@example.com downloaded correctly ok 12 - Asset sle-12-SP3firstname.lastname@example.org downloaded correctly <5> [i] Worker 26308 stopped # Cache minion worker stopped <5> [i] Worker 26309 stopped # Cache minion worker stopped Bailout called. Further testing stopped: test 't/25-cache-service.t' exceeds runtime limit of '90' seconds Bail out! test 't/25-cache-service.t' exceeds runtime limit of '90' seconds FAILED--Further testing stopped: test 't/25-cache-service.t' exceeds runtime limit of '90' seconds
count_fail_ratio prove -l -v t/25-cache-service.t using https://github.com/okurz/scripts/blob/master/count_fail_ratio with default 20 runs and all 20 runs fail consistently in the above step.
circleCI already runs Leap 15.3 as well https://github.com/os-autoinst/openQA/blob/master/container/devel:openQA:ci/base/Dockerfile#L7 and that works just fine there. If I bump the timeout 90s->300s then tests pass fine.
- AC1: Time limit of 90s is not exceeded
- Run tests in a clean container e.g.
toolbox -u -i registry.opensuse.org/opensuse/leap:15.3 -t leap15.3or similar
- Maybe files left over from another test run?
- Consider a
git statuscheck like we have for os-autoinst
- Investigate what's slowing the test down e.g. bisect test runtime from when the timeout was introduced or changed
#1 Updated by cdywan about 1 year ago
- Subject changed from t/25-cache-service.t fails (for me, on Leap 15.3) running into timeout consistently (longer timeouts work) to t/25-cache-service.t fails exceeding 90s timeout consistently size:M
- Description updated (diff)
- Status changed from New to Workable
#5 Updated by kraih about 1 year ago
I went back to January 2021 in 2 week intervals and i've been unable to measure any big jumps in test runtime. For me it is very consistently around 103 seconds on a very fast machine (with 2-3 seconds variance here and there).
All tests successful. Files=1, Tests=23, 103 wallclock secs ( 0.07 usr 0.01 sys + 15.86 cusr 1.72 csys = 17.66 CPU) Result: PASS
#8 Updated by okurz about 1 year ago
Honestly, i suspect that everyone on the team just runs tests with
OPENQA_TEST_TIMEOUT_DISABLElocally now, and that's why the values don't match reality anymore. :)
well, I don't so this is how I saw the issue ;) Actually I assume that most just don't run all the tests but rely on the CI.
Good that you tried to bisect. The question could be if there actually is any older state that works because if not that makes it more likely that other dependencies have introduced the change in runtime.