action #158422
closedcoordination #154777: [saga][epic] Shareable os-autoinst and test distribution plugins
coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributions
flaky sporadic test failures t/ui/13-admin.t
0%
Description
Updated by okurz 7 months ago
I ran count-fail-ratio make coverage KEEP_DB=1 TESTS=t/ui/13-admin.t
and found
## count-fail-ratio: Run: 20. Fails: 1. Fail ratio 5.00±9.55%
## mean runtime: 84.7±2.0 ms
with the one failure
t/ui/13-admin.t .. 14/?
# Failed test 'exactly one key present'
# at t/ui/13-admin.t line 681.
# got: '0'
# expected: '1'
# Failed test 'new key has expiration date'
# at t/ui/13-admin.t line 682.
# ''
# doesn't match '(?^:\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})'
# Looks like you failed 2 tests of 2.
# Failed test 'create key with expiration date'
# at t/ui/13-admin.t line 683.
# Failed test 'two keys present now'
# at t/ui/13-admin.t line 689.
# got: '1'
# expected: '2'
# Looks like you failed 1 test of 3.
# Failed test 'create key without expiration date'
# at t/ui/13-admin.t line 691.
# Looks like you failed 2 tests of 4.
t/ui/13-admin.t .. 15/?
# Failed test 'Manage API keys'
# at t/ui/13-admin.t line 692.
t/ui/13-admin.t .. 16/? # Looks like you failed 1 test of 16.
t/ui/13-admin.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/16 subtests
so not the same problem as the one in the pull request. Should try again with "make test"
Updated by openqa_review 7 months ago
- Due date set to 2024-04-17
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz 7 months ago
runs=200 OPENQA_TEST_TIMEOUT_DISABLE=1 count-fail-ratio make test KEEP_DB=1 TESTS=t/ui/13-admin.t
completed:
count-fail-ratio: Run: 200. Fails: 4. Fail ratio 2.00±1.94%, mean runtime: 39.0±3.0 s. Reproduced the error from #158422-1 four times but not the one from https://github.com/os-autoinst/openQA/pull/5557#issuecomment-2031627945
Run again with make coverage
with runs=200 count-fail-ratio make coverage KEEP_DB=1 TESTS=t/ui/13-admin.t
and found:
count-fail-ratio: Run: 200. Fails: 7. Fail ratio 3.50±2.54%, mean runtime: 80 s
so slightly higher fail ratio with double runtime. I guess except for cutting down t/ui/13-admin.t for quicker reproduction of the observed running 200 runs with make test
is the best reproducer.
Updated by okurz 7 months ago
I could reproduce the error with a stripped down test file taking "only" 9s. I tried with … or die explain $tbody
to show more details but that is apparently never called. Trying again with "diag explain". Also I tried without success wait_for_element(selector => '#api-keys-tbody > tr', description => 'key row present');
which ran into timeouts so actually waiting didn't help. Seems in this case the table is actually never updated to show an entry.