coordination #154777: [saga][epic] Shareable os-autoinst and test distribution plugins
coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributions
Added by okurz about 1 month ago. Updated about 1 month ago.
0%
Description
I ran count-fail-ratio make coverage KEEP_DB=1 TESTS=t/ui/13-admin.t and found
## count-fail-ratio: Run: 20. Fails: 1. Fail ratio 5.00±9.55%
## mean runtime: 84.7±2.0 ms
with the one failure
t/ui/13-admin.t .. 14/?
# Failed test 'exactly one key present'
# at t/ui/13-admin.t line 681.
# got: '0'
# expected: '1'
# Failed test 'new key has expiration date'
# at t/ui/13-admin.t line 682.
# ''
# doesn't match '(?^:\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})'
# Looks like you failed 2 tests of 2.
# Failed test 'create key with expiration date'
# at t/ui/13-admin.t line 683.
# Failed test 'two keys present now'
# at t/ui/13-admin.t line 689.
# got: '1'
# expected: '2'
# Looks like you failed 1 test of 3.
# Failed test 'create key without expiration date'
# at t/ui/13-admin.t line 691.
# Looks like you failed 2 tests of 4.
t/ui/13-admin.t .. 15/?
# Failed test 'Manage API keys'
# at t/ui/13-admin.t line 692.
t/ui/13-admin.t .. 16/? # Looks like you failed 1 test of 16.
t/ui/13-admin.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/16 subtests
so not the same problem as the one in the pull request. Should try again with "make test"
Setting due date based on mean cycle time of SUSE QE Tools
runs=200 OPENQA_TEST_TIMEOUT_DISABLE=1 count-fail-ratio make test KEEP_DB=1 TESTS=t/ui/13-admin.t completed:
count-fail-ratio: Run: 200. Fails: 4. Fail ratio 2.00±1.94%, mean runtime: 39.0±3.0 s. Reproduced the error from #158422-1 four times but not the one from https://github.com/os-autoinst/openQA/pull/5557#issuecomment-2031627945
Run again with make coverage with runs=200 count-fail-ratio make coverage KEEP_DB=1 TESTS=t/ui/13-admin.t and found:
count-fail-ratio: Run: 200. Fails: 7. Fail ratio 3.50±2.54%, mean runtime: 80 s
so slightly higher fail ratio with double runtime. I guess except for cutting down t/ui/13-admin.t for quicker reproduction of the observed running 200 runs with make test is the best reproducer.
I could reproduce the error with a stripped down test file taking "only" 9s. I tried with … or die explain $tbody to show more details but that is apparently never called. Trying again with "diag explain". Also I tried without success wait_for_element(selector => '#api-keys-tbody > tr', description => 'key row present'); which ran into timeouts so actually waiting didn't help. Seems in this case the table is actually never updated to show an entry.