QA » openQA Project

I ran count-fail-ratio make coverage KEEP_DB=1 TESTS=t/ui/13-admin.t and found

## count-fail-ratio: Run: 20. Fails: 1. Fail ratio 5.00±9.55%
## mean runtime: 84.7±2.0 ms

with the one failure

t/ui/13-admin.t .. 14/?
        #   Failed test 'exactly one key present'
        #   at t/ui/13-admin.t line 681.
        #          got: '0'
        #     expected: '1'

        #   Failed test 'new key has expiration date'
        #   at t/ui/13-admin.t line 682.
        #                   ''
        #     doesn't match '(?^:\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})'
        # Looks like you failed 2 tests of 2.

    #   Failed test 'create key with expiration date'
    #   at t/ui/13-admin.t line 683.

        #   Failed test 'two keys present now'
        #   at t/ui/13-admin.t line 689.
        #          got: '1'
        #     expected: '2'
        # Looks like you failed 1 test of 3.

    #   Failed test 'create key without expiration date'
    #   at t/ui/13-admin.t line 691.
    # Looks like you failed 2 tests of 4.
t/ui/13-admin.t .. 15/?
#   Failed test 'Manage API keys'
#   at t/ui/13-admin.t line 692.
t/ui/13-admin.t .. 16/? # Looks like you failed 1 test of 16.
t/ui/13-admin.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/16 subtests

so not the same problem as the one in the pull request. Should try again with "make test"

runs=200 OPENQA_TEST_TIMEOUT_DISABLE=1 count-fail-ratio make test KEEP_DB=1 TESTS=t/ui/13-admin.t completed:

count-fail-ratio: Run: 200. Fails: 4. Fail ratio 2.00±1.94%, mean runtime: 39.0±3.0 s. Reproduced the error from #158422-1 four times but not the one from https://github.com/os-autoinst/openQA/pull/5557#issuecomment-2031627945

Run again with make coverage with runs=200 count-fail-ratio make coverage KEEP_DB=1 TESTS=t/ui/13-admin.t and found:

count-fail-ratio: Run: 200. Fails: 7. Fail ratio 3.50±2.54%, mean runtime: 80 s

so slightly higher fail ratio with double runtime. I guess except for cutting down t/ui/13-admin.t for quicker reproduction of the observed running 200 runs with make test is the best reproducer.

I could reproduce the error with a stripped down test file taking "only" 9s. I tried with … or die explain $tbody to show more details but that is apparently never called. Trying again with "diag explain". Also I tried without success wait_for_element(selector => '#api-keys-tbody > tr', description => 'key row present'); which ran into timeouts so actually waiting didn't help. Seems in this case the table is actually never updated to show an entry.