openQA Project (public)

Just for reference: https://github.com/os-autoinst/openQA/pull/4037

I've been using OpenQA::Benchmark::Stopwatch to compare only the time of the test itself (and not db initialization and generation of the final coverage report) and even than there's quite a difference. Without coverage it takes ~ 2.990 s and with it takes ~ 12.418. It would make most sense to simply disable coverage tracking for this test in forked processes because we really shouldn't need it. (Nothing should only be covered by this test.)

Disabling the coverage report (Devel::Cover::report()) in the forked web UI and websocket server processes (the only forked processes which currently report coverage) doesn't make much of a difference. After that the time spend on the different sub tests looks like this:

coverage generally enabled but no Devel::Cover::report():
NAME                        TIME        CUMULATIVE      PERCENTAGE
 wait for workers to be idle 3.066       3.066           25.964%
 assign and run jobs         6.031       9.097           51.072%
 stop all workers            2.712       11.809          22.964%

coverage completely disabled:
NAME                        TIME        CUMULATIVE      PERCENTAGE
 wait for workers to be idle 2.125       2.125           70.712%
 assign and run jobs         0.853       2.978           28.378%
 stop all workers            0.027       3.006           0.910%

The functionality tested here is Perl-code extensive so maybe the coverage analysis slows down things even without reporting coverage data from forked processes.

So for now I would just apply OpenQA::Test::TimeLimit::scale_timeout here (PR: https://github.com/os-autoinst/openQA/pull/4070).

Created another PR: https://github.com/os-autoinst/openQA/pull/4072
See also https://github.com/os-autoinst/openQA/pull/4070#issuecomment-885688115 and previous comments for an explanation.

PR to test stability: https://github.com/os-autoinst/openQA/pull/4080

The test is now definitely more stable than before but still failed on the 163th run within CircleCI:

Rerun 163 of 200 …
[12:32:25] t/43-scheduling-and-worker-scalability.t .. 
[12:32:25] t/43-scheduling-and-worker-scalability.t .. 1/? 

    #   Failed test 'all jobs passed'
    #   at t/43-scheduling-and-worker-scalability.t line 220.
    #          got: '4'
    #     expected: '5'
    # All jobs:
    #  - id: 1, state: done, result: passed, reason: none
    #  - id: 2, state: done, result: incomplete, reason: setup failure: isotovideo can not be started
    #  - id: 3, state: done, result: passed, reason: none
    #  - id: 4, state: done, result: passed, reason: none
    #  - id: 5, state: done, result: passed, reason: none
    # Looks like you failed 1 test of 3.
[12:32:25] t/43-scheduling-and-worker-scalability.t .. 2/? 
#   Failed test 'assign and run jobs'
#   at t/43-scheduling-and-worker-scalability.t line 222.
# Looks like you failed 1 test of 4.

I haven't seen this problem before. Note that isotovideo is just a dummy shell script within this test. Maybe we're hitting some limit here (which we wouldn't if not running the test for the 163th time)?

I have already prepared a change for considering the test no longer unstable: https://github.com/Martchus/openQA/pull/new/unstable-43-scheduling-and-worker-scalability

After a re-run it fails even on an earlier attempt. This time it looks more like the timeout is nevertheless too low:

Rerun 46 of 200 …
[14:37:36] t/43-scheduling-and-worker-scalability.t .. 1/? Mojo::Reactor::Poll: I/O watcher failed: Not an ARRAY reference at /home/squamata/project/script/../lib/OpenQA/Worker/Job.pm line 1129.

    #   Failed test 'all jobs done'
    #   at t/43-scheduling-and-worker-scalability.t line 219.
    #          got: '4'
    #     expected: '5'

    #   Failed test 'all jobs passed'
    #   at t/43-scheduling-and-worker-scalability.t line 220.
    #          got: '4'
    #     expected: '5'
    # All jobs:
    #  - id: 1, state: done, result: passed, reason: none
    #  - id: 2, state: done, result: passed, reason: none
    #  - id: 3, state: uploading, result: none, reason: none
    #  - id: 4, state: done, result: passed, reason: none
    #  - id: 5, state: done, result: passed, reason: none
    # Looks like you failed 2 tests of 3.
[14:37:36] t/43-scheduling-and-worker-scalability.t .. 2/? 
#   Failed test 'assign and run jobs'
#   at t/43-scheduling-and-worker-scalability.t line 222.

    #   Failed test 'all workers offline'
    #   at t/43-scheduling-and-worker-scalability.t line 236.
    # 0
    # Looks like you failed 1 test of 1.
[14:37:36] t/43-scheduling-and-worker-scalability.t .. 3/? 
#   Failed test 'stop all workers'
#   at t/43-scheduling-and-worker-scalability.t line 237.
# Looks like you failed 2 tests of 4.