QA » openQA Project

https://codecov.io/gh/os-autoinst/openQA/pull/3606/changes shows more unreliable coverage changes:

In t/lib/OpenQA/Test/Utils.pm the method sub setup_worker shows completely missing coverage now and sub unstable_worker partial coverage:

sub setup_worker {
    my ($worker, $host) = @_;
    $worker->settings->webui_hosts([]);
    $worker->settings->webui_host_specific_settings({});
    push(@{$worker->settings->webui_hosts}, $host);
    $worker->settings->webui_host_specific_settings->{$host} = {};
    $worker->log_setup_info;
…
sub unstable_worker {
…
        _setup_sub_process 'openqa-worker-unstable';
        my $worker = OpenQA::Worker->new(
        setup_worker($worker, $host);
        $worker->init();
        if ($ticks < 0) {
            Mojo::IOLoop->singleton->one_tick for (0 .. $ticks);
        Devel::Cover::report() if Devel::Cover->can('report');

so maybe the method was called but we could not successfully finish the last line which could explain the missing report.

I could reproduce this in my local. But I am not sure how to fix it. Could we simply use "uncoverable" to the flaky part, such as sub setup_worker and unstable_worker? or we should find out why sometimes these codes are not covered? I did not find out why this happened.

Commands used:

make coverage SCHEDULER_FULLSTACK=1 TESTS=t/05-scheduler-full.t | grep '^t/05-scheduler-full'
DIE_ON_FAIL=1 make test-with-database SCHEDULER_FULLSTACK=1 TESTS=t/05-scheduler-full.t

So I think the problem here is that the sigchild handler never completes. A simple note I added wasn't even showing. There's existing code checking if the worker's available via the Workers table but that's not useful for this purpose. The function at best makes it to the initial IOLoop invocation, before it gets to the explicit Devel::Cover::report call.
Having written down all of that I realize this is with $ticks=-1 which means the IOLoop blocks forever...🤦 Always forcing n ticks doesn't seem to help much, but moving up the report call at least gets coverage up to 55.4%.

https://github.com/os-autoinst/openQA/pull/3655 has some of my attempts at improving this, and although I'm not making a lot of headway here maybe someone else will have more ideas

https://github.com/os-autoinst/openQA/pull/3686

We discussed this in the weekly and I took some notes on how to tackle this better, since the PR review had raised some interesting questions.

• What does unstable_worker do?
• Can we use an event on the worker, like the current job?
• Where does it usually stop?
• Why do we want the infinite worker?
• An "unstable" worker can't process jobs?
• The original idea of the test was conceived to behave differently between runs - a counter example would be the scalability test which overlaps with this one.
• Maybe delete unstable worker tests, see what the coverage looks like afterwards and write new tests for what's missing with defined use cases for issues that we know to expect. We have more knowledge since the test was introduced.

That last part is what I'll look into, rather than trying to fixup a test don't really consider a good approach.

cdywan wrote:

• Maybe delete unstable worker tests, see what the coverage looks like afterwards and write new tests for what's missing with defined use cases for issues that we know to expect. We have more knowledge since the test was introduced.

https://github.com/os-autoinst/openQA/pull/3695

With #95662 resolved – thx tinita – codecov reports should be more consistent and it should be easier to fix the coverage.

okurz wrote:

With #95662 resolved – thx tinita – codecov reports should be more consistent and it should be easier to fix the coverage.

Excellent, that was really fast.