action #111251: Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M - openQA Project - openSUSE Project Management Tool

Actions

Copy link

action #111251

closed

coordination #109668: [saga][epic] Stable and updated non-qemu backends for SLE validation

coordination #109740: [epic] Stable os-autoinst unit tests with good coverage

Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M

Added by okurz almost 2 years ago. Updated almost 2 years ago.

Status:

Resolved

Priority:

Low

Assignee:

tinita

Category:

Feature requests

Target version:

Ready

Start date:

2022-05-18

Due date:

% Done:

Estimated time:

Description

Acceptance criteria¶

AC1: the path OpenQA/ is listed in codecov.yml under "fully_covered"

Suggestions¶

Take a look into https://app.codecov.io/gh/os-autoinst/os-autoinst/ about what is currently covered.
Add unit test statement coverage for that to bump the target

Files

Download all files

coverage.tar (5.29 MB) coverage.tar	Devel::Cover 2022-06-03	tinita, 2022-06-10 15:40
devel-cover-numbers-off.png (91.5 KB) devel-cover-numbers-off.png	Screenshot of the report	tinita, 2022-06-10 15:43
coverage-needle-downloader.png (76.5 KB) coverage-needle-downloader.png		tinita, 2022-06-22 13:23

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by okurz almost 2 years ago

Copied to action #111254: Cover code of os-autoinst path backend/ fully (statement coverage) size:M added

Actions

Copy link

Updated by mkittler almost 2 years ago

Subject changed from Increase code coverage of os-autoinst path OpenQA/ to Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M
Status changed from New to Workable

Actions

Copy link

Updated by tinita almost 2 years ago

Status changed from Workable to In Progress
Assignee set to tinita

Actions

Copy link

Updated by tinita almost 2 years ago

Looking at https://app.codecov.io/gh/os-autoinst/os-autoinst/blob/master/OpenQA/NamedIOSelect.pm it has 72% line coverage.
Devel::Cover says 80% statement coverage https://github.com/os-autoinst/os-autoinst/runs/6675762690?check_suite_focus=true

------------------------------------------ ------ ------
File                                         stmt  total
------------------------------------------ ------ ------
/dev/null                                    75.0   75.0  # ???
/opt/OpenQA/Benchmark/Stopwatch.pm          100.0  100.0
/opt/OpenQA/Commands.pm                      96.8   96.8
/opt/OpenQA/Exceptions.pm                   100.0  100.0
/opt/OpenQA/Isotovideo/CommandHandler.pm     92.8   92.8
/opt/OpenQA/Isotovideo/Interface.pm         100.0  100.0
/opt/OpenQA/Isotovideo/NeedleDownloader.pm   90.7   90.7
/opt/OpenQA/Isotovideo/Utils.pm              96.1   96.1
/opt/OpenQA/NamedIOSelect.pm                 80.0   80.0  # <---

Locally I get 100% though:

% tools/invoke-tests --coverage t/36-openqa-namedioselect.t
% cover -report html_minimal  cover_db
----------------------- ------ ------
File                      stmt  total
----------------------- ------ ------
OpenQA/NamedIOSelect.pm  100.0  100.0
Total                    100.0  100.0
----------------------- ------ ------

And just for double checking, I added debugging to the allegedly not covered functions and they are executed.

Actions

Copy link

Updated by tinita almost 2 years ago

And if I run all tests, I also get 80% locally. I suspect the old problem of having coverage for the same module under different paths, and at the end Devel::Cover decides only for one of them.

I tracked it down to these two tests, and the order in which they called, matters:

% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t 36-openqa-namedioselect.t
% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t

----------------------------- ------ ------
File                            stmt  total
----------------------------- ------ ------
OpenQA/Benchmark/Stopwatch.pm    6.9    6.9
OpenQA/Isotovideo/Utils.pm      15.4   15.4
autotest.pm                     16.6   16.6
bmwqemu.pm                      25.0   25.0
cv.pm                           89.2   89.2
imgsearch                       94.1   94.1
log.pm                          34.1   34.1
needle.pm                       19.0   19.0
ppmclibs/blib/lib/tinycv.pm     74.1   74.1
signalblocker.pm                31.0   31.0
Total                           30.5   30.5
----------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t 35-imgsearch.t
--------------------------------------------------------------- ------ ------
File                                                              stmt  total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm  100.0  100.0
OpenQA/Benchmark/Stopwatch.pm                                      6.9    6.9
OpenQA/Isotovideo/Utils.pm                                        15.4   15.4
autotest.pm                                                       16.6   16.6
bmwqemu.pm                                                        25.0   25.0
cv.pm                                                             89.2   89.2
imgsearch                                                         94.1   94.1
log.pm                                                            34.1   34.1
needle.pm                                                         19.0   19.0
ppmclibs/blib/lib/tinycv.pm                                       74.1   74.1
signalblocker.pm                                                  31.0   31.0
Total                                                             32.4   32.4
--------------------------------------------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t
--------------------------------------------------------------- ------ ------
File                                                              stmt  total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm  100.0  100.0
Total                                                            100.0  100.0
--------------------------------------------------------------- ------ ------

Actions

Copy link

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2073 ci: Upload coverage HTML as an artifact (merged)

Actions

Copy link

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2074 Fix code coverage and run tests from root dir

Actions

Copy link

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2074 merged.

Actions

Copy link Download all files

Updated by tinita almost 2 years ago

File coverage.tar coverage.tar added
File devel-cover-numbers-off.png devel-cover-numbers-off.png added

I noticed something weird with the Devel::Cover report artifact, but this happened already before my last PR.
See the attached report in coverage.tar and the screenshot.
The numbers are a bit off and sub ... are missing completely from the whole report.
Line 245 is covered and line 246 not, that doesn't make sense.
In reality the code $self->status('finished'); is on line 267.

I would at least like to reproduce that locally.

Actions

Copy link

#10

Updated by tinita almost 2 years ago

I could not reproduce it by running tools/invoke-tests --coverage; cover -report html_minimal cover_db, but with running the stuff from tools/container_run_ci in registry.opensuse.org/devel/openqa/containers15.3/os-autoinst_dev I can see it.

Now testing things in between to find out what makes the difference.

Actions

Copy link

#11

Updated by tinita almost 2 years ago

We are now using html_minimal instead of html_basic:
https://github.com/os-autoinst/os-autoinst/pull/2088 - Fix HTML Devel::Cover report (merged)

I created an upstream issue:
https://github.com/pjcj/Devel--Cover/issues/300

Actions

Copy link

#12

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2090 - Cover OpenQA::Commands completely (merged)

Actions

Copy link

#13

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2092 - ci: Cover OpenQA::Isotovideo::CommandHandler

Actions

Copy link

#14

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2092 merged

Actions

Copy link

#15

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2094 - ci: Cover OpenQA::Isotovideo::Utils

Actions

Copy link

#16

Updated by tinita almost 2 years ago

File coverage-needle-downloader.png coverage-needle-downloader.png added

https://github.com/os-autoinst/os-autoinst/pull/2094 merged

Yesterday I ran t/21-needle-downloader.t and discovered that my local coverage of this code block is completely the opposite of what happens in github actions.

    try {
       $download_res = $self->ua->get($download_url)->result;
       if (!$download_res->is_success) {
           my $return_code = $download_res->code;
           bmwqemu::diag("failed to download $download_url, server returned $return_code");
           $download_res = undef;
        }
    }
    catch {
       bmwqemu::fctinfo("internal error occurred when downloading $download_url: $_");
   };

Attached a screenshot of the two coverage reports. The one on the right is my local one.
Need to find out what is going wrong there and why the test still passes.

Actions

Copy link

#17