Project

General

Profile

action #111251

coordination #109668: [saga][epic] Stable and updated non-qemu backends for SLE validation

coordination #109740: [epic] Stable os-autoinst unit tests with good coverage

Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M

Added by okurz about 1 month ago. Updated 3 days ago.

Status:
In Progress
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2022-05-18
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Acceptance criteria

  • AC1: the path OpenQA/ is listed in codecov.yml under "fully_covered"

Suggestions

coverage.tar (5.29 MB) coverage.tar Devel::Cover 2022-06-03 tinita, 2022-06-10 15:40
devel-cover-numbers-off.png (91.5 KB) devel-cover-numbers-off.png Screenshot of the report tinita, 2022-06-10 15:43
coverage-needle-downloader.png (76.5 KB) coverage-needle-downloader.png tinita, 2022-06-22 13:23
13376
13451

Related issues

Copied to openQA Project - action #111254: Cover code of os-autoinst path backend/ fully (statement coverage) size:MWorkable2022-05-18

History

#1 Updated by okurz about 1 month ago

  • Copied to action #111254: Cover code of os-autoinst path backend/ fully (statement coverage) size:M added

#2 Updated by mkittler about 1 month ago

  • Subject changed from Increase code coverage of os-autoinst path OpenQA/ to Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M
  • Status changed from New to Workable

#3 Updated by tinita 25 days ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita

#4 Updated by tinita 25 days ago

Looking at https://app.codecov.io/gh/os-autoinst/os-autoinst/blob/master/OpenQA/NamedIOSelect.pm it has 72% line coverage.
Devel::Cover says 80% statement coverage https://github.com/os-autoinst/os-autoinst/runs/6675762690?check_suite_focus=true

------------------------------------------ ------ ------
File                                         stmt  total
------------------------------------------ ------ ------
/dev/null                                    75.0   75.0  # ???
/opt/OpenQA/Benchmark/Stopwatch.pm          100.0  100.0
/opt/OpenQA/Commands.pm                      96.8   96.8
/opt/OpenQA/Exceptions.pm                   100.0  100.0
/opt/OpenQA/Isotovideo/CommandHandler.pm     92.8   92.8
/opt/OpenQA/Isotovideo/Interface.pm         100.0  100.0
/opt/OpenQA/Isotovideo/NeedleDownloader.pm   90.7   90.7
/opt/OpenQA/Isotovideo/Utils.pm              96.1   96.1
/opt/OpenQA/NamedIOSelect.pm                 80.0   80.0  # <---

Locally I get 100% though:

% tools/invoke-tests --coverage t/36-openqa-namedioselect.t
% cover -report html_minimal  cover_db
----------------------- ------ ------
File                      stmt  total
----------------------- ------ ------
OpenQA/NamedIOSelect.pm  100.0  100.0
Total                    100.0  100.0
----------------------- ------ ------

And just for double checking, I added debugging to the allegedly not covered functions and they are executed.

#5 Updated by tinita 25 days ago

And if I run all tests, I also get 80% locally. I suspect the old problem of having coverage for the same module under different paths, and at the end Devel::Cover decides only for one of them.

I tracked it down to these two tests, and the order in which they called, matters:

% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t 36-openqa-namedioselect.t
% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t

----------------------------- ------ ------
File                            stmt  total
----------------------------- ------ ------
OpenQA/Benchmark/Stopwatch.pm    6.9    6.9
OpenQA/Isotovideo/Utils.pm      15.4   15.4
autotest.pm                     16.6   16.6
bmwqemu.pm                      25.0   25.0
cv.pm                           89.2   89.2
imgsearch                       94.1   94.1
log.pm                          34.1   34.1
needle.pm                       19.0   19.0
ppmclibs/blib/lib/tinycv.pm     74.1   74.1
signalblocker.pm                31.0   31.0
Total                           30.5   30.5
----------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t 35-imgsearch.t
--------------------------------------------------------------- ------ ------
File                                                              stmt  total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm  100.0  100.0
OpenQA/Benchmark/Stopwatch.pm                                      6.9    6.9
OpenQA/Isotovideo/Utils.pm                                        15.4   15.4
autotest.pm                                                       16.6   16.6
bmwqemu.pm                                                        25.0   25.0
cv.pm                                                             89.2   89.2
imgsearch                                                         94.1   94.1
log.pm                                                            34.1   34.1
needle.pm                                                         19.0   19.0
ppmclibs/blib/lib/tinycv.pm                                       74.1   74.1
signalblocker.pm                                                  31.0   31.0
Total                                                             32.4   32.4
--------------------------------------------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t
--------------------------------------------------------------- ------ ------
File                                                              stmt  total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm  100.0  100.0
Total                                                            100.0  100.0
--------------------------------------------------------------- ------ ------

#6 Updated by tinita 24 days ago

https://github.com/os-autoinst/os-autoinst/pull/2073 ci: Upload coverage HTML as an artifact (merged)

#7 Updated by tinita 24 days ago

https://github.com/os-autoinst/os-autoinst/pull/2074 Fix code coverage and run tests from root dir

#9 Updated by tinita 16 days ago

13376

I noticed something weird with the Devel::Cover report artifact, but this happened already before my last PR.
See the attached report in coverage.tar and the screenshot.
The numbers are a bit off and sub ... are missing completely from the whole report.
Line 245 is covered and line 246 not, that doesn't make sense.
In reality the code $self->status('finished'); is on line 267.

I would at least like to reproduce that locally.

#10 Updated by tinita 11 days ago

I could not reproduce it by running tools/invoke-tests --coverage; cover -report html_minimal cover_db, but with running the stuff from tools/container_run_ci in registry.opensuse.org/devel/openqa/containers15.3/os-autoinst_dev I can see it.

Now testing things in between to find out what makes the difference.

#11 Updated by tinita 11 days ago

We are now using html_minimal instead of html_basic:
https://github.com/os-autoinst/os-autoinst/pull/2088 - Fix HTML Devel::Cover report (merged)

I created an upstream issue:
https://github.com/pjcj/Devel--Cover/issues/300

#12 Updated by tinita 10 days ago

https://github.com/os-autoinst/os-autoinst/pull/2090 - Cover OpenQA::Commands completely (merged)

#13 Updated by tinita 9 days ago

https://github.com/os-autoinst/os-autoinst/pull/2092 - ci: Cover OpenQA::Isotovideo::CommandHandler

#15 Updated by tinita 9 days ago

https://github.com/os-autoinst/os-autoinst/pull/2094 - ci: Cover OpenQA::Isotovideo::Utils

#16 Updated by tinita 4 days ago

13451

https://github.com/os-autoinst/os-autoinst/pull/2094 merged

Yesterday I ran t/21-needle-downloader.t and discovered that my local coverage of this code block is completely the opposite of what happens in github actions.

    try {
       $download_res = $self->ua->get($download_url)->result;
       if (!$download_res->is_success) {
           my $return_code = $download_res->code;
           bmwqemu::diag("failed to download $download_url, server returned $return_code");
           $download_res = undef;
        }
    }
    catch {
       bmwqemu::fctinfo("internal error occurred when downloading $download_url: $_");
   };

Attached a screenshot of the two coverage reports. The one on the right is my local one.
Need to find out what is going wrong there and why the test still passes.

#17 Updated by tinita 4 days ago

https://github.com/os-autoinst/os-autoinst/pull/2097 - ci: Cover OpenQA::Isotovideo::NeedleDownloader

#19 Updated by tinita 4 days ago

https://github.com/os-autoinst/os-autoinst/pull/2098 - ci: Cover OpenQA::Qemu::MutParams

#20 Updated by tinita 4 days ago

#22 Updated by tinita 3 days ago

https://github.com/os-autoinst/os-autoinst/pull/2101 - Remove unused method gen_runfile (merged)

Also available in: Atom PDF