Project

General

Profile

Actions

action #111251

closed

coordination #109668: [saga][epic] Stable and updated non-qemu backends for SLE validation

coordination #109740: [epic] Stable os-autoinst unit tests with good coverage

Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M

Added by okurz over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2022-05-18
Due date:
% Done:

0%

Estimated time:

Description

Acceptance criteria

  • AC1: the path OpenQA/ is listed in codecov.yml under "fully_covered"

Suggestions


Files

coverage.tar (5.29 MB) coverage.tar Devel::Cover 2022-06-03 tinita, 2022-06-10 15:40
devel-cover-numbers-off.png (91.5 KB) devel-cover-numbers-off.png Screenshot of the report tinita, 2022-06-10 15:43
coverage-needle-downloader.png (76.5 KB) coverage-needle-downloader.png tinita, 2022-06-22 13:23

Related issues 1 (0 open1 closed)

Copied to openQA Project - action #111254: Cover code of os-autoinst path backend/ fully (statement coverage) size:MResolvedmkittler2022-05-18

Actions
Actions #1

Updated by okurz over 2 years ago

  • Copied to action #111254: Cover code of os-autoinst path backend/ fully (statement coverage) size:M added
Actions #2

Updated by mkittler over 2 years ago

  • Subject changed from Increase code coverage of os-autoinst path OpenQA/ to Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M
  • Status changed from New to Workable
Actions #3

Updated by tinita over 2 years ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita
Actions #4

Updated by tinita over 2 years ago

Looking at https://app.codecov.io/gh/os-autoinst/os-autoinst/blob/master/OpenQA/NamedIOSelect.pm it has 72% line coverage.
Devel::Cover says 80% statement coverage https://github.com/os-autoinst/os-autoinst/runs/6675762690?check_suite_focus=true

------------------------------------------ ------ ------
File                                         stmt  total
------------------------------------------ ------ ------
/dev/null                                    75.0   75.0  # ???
/opt/OpenQA/Benchmark/Stopwatch.pm          100.0  100.0
/opt/OpenQA/Commands.pm                      96.8   96.8
/opt/OpenQA/Exceptions.pm                   100.0  100.0
/opt/OpenQA/Isotovideo/CommandHandler.pm     92.8   92.8
/opt/OpenQA/Isotovideo/Interface.pm         100.0  100.0
/opt/OpenQA/Isotovideo/NeedleDownloader.pm   90.7   90.7
/opt/OpenQA/Isotovideo/Utils.pm              96.1   96.1
/opt/OpenQA/NamedIOSelect.pm                 80.0   80.0  # <---

Locally I get 100% though:

% tools/invoke-tests --coverage t/36-openqa-namedioselect.t
% cover -report html_minimal  cover_db
----------------------- ------ ------
File                      stmt  total
----------------------- ------ ------
OpenQA/NamedIOSelect.pm  100.0  100.0
Total                    100.0  100.0
----------------------- ------ ------

And just for double checking, I added debugging to the allegedly not covered functions and they are executed.

Actions #5

Updated by tinita over 2 years ago

And if I run all tests, I also get 80% locally. I suspect the old problem of having coverage for the same module under different paths, and at the end Devel::Cover decides only for one of them.

I tracked it down to these two tests, and the order in which they called, matters:

% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t 36-openqa-namedioselect.t
% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t

----------------------------- ------ ------
File                            stmt  total
----------------------------- ------ ------
OpenQA/Benchmark/Stopwatch.pm    6.9    6.9
OpenQA/Isotovideo/Utils.pm      15.4   15.4
autotest.pm                     16.6   16.6
bmwqemu.pm                      25.0   25.0
cv.pm                           89.2   89.2
imgsearch                       94.1   94.1
log.pm                          34.1   34.1
needle.pm                       19.0   19.0
ppmclibs/blib/lib/tinycv.pm     74.1   74.1
signalblocker.pm                31.0   31.0
Total                           30.5   30.5
----------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t 35-imgsearch.t
--------------------------------------------------------------- ------ ------
File                                                              stmt  total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm  100.0  100.0
OpenQA/Benchmark/Stopwatch.pm                                      6.9    6.9
OpenQA/Isotovideo/Utils.pm                                        15.4   15.4
autotest.pm                                                       16.6   16.6
bmwqemu.pm                                                        25.0   25.0
cv.pm                                                             89.2   89.2
imgsearch                                                         94.1   94.1
log.pm                                                            34.1   34.1
needle.pm                                                         19.0   19.0
ppmclibs/blib/lib/tinycv.pm                                       74.1   74.1
signalblocker.pm                                                  31.0   31.0
Total                                                             32.4   32.4
--------------------------------------------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t
--------------------------------------------------------------- ------ ------
File                                                              stmt  total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm  100.0  100.0
Total                                                            100.0  100.0
--------------------------------------------------------------- ------ ------
Actions #6

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2073 ci: Upload coverage HTML as an artifact (merged)

Actions #7

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2074 Fix code coverage and run tests from root dir

Updated by tinita over 2 years ago

I noticed something weird with the Devel::Cover report artifact, but this happened already before my last PR.
See the attached report in coverage.tar and the screenshot.
The numbers are a bit off and sub ... are missing completely from the whole report.
Line 245 is covered and line 246 not, that doesn't make sense.
In reality the code $self->status('finished'); is on line 267.

I would at least like to reproduce that locally.

Actions #10

Updated by tinita over 2 years ago

I could not reproduce it by running tools/invoke-tests --coverage; cover -report html_minimal cover_db, but with running the stuff from tools/container_run_ci in registry.opensuse.org/devel/openqa/containers15.3/os-autoinst_dev I can see it.

Now testing things in between to find out what makes the difference.

Actions #11

Updated by tinita over 2 years ago

We are now using html_minimal instead of html_basic:
https://github.com/os-autoinst/os-autoinst/pull/2088 - Fix HTML Devel::Cover report (merged)

I created an upstream issue:
https://github.com/pjcj/Devel--Cover/issues/300

Actions #12

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2090 - Cover OpenQA::Commands completely (merged)

Actions #13

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2092 - ci: Cover OpenQA::Isotovideo::CommandHandler

Actions #15

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2094 - ci: Cover OpenQA::Isotovideo::Utils

Actions #16

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2094 merged

Yesterday I ran t/21-needle-downloader.t and discovered that my local coverage of this code block is completely the opposite of what happens in github actions.

    try {
       $download_res = $self->ua->get($download_url)->result;
       if (!$download_res->is_success) {
           my $return_code = $download_res->code;
           bmwqemu::diag("failed to download $download_url, server returned $return_code");
           $download_res = undef;
        }
    }
    catch {
       bmwqemu::fctinfo("internal error occurred when downloading $download_url: $_");
   };

Attached a screenshot of the two coverage reports. The one on the right is my local one.
Need to find out what is going wrong there and why the test still passes.

Actions #17

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2097 - ci: Cover OpenQA::Isotovideo::NeedleDownloader

Actions #19

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2098 - ci: Cover OpenQA::Qemu::MutParams

Actions #20

Updated by tinita over 2 years ago

Actions #22

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2101 - Remove unused method gen_runfile (merged)

Actions #23

Updated by tinita over 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2103 - ci: Fully cover OpenQA::Qemu::Proc (merged)

https://github.com/os-autoinst/os-autoinst/pull/2105 - ci: Mark OpenQA directory as fully covered

Actions #24

Updated by tinita over 2 years ago

  • Status changed from In Progress to Feedback
Actions #25

Updated by tinita over 2 years ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF