action #111251
closedcoordination #109668: [saga][epic] Stable and updated non-qemu backends for SLE validation
coordination #109740: [epic] Stable os-autoinst unit tests with good coverage
Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M
Description
Acceptance criteria¶
- AC1: the path OpenQA/ is listed in codecov.yml under "fully_covered"
Suggestions¶
- Take a look into https://app.codecov.io/gh/os-autoinst/os-autoinst/ about what is currently covered.
- Add unit test statement coverage for that to bump the target
Files
Updated by okurz over 2 years ago
- Copied to action #111254: Cover code of os-autoinst path backend/ fully (statement coverage) size:M added
Updated by mkittler over 2 years ago
- Subject changed from Increase code coverage of os-autoinst path OpenQA/ to Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M
- Status changed from New to Workable
Updated by tinita over 2 years ago
- Status changed from Workable to In Progress
- Assignee set to tinita
Updated by tinita over 2 years ago
Looking at https://app.codecov.io/gh/os-autoinst/os-autoinst/blob/master/OpenQA/NamedIOSelect.pm it has 72% line coverage.
Devel::Cover says 80% statement coverage https://github.com/os-autoinst/os-autoinst/runs/6675762690?check_suite_focus=true
------------------------------------------ ------ ------
File stmt total
------------------------------------------ ------ ------
/dev/null 75.0 75.0 # ???
/opt/OpenQA/Benchmark/Stopwatch.pm 100.0 100.0
/opt/OpenQA/Commands.pm 96.8 96.8
/opt/OpenQA/Exceptions.pm 100.0 100.0
/opt/OpenQA/Isotovideo/CommandHandler.pm 92.8 92.8
/opt/OpenQA/Isotovideo/Interface.pm 100.0 100.0
/opt/OpenQA/Isotovideo/NeedleDownloader.pm 90.7 90.7
/opt/OpenQA/Isotovideo/Utils.pm 96.1 96.1
/opt/OpenQA/NamedIOSelect.pm 80.0 80.0 # <---
Locally I get 100% though:
% tools/invoke-tests --coverage t/36-openqa-namedioselect.t
% cover -report html_minimal cover_db
----------------------- ------ ------
File stmt total
----------------------- ------ ------
OpenQA/NamedIOSelect.pm 100.0 100.0
Total 100.0 100.0
----------------------- ------ ------
And just for double checking, I added debugging to the allegedly not covered functions and they are executed.
Updated by tinita over 2 years ago
And if I run all tests, I also get 80% locally. I suspect the old problem of having coverage for the same module under different paths, and at the end Devel::Cover decides only for one of them.
I tracked it down to these two tests, and the order in which they called, matters:
% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t 36-openqa-namedioselect.t
% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t
----------------------------- ------ ------
File stmt total
----------------------------- ------ ------
OpenQA/Benchmark/Stopwatch.pm 6.9 6.9
OpenQA/Isotovideo/Utils.pm 15.4 15.4
autotest.pm 16.6 16.6
bmwqemu.pm 25.0 25.0
cv.pm 89.2 89.2
imgsearch 94.1 94.1
log.pm 34.1 34.1
needle.pm 19.0 19.0
ppmclibs/blib/lib/tinycv.pm 74.1 74.1
signalblocker.pm 31.0 31.0
Total 30.5 30.5
----------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t 35-imgsearch.t
--------------------------------------------------------------- ------ ------
File stmt total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm 100.0 100.0
OpenQA/Benchmark/Stopwatch.pm 6.9 6.9
OpenQA/Isotovideo/Utils.pm 15.4 15.4
autotest.pm 16.6 16.6
bmwqemu.pm 25.0 25.0
cv.pm 89.2 89.2
imgsearch 94.1 94.1
log.pm 34.1 34.1
needle.pm 19.0 19.0
ppmclibs/blib/lib/tinycv.pm 74.1 74.1
signalblocker.pm 31.0 31.0
Total 32.4 32.4
--------------------------------------------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t
--------------------------------------------------------------- ------ ------
File stmt total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm 100.0 100.0
Total 100.0 100.0
--------------------------------------------------------------- ------ ------
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2073 ci: Upload coverage HTML as an artifact (merged)
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2074 Fix code coverage and run tests from root dir
Updated by tinita over 2 years ago
Updated by tinita over 2 years ago
- File coverage.tar coverage.tar added
- File devel-cover-numbers-off.png devel-cover-numbers-off.png added
I noticed something weird with the Devel::Cover report artifact, but this happened already before my last PR.
See the attached report in coverage.tar and the screenshot.
The numbers are a bit off and sub ...
are missing completely from the whole report.
Line 245 is covered and line 246 not, that doesn't make sense.
In reality the code $self->status('finished');
is on line 267.
I would at least like to reproduce that locally.
Updated by tinita over 2 years ago
I could not reproduce it by running tools/invoke-tests --coverage; cover -report html_minimal cover_db
, but with running the stuff from tools/container_run_ci in registry.opensuse.org/devel/openqa/containers15.3/os-autoinst_dev I can see it.
Now testing things in between to find out what makes the difference.
Updated by tinita over 2 years ago
We are now using html_minimal instead of html_basic:
https://github.com/os-autoinst/os-autoinst/pull/2088 - Fix HTML Devel::Cover report (merged)
I created an upstream issue:
https://github.com/pjcj/Devel--Cover/issues/300
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2090 - Cover OpenQA::Commands completely (merged)
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2092 - ci: Cover OpenQA::Isotovideo::CommandHandler
Updated by tinita over 2 years ago
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2094 - ci: Cover OpenQA::Isotovideo::Utils
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2094 merged
Yesterday I ran t/21-needle-downloader.t
and discovered that my local coverage of this code block is completely the opposite of what happens in github actions.
try {
$download_res = $self->ua->get($download_url)->result;
if (!$download_res->is_success) {
my $return_code = $download_res->code;
bmwqemu::diag("failed to download $download_url, server returned $return_code");
$download_res = undef;
}
}
catch {
bmwqemu::fctinfo("internal error occurred when downloading $download_url: $_");
};
Attached a screenshot of the two coverage reports. The one on the right is my local one.
Need to find out what is going wrong there and why the test still passes.
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2097 - ci: Cover OpenQA::Isotovideo::NeedleDownloader
Updated by tinita over 2 years ago
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2098 - ci: Cover OpenQA::Qemu::MutParams
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2099 - ci: Cover CARP_TRACE methods
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2098 and https://github.com/os-autoinst/os-autoinst/pull/2099 merged
https://github.com/os-autoinst/os-autoinst/pull/2100 - Remove unused method gen_qemu_img_commit (merged)
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2101 - Remove unused method gen_runfile (merged)
Updated by tinita over 2 years ago
https://github.com/os-autoinst/os-autoinst/pull/2103 - ci: Fully cover OpenQA::Qemu::Proc (merged)
https://github.com/os-autoinst/os-autoinst/pull/2105 - ci: Mark OpenQA directory as fully covered
Updated by tinita over 2 years ago
- Status changed from Feedback to Resolved
https://github.com/os-autoinst/os-autoinst/pull/2105 merged, AC fulfilled