action #111251: Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M - openQA Project - openSUSE Project Management Tool

Custom queries

All 'new' issues w/o assignee, sorted by version/priority
All auto_review tickets
All auto_review+force_result tickets
openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (ready issues)
QE tools team - backlog (w/o infra)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE Tools team - due soon
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (w/o infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #111251

closed

coordination #109668: [saga][epic] Stable and updated non-qemu backends for SLE validation

coordination #109740: [epic] Stable os-autoinst unit tests with good coverage

Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M

Added by okurz almost 2 years ago. Updated almost 2 years ago.

Status:

Resolved

Priority:

Low

Assignee:

tinita

Category:

Feature requests

Target version:

Ready

Start date:

2022-05-18

Due date:

% Done:

Estimated time:

Description

Acceptance criteria¶

AC1: the path OpenQA/ is listed in codecov.yml under "fully_covered"

Suggestions¶

Take a look into https://app.codecov.io/gh/os-autoinst/os-autoinst/ about what is currently covered.
Add unit test statement coverage for that to bump the target

Files

Download all files

coverage.tar (5.29 MB) coverage.tar	Devel::Cover 2022-06-03	tinita, 2022-06-10 15:40
devel-cover-numbers-off.png (91.5 KB) devel-cover-numbers-off.png	Screenshot of the report	tinita, 2022-06-10 15:43
coverage-needle-downloader.png (76.5 KB) coverage-needle-downloader.png		tinita, 2022-06-22 13:23

Related issues 1 (0 open — 1 closed)

Copied to openQA Project - action #111254: Cover code of os-autoinst path backend/ fully (statement coverage) size:M

Resolved

mkittler

2022-05-18

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by okurz almost 2 years ago

Copied to action #111254: Cover code of os-autoinst path backend/ fully (statement coverage) size:M added

Actions

Copy link

Updated by mkittler almost 2 years ago

Subject changed from Increase code coverage of os-autoinst path OpenQA/ to Cover code of os-autoinst path OpenQA/ fully (statement coverage) size:M
Status changed from New to Workable

Actions

Copy link

Updated by tinita almost 2 years ago

Status changed from Workable to In Progress
Assignee set to tinita

Actions

Copy link

Updated by tinita almost 2 years ago

Looking at https://app.codecov.io/gh/os-autoinst/os-autoinst/blob/master/OpenQA/NamedIOSelect.pm it has 72% line coverage.
Devel::Cover says 80% statement coverage https://github.com/os-autoinst/os-autoinst/runs/6675762690?check_suite_focus=true

------------------------------------------ ------ ------
File                                         stmt  total
------------------------------------------ ------ ------
/dev/null                                    75.0   75.0  # ???
/opt/OpenQA/Benchmark/Stopwatch.pm          100.0  100.0
/opt/OpenQA/Commands.pm                      96.8   96.8
/opt/OpenQA/Exceptions.pm                   100.0  100.0
/opt/OpenQA/Isotovideo/CommandHandler.pm     92.8   92.8
/opt/OpenQA/Isotovideo/Interface.pm         100.0  100.0
/opt/OpenQA/Isotovideo/NeedleDownloader.pm   90.7   90.7
/opt/OpenQA/Isotovideo/Utils.pm              96.1   96.1
/opt/OpenQA/NamedIOSelect.pm                 80.0   80.0  # <---

Locally I get 100% though:

% tools/invoke-tests --coverage t/36-openqa-namedioselect.t
% cover -report html_minimal  cover_db
----------------------- ------ ------
File                      stmt  total
----------------------- ------ ------
OpenQA/NamedIOSelect.pm  100.0  100.0
Total                    100.0  100.0
----------------------- ------ ------

And just for double checking, I added debugging to the allegedly not covered functions and they are executed.

Actions

Copy link

Updated by tinita almost 2 years ago

And if I run all tests, I also get 80% locally. I suspect the old problem of having coverage for the same module under different paths, and at the end Devel::Cover decides only for one of them.

I tracked it down to these two tests, and the order in which they called, matters:

% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t 36-openqa-namedioselect.t
% cd t && "$prove_path" $PROVE_ARGS -r 35-imgsearch.t

----------------------------- ------ ------
File                            stmt  total
----------------------------- ------ ------
OpenQA/Benchmark/Stopwatch.pm    6.9    6.9
OpenQA/Isotovideo/Utils.pm      15.4   15.4
autotest.pm                     16.6   16.6
bmwqemu.pm                      25.0   25.0
cv.pm                           89.2   89.2
imgsearch                       94.1   94.1
log.pm                          34.1   34.1
needle.pm                       19.0   19.0
ppmclibs/blib/lib/tinycv.pm     74.1   74.1
signalblocker.pm                31.0   31.0
Total                           30.5   30.5
----------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t 35-imgsearch.t
--------------------------------------------------------------- ------ ------
File                                                              stmt  total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm  100.0  100.0
OpenQA/Benchmark/Stopwatch.pm                                      6.9    6.9
OpenQA/Isotovideo/Utils.pm                                        15.4   15.4
autotest.pm                                                       16.6   16.6
bmwqemu.pm                                                        25.0   25.0
cv.pm                                                             89.2   89.2
imgsearch                                                         94.1   94.1
log.pm                                                            34.1   34.1
needle.pm                                                         19.0   19.0
ppmclibs/blib/lib/tinycv.pm                                       74.1   74.1
signalblocker.pm                                                  31.0   31.0
Total                                                             32.4   32.4
--------------------------------------------------------------- ------ ------
% cd t && "$prove_path" $PROVE_ARGS -r 36-openqa-namedioselect.t
--------------------------------------------------------------- ------ ------
File                                                              stmt  total
--------------------------------------------------------------- ------ ------
.../tina/openqa-devel/repos/os-autoinst/OpenQA/NamedIOSelect.pm  100.0  100.0
Total                                                            100.0  100.0
--------------------------------------------------------------- ------ ------

Actions

Copy link

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2073 ci: Upload coverage HTML as an artifact (merged)

Actions

Copy link

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2074 Fix code coverage and run tests from root dir

Actions

Copy link

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2074 merged.

Actions

Copy link Download all files

Updated by tinita almost 2 years ago

File coverage.tar coverage.tar added
File devel-cover-numbers-off.png devel-cover-numbers-off.png added

I noticed something weird with the Devel::Cover report artifact, but this happened already before my last PR.
See the attached report in coverage.tar and the screenshot.
The numbers are a bit off and sub ... are missing completely from the whole report.
Line 245 is covered and line 246 not, that doesn't make sense.
In reality the code $self->status('finished'); is on line 267.

I would at least like to reproduce that locally.

Actions

Copy link

#10

Updated by tinita almost 2 years ago

I could not reproduce it by running tools/invoke-tests --coverage; cover -report html_minimal cover_db, but with running the stuff from tools/container_run_ci in registry.opensuse.org/devel/openqa/containers15.3/os-autoinst_dev I can see it.

Now testing things in between to find out what makes the difference.

Actions

Copy link

#11

Updated by tinita almost 2 years ago

We are now using html_minimal instead of html_basic:
https://github.com/os-autoinst/os-autoinst/pull/2088 - Fix HTML Devel::Cover report (merged)

I created an upstream issue:
https://github.com/pjcj/Devel--Cover/issues/300

Actions

Copy link

#12

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2090 - Cover OpenQA::Commands completely (merged)

Actions

Copy link

#13

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2092 - ci: Cover OpenQA::Isotovideo::CommandHandler

Actions

Copy link

#14

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2092 merged

Actions

Copy link

#15

Updated by tinita almost 2 years ago

https://github.com/os-autoinst/os-autoinst/pull/2094 - ci: Cover OpenQA::Isotovideo::Utils

Actions

Copy link

#16

Updated by tinita almost 2 years ago

File coverage-needle-downloader.png coverage-needle-downloader.png added

https://github.com/os-autoinst/os-autoinst/pull/2094 merged

Yesterday I ran t/21-needle-downloader.t and discovered that my local coverage of this code block is completely the opposite of what happens in github actions.

    try {
       $download_res = $self->ua->get($download_url)->result;
       if (!$download_res->is_success) {
           my $return_code = $download_res->code;
           bmwqemu::diag("failed to download $download_url, server returned $return_code");
           $download_res = undef;
        }
    }
    catch {
       bmwqemu::fctinfo("internal error occurred when downloading $download_url: $_");
   };

Attached a screenshot of the two coverage reports. The one on the right is my local one.
Need to find out what is going wrong there and why the test still passes.

Actions

Copy link

#17