QA » openQA Project

https://github.com/os-autoinst/os-autoinst/pull/2271 and https://github.com/os-autoinst/os-autoinst/pull/2272

https://github.com/os-autoinst/os-autoinst/pull/2271 reached 100% with the following removed:

          - bmwqemu.pm
          - log.pm
          - mmapi.pm
          - osutils.pm
          - signalblocker.pm
          - t
          - xt

the other was still at 98.94%, updating with https://github.com/os-autoinst/os-autoinst/pull/2272#issuecomment-1450861262 to only exclude t/

I think one reason could be the flaky test modules under t/:

t/data/tests/bar/baz.pm
t/data/tests/foo.pm

Sometimes the first is covered and the second not at all, dropping out of the coverage report completely; sometimes it's the other way round. Maybe this results in this 99.x percentage somehow, depending on codecov is calculating here.
I assume that if a file is not in coverage at all, it doesn't get considered for the complete coverage result.
e.g. we currently don't have t/data/tests/foo.pm covered in the current master, still it says 100% for that directory

waiting for https://github.com/os-autoinst/os-autoinst/pull/2277 first

https://github.com/os-autoinst/os-autoinst/pull/2278

See also https://github.com/os-autoinst/os-autoinst/pull/2284

I was able to reproduce locally:

• Devel::Cover reports less than 100% coverage for 14-isotovideo.t
• Still Devel::Cover reports every line as green
• The lines which make the differences are the "uncoverable statement" lines
• Devel::Cover::Report::Codecovbash reports those lines as uncovered

So I will check if something changed recently in Devel::Cover

Actually the uncoverable lines are reported as undef, which means it doesn't count as code to cover, while 0 means not covered.

I just found something else:
All current pull requests have no coverage report because
https://github.com/os-autoinst/os-autoinst/pull/2289/checks?check_run_id=12439118095

Missing Base Report
Unable to compare commits because the base of the pull request did not upload a coverage report.

and that's because the last master run failed:
https://github.com/os-autoinst/os-autoinst/actions/runs/4481994082/jobs/7879448387

[2023-03-21T17:37:55.074Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 500 - Unknown error, please try again later
95
[2023-03-21T17:37:55.075Z] ['verbose'] The error stack is: Error: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 500 - Unknown error, please try again later

So we have the codecov upload error on master, which should actually have been fixed by using the token.

I have restarted the test, codecov upload succeeded, and now I get a proper report for my test PR.

https://github.com/os-autoinst/os-autoinst/pull/2292
Should fix two issues:

• flaky test modules (e.g. see message Devel::Cover: merging data for t/data/tests/bar/module2.pm into t/data/tests/module1.pm in https://github.com/os-autoinst/os-autoinst/actions/runs/4583543527/jobs/8094455041)
• fully_covered

I tried to investigate more on the issue that having t/ in fully_covered results in less than 100%, while having t/*.t, t/data, t/fake results in 100%.
To investigate more one would probably have to remove test by test and observe what happens.
Running just one test, e.g. t/24-myjsonrpc.t and putting t/ in fully_covered works just fine.

Todo: try to reproduce the clash of the modules with the same content in a simple reproducer and file a bug (or talk to Paul personally in 3 weeks at the toolchain summit ;-)

https://github.com/os-autoinst/os-autoinst/pull/2292 merged

https://github.com/os-autoinst/os-autoinst/pull/2294 ci: Add slash to directories in codecov.yml (merged)

Apparently if one only specifies t then it counts all files
starting with t including testapi.pm and tools/update-deps etc.

It would be really good if there was a page on the codecov report where
it says which files it considered for the several groups.

tinita wrote:

https://github.com/os-autoinst/os-autoinst/pull/2294 ci: Add slash to directories in codecov.yml

Apparently if one only specifies t then it counts all files
starting with t including testapi.pm and tools/update-deps etc.

It would be really good if there was a page on the codecov report where
it says which files it considered for the several groups.

I wonder if that behavior changed on the side of codecov in the past month. But regardless great that you could find this!

I created the issue for the modules with the same content: https://github.com/pjcj/Devel--Cover/issues/322
I think it's probable that this might not be easy to fix and we have a very special, uncommon case anyway that was easy to fix on our side.

Now reproducing the case where the coverage for files with "# uncoverable statement" is less than 100% even when all lines are green in the report.

Ok, so I reduced the reproducer regarding uncoverable to a few lines and reported it here: https://github.com/pjcj/Devel--Cover/issues/323
Basically it looks like Devel::Cover counts lines as uncovered, if they are covered && marked as uncoverable.
To add to the confusion, the line is still marked as green in the HTML report, so there is no quick way to find out which is the actual line that is responsible for less than 100% coverage.

It could be that the behaviour was always like that, but we didn't notice, because it doesn't affect the codecov reports, we had to actively look into the test logs.

Just a PR to correct all uncoverable markers in t/*.t files for now: https://github.com/os-autoinst/os-autoinst/pull/2297

https://github.com/os-autoinst/os-autoinst/pull/2297 merged