openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-03-01T20:06:09ZopenSUSE Project Management Tool
Redmine openQA Project - action #125237 (Resolved): os-autoinst codecov check "fully_covered" returns 99%...https://progress.opensuse.org/issues/1252372023-03-01T20:06:09Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>So it looks like the problem first mentioned in <a href="https://github.com/os-autoinst/os-autoinst/pull/2260#issuecomment-1448414226" class="external">https://github.com/os-autoinst/os-autoinst/pull/2260#issuecomment-1448414226</a> shows up in other pull requests as well now. Who can understand why codecov says that we have 99.28% coverage in the "fully_covered" section as configured in <a href="https://github.com/os-autoinst/os-autoinst/blob/master/codecov.yml#L18" class="external">https://github.com/os-autoinst/os-autoinst/blob/master/codecov.yml#L18</a> even though <a href="https://app.codecov.io/gh/os-autoinst/os-autoinst/pull/2270/tree" class="external">https://app.codecov.io/gh/os-autoinst/os-autoinst/pull/2270/tree</a> shows all referenced paths to be covered by 100%?</p>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<p>Seems to happen in all pull requests that are either opened anew or updated</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> It is known why the actual percentage doesn't match the expected 100%</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Bisect like it is done in <a href="https://github.com/os-autoinst/os-autoinst/pull/2271" class="external">https://github.com/os-autoinst/os-autoinst/pull/2271</a> and similar PRs</li>
<li>Click on "View details" in merged PRs to see the checks</li>
</ul>
openQA Project - action #89899 (Resolved): Fix flaky coverage - t/ui/27-plugin_obs_rsync_status_d...https://progress.opensuse.org/issues/898992021-03-11T07:00:28Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>See <a class="issue tracker-6 status-3 priority-4 priority-default closed child parent" title="coordination: [epic] Let's make codecov reports reliable (Resolved)" href="https://progress.opensuse.org/issues/55364">#55364</a> : codecov reports often report about coverage changes which are obviously not related to the actual changes of a PR, e.g. when documentation is changed. We can already trust our coverage analysis more but should have only coverage changes reported for actual changes we introduced in a pull request.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> t/ui/27-plugin_obs_rsync_status_details.t does not appear anymore as changing code coverage in unrelated changes</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Try to reproduce locally with <code>rm -rf cover_db/ && make coverage KEEP_DB=1 TESTS=t/ui/27-plugin_obs_rsync_status_details.t</code></li>
<li>check coverage details in generated html report, e.g. call <code>firefox cover_db/coverage.html</code></li>
<li>Fix uncovered lines with "uncoverable" statements, see previous commits adding these comments or look into <a href="https://metacpan.org/pod/Devel::Cover#UNCOVERABLE-CRITERIA" class="external">https://metacpan.org/pod/Devel::Cover#UNCOVERABLE-CRITERIA</a> or other means</li>
<li>retry multiple times to check for flakyness</li>
</ul>