action #99594
closedFix flaky coverage - t/lib/OpenQA/Test/FullstackUtils.pm size:M
Description
Observation¶
Flaky coverage in https://github.com/os-autoinst/openQA/pull/4256#issuecomment-932021197. No logs available.
Acceptance criteria¶
- AC1: t/lib/OpenQA/Test/FullstackUtils.pm is not showing up in codecov reports for multiple PRs with unrelated changes
Suggestions¶
- Find an occurence and preserve the results
- Try
full-stack.t
orscheduler-full.t
Updated by okurz about 3 years ago
- Copied to action #99597: Fix flaky coverage - lib/OpenQA/Worker/WebUIConnection.pm size:M added
Updated by livdywan about 3 years ago
- Subject changed from Fix flaky coverage - t/lib/OpenQA/Test/FullstackUtils.pm to Fix flaky coverage - t/lib/OpenQA/Test/FullstackUtils.pm size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by tinita about 3 years ago
- Status changed from Workable to Feedback
- Assignee set to tinita
Monitoring new pull requests to catch a new occurrence of this coverage decrease, in order to save the codecov.io report and see which code actually wasn't covered.
Updated by tinita about 3 years ago
I found a merged PR where the coverage decreased by 5%: https://github.com/os-autoinst/openQA/pull/4257
That's why we regularly see an increase of 5% now
https://app.codecov.io/gh/os-autoinst/openQA/compare/4257/changes
The lines marked with ---
lost coverage:
92 sub _bail_with_result_panel_contents {
--- 93 my ($result_panel_contents) = @_;
--- 94 diag("full result panel contents:\n$result_panel_contents");
--- 95 javascript_console_has_no_warnings_or_errors;
96 BAIL_OUT "Expected result not found"; # uncoverable statement
97 }
...
113 if ($looking_for_result && $status_text =~ qr/Result: (.*) finished/) {
--- 114 diag("stopped waiting for '$result_panel', result turned out to be '$1'");
--- 115 return _bail_with_result_panel_contents $status_text;
116 }
Updated by tinita about 3 years ago
- Status changed from Feedback to In Progress
I identified the problem: for certain tests like t/full-stack.t
, we retry the test.
When the test fails, the _bail_with_result_panel_contents
will be called and recorded as covered.
When the test succeeds in one of the retries, _bail_with_result_panel_contents
is never called, but it is still recorded, as Devel::Cover keeps coverage from previous tests.
This way we see a subroutine as covered which is only called when a test fails.
One solution would be to just delete the cover_db
folder in tools/retry
, but that would also remove coverage data from previous other tests which we want to keep.
Updated by tinita about 3 years ago
https://github.com/os-autoinst/openQA/pull/4294 Mark uncoverable bailout code paths - merged
Updated by livdywan about 3 years ago
tinita wrote:
One solution would be to just delete the
cover_db
folder intools/retry
, but that would also remove coverage data from previous other tests which we want to keep.
Here's an idea. We already define COVERDB_SUFFIX
, and we can see that in tools/retry
. So we can delete the correct db after an iteration. It doesn't seem 100% clean, but not sure how else we could perform such cleanup.
Updated by openqa_review about 3 years ago
- Due date set to 2021-10-27
Setting due date based on mean cycle time of SUSE QE Tools
Updated by tinita about 3 years ago
Updated by tinita about 3 years ago
PR https://github.com/os-autoinst/openQA/pull/4299 t: Add a hook to tools/retry to delete coverage data
edit: merged 3pm
Updated by tinita about 3 years ago
- Status changed from In Progress to Resolved
Resolving, as I don't see FullstackUtils.pm
in the codecov reports anymore.