Project

General

Profile

Actions

action #99594

closed

Fix flaky coverage - t/lib/OpenQA/Test/FullstackUtils.pm size:M

Added by okurz about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2021-10-01
Due date:
2021-10-27
% Done:

0%

Estimated time:

Description

Observation

Flaky coverage in https://github.com/os-autoinst/openQA/pull/4256#issuecomment-932021197. No logs available.

Acceptance criteria

  • AC1: t/lib/OpenQA/Test/FullstackUtils.pm is not showing up in codecov reports for multiple PRs with unrelated changes

Suggestions

  • Find an occurence and preserve the results
  • Try full-stack.t or scheduler-full.t

Related issues 1 (0 open1 closed)

Copied to openQA Project - action #99597: Fix flaky coverage - lib/OpenQA/Worker/WebUIConnection.pm size:MResolvedtinita2021-10-012021-10-29

Actions
Actions #1

Updated by okurz about 3 years ago

  • Copied to action #99597: Fix flaky coverage - lib/OpenQA/Worker/WebUIConnection.pm size:M added
Actions #2

Updated by livdywan about 3 years ago

  • Subject changed from Fix flaky coverage - t/lib/OpenQA/Test/FullstackUtils.pm to Fix flaky coverage - t/lib/OpenQA/Test/FullstackUtils.pm size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #3

Updated by tinita about 3 years ago

  • Status changed from Workable to Feedback
  • Assignee set to tinita

Monitoring new pull requests to catch a new occurrence of this coverage decrease, in order to save the codecov.io report and see which code actually wasn't covered.

Actions #4

Updated by tinita about 3 years ago

I found a merged PR where the coverage decreased by 5%: https://github.com/os-autoinst/openQA/pull/4257

That's why we regularly see an increase of 5% now

https://app.codecov.io/gh/os-autoinst/openQA/compare/4257/changes
The lines marked with --- lost coverage:

     92 sub _bail_with_result_panel_contents {
---  93     my ($result_panel_contents) = @_;
---  94     diag("full result panel contents:\n$result_panel_contents");
---  95     javascript_console_has_no_warnings_or_errors;
     96     BAIL_OUT "Expected result not found";    # uncoverable statement
     97 }
...
    113         if ($looking_for_result && $status_text =~ qr/Result: (.*) finished/) {
--- 114             diag("stopped waiting for '$result_panel', result turned out to be '$1'");
--- 115             return _bail_with_result_panel_contents $status_text;
    116         }
Actions #5

Updated by tinita about 3 years ago

  • Status changed from Feedback to In Progress

I identified the problem: for certain tests like t/full-stack.t, we retry the test.

When the test fails, the _bail_with_result_panel_contents will be called and recorded as covered.

When the test succeeds in one of the retries, _bail_with_result_panel_contents is never called, but it is still recorded, as Devel::Cover keeps coverage from previous tests.

This way we see a subroutine as covered which is only called when a test fails.

One solution would be to just delete the cover_db folder in tools/retry, but that would also remove coverage data from previous other tests which we want to keep.

Actions #6

Updated by tinita about 3 years ago

https://github.com/os-autoinst/openQA/pull/4294 Mark uncoverable bailout code paths - merged

Actions #7

Updated by livdywan about 3 years ago

tinita wrote:

One solution would be to just delete the cover_db folder in tools/retry, but that would also remove coverage data from previous other tests which we want to keep.

Here's an idea. We already define COVERDB_SUFFIX, and we can see that in tools/retry. So we can delete the correct db after an iteration. It doesn't seem 100% clean, but not sure how else we could perform such cleanup.

https://github.com/os-autoinst/openQA/pull/4295

Actions #8

Updated by openqa_review about 3 years ago

  • Due date set to 2021-10-27

Setting due date based on mean cycle time of SUSE QE Tools

Actions #10

Updated by tinita about 3 years ago

PR https://github.com/os-autoinst/openQA/pull/4299 t: Add a hook to tools/retry to delete coverage data

edit: merged 3pm

Actions #11

Updated by tinita about 3 years ago

  • Status changed from In Progress to Resolved

Resolving, as I don't see FullstackUtils.pm in the codecov reports anymore.

Actions

Also available in: Atom PDF