action #89899
closed
coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
coordination #55364: [epic] Let's make codecov reports reliable
Fix flaky coverage - t/ui/27-plugin_obs_rsync_status_details.t
Added by okurz about 3 years ago.
Updated about 3 years ago.
Category:
Feature requests
Description
Motivation¶
See #55364 : codecov reports often report about coverage changes which are obviously not related to the actual changes of a PR, e.g. when documentation is changed. We can already trust our coverage analysis more but should have only coverage changes reported for actual changes we introduced in a pull request.
Acceptance criteria¶
- AC1: t/ui/27-plugin_obs_rsync_status_details.t does not appear anymore as changing code coverage in unrelated changes
Suggestions¶
- Try to reproduce locally with
rm -rf cover_db/ && make coverage KEEP_DB=1 TESTS=t/ui/27-plugin_obs_rsync_status_details.t
- check coverage details in generated html report, e.g. call
firefox cover_db/coverage.html
- Fix uncovered lines with "uncoverable" statements, see previous commits adding these comments or look into https://metacpan.org/pod/Devel::Cover#UNCOVERABLE-CRITERIA or other means
- retry multiple times to check for flakyness
- Copied from action #80274: Fix flaky coverage - t/lib/OpenQA/Test/Utils.pm size:M added
- Related to action #89935: t/ui/27-plugin_obs_rsync_status_details.t fails in circleCI, master branch even added
- Status changed from Workable to Blocked
- Assignee set to kraih
- Status changed from Blocked to Workable
- Due date set to 2021-03-30
Setting due date based on mean cycle time of SUSE QE Tools
- Status changed from Workable to In Progress
- Status changed from In Progress to Feedback
While running this test countless times in the past few days, i have seen it time out twice on Circle CI. That could have been bad luck (maybe we just need to give it a few extra seconds for very slow Circle CI runs?) or there is still a rare logic bug hidden somewhere that prevents the expected result from appearing in the UI. So we'll have to keep an eye on this test and maybe create a followup ticket to collect more information.
Here you can focus on just code coverage. We will see from maybe a couple more PRs if the coverage changes for unrelated changes. For general stability of the test module you still have #89935
- Due date changed from 2021-03-30 to 2021-04-09
kraih wrote:
While running this test countless times in the past few days, i have seen it time out twice on Circle CI. That could have been bad luck (maybe we just need to give it a few extra seconds for very slow Circle CI runs?) or there is still a rare logic bug hidden somewhere that prevents the expected result from appearing in the UI. So we'll have to keep an eye on this test and maybe create a followup ticket to collect more information.
Do we consider the coverage "stable" now? I agree with @okurz that this isn't about the test itself but the % we get.
- Status changed from Feedback to Resolved
I asked in the team chat and got no objections to consider this solved
- Due date deleted (
2021-04-09)
Also available in: Atom
PDF