Project

General

Profile

Actions
Copy link

action #102428

closed

Provide "fail-rate" alerting with ratio_mm_failed 5.360 size:M

Added by livdywan over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
kraih
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2021-07-28
Due date:
2021-12-07
% Done:

0%

Estimated time:

Description

Observation

The Ratio of multi-machine tests is alerting.

ratio_mm_failed 5.360

Suggestion

  • Investigate what caused the ratio to turn
  • Maybe the investigation work on #101271 caused a higher fail-ratio

Related issues 2 (0 open — 2 closed)

Copied from openQA Project - action #96191: Provide "fail-rate" of tests, especially multi-machine, in grafana size:MResolvedokurz2021-07-282021-09-29

Actions
Copied to openQA Project - action #103425: Ratio of multi-machine tests alerting with ratio_mm_failed 5.280 size:MResolvedmkittler

Actions
Actions
Copy link
 #1

Updated by livdywan over 2 years ago

  • Copied from action #96191: Provide "fail-rate" of tests, especially multi-machine, in grafana size:M added
Actions
Copy link
 #2

Updated by okurz over 2 years ago

  • Priority changed from Normal to Urgent
Actions
Copy link
 #3

Updated by okurz over 2 years ago

The alert turned green again for now. Maybe the investigation work on #101271 caused a higher fail-ratio. Still, this should be investigated in more detail, e.g. take a look for the specific reason why multi-machine tests failed in the past two days.

Actions
Copy link
 #4

Updated by livdywan over 2 years ago

  • Subject changed from Provide "fail-rate" alerting with ratio_mm_failed 5.360 to Provide "fail-rate" alerting with ratio_mm_failed 5.360 size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #5

Updated by mkittler over 2 years ago

The alert was firing today again (but has now already went off again).

Actions
Copy link
 #6

Updated by kraih over 2 years ago

  • Assignee set to kraih
Actions
Copy link
 #7

Updated by kraih over 2 years ago

  • Status changed from Workable to In Progress

Looking through the failed jobs.

Actions
Copy link
 #8

Updated by openqa_review over 2 years ago

  • Due date set to 2021-12-07

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #9

Updated by kraih over 2 years ago

So far i'm not seeing any patterns, the reasons for why the tests failed are very diverse. Only recent-ish change is that there is more active test development for one of our newer products.

Actions
Copy link
 #10

Updated by kraih over 2 years ago

I'll take a closer look at two 24 hour time frames, 2 days ago vs 30 days ago (when the ratio was around 3).

Actions
Copy link
 #11

Updated by kraih over 2 years ago

  • Status changed from In Progress to Feedback

Checked about 200 multi machine jobs and the only noticeable change has really been new products being tested (and the tests being in active development). So i'm fairly certain there have been no larger underlying issues here, and we should (at least temporarily) consider increasing the alert value.

Actions
Copy link
 #12

Updated by kraih over 2 years ago

  • Status changed from Feedback to Resolved
Actions
Copy link
 #13

Updated by livdywan over 2 years ago

  • Copied to action #103425: Ratio of multi-machine tests alerting with ratio_mm_failed 5.280 size:M added
Actions
Copy link

Also available in: Atom PDF