Project

General

Profile

action #117172

Flaky alert about infrastructure packet loss

Added by okurz 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2022-09-25
Due date:
% Done:

0%

Estimated time:

Description

See http://monitor.qa.suse.de/d/EML0bpuGk/monitoring?tab=alert&viewPanel=4&orgId=1 in particular about an alert on 25. September 2022 03:40:44 CEST which turned back to green just 1m later. We should such flaky alert reports


Related issues

Copied to openQA Infrastructure - action #118375: Do not alert about "packet loss" if hosts are downNew

History

#1 Updated by okurz 2 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz

mkittler prepared https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/745 to bump the alerting time period.

#2 Updated by okurz 2 months ago

  • Due date set to 2022-10-11
  • Status changed from In Progress to Feedback

I was thinking if we should not increase the alerting time threshold even more but maybe 15m is ok for now.

#3 Updated by mkittler 2 months ago

  • Assignee changed from okurz to mkittler

I've seen the additional alert from today but it was before my SR has been merged. I'd wait a few days to see whether it helped.

#4 Updated by mkittler 2 months ago

  • Status changed from Feedback to Resolved

It hasn't happened again so I'm resolving the ticket for now.

#5 Updated by cdywan about 2 months ago

  • Status changed from Resolved to Feedback

The alert was just live for 3 minutes. Did we actually increase it to 15 minutes as per #117172#note-2 or is there another alert involved here? Something doesn't seem to work correctly here.

#7 Updated by okurz about 2 months ago

  • Copied to action #118375: Do not alert about "packet loss" if hosts are down added

#8 Updated by okurz about 2 months ago

  • Due date deleted (2022-10-11)
  • Status changed from Feedback to Resolved

MR merged, rolled out and effective. Further improvements put into #118375

Also available in: Atom PDF