Project

General

Profile

Actions
Copy link

action #122848

closed

openQA Project - coordination #109846: [epic] Ensure all our database tables accomodate enough data, e.g. bigint for ids

coordination #113674: [epic] Configure I/O alerts again for the webui after migrating to the "unified alerting" in grafana size:M

Configure grouped alerts in Grafana correctly size:M

Added by livdywan over 1 year ago. Updated 12 months ago.

Status:
Resolved
Priority:
Low
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-01-09
Due date:
% Done:

0%

Estimated time:
Tags:
infra

Description

Summary

The migration in #112845 introduces repeating panels as well as a new concept of grouped alerts. We should know how that impacts existing alert handling and potentially take advantage of it where beneficial.

Acceptance criteria

  • AC1: grouping of alerts is properly configured and understood

Suggestions

  • Look into alert grouping
  • Confirm that alert group is what we think it is
  • Monitor alert emails and see how they can be affected by grouping
  • Figure out how to avoid multiple redundant alert emails e.g. several emails like [FIRING:1] (Failed systemd services alert (except openqa) about the same issue (the alert was only in the state "Alerting" once on the day we've got the redundant mails) with no changes in-between OR confirm if there was a change that's somehow not reflected in the email

Related issues 3 (1 open — 2 closed)

Related to openQA Infrastructure - action #138044: Grouped seemingly unrelated alert emails are confusing size:MRejectedokurz2023-10-09

Actions
Related to openQA Infrastructure - action #159639: [alert] "web UI: Too many 5xx HTTP responses alert" size:SWorkable2024-04-26

Actions
Blocked by openQA Infrastructure - action #122845: Migrate our Grafana setup to "unified alerting"Resolvednicksinger2023-01-09

Actions
Actions
Copy link
 #1

Updated by livdywan over 1 year ago

  • Blocked by action #122845: Migrate our Grafana setup to "unified alerting" added
Actions
Copy link
 #2

Updated by livdywan over 1 year ago

  • Status changed from New to Blocked
Actions
Copy link
 #3

Updated by livdywan over 1 year ago

  • Target version set to future

This is probably not important for now as we discussed, so I'm putting it in future. Eventually we'll need to answer questions regarding this.

Actions
Copy link
 #4

Updated by okurz about 1 year ago

  • Tags set to infra
  • Status changed from Blocked to New
  • Target version changed from future to Ready
Actions
Copy link
 #5

Updated by livdywan about 1 year ago

  • Subject changed from Configure grouped alerts in Grafana correctly to Configure grouped alerts in Grafana correctly size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #6

Updated by okurz 12 months ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz

Grouped alerts are working in a way that if multiple alerts are firing they are grouped hence preventing a too high number of alert notification messages. So it's properly configured and I have a good feeling of understanding hence IMHO we can resolve

Actions
Copy link
 #7

Updated by tinita 6 months ago

  • Related to action #138044: Grouped seemingly unrelated alert emails are confusing size:M added
Actions
Copy link
 #8

Updated by okurz about 21 hours ago

  • Related to action #159639: [alert] "web UI: Too many 5xx HTTP responses alert" size:S added
Actions
Copy link

Also available in: Atom PDF