action #122848
closedopenQA Project - coordination #109846: [epic] Ensure all our database tables accomodate enough data, e.g. bigint for ids
coordination #113674: [epic] Configure I/O alerts again for the webui after migrating to the "unified alerting" in grafana size:M
Configure grouped alerts in Grafana correctly size:M
0%
Description
Summary¶
The migration in #112845 introduces repeating panels as well as a new concept of grouped alerts. We should know how that impacts existing alert handling and potentially take advantage of it where beneficial.
Acceptance criteria¶
- AC1: grouping of alerts is properly configured and understood
Suggestions¶
- Look into alert grouping
- Confirm that alert group is what we think it is
- Monitor alert emails and see how they can be affected by grouping
- Figure out how to avoid multiple redundant alert emails e.g. several emails like
[FIRING:1] (Failed systemd services alert (except openqa)
about the same issue (the alert was only in the state "Alerting" once on the day we've got the redundant mails) with no changes in-between OR confirm if there was a change that's somehow not reflected in the email
Updated by livdywan over 1 year ago
- Blocked by action #122845: Migrate our Grafana setup to "unified alerting" added
Updated by livdywan over 1 year ago
- Target version set to future
This is probably not important for now as we discussed, so I'm putting it in future. Eventually we'll need to answer questions regarding this.
Updated by okurz over 1 year ago
- Tags set to infra
- Status changed from Blocked to New
- Target version changed from future to Ready
Updated by livdywan over 1 year ago
- Subject changed from Configure grouped alerts in Grafana correctly to Configure grouped alerts in Grafana correctly size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by okurz over 1 year ago
- Status changed from Workable to Resolved
- Assignee set to okurz
Grouped alerts are working in a way that if multiple alerts are firing they are grouped hence preventing a too high number of alert notification messages. So it's properly configured and I have a good feeling of understanding hence IMHO we can resolve
Updated by tinita 12 months ago
- Related to action #138044: Grouped seemingly unrelated alert emails are confusing size:M added
Updated by okurz 5 months ago
- Related to action #159639: [alert] "web UI: Too many 5xx HTTP responses alert" size:S added