Project

General

Profile

Actions

action #122848

closed

openQA Project - coordination #109846: [epic] Ensure all our database tables accomodate enough data, e.g. bigint for ids

coordination #113674: [epic] Configure I/O alerts again for the webui after migrating to the "unified alerting" in grafana size:M

Configure grouped alerts in Grafana correctly size:M

Added by livdywan over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
-
Target version:
Start date:
2023-01-09
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Summary

The migration in #112845 introduces repeating panels as well as a new concept of grouped alerts. We should know how that impacts existing alert handling and potentially take advantage of it where beneficial.

Acceptance criteria

  • AC1: grouping of alerts is properly configured and understood

Suggestions

  • Look into alert grouping
  • Confirm that alert group is what we think it is
  • Monitor alert emails and see how they can be affected by grouping
  • Figure out how to avoid multiple redundant alert emails e.g. several emails like [FIRING:1] (Failed systemd services alert (except openqa) about the same issue (the alert was only in the state "Alerting" once on the day we've got the redundant mails) with no changes in-between OR confirm if there was a change that's somehow not reflected in the email

Related issues 3 (0 open3 closed)

Related to openQA Infrastructure - action #138044: Grouped seemingly unrelated alert emails are confusing size:MRejectedokurz2023-10-09

Actions
Related to openQA Infrastructure - action #159639: [alert] "web UI: Too many 5xx HTTP responses alert" size:SResolveddheidler2024-04-26

Actions
Blocked by openQA Infrastructure - action #122845: Migrate our Grafana setup to "unified alerting"Resolvednicksinger2023-01-09

Actions
Actions #1

Updated by livdywan over 1 year ago

  • Blocked by action #122845: Migrate our Grafana setup to "unified alerting" added
Actions #2

Updated by livdywan over 1 year ago

  • Status changed from New to Blocked
Actions #3

Updated by livdywan over 1 year ago

  • Target version set to future

This is probably not important for now as we discussed, so I'm putting it in future. Eventually we'll need to answer questions regarding this.

Actions #4

Updated by okurz over 1 year ago

  • Tags set to infra
  • Status changed from Blocked to New
  • Target version changed from future to Ready
Actions #5

Updated by livdywan over 1 year ago

  • Subject changed from Configure grouped alerts in Grafana correctly to Configure grouped alerts in Grafana correctly size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #6

Updated by okurz about 1 year ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz

Grouped alerts are working in a way that if multiple alerts are firing they are grouped hence preventing a too high number of alert notification messages. So it's properly configured and I have a good feeling of understanding hence IMHO we can resolve

Actions #7

Updated by tinita 8 months ago

  • Related to action #138044: Grouped seemingly unrelated alert emails are confusing size:M added
Actions #8

Updated by okurz about 2 months ago

  • Related to action #159639: [alert] "web UI: Too many 5xx HTTP responses alert" size:S added
Actions

Also available in: Atom PDF