Project

General

Profile

action #133130

Updated by okurz almost 1 year ago

## Observation 

 Received the following alert emails: 
 * sapworker1: host up alert 
 * sapworker1: OpenQA Ping time alert 
 * sapworker2: host up alert 
 * ... 
 * sapworker3: OpenQA Ping time alert 
 * sapworker3: Ping time alert 
 * Average Ping time (ms) alert 

 all for a singular reason: Problem with the Frankencampus network. Can we group alerts and also not have host up and openQA ping time *and* ping time alerts? 

 ## Acceptance criteria 
 * **AC1:** Grouped alerts, grafana supports this! 
 * **AC2:** No ping time alerts if there is a corresponding host up alert, at least the ping time should come much later than the host up 

 ## Suggestions 
 * Look into "grafana alert grouping" and configure alerts accordingly 

 ## Rollback steps 
 * Remove according silences from https://monitor.qa.suse.de/alerting/silences either referencing this ticket or anything concerning "host up" or "ping time"

Back