action #133130
closed
Lots of alerts for a single cause. Can we group and de-duplicate?
Added by okurz 10 months ago.
Updated 4 months ago.
Description
Observation¶
Received the following alert emails:
- sapworker1: host up alert
- sapworker1: OpenQA Ping time alert
- sapworker2: host up alert
- ...
- sapworker3: OpenQA Ping time alert
- sapworker3: Ping time alert
- Average Ping time (ms) alert
all for a singular reason: Problem with the Frankencampus network. Can we group alerts and also not have host up and openQA ping time and ping time alerts?
Acceptance criteria¶
- AC1: Grouped alerts, grafana supports this!
- AC2: No separate ping time alerts if there is a corresponding host up alert, at least the ping time should come much later than the host up
Suggestions¶
Rollback steps¶
- Copied from action #133127: Frankencampus network broken + GitlabCi failed --> uploading artefacts added
- Related to action #132788: [alert][flaky] QA-Power8-5-kvm: QA network infrastructure Ping time alert added
- Description updated (diff)
- Subject changed from Lots of alerts for a single cause. Can we group and de-duplicate? to Lots of alerts for a single cause. Can we group and de-duplicate? size:M
- Description updated (diff)
- Status changed from New to Workable
- Assignee set to nicksinger
- Status changed from Workable to Feedback
I modified our "Notification policy" to group by "hostname". It waits for 30s to send out all alerts by group which might not be enough but I don't know how I could test it. Setting to "feedback" to just wait for now until something bigger breaks.
- Priority changed from Urgent to Normal
We discussed it a couple times now but unfortunately without updating the ticket. I'm lowering the urgency to reflect the fact that this is mostly done with the changes Nick already implemented.
- Copied to action #133991: Cover same metric for different hosts with a single alert rule added
- Copied to deleted (action #133991: Cover same metric for different hosts with a single alert rule)
- Related to action #133991: Cover same metric for different hosts with a single alert rule added
livdywan wrote:
We discussed it a couple times now but unfortunately without updating the ticket. I'm lowering the urgency to reflect the fact that this is mostly done with the changes Nick already implemented.
As mentioned we already group by hostname (which is a already present label in our setup). I've additionally created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/935 to introduce a common label for every instance of "host_up" alerts and created a follow-up #133991 to cover reducing the amount of alert rule instances per metric.
- Status changed from Feedback to Resolved
I've added another "nested policy" below our general __contacts__ =~ .*"osd-admins".*-policy (which does the grouping by hostname). If the label of an alert is alert = host_up, it will group by alert. Not sure how this behaves in combination with the parent grouping but IIUC this is an "overwrite" so it should not interfere.
Resolving for now. If we see it not working out we can reopen it.
- Status changed from Resolved to Workable
- Assignee deleted (
nicksinger)
We decided to re-open because we couldn't tellif this was alerting and maybe it "should have"?
- Copied to action #135470: Grafana: Average Ping time (ms) alert with unexpanded variable "${tag_url}", which machine is this about? size:M added
- Target version changed from Ready to Tools - Next
- Target version changed from Tools - Next to Ready
- Related to action #138044: Grouped seemingly unrelated alert emails are confusing size:M added
- Target version changed from Ready to Tools - Next
- Target version changed from Tools - Next to Ready
- Subject changed from Lots of alerts for a single cause. Can we group and de-duplicate? size:M to Lots of alerts for a single cause. Can we group and de-duplicate?
- Status changed from Workable to New
Maybe we should re-estimate this.
- Status changed from New to Resolved
- Assignee set to nicksinger
In the daily infra call we tried to estimate this ticket again but we did not understand the comment
We decided to re-open because we couldn't tellif this was alerting and maybe it "should have"?
Also we checked the mailing list and did not found many occurences of grouping at all. Examples like https://mailman.suse.de/mlarch/SuSE/osd-admins/2023/osd-admins.2023.11/msg00120.html with subject "FIRING:2" don't look that helpful but we wouldn't know what to do.
Obviously after this long time if you see a related issue please better reopen a linked ticket with explicit description, not reopen.
Also available in: Atom
PDF