action #98919
openImprove alert handling - slack notifications
0%
Description
Acceptance criteria¶
- AC1: API access to Slack is available to the team
- AC2: Grafana alerts trigger Slack notifications
- AC3: All gitlab CI pipeline failures that we care about also end up in Slack notifications
Updated by okurz over 2 years ago
- Copied from action #98916: Improve alert handling - weekly alert duty added
Updated by nicksinger over 2 years ago
- Status changed from New to In Progress
https://grafana.com/docs/grafana/latest/alerting/old-alerting/notifications/#slack mentions the native integration in grafana. I'm now following https://api.slack.com/bot-usershttps://api.slack.com/bot-users to request a bot user for our workspace and channel from the "workspace admins".
Updated by nicksinger over 2 years ago
- Status changed from In Progress to Blocked
Request to install openQA monitoring: Hello dear Admins! For our tools-team inside the QA department we want to try out slack as notification platform for our Alerts (see https://progress.opensuse.org/issues/98919). I kindly request this bot account so we could try out what is offers and if it fits our needs.
Request now pending.
Updated by okurz over 2 years ago
I see one problem with grafana alerts in slack: We would still need to react to a lot of different projects by email, e.g. gitlab CI pipelines. Is it realistic to find slack notifications for all of these as well?
Updated by nicksinger over 2 years ago
I asked in https://suse.slack.com/archives/C029GP8DCDA/p1632489599077000 if there is anything more I need to provide.
I see one problem with grafana alerts in slack: We would still need to react to a lot of different projects by email, e.g. gitlab CI pipelines. Is it realistic to find slack notifications for all of these as well?
I'd start simple now by moving one single source into slack. If it works out we can see if we can implement notifications for other projects/sources too
Updated by okurz over 2 years ago
I agree. I don't see this as a stopper. Merely something to keep in mind. Also, so far grafana alerts are likely more critical. Pipeline failures are either concerning surrounding, less critical ecosystem tools or delivery pipelines that can be monitored as part of development work.
Updated by nicksinger over 2 years ago
okurz wrote:
I agree. I don't see this as a stopper. Merely something to keep in mind. Also, so far grafana alerts are likely more critical. Pipeline failures are either concerning surrounding, less critical ecosystem tools or delivery pipelines that can be monitored as part of development work.
Another Idea which comes to mind: check pipeline-status by grafana and trigger alerts this way. Unfortunately I didn't receive a reply in #help-slack, I asked a second time before I will try to reach out to my manager with this.
Updated by nicksinger over 2 years ago
I got an answer from Andrew Rutherford in #help-slack and also in a private message. It takes some time before this process is completed but it is not stuck according to him.
Updated by okurz over 2 years ago
- Due date changed from 2021-10-04 to 2022-02-04
- Priority changed from Normal to Low
ok, fine. We can wait. Updating prio and due-date accordingly.
Updated by okurz over 2 years ago
- Related to action #102059: Integrate the Slack feed notifications feature for progress queries added
Updated by okurz over 2 years ago
- Status changed from Blocked to Feedback
@nicksinger where can we check the current status? Currently reading the latest comments here there is no other ticket to track, right? So in that case "Feedback" would be better meaning that recurringly we need to check again what's the current status.
Updated by nicksinger over 2 years ago
Feedback is fine for me :) I asked "Andrew Rutherford" once again. Apparently some mail got missing which he forwarded again to me with two documents with questions to fill out. One word-file, one powerpoint-file. After I fill these there seems to be some kind of committee deciding if the app can be approved. I will update you once the further process is clear to me.
Updated by nicksinger over 2 years ago
- Status changed from Feedback to Workable
Updated by okurz over 2 years ago
- Description updated (diff)
- Due date deleted (
2022-02-04) - Assignee deleted (
nicksinger) - Target version changed from Ready to future
Updated by okurz over 2 years ago
- Parent task deleted (
#98673)
Deleting parent task connection to conclude the parent without this future ticket.
Updated by okurz about 2 years ago
We could try something simpler. Every Slack channel can have an auto-generated email address assigned. To do that click on the title of the room on top, "Integrations", on bottom "Send emails to this channel" and copy the email address. We could configure this address as alert address or add the email to a mailing list like osd-admins@suse.de
Updated by okurz almost 2 years ago
- Related to action #91605: notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) size:M added