Project

General

Profile

action #98919

Improve alert handling - slack notifications

Added by okurz 8 months ago. Updated 3 months ago.

Status:
Workable
Priority:
Low
Assignee:
-
Target version:
Start date:
2021-09-20
Due date:
% Done:

0%

Estimated time:

Description

Acceptance criteria

  • AC1: API access to Slack is available to the team
  • AC2: Grafana alerts trigger Slack notifications
  • AC3: All gitlab CI pipeline failures that we care about also end up in Slack notifications

Related issues

Related to QA - action #102059: Integrate the Slack feed notifications feature for progress queriesResolved2021-11-082021-11-19

Related to openQA Project - action #91605: notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) size:MResolved

Copied from QA - action #98916: Improve alert handling - weekly alert dutyResolved

History

#1 Updated by okurz 8 months ago

  • Copied from action #98916: Improve alert handling - weekly alert duty added

#2 Updated by nicksinger 8 months ago

  • Status changed from New to In Progress

https://grafana.com/docs/grafana/latest/alerting/old-alerting/notifications/#slack mentions the native integration in grafana. I'm now following https://api.slack.com/bot-usershttps://api.slack.com/bot-users to request a bot user for our workspace and channel from the "workspace admins".

#3 Updated by nicksinger 8 months ago

  • Status changed from In Progress to Blocked

Request to install openQA monitoring: Hello dear Admins! For our tools-team inside the QA department we want to try out slack as notification platform for our Alerts (see https://progress.opensuse.org/issues/98919). I kindly request this bot account so we could try out what is offers and if it fits our needs.

Request now pending.

#4 Updated by okurz 8 months ago

I see one problem with grafana alerts in slack: We would still need to react to a lot of different projects by email, e.g. gitlab CI pipelines. Is it realistic to find slack notifications for all of these as well?

#5 Updated by nicksinger 8 months ago

I asked in https://suse.slack.com/archives/C029GP8DCDA/p1632489599077000 if there is anything more I need to provide.

I see one problem with grafana alerts in slack: We would still need to react to a lot of different projects by email, e.g. gitlab CI pipelines. Is it realistic to find slack notifications for all of these as well?

I'd start simple now by moving one single source into slack. If it works out we can see if we can implement notifications for other projects/sources too

#6 Updated by okurz 8 months ago

I agree. I don't see this as a stopper. Merely something to keep in mind. Also, so far grafana alerts are likely more critical. Pipeline failures are either concerning surrounding, less critical ecosystem tools or delivery pipelines that can be monitored as part of development work.

#7 Updated by nicksinger 8 months ago

okurz wrote:

I agree. I don't see this as a stopper. Merely something to keep in mind. Also, so far grafana alerts are likely more critical. Pipeline failures are either concerning surrounding, less critical ecosystem tools or delivery pipelines that can be monitored as part of development work.

Another Idea which comes to mind: check pipeline-status by grafana and trigger alerts this way. Unfortunately I didn't receive a reply in #help-slack, I asked a second time before I will try to reach out to my manager with this.

#8 Updated by nicksinger 8 months ago

I got an answer from Andrew Rutherford in #help-slack and also in a private message. It takes some time before this process is completed but it is not stuck according to him.

#9 Updated by okurz 8 months ago

  • Due date changed from 2021-10-04 to 2022-02-04
  • Priority changed from Normal to Low

ok, fine. We can wait. Updating prio and due-date accordingly.

#10 Updated by okurz 7 months ago

  • Related to action #102059: Integrate the Slack feed notifications feature for progress queries added

#11 Updated by okurz 6 months ago

  • Status changed from Blocked to Feedback

nicksinger where can we check the current status? Currently reading the latest comments here there is no other ticket to track, right? So in that case "Feedback" would be better meaning that recurringly we need to check again what's the current status.

#12 Updated by nicksinger 6 months ago

Feedback is fine for me :) I asked "Andrew Rutherford" once again. Apparently some mail got missing which he forwarded again to me with two documents with questions to fill out. One word-file, one powerpoint-file. After I fill these there seems to be some kind of committee deciding if the app can be approved. I will update you once the further process is clear to me.

#13 Updated by nicksinger 5 months ago

  • Status changed from Feedback to Workable

#14 Updated by okurz 4 months ago

  • Description updated (diff)
  • Due date deleted (2022-02-04)
  • Assignee deleted (nicksinger)
  • Target version changed from Ready to future

#15 Updated by okurz 4 months ago

  • Parent task deleted (#98673)

Deleting parent task connection to conclude the parent without this future ticket.

#16 Updated by okurz 3 months ago

We could try something simpler. Every Slack channel can have an auto-generated email address assigned. To do that click on the title of the room on top, "Integrations", on bottom "Send emails to this channel" and copy the email address. We could configure this address as alert address or add the email to a mailing list like osd-admins@suse.de

#17 Updated by okurz 18 days ago

  • Related to action #91605: notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) size:M added

Also available in: Atom PDF