Project

General

Profile

action #98919

action #98673: [retro] Unhandled alert about job queue for nearly a day, users brought it up in chat, should have been picked up sooner size:S

Improve alert handling - slack notifications

Added by okurz about 1 month ago. Updated 22 days ago.

Status:
Blocked
Priority:
Low
Assignee:
Target version:
Start date:
2021-09-20
Due date:
2022-02-04
% Done:

0%

Estimated time:

Related issues

Copied from QA - action #98916: Improve alert handling - weekly alert dutyResolved2021-10-04

History

#1 Updated by okurz about 1 month ago

  • Copied from action #98916: Improve alert handling - weekly alert duty added

#2 Updated by nicksinger about 1 month ago

  • Status changed from New to In Progress

https://grafana.com/docs/grafana/latest/alerting/old-alerting/notifications/#slack mentions the native integration in grafana. I'm now following https://api.slack.com/bot-usershttps://api.slack.com/bot-users to request a bot user for our workspace and channel from the "workspace admins".

#3 Updated by nicksinger about 1 month ago

  • Status changed from In Progress to Blocked

Request to install openQA monitoring: Hello dear Admins! For our tools-team inside the QA department we want to try out slack as notification platform for our Alerts (see https://progress.opensuse.org/issues/98919). I kindly request this bot account so we could try out what is offers and if it fits our needs.

Request now pending.

#4 Updated by okurz about 1 month ago

I see one problem with grafana alerts in slack: We would still need to react to a lot of different projects by email, e.g. gitlab CI pipelines. Is it realistic to find slack notifications for all of these as well?

#5 Updated by nicksinger about 1 month ago

I asked in https://suse.slack.com/archives/C029GP8DCDA/p1632489599077000 if there is anything more I need to provide.

I see one problem with grafana alerts in slack: We would still need to react to a lot of different projects by email, e.g. gitlab CI pipelines. Is it realistic to find slack notifications for all of these as well?

I'd start simple now by moving one single source into slack. If it works out we can see if we can implement notifications for other projects/sources too

#6 Updated by okurz about 1 month ago

I agree. I don't see this as a stopper. Merely something to keep in mind. Also, so far grafana alerts are likely more critical. Pipeline failures are either concerning surrounding, less critical ecosystem tools or delivery pipelines that can be monitored as part of development work.

#7 Updated by nicksinger 29 days ago

okurz wrote:

I agree. I don't see this as a stopper. Merely something to keep in mind. Also, so far grafana alerts are likely more critical. Pipeline failures are either concerning surrounding, less critical ecosystem tools or delivery pipelines that can be monitored as part of development work.

Another Idea which comes to mind: check pipeline-status by grafana and trigger alerts this way. Unfortunately I didn't receive a reply in #help-slack, I asked a second time before I will try to reach out to my manager with this.

#8 Updated by nicksinger 22 days ago

I got an answer from Andrew Rutherford in #help-slack and also in a private message. It takes some time before this process is completed but it is not stuck according to him.

#9 Updated by okurz 22 days ago

  • Due date changed from 2021-10-04 to 2022-02-04
  • Priority changed from Normal to Low

ok, fine. We can wait. Updating prio and due-date accordingly.

Also available in: Atom PDF