Project

General

Profile

Actions

action #175210

closed

[o3][zabbix] reconsider e-mail notification settings size:S

Added by jbaier_cz about 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Category:
-
Start date:
2024-12-12
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

The current configuration is to receive an e-mail for a problem if:

  1. The problem is there for at least for 15 minutes
  2. The problem severity is at least Average (i.e. Average, High or Critical); we are not informed about Information and Warning via e-mail

Maybe we want to reevaluated those rules.

Acceptance criteria

  • AC1: We receive an email on "WARNING" conditions from zabbix

Suggestion

  • Reconsider settings (see #174316 for motivation)
  • Create SD ticket to adjust the settings (see #132752 for additional info)

Related issues 2 (0 open2 closed)

Related to openQA Infrastructure (public) - action #132752: Use proper bot account for notifications in zabbix.suse.de size:MResolvedjbaier_cz2023-07-14

Actions
Related to openQA Infrastructure (public) - action #174316: [o3][zabbix][alert] no email about zabbix alerts including storage and cpu load size:SResolvedjbaier_cz2024-12-12

Actions
Actions #1

Updated by jbaier_cz about 2 months ago

  • Related to action #132752: Use proper bot account for notifications in zabbix.suse.de size:M added
Actions #2

Updated by jbaier_cz about 2 months ago

  • Related to action #174316: [o3][zabbix][alert] no email about zabbix alerts including storage and cpu load size:S added
Actions #3

Updated by robert.richardson about 2 months ago

  • Subject changed from [o3][zabbix] reconsider e-mail notification settings to [o3][zabbix] reconsider e-mail notification settings size: S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #4

Updated by okurz about 1 month ago

  • Subject changed from [o3][zabbix] reconsider e-mail notification settings size: S to [o3][zabbix] reconsider e-mail notification settings size:S
Actions #5

Updated by robert.richardson about 1 month ago

  • Assignee set to robert.richardson
Actions #6

Updated by robert.richardson about 1 month ago

  • Status changed from Workable to In Progress
Actions #7

Updated by robert.richardson about 1 month ago

  • Status changed from In Progress to Feedback

I'm unsure on how to proceed, on https://progress.opensuse.org/projects/openqav3/wiki/Wiki#Monitoring it is stated that

Alert notification is configured via trigger action in a special Infra-owned RO bot account. E-mail notification is in place for average problems and higher.

which sounds to me like i need to open a SD ticket similar to this one to increase the notification verbosity?

however further below it says

Configuration of alerts is done in /etc/munin/munin.conf

i checked /etc/munin/munin.conf and it has this setting commented out

# Use this to always send an email, not oly when the status changes
#contact.o3admins.always_send warning critical

Would it be enough to uncomment this, (maybe it should be contact.o3admins.always_send warning average high critical instead), or do i need to open a sd ticket ?

Actions #8

Updated by jbaier_cz about 1 month ago

Munin and Zabbix are two different monitoring platforms. This ticket is about Zabbix where the alerting is done as described in #132752. As we do not have credentials for the bot user to change the setting, SD ticket is needed. Changing /etc/munin/munin.conf will not (should not) affect Zabbix in any way.

Actions #9

Updated by robert.richardson about 1 month ago ยท Edited

  • Status changed from Feedback to Blocked

jbaier_cz wrote in #note-8:

Munin and Zabbix are two different monitoring platforms. This ticket is about Zabbix where the alerting is done as described in #132752. As we do not have credentials for the bot user to change the setting, SD ticket is needed. Changing /etc/munin/munin.conf will not (should not) affect Zabbix in any way.

ok i see, thanks
created a sd ticket to change the setting and will block the current ticket on that

Actions #10

Updated by robert.richardson about 1 month ago

  • Status changed from Blocked to Feedback

The SD ticket mentioned above is resolved, waiting for the first mail with "WARNING" Severity to close this issue.

Actions #11

Updated by okurz about 1 month ago

I got an email with subject "Problem: Test trigger for #175210", looks good I guess?

Actions #12

Updated by jbaier_cz about 1 month ago

I created that trigger to test the settings and forgot it already in the mean time. Glad that the alerting worked :)

Actions #13

Updated by robert.richardson about 1 month ago

  • Status changed from Feedback to Resolved

Resolved

Actions

Also available in: Atom PDF