action #132815
closed[alert][flaky][o3] Multiple flaky zabbix alerts related to o3
Added by okurz 10 months ago. Updated 10 months ago.
0%
Description
Observation¶
Multiple emails received, see https://mailman.suse.de/mlarch/SuSE/o3-admins/2023/o3-admins.2023.07/maillist.html
Rollback steps¶
- enable "Load average is too high" trigger
Updated by jbaier_cz 10 months ago
- Status changed from In Progress to Feedback
After brief investigation I find out that the notification settings inside zabbix were overly verbose. I reconfigured it to alert only on issues which are "Averege" and higher (at this moment, this should include severe filesystem and accessibility problems).
Updated by okurz 10 months ago
- Related to action #132278: Basic o3 http response alert on zabbix size:M added
Updated by okurz 10 months ago
- Related to coordination #132275: [epic] Better o3 monitoring added
Updated by okurz 10 months ago
- Related to action #131150: Add alarms for partition usage on o3 size:M added
Updated by okurz 10 months ago
From today: "Problem: Load average is too high (per CPU load over 1.5 for 5m)". Please look up the according threshold that we defined for OSD in grafana and I suggest to apply that here as well, something like in the range of 20-40. 1.5 is way too restrictive for our setup.
Updated by jbaier_cz 10 months ago
- Description updated (diff)
- Status changed from Feedback to Blocked
okurz wrote:
From today: "Problem: Load average is too high (per CPU load over 1.5 for 5m)".
Hm, I see. So we are hitting also the average trigger for CPU... Blocking this on SD-127003 (to be able to change the threshold). In the mean time, I will deactivate the CPU load trigger warning to suppress the emails.
Updated by tinita 10 months ago
okurz wrote:
From today: "Problem: Load average is too high (per CPU load over 1.5 for 5m)". Please look up the according threshold that we defined for OSD in grafana and I suggest to apply that here as well, something like in the range of 20-40. 1.5 is way too restrictive for our setup.
20-40? Note that the current threshold 1.5 is the average per CPU, so 15 for 10 CPUs. Maybe set it to 4? (As soon as we get access).
Updated by jbaier_cz 10 months ago
- Status changed from In Progress to Resolved
Threshold for alerts were adjusted: CPU load should trigger on 4 times the cpu cores (that should be load over 40 in our case), storage triggers should warn us around 90% of space used, the same holds for available memory and finally, we have some availability triggers. I did not see any alert since this morning, so I am considering this resolved for now. Any additional improvement should be made (and will be made) in other tickets.