Added by mkittler over 1 year ago. Updated over 1 year ago.
0%
Description
They were ok again on the same day but we should investigate what happened. The problematic services were web UI host services and other services on that host like postgresql.service alert. So maybe a problem on OSD itself.
I've been looking at https://stats.openqa-monitor.qa.suse.de/d/webuiSyS/webui-systemd-services?editPanel=13&tab=alert&orgId=1&from=now-7d&to=now and strangely the state history doesn't show any "Alerting" entries but I suppose the mail with "[No Data]" subject corresponds to the "NO DATA" entry (yellow question mark) from Nov. 6, 2022 03:37:03. There are also more of those "NO DATA" entries and there were also mails about them (but we likely haven't looked into them at the time, at least I haven't found a reply to those I've checked).
I suppose it can be normal that there's shortly no data. We normally work around it by setting "If no data or all values are null" to "Keep last state" but for these alerts it is set to "No data". So I suggest to consistently set this to "Keep last state".
I don't think there was anything wrong with those services as it was just a no data alert.
This SR should prevent those notifications in the future: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/768
The SR has been merged and changes are effective in Grafana.
Sorry, I still think https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/768/diffs#2bbeaa6f546d17e656be75c46f57589662264bea_982_981 is wrong. Please see #71098 for why we introduced those explicit "no data" alerts.
Please take a close look at my MR https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/771 to fix that again.
I thought this was only about https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/768/diffs?diff_id=139700#e3826da6725e3d4c3e0febb6a40114acc83256ec_966_965. I've merged your MR.