[osd][alert] NTP offset alert
[osd-admins] [Alerting] NTP offset alert ￼ From: Grafana <firstname.lastname@example.org> To: email@example.com Sender: osd-admins <firstname.lastname@example.org> List-Id: <osd-admins.suse.de> Date: 10/09/2020 17.17 */[Alerting] NTP offset alert/* *Metric name* *Value* Stratum 2 16824.203
Time seem to have been off by up to 33.6s. The alert recovered itself after 2.5h
Based on the observation that communication between workers and webui did not break I assume that either the limit is higher or workers were shifted in time just as well as webui :) Nevertheless we want the alert to trigger before communication breaks down, right? Also we should have a proper time for logs, etc. And 30s is a lot
#4 Updated by nicksinger 3 months ago
- Status changed from Rejected to In Progress
- Assignee changed from okurz to nicksinger
Just yesterday we saw the same alert again which also again recovered after 2.5h. I saw that we only have suse internal hosts configured as ntp upstream servers so adding yet another one but from the ntp pool might ease the issue? It could also rule out that we have some different issue on our host.
Besides that I also saw that we still use ntpd on OSD. Chronie is installed but not used and might be yet another measure we could try out.
#6 Updated by nicksinger 3 months ago
- Status changed from In Progress to Feedback
gnah, damn yeah. I confuse them all the time :)
I've created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/393 for now to (1) reflect the current setup and (2) because it was the simplest experiment we can conduct right now.
- Status changed from Feedback to Resolved
perfect. MR merged. I checked that it was applied correctly and checked logs on osd. ntpd started up fine again and is using the SUSE internal NTP server by default. It would probably take some months for any next incident to see if the fallback to external NTP servers works so this should be good enough for now.