action #71224
closed[osd][alert] NTP offset alert
0%
Description
Observation¶
[osd-admins] [Alerting] NTP offset alert

From: Grafana <osd-admins@suse.de>
To: osd-admins@suse.de
Sender: osd-admins <osd-admins-bounces+okurz=suse.de@suse.de>
List-Id: <osd-admins.suse.de>
Date: 10/09/2020 17.17
*/[Alerting] NTP offset alert/*
*Metric name*
*Value*
Stratum 2
16824.203
Time seem to have been off by up to 33.6s. The alert recovered itself after 2.5h
Updated by okurz over 3 years ago
- Status changed from New to Rejected
- Assignee set to okurz
Checked in journal of osd but have found nothing unusual. In this rare case I would actually not do anything and also not change the alert.
Updated by nicksinger over 3 years ago
I remember that a correct time is mainly required for API keys to work. Can we actually somehow calculate/look up the exact value when API keys will start to break?
Updated by okurz over 3 years ago
Based on the observation that communication between workers and webui did not break I assume that either the limit is higher or workers were shifted in time just as well as webui :) Nevertheless we want the alert to trigger before communication breaks down, right? Also we should have a proper time for logs, etc. And 30s is a lot
Updated by nicksinger over 3 years ago
- Status changed from Rejected to In Progress
- Assignee changed from okurz to nicksinger
Just yesterday we saw the same alert again which also again recovered after 2.5h. I saw that we only have suse internal hosts configured as ntp upstream servers so adding yet another one but from the ntp pool might ease the issue? It could also rule out that we have some different issue on our host.
Besides that I also saw that we still use ntpd on OSD. Chronie is installed but not used and might be yet another measure we could try out.
Updated by okurz over 3 years ago
I am glad you found this older ticket :) And good idea about consolidating the ntp/chrony thing. I think it's not the first time you confuse "chrony" and "cronie" ;)
Updated by nicksinger over 3 years ago
- Status changed from In Progress to Feedback
gnah, damn yeah. I confuse them all the time :)
I've created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/393 for now to (1) reflect the current setup and (2) because it was the simplest experiment we can conduct right now.
Updated by okurz over 3 years ago
- Status changed from Feedback to Resolved
perfect. MR merged. I checked that it was applied correctly and checked logs on osd. ntpd started up fine again and is using the SUSE internal NTP server by default. It would probably take some months for any next incident to see if the fallback to external NTP servers works so this should be good enough for now.