Project

General

Profile

action #71224

[osd][alert] NTP offset alert

Added by okurz 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2020-09-11
Due date:
% Done:

0%

Estimated time:

Description

Observation

[osd-admins] [Alerting] NTP offset alert

From:   Grafana <osd-admins@suse.de>
To: osd-admins@suse.de
Sender: osd-admins <osd-admins-bounces+okurz=suse.de@suse.de>
List-Id:    <osd-admins.suse.de>
Date:   10/09/2020 17.17

*/[Alerting] NTP offset alert/*

*Metric name*
*Value*
Stratum 2
16824.203

http://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?fullscreen&edit&tab=alert&panelId=86&orgId=1

Time seem to have been off by up to 33.6s. The alert recovered itself after 2.5h

History

#1 Updated by okurz 4 months ago

  • Status changed from New to Rejected
  • Assignee set to okurz

Checked in journal of osd but have found nothing unusual. In this rare case I would actually not do anything and also not change the alert.

#2 Updated by nicksinger 4 months ago

I remember that a correct time is mainly required for API keys to work. Can we actually somehow calculate/look up the exact value when API keys will start to break?

#3 Updated by okurz 4 months ago

Based on the observation that communication between workers and webui did not break I assume that either the limit is higher or workers were shifted in time just as well as webui :) Nevertheless we want the alert to trigger before communication breaks down, right? Also we should have a proper time for logs, etc. And 30s is a lot

#4 Updated by nicksinger 3 months ago

  • Status changed from Rejected to In Progress
  • Assignee changed from okurz to nicksinger

Just yesterday we saw the same alert again which also again recovered after 2.5h. I saw that we only have suse internal hosts configured as ntp upstream servers so adding yet another one but from the ntp pool might ease the issue? It could also rule out that we have some different issue on our host.
Besides that I also saw that we still use ntpd on OSD. Chronie is installed but not used and might be yet another measure we could try out.

#5 Updated by okurz 3 months ago

I am glad you found this older ticket :) And good idea about consolidating the ntp/chrony thing. I think it's not the first time you confuse "chrony" and "cronie" ;)

#6 Updated by nicksinger 3 months ago

  • Status changed from In Progress to Feedback

gnah, damn yeah. I confuse them all the time :)
I've created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/393 for now to (1) reflect the current setup and (2) because it was the simplest experiment we can conduct right now.

#7 Updated by okurz 3 months ago

  • Status changed from Feedback to Resolved

perfect. MR merged. I checked that it was applied correctly and checked logs on osd. ntpd started up fine again and is using the SUSE internal NTP server by default. It would probably take some months for any next incident to see if the fallback to external NTP servers works so this should be good enough for now.

Also available in: Atom PDF