QA (public) » openQA Project (public) » openQA Infrastructure (public)

monitor.qa.suse.de and stats.openqa-monitor.qa.suse.de seems to work - any remaining steps wrt making sure renewal happens automatically?

I suggest to add monitoring, same as we already have for OSD

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/634 should be a start to bring in the according telegraf config. For this we should read out the domain list from pillars but I realized that pillars have not been updated on monitor.qa since 2021-04 so that should be the first thing to check. Then in grafana copy the certificate related panel from the webui dashboard into https://monitor.qa.suse.de/d/EML0bpuGk/monitoring

I've opened https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639 as a first draft how I would imagine the monitoring. It should be more generic and work for every host we eventually add in the future. I still need to figure out how to properly loop over the SANs of the hosts.txt entries per host to have a proper check for each of them (this is what happens in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639/diffs#85e7a4e39662e846e9a7e8c1660d41d28a0389c3_0_8) - it might be even possible to shrink the certificates.conf down to a single [[inputs.x509_cert]] section

nicksinger wrote:

Monitoring is left to do. Will carry this over into the new year. If anybody is bored feel free to take over :) Otherwise I will add the monitoring in the new year. Our current certificate is valid until the 20th of Jan so we have headroom to implement this monitoring before the current cert expires

Apparently we're at Feb 14 now, which suggests automated renewal worked at least once.

cdywan wrote:

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639 is still in progress, I'm starting to get worried that we'll see alerts for broken certificates afterall... bumping the due date to tomorrow in any case

@nicksinger Please prepare manual steps to do the update tomorrow

Just to make it clear here in the ticket too:

AC4: The certificates are automatically refreshed
is already done. Therefore we don't need to worry about an expiring certificate and we won't see any alerts because this is why this ticket is still open - creating monitoring/alerts for it. I polished up https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/639 and think it can be merged now to receive the metrics. A dashboard for grafana is still left to do but I'd like to cover it in a separate MR

I came up with a new dashboard including alerts and proper instructions here: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/640
It might not work after merging because I couldn't really test the template for the dashboard.