openQA Project (public) » openQA Infrastructure (public)

There seems to be some grace period in place, e.g. we've got "Pending (error)" on 2023-04-23 03:40:58 and then "Normal (nodata)"/"Normal (missingseries)"¹ three minutes later on 2023-04-23 03:43:50 with no "Error" in between. The same was also the case the week before and another week before.

After that week on 2023-04-30 03:38:44 there was just "Error" with no preceding "Pending (error)". As if there was no grace period at all anymore. I suppose I'll have to find out what changed between 2023-04-23 03:40:58 and 2023-04-30 03:38:4.

The state history also looks weirdly broken. Apparently some parts of the internally used JSON cannot be parsed by the fronted so the data is torn into two distinct tables. Maybe something worth reporting upstream.

¹ Not sure what's the difference between those two. Note that having no data is actually normal for this dummy query as it is really just about alerting in the error case. So configuring execErrState: OK would mean it would be completely useless and if we wanted that we should remove it completely.

Judging by the output of openqa-monitor:/home/martchus # cat /var/log/zypp/history | grep -i grafana the Grafana version 9.4.7 has already been installed over one week before the 04-23. So the service has already been running at 9.4.7 when the issue has not yet been present (at least not within the time frame we know the state history of).

Regarding configurable grace period for execution error https://github.com/grafana/grafana/issues/55320 should help, closed 3 weeks ago but it's unclear in which release this should show up. https://github.com/grafana/grafana/pull/65574 says it's part of milestone 9.5.0. https://build.opensuse.org/package/show/openSUSE:Factory/grafana is already 9.5.1 . https://build.opensuse.org/package/show/devel:openQA:monitoring/grafana looks kinda broken with source archives for both 9.4.7 and 9.5.1 included.

I called osc linkpac --force openSUSE:Factory grafana devel:openQA:monitoring but the package sources still show 9.4.7. Maybe we need to remove and then recreate the link. In https://build.opensuse.org/package/live_build_log/devel:openQA:monitoring/grafana/15.4/x86_64 I still saw it building 9.4.7 so I called

osc rdelete -m "Delete broken old 9.4.7 package (poo#128870)" devel:openQA:monitoring grafana && osc linkpac --force openSUSE:Factory grafana devel:openQA:monitoring

EDIT: As soon as the package is built and deployed on monitor.qa.suse.de this should automatically solve the problem as the release notes state:

Grafana Alerting rules with NoDataState configuration set to Alerting will now respect "For" duration.

EDIT: https://monitor.qa.suse.de/ now has grafana 9.5.1 which should have the fix for this ticket.

EDIT: I triggered a reboot of monitor.qa.suse.de but that still triggered an alert in the period https://monitor.qa.suse.de/d/EML0bpuGk/monitoring?viewPanel=2&orgId=1&editPanel=2&tab=alert&from=1683656412889&to=1683656698684 . The alert is currently configured to "Evaluate Every 10s For 20m". How about we try to change that to "Evaluate every 20m for 60m"? The description in https://github.com/grafana/grafana/issues/55320#issue-1376161291 brought me to another idea: Configuring a special notification policy: On https://monitor.qa.suse.de/alerting/routes?tab=notification_policies# I created a new nested policy below the "osd-admins" one: alertname = InfluxDB not reachable, Delivered to osd-admins, Wait 20m to group instances. Rebooted 4 times and no alert triggered. Maybe that helped?

I suggest you crosscheck what I did and if you agree you can resolve the ticket assuming that we have fixed the problem and if not then we will be informed :)

Maybe the silence does not cover all use-cases? We have another alert

Some progress was made upstream: https://github.com/grafana/grafana/issues/16290#issuecomment-2100141960

So I guess we should wait for the next Grafana release and update the packaging and see whether it helps.