QA » openQA Project » openQA Infrastructure

going trhoght journal on ow14 - no issues present , it looks like monitoring lost connection to ow14 and nothing else

Discussed with nicksinger. The general silencing does not make sense as the "host up" alert was actually the only real alert where we would care about data because "average_response_ms" never returns a value if there is no response. However, it looks like we never challenged that query design which was in since 2019 in salt-states-openqa commit 5ae5356. So I deleted the generic silence again. Also openqa-piworker just reappeared after dheidler fixed the network config so I also unsilenced the specific alert about openqa-piworker.

Looking back into my email archive over the past days I could only find "FIRING.DatasourceNoData.*host up alert" which are the good ones. So it seems we never had an unintended message about NoData. Still, we can try to improve the alert by switching to "result_code" checking which apparently yields 0 in case of successful ping response and 1 otherwise. We changed the alert but as we already know the alert configuration is not saved in the exported json file. So that is something to continue … or manually change *all ping alerts to use "last of packets_received, alert if max is below 1"

grafana 9.3.6 was built in https://build.opensuse.org/package/show/server:monitoring/grafana but not yet published so we can monitor http://download.opensuse.org/repositories/server:/monitoring/15.4/x86_64/?P=grafana* and upgrade as soon as published.

cdywan wrote:

We decided to rollback the alerts for now. This can be done by adjusting the config file so I'll take care of that step

For the record the way we write the config file it will be overridden fully which is why I decided against side-stepping salt and prepared an MR for it.

The rollback had the effect that apparently the alert list on grafana does not show "unified alerting" alerts anymore and we receive emails in the old format again, like subject "[Alerting]" and "[Ok]" instead of "[FIRING]" and "[RESOLVED]". However https://monitor.qa.suse.de/alerting/list shows an error about "Failed to load Grafana rules state: 404 from rule state endpoint. Perhaps ruler API is not enabled". However that might also come from the automatic upgrade to grafana 9.3.6 which happened over night so I suggest to do a web research for that error message, enable a simple boolean option in the grafana config, restart the service and have it hopefully fixed.

I'm not sure where you're seeing that error. It seems to be related to Loki which I don't thing we even use. I don't think there's a simple boolean flag to enable (unless we would actually use Loki).