QA » openQA Project » openQA Infrastructure

The alert might look like flaky because we only look at the incompletes within a certain time period. Maybe we are hardly able to change that. However it's important that we find out what these incompletes are. One can simply call openqa-review manually to find out

The graphs of the alerts don't show that the number of (new) incompletes goes over the threshold.

I suspect Grafana could simply not access InfluxDB. Hence it produced the alert but it is a false alert.

Telegraf was also restarted at the same time the alert mails were sent (28.02.21 03:32):

systemctl status telegraf.service 
● telegraf.service - The plugin-driven server agent for reporting metrics into InfluxDB
   Loaded: loaded (/usr/lib/systemd/system/telegraf.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-02-28 03:32:27 CET; 1 day 7h ago

That InfluxDB was shortly unavailable would also explain why everything was good again and OK mails were sent in the next minute.

Find out what's on 8086 and why it refuses connections

8086 is just the port InfluxDB is listening on (running on openqa-monitor.qa.suse.de).

Investigate logfiles on openqa-monitor.qa.suse.de

I had a look into the journal of InfluxDB on openqa-monitor.qa.suse.de but unfortunately it logs every query so it is quite cluttered and I couldn't find anything obvious. In fact it looks like InfluxDB would actually respond to other queries within that time frame.

Since both alerts were effectively no data alerts and the issue resolved itself in a minute I don't think it is worth looking into it and I also wouldn't know what else to check.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/457

cdywan wrote:

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/457

MR got merged, let's see how well this works

I've not observed this particular alert recently.