action #128870
Updated by okurz over 1 year ago
## Observation See http://stats.openqa-monitor.qa.suse.de/d/EML0bpuGk?orgId=1&viewPanel=2 ## Reproducible every time the monitoring host reboots, e.g. weekly reboot maintenance window in Sunday morning ## Suggestions * We already configured all other alerts to not alert on no data and execution error, only for this specific alert we did not do that so that we are actually alerted if there is no data arriving in influxdb. Maybe we can still select to treat "ExecutionError" as ok and only alert on NoDataError * Crosscheck the state history in https://stats.openqa-monitor.qa.suse.de/d/EML0bpuGk/monitoring?orgId=1&viewPanel=2&from=1684012870771&to=1684068285405&editPanel=2&tab=alert# . It seems in 2023-04 the alert seems to have worked fine with the "pending" state * track upstream issue https://github.com/grafana/grafana/issues/16290 which is still open ## Rollback actions * remove silence for "InfluxDB not reachable"