Added by livdywan 3 months ago. Updated 3 months ago.
0%
Description
Sep 18 11:09:55 monitor systemd[1]: Failed to start Grafana instance.
Sep 18 11:09:55 monitor systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Sep 18 11:09:55 monitor systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 11:09:55 monitor grafana[9230]: Error: ✗ alert rules: a conflicting alert rule is found: rule title under the same organisation and folder should be unique
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1268 should fix it, merged fast.
So the MR of @okurz didn't fix it. Oli remembered that we recently added worker36 back but according alerts where never updated when I switched them to generated UID based ones. So we had the same alert (name) with different UIDs in our database. We followed https://gitlab.suse.de/openqa/salt-states-openqa#further-remarks to remove every existing reference to "worker36" in the database and restarted grafana which now works again, so reducing the priority. To resolve we should ensure the same will not happen again with worker37-39 (and possibly others).
we still have the w36 dashboard file not included in provisioning files so I am not sure if the previous solution worked. Continuing.
I included the deployment file and it's fine. Removing old rules for w37-w39.
systemctl stop grafana-server && for host in worker37 worker38 worker39 ; do sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from alert_instance where rule_uid = 'memory_usage_alert_$host';" && sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from annotation where text regexp '.*_$host.*';" && sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from provenance_type where record_key regexp '.*_$host.*';" && sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from alert_rule_version where rule_uid regexp '.*_$host.*';" && sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from alert_rule where uid regexp '.*_$host.*';" ; done && systemctl start grafana-server
this seems to have done it.