action #166979
closedGrafana aka monitor.qa.suse.de reporting Bad Gateway error
0%
Description
Updated by livdywan 3 months ago
Sep 18 11:09:55 monitor systemd[1]: Failed to start Grafana instance.
Sep 18 11:09:55 monitor systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Sep 18 11:09:55 monitor systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 11:09:55 monitor grafana[9230]: Error: ✗ alert rules: a conflicting alert rule is found: rule title under the same organisation and folder should be unique
Updated by livdywan 3 months ago
- Status changed from In Progress to Feedback
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1268 should fix it, merged fast.
Updated by nicksinger 3 months ago
- Priority changed from Urgent to Normal
So the MR of @okurz didn't fix it. Oli remembered that we recently added worker36 back but according alerts where never updated when I switched them to generated UID based ones. So we had the same alert (name) with different UIDs in our database. We followed https://gitlab.suse.de/openqa/salt-states-openqa#further-remarks to remove every existing reference to "worker36" in the database and restarted grafana which now works again, so reducing the priority. To resolve we should ensure the same will not happen again with worker37-39 (and possibly others).
Updated by okurz 3 months ago · Edited
- Status changed from Workable to In Progress
- Assignee set to okurz
we still have the w36 dashboard file not included in provisioning files so I am not sure if the previous solution worked. Continuing.
I included the deployment file and it's fine. Removing old rules for w37-w39.
Updated by okurz 3 months ago
- Status changed from In Progress to Resolved
systemctl stop grafana-server && for host in worker37 worker38 worker39 ; do sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from alert_instance where rule_uid = 'memory_usage_alert_$host';" && sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from annotation where text regexp '.*_$host.*';" && sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from provenance_type where record_key regexp '.*_$host.*';" && sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from alert_rule_version where rule_uid regexp '.*_$host.*';" && sudo -u grafana sqlite3 /var/lib/grafana/grafana.db "delete from alert_rule where uid regexp '.*_$host.*';" ; done && systemctl start grafana-server
this seems to have done it.
Updated by livdywan 3 months ago
- Copied to action #167257: Grafana aka monitor.qa.suse.de reporting Bad Gateway error - again size:S added