action #153346
Updated by okurz 4 months ago
## Observation
We are requesting https://stats.openqa-monitor.qa.suse.de/api/alerts?state=alerting in our post-deploy monitor step in osd-deployment.
However, that returns nothing, and even without a filter https://stats.openqa-monitor.qa.suse.de/api/alerts it returns an empty list.
After the migration to the unified alerting we probably need to use new API calls.
It's not obvious which call it would be though. See https://grafana.com/docs/grafana/latest/developers/
Our goal is to detect broken deployments, i.e. new alerts firing after deployment.
## Suggestions
* Read how https://gitlab.suse.de/openqa/osd-deployment/-/blob/master/.gitlab-ci.yml?ref_type=heads#L19
* I found this [openapi editor url](https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/pkg/services/ngalert/api/tooling/post.json) which lists some API calls that I can't find in the documentation. e.g. https://stats.openqa-monitor.qa.suse.de/api/alertmanager/grafana/api/v2/alerts does return something, but not much. But it includes the current silenced one https://monitor.qa.suse.de/alerting/grafana/2Z025iB4km/view
* Check https://gitlab.suse.de/openqa/salt-states-openqa#updating-alert-rules-with-the-help-of-the-grafana-web-ui
* Update our check, maybe extend the jq hacking we have
* Crosscheck that what we did in the past actually worked
Back