openQA Project (public) » openQA Infrastructure (public)

Observation¶

Our alerts operate on different levels of a system. Starting from general checks like a host is "up" and network is reachable up to checking output of services (e.g. worker minions). If machines are down this means of course that its services are down too resulting in a lot of mails/alerts. We should introduce a way to disable more sophisticated checks if basic ones already fail.

Acceptance criteria¶

• AC1: Offline machines create just a single alert/e-mail
  • AC1.1: We get reminded or have an overview about the current status

Suggestions¶

• Make use of the "automatic actions pipeline/dashboard" (https://stats.openqa-monitor.qa.suse.de/d/1bNU0StZz/automatic-actions) to trigger pausing alerts via http api (https://grafana.com/docs/grafana/v9.0/developers/http_api/alerting/#pause-alert-by-id)
  1. Create a pipeline in gitlab which pauses alerts and can be triggered by a webhook
  2. Create an grafana alert channel similar to the automatic arm reboot (see https://stats.openqa-monitor.qa.suse.de/alerting/notification/3/edit)
  3. Create an according panel with the webhook alert channel configured to trigger if a machine is down
• Alternatively: Try if we can make use of grafana 9 and it's newly created unified alerting (https://grafana.com/blog/2022/06/14/grafana-alerting-explore-our-latest-updates-in-grafana-9/). This automatically groups alerts, sends reminders and might even have more features to make alerts depend on each other
• Possibly a combination of both (e.g. make a specific group, based on tags, of alerts trigger a specific pipeline silencing the whole group/tag)