action #89497
Updated by livdywan about 4 years ago
## Observation
```
[Alerting] Failed systemd services alert (except openqa.suse.de)
Check failed systemd services on hosts with `systemctl --failed`. Hint: Go to parent dashboard https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services to see a list of affected hosts.
Metric name
Value
Sum of failed systemd services
1.000
```
Alerting at 12:48Z, OK at 12.49Z.
Once more alerting at 13.03Z, OK at 13.07Z
Once more alerting at 13.27Z, OK at 13.37Z
Once more alerting at 13.49Z, OK at 13.52Z
Once more alerting at 2.30Z, OK at 2.31Z
Once more alerting at 2.37Z, OK at 2.40Z on *openqaworker13*
## Suggestion
- Investigate how to include failed services in the alert notification (to make it clear if new alerts are flaky or new issues)
- Observe [failed systemd services](https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services)
- Check what `sudo salt -l error --no-color -C 'G@roles:worker' cmd.run "/etc/telegraf/scripts/systemd_failed.sh"` says