action #89497

Updated by cdywan 7 months ago

## Observation

[Alerting] Failed systemd services alert (except

Check failed systemd services on hosts with `systemctl --failed`. Hint: Go to parent dashboard to see a list of affected hosts.
Metric name

Sum of failed systemd services


Alerting at 12:48Z, OK at 12.49Z.
Once more alerting at 13.03Z, OK at 13.07Z
Once more alerting at 13.27Z, OK at 13.37Z
Once more alerting at 13.49Z, OK at 13.52Z
Once more alerting at 2.30Z, OK at 2.31Z
Once more alerting at 2.37Z, OK at 2.40Z on *openqaworker13*

## Suggestion
- Investigate how to include failed services in the alert notification (to make it clear if new alerts are flaky or new issues)
- Observe [failed systemd services](
- Check what `sudo salt -l error --no-color -C 'G@roles:worker' "/etc/telegraf/scripts/"` says