QA » openQA Project » openQA Infrastructure

Observation¶

[Alerting] Failed systemd services alert (except openqa.suse.de)

Check failed systemd services on hosts with `systemctl --failed`. Hint: Go to parent dashboard https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services to see a list of affected hosts.
Metric name

Value
Sum of failed systemd services

1.000

Alerting at 12:48Z, OK at 12.49Z.
Once more alerting at 13.03Z, OK at 13.07Z
Once more alerting at 13.27Z, OK at 13.37Z
Once more alerting at 13.49Z, OK at 13.52Z
Once more alerting at 2.30Z, OK at 2.31Z
Once more alerting at 2.37Z, OK at 2.40Z on openqaworker13

Suggestion¶

• Investigate how to include failed services in the alert notification (to make it clear if new alerts are flaky or new issues)
• Observe failed systemd services
• Check what sudo salt -l error --no-color -C 'G@roles:worker' cmd.run "/etc/telegraf/scripts/systemd_failed.sh" says