action #89497
closedflaky Failed systemd services alert (except openqa.suse.de)
0%
Description
Observation¶
[Alerting] Failed systemd services alert (except openqa.suse.de)
Check failed systemd services on hosts with `systemctl --failed`. Hint: Go to parent dashboard https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services to see a list of affected hosts.
Metric name
Value
Sum of failed systemd services
1.000
Alerting at 12:48Z, OK at 12.49Z.
Once more alerting at 13.03Z, OK at 13.07Z
Once more alerting at 13.27Z, OK at 13.37Z
Once more alerting at 13.49Z, OK at 13.52Z
Once more alerting at 2.30Z, OK at 2.31Z
Once more alerting at 2.37Z, OK at 2.40Z on openqaworker13
Suggestion¶
- Investigate how to include failed services in the alert notification (to make it clear if new alerts are flaky or new issues)
- Observe failed systemd services
- Check what
sudo salt -l error --no-color -C 'G@roles:worker' cmd.run "/etc/telegraf/scripts/systemd_failed.sh"says
Updated by livdywan almost 4 years ago
- Related to action #88900: openqaworker13 was unreachable added
Updated by livdywan almost 4 years ago
- Status changed from New to Feedback
It looks like the "flaky" alerts have been caused by manual reboots i.e. #88900 and the offending unit was var-lib-openqa-share.mount
Updated by livdywan almost 4 years ago
Status changed from Feedback to Workable
Can you please state why you changed this back to Workable?
Updated by okurz almost 4 years ago
There was no assignee and I would not know how we would ever look into tickets that are in "Feedback" but without assignee. If you consider that "Workable" sounds too much "workable", well then "New" would be ok for me :)
Updated by livdywan almost 4 years ago
I observe 9 alerts Failed systemd services alert (except openqa.suse.de) between March 6, 2.14Z and March 7, 11.05Z. Suspected to be due to automatic reboots after updates, alerting 2-3 minutes each.
Updated by livdywan almost 4 years ago
This will probably become more practical to verify with https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/459
Updated by okurz almost 4 years ago
- Status changed from Workable to In Progress
- Assignee set to nicksinger
Updated by okurz over 3 years ago
- Due date set to 2021-03-31
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger over 3 years ago
- Status changed from In Progress to Feedback
I will set this to feedback to give it some more days to check if the "flakiness" is gone by bumping the "pending timeout" to 15m
Updated by livdywan over 3 years ago
- Status changed from Feedback to Resolved