Project

General

Profile

action #89497

flaky Failed systemd services alert (except openqa.suse.de)

Added by cdywan 5 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2021-03-04
Due date:
2021-03-31
% Done:

0%

Estimated time:

Description

Observation

[Alerting] Failed systemd services alert (except openqa.suse.de)

Check failed systemd services on hosts with `systemctl --failed`. Hint: Go to parent dashboard https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services to see a list of affected hosts.
Metric name

Value
Sum of failed systemd services

1.000

Alerting at 12:48Z, OK at 12.49Z.
Once more alerting at 13.03Z, OK at 13.07Z
Once more alerting at 13.27Z, OK at 13.37Z
Once more alerting at 13.49Z, OK at 13.52Z
Once more alerting at 2.30Z, OK at 2.31Z
Once more alerting at 2.37Z, OK at 2.40Z on openqaworker13

Suggestion

  • Investigate how to include failed services in the alert notification (to make it clear if new alerts are flaky or new issues)
  • Observe failed systemd services
  • Check what sudo salt -l error --no-color -C 'G@roles:worker' cmd.run "/etc/telegraf/scripts/systemd_failed.sh" says

Related issues

Related to openQA Infrastructure - action #88900: openqaworker13 was unreachableResolved2021-02-22

History

#1 Updated by cdywan 5 months ago

  • Description updated (diff)

#2 Updated by cdywan 5 months ago

  • Description updated (diff)

#3 Updated by cdywan 5 months ago

  • Description updated (diff)

#4 Updated by cdywan 5 months ago

  • Description updated (diff)

#5 Updated by cdywan 5 months ago

#6 Updated by cdywan 5 months ago

  • Status changed from New to Feedback

It looks like the "flaky" alerts have been caused by manual reboots i.e. #88900 and the offending unit was var-lib-openqa-share.mount

#7 Updated by okurz 5 months ago

  • Status changed from Feedback to Workable

#8 Updated by cdywan 5 months ago

Status changed from Feedback to Workable

Can you please state why you changed this back to Workable?

#9 Updated by okurz 5 months ago

There was no assignee and I would not know how we would ever look into tickets that are in "Feedback" but without assignee. If you consider that "Workable" sounds too much "workable", well then "New" would be ok for me :)

#10 Updated by cdywan 5 months ago

I observe 9 alerts Failed systemd services alert (except openqa.suse.de) between March 6, 2.14Z and March 7, 11.05Z. Suspected to be due to automatic reboots after updates, alerting 2-3 minutes each.

#11 Updated by cdywan 5 months ago

This will probably become more practical to verify with https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/459

#12 Updated by cdywan 5 months ago

  • Description updated (diff)

#13 Updated by okurz 4 months ago

  • Status changed from Workable to In Progress
  • Assignee set to nicksinger

#14 Updated by okurz 4 months ago

  • Due date set to 2021-03-31

Setting due date based on mean cycle time of SUSE QE Tools

#15 Updated by nicksinger 4 months ago

  • Status changed from In Progress to Feedback

I will set this to feedback to give it some more days to check if the "flakiness" is gone by bumping the "pending timeout" to 15m

#16 Updated by cdywan 4 months ago

  • Status changed from Feedback to Resolved

I've seen the alert on April 7, where it was caused by the os-autoinst-openvswitch service on openqaworker-arm-1 and on April 1 where it was "expected" due to okurz working on #81058 so I'd say this is not flaky at this point.

Also available in: Atom PDF