Project

General

Profile

Actions

action #89497

closed

flaky Failed systemd services alert (except openqa.suse.de)

Added by livdywan over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2021-03-04
Due date:
2021-03-31
% Done:

0%

Estimated time:

Description

Observation

[Alerting] Failed systemd services alert (except openqa.suse.de)

Check failed systemd services on hosts with `systemctl --failed`. Hint: Go to parent dashboard https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services to see a list of affected hosts.
Metric name

Value
Sum of failed systemd services

1.000

Alerting at 12:48Z, OK at 12.49Z.
Once more alerting at 13.03Z, OK at 13.07Z
Once more alerting at 13.27Z, OK at 13.37Z
Once more alerting at 13.49Z, OK at 13.52Z
Once more alerting at 2.30Z, OK at 2.31Z
Once more alerting at 2.37Z, OK at 2.40Z on openqaworker13

Suggestion

  • Investigate how to include failed services in the alert notification (to make it clear if new alerts are flaky or new issues)
  • Observe failed systemd services
  • Check what sudo salt -l error --no-color -C 'G@roles:worker' cmd.run "/etc/telegraf/scripts/systemd_failed.sh" says

Related issues 1 (0 open1 closed)

Related to openQA Infrastructure - action #88900: openqaworker13 was unreachableResolvedmkittler2021-02-22

Actions
Actions #1

Updated by livdywan over 3 years ago

  • Description updated (diff)
Actions #2

Updated by livdywan over 3 years ago

  • Description updated (diff)
Actions #3

Updated by livdywan over 3 years ago

  • Description updated (diff)
Actions #4

Updated by livdywan over 3 years ago

  • Description updated (diff)
Actions #5

Updated by livdywan over 3 years ago

Actions #6

Updated by livdywan over 3 years ago

  • Status changed from New to Feedback

It looks like the "flaky" alerts have been caused by manual reboots i.e. #88900 and the offending unit was var-lib-openqa-share.mount

Actions #7

Updated by okurz over 3 years ago

  • Status changed from Feedback to Workable
Actions #8

Updated by livdywan over 3 years ago

Status changed from Feedback to Workable

Can you please state why you changed this back to Workable?

Actions #9

Updated by okurz over 3 years ago

There was no assignee and I would not know how we would ever look into tickets that are in "Feedback" but without assignee. If you consider that "Workable" sounds too much "workable", well then "New" would be ok for me :)

Actions #10

Updated by livdywan over 3 years ago

I observe 9 alerts Failed systemd services alert (except openqa.suse.de) between March 6, 2.14Z and March 7, 11.05Z. Suspected to be due to automatic reboots after updates, alerting 2-3 minutes each.

Actions #11

Updated by livdywan over 3 years ago

This will probably become more practical to verify with https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/459

Actions #12

Updated by livdywan over 3 years ago

  • Description updated (diff)
Actions #13

Updated by okurz over 3 years ago

  • Status changed from Workable to In Progress
  • Assignee set to nicksinger
Actions #14

Updated by okurz over 3 years ago

  • Due date set to 2021-03-31

Setting due date based on mean cycle time of SUSE QE Tools

Actions #15

Updated by nicksinger over 3 years ago

  • Status changed from In Progress to Feedback

I will set this to feedback to give it some more days to check if the "flakiness" is gone by bumping the "pending timeout" to 15m

Actions #16

Updated by livdywan over 3 years ago

  • Status changed from Feedback to Resolved

I've seen the alert on April 7, where it was caused by the os-autoinst-openvswitch service on openqaworker-arm-1 and on April 1 where it was "expected" due to @okurz working on #81058 so I'd say this is not flaky at this point.

Actions

Also available in: Atom PDF