Project

General

Profile

coordination #96447

[epic] Failed systemd services and job age alerts

Added by cdywan almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Target version:
openQA Project - Ready
Start date:
2021-08-04
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Observation

  • Alerts Disk I/O time for /dev/vdd (/results) handled in #96554
  • Alerts for Job age (scheduled)
  • Alerts for Failed systemd services
    • This is about the alert from 02.08.21 01:17 (the one from 05.08.21 07:26 was caused by a user's misconfiguration).

Suggestion

  • Bump our thresholds
  • Investigate if our average load has increased immensely e.g. new test groups being scheduled
  • Look at systemd journal while the alert is running (short of having #96551)
  • Check if we have data on reduced heat/ power in server room 2
  • Job age (scheduled) (median) is likely due to issues with the WORKER_CLASS of https://openqa.suse.de/tests/6513484

Subtasks

action #96551: Persistent records of systemd journal size:SResolvedokurz

action #96554: Mitigate on-going disk I/O alerts size:MResolvedmkittler

action #97043: job queue hitting new record 14k jobsResolvedokurz

openQA Project - action #105064: Reduce verbosity of openQA logging to improve performance and reduce storage requirements size:MResolvedokurz

action #105373: Ask to increase OSD /srv so that we can save enough logs+DBResolvedokurz

Related issues

Related to openQA Infrastructure - action #96710: Error `Can't call method "write" on an undefined value` shows up in worker log leading to incompletesResolved2021-08-102021-08-31

Related to openQA Project - action #96557: jobs run into MAX_SETUP_TIME, one hour between 'Downloading' and 'Download processed' and no useful output in between auto_review:"timeout: setup exceeded MAX_SETUP_TIME":retryResolved2021-08-042021-08-19

Related to openQA Infrastructure - action #96552: Persistent records of I/O usage by process size:MWorkable2021-08-04

Related to openQA Project - action #99246: Published QCOW images appear to be uncompressedResolved2021-09-242021-10-09

History

#1 Updated by cdywan almost 2 years ago

Disk I/O time for /dev/vde (/space-slow) with vde: read 20068.460 was alterting again for a minute

#2 Updated by cdywan almost 2 years ago

  • Description updated (diff)

#3 Updated by cdywan almost 2 years ago

Disk I/O time for /dev/vdd (/results) once more alerting with vdd: write 13648.552 for 9 minutes.

#4 Updated by okurz almost 2 years ago

  • Priority changed from Normal to Urgent

bumping to "urgent" to counter alarm fatigue.

#5 Updated by cdywan almost 2 years ago

  • Subject changed from Disk I/O and job age schedule alerts to [epic] Disk I/O and job age schedule alerts
  • Description updated (diff)

#6 Updated by cdywan almost 2 years ago

  • Status changed from New to Workable

#7 Updated by mkittler almost 2 years ago

  • Subject changed from [epic] Disk I/O and job age schedule alerts to [epic] Failed systemd services and job age alerts
  • Description updated (diff)

#8 Updated by cdywan almost 2 years ago

  • Related to action #96710: Error `Can't call method "write" on an undefined value` shows up in worker log leading to incompletes added

#9 Updated by cdywan almost 2 years ago

  • Related to action #96557: jobs run into MAX_SETUP_TIME, one hour between 'Downloading' and 'Download processed' and no useful output in between auto_review:"timeout: setup exceeded MAX_SETUP_TIME":retry added

#11 Updated by cdywan almost 2 years ago

  • Related to action #96552: Persistent records of I/O usage by process size:M added

#12 Updated by mkittler over 1 year ago

Not sure how workable this ticket is. There's only one point left of the ones mentioned under observation and retrospectively it is hard to tell what was wrong at the time and we handled future alerts of the same kind. I get that this is an epic but the ticket title and description don't relate with the remaining sub tasks in a meaningful way.

#13 Updated by okurz over 1 year ago

  • Tracker changed from action to coordination

#14 Updated by okurz over 1 year ago

  • Related to action #99246: Published QCOW images appear to be uncompressed added

#15 Updated by okurz over 1 year ago

  • Status changed from Workable to Blocked
  • Assignee set to okurz

Looks like this could also be related to #99246 . I think see have all the mentioned ideas covered in follow up tasks or resolved already. Tracking remaining subtask

#16 Updated by okurz over 1 year ago

  • Status changed from Blocked to Resolved

All subtasks resolved, all tasks and goals covered.

Also available in: Atom PDF