coordination #96447
[epic] Failed systemd services and job age alerts
Start date:
2021-08-04
Due date:
% Done:
100%
Estimated time:
(Total: 0.00 h)
Description
Observation¶
Alertshandled in #96554Disk I/O time for /dev/vdd (/results)
Alerts forJob age (scheduled)
- Alerts for
Failed systemd services
- This is about the alert from 02.08.21 01:17 (the one from 05.08.21 07:26 was caused by a user's misconfiguration).
Suggestion¶
- Bump our thresholds
- Investigate if our average load has increased immensely e.g. new test groups being scheduled
- Look at systemd journal while the alert is running (short of having #96551)
- Check if we have data on reduced heat/ power in server room 2
Job age (scheduled) (median)
is likely due to issues with theWORKER_CLASS
of https://openqa.suse.de/tests/6513484
Subtasks
Related issues
History
#8
Updated by cdywan 10 months ago
- Related to action #96710: Error `Can't call method "write" on an undefined value` shows up in worker log leading to incompletes added
#9
Updated by cdywan 10 months ago
- Related to action #96557: jobs run into MAX_SETUP_TIME, one hour between 'Downloading' and 'Download processed' and no useful output in between auto_review:"timeout: setup exceeded MAX_SETUP_TIME":retry added
#11
Updated by cdywan 9 months ago
- Related to action #96552: Persistent records of I/O usage by process size:M added
#12
Updated by mkittler 9 months ago
Not sure how workable this ticket is. There's only one point left of the ones mentioned under observation and retrospectively it is hard to tell what was wrong at the time and we handled future alerts of the same kind. I get that this is an epic but the ticket title and description don't relate with the remaining sub tasks in a meaningful way.
#14
Updated by okurz 7 months ago
- Related to action #99246: Published QCOW images appear to be uncompressed added