action #163394
openConsider extending our logging of broken workers in grafana (Better understand "Broken workers alert" retroactively)
0%
Description
Observation¶
The "Broken workers alert" (https://stats.openqa-monitor.qa.suse.de/alerting/grafana/dZ025mf4z/view?orgId=1) is hard to understand retroactively because we only collect the total amount of broken workers and not their names. https://openqa.suse.de/admin/workers only has the current status of a worker and no history.
Suggestions¶
- Consider extending our metrics to also collect the worker(instance) with its state. This allows us to further understand past issues.
- Include these information into alert messages if possible. Try to not introduce new alerts.
Updated by okurz 8 months ago
- Tags set to infra, monitoring, grafana, telegraf, influxdb
- Category set to Feature requests
- Target version set to future
More information would indeed be helpful. However that would also increase the space we need to store for monitoring data. Also very likely the reason for more broken workers is the system load limit which keeps workers in "broken" until the system load is again under the configured limit. I would rather like to improve that behavior in openQA itself
Updated by jbaier_cz 22 days ago
- Related to action #176763: [alert] Flaky broken workers alert size:S added
Updated by jbaier_cz 18 days ago
Maybe we can at least report the names of the broken workers? https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1374