QA (public) » openQA Project (public) » openQA Infrastructure (public)

Motivation¶

In #41882 we identified arm machines being completely unresponsive and are automatically detecting these situations and recover. But there are also cases when systems are I/O blocked, the machine still responds to ping but is not "usable". In this situation the machine can still have openQA jobs assigned that are then stuck for many hours. Also the machine is not detected as broken in grafana hence never recovered automatically. We should detect a situation like this and recover automatically.

Acceptance criteria¶

• AC1: Machines in I/O blocked stated for multiple minutes/hours are detected and recovered, e.g. with reboot, similar/same as "worker completely down"

Suggestions¶

• Check if there are already measurements available in grafana that could be used to trigger alerts which then trigger the reboot actions same as https://stats.openqa-monitor.qa.suse.de/d/1bNU0StZz/automatic-actions?orgId=1
• If not, find an additional measurement/alert for this purpose
• Ensure the alerts and notification configurations are covered in salt