QA » openQA Project

Observation¶

https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=1694660738324&to=1694756474277&viewPanel=19 shows an increase of inactive minion jobs at an alarming rate. https://openqa.suse.de/minion/jobs?state=active shows 4 jobs for task "hook_script" active most of the time it seems, e.g. it showed when looking:

• https://openqa.suse.de/minion/jobs?id=8485276
• https://openqa.suse.de/minion/jobs?id=8485208
• https://openqa.suse.de/minion/jobs?id=8485171
• https://openqa.suse.de/minion/jobs?id=8485099

I triggered a retry on the last three assuming they were stuck but I think that was a wrong assessment.

As a consequence also many investigation jobs and such have not been executed.

Acceptance criteria¶

• AC1: https://openqa.suse.de/tests/12128032#comments has investigation job results shown
• AC2: An alert for https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=1694660738324&to=1694756474277&viewPanel=19 has been considered
• AC3: https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=1694660738324&to=1694756474277&viewPanel=19 "inactive" is back to normal levels

Suggestions¶

• Mitigate and monitor the situation, e.g. manually stopping/retrying any stuck minion jobs, restarting openqa-gru, etc.
• Follow the above minion job URLs to see what the state was
• Crosscheck the current state on the system, e.g. check the process table, attach strace for blocked processes, etc.
• Consider running more than 4 jobs:

# cat /usr/share/openqa/script/openqa-gru 
#!/bin/sh -e
[ "$1" = "-h" ] || [ "$1" = "--help" ] && echo "Start openQA GRU service" && exit
exec "$(dirname "$0")"/openqa gru -m production run --reset-locks --jobs 4 --spare 2 --spare-min-priority 10 "$@"