action #137603
closed[alert] Queue: State (SUSE) - too few jobs executed alert size:S
0%
Description
Observation¶
Queue: State (SUSE) - too few jobs executed alert
Summary
Too few openQA jobs are executed
Description
Not enough openQA jobs are assigned to workers and executed while many
scheduled jobs exist in the scheduled state.see https://progress.opensuse.org/issues/135122 for details
Values
E0=17 E1=273
Labels
alertname Queue: State (SUSE) - too few jobs executed alert
grafana_folder SaltView dashboard http://stats.openqa-monitor.qa.suse.de/d/nRDab3Jiz?orgId=1
View panel
http://stats.openqa-monitor.qa.suse.de/d/nRDab3Jiz?orgId=1&viewPanel=9Observed 38s before this notification was delivered, at 2023-10-09
04:30:00 +0200 CEST
Might be related to #137600, might be just a normal behavior over weekend. The alert is currently in a resolved state.
Suggestions¶
- "too few jobs" means < 20 running, > 100 scheduled - are these numbers sensible? Reduce/increase?
Updated by tinita 7 months ago
from /var/log/openqa_scheduler around that time:
[2023-10-09T04:29:07.965461+02:00] [debug] [pid:1683] Scheduling: Free workers: 927/949; Scheduled jobs: 273
[2023-10-09T04:29:08.614353+02:00] [debug] [pid:1683] Skipping 273 jobs because of no free workers for requested worker classes (qemu_ppc64le,tap:107,spvm_ppc64le:86,hmc_ppc64le-1disk:36,svirt-vmware:22,qemu_ppc64le-large-mem,tap:8,openqaworker14,qemu_x86_64,tap:3,hmc_ppc64le:2,hmc_ppc64le_sap:2,nue,qemu_x86_64,tap:2,prg_office,qemu_ppc64le,qemu_x86_64:2,qemu_x86_64_no_tmpfs:2,hmc_ppc64le-4disk:1)
And that simply goes on until now. We apparently just didn't have very much jobs that actually found a matching worker.
Updated by kraih 7 months ago
I just made the same observation while manually reviewing monitoring data. The system has actually been very responsive over the weekend and picked up new jobs almost immediately. This is exactly how it should be.
Updated by kraih 7 months ago
okurz wrote in #note-4:
Then we should ensure that there is no alert in such cases, right?
The real problem is the 273 stuck scheduled jobs, which seem to be mostly ppc64le variations with no available workers. So the alert itself is valid, but there are other tickets about the problem already.
Updated by okurz 7 months ago
- Related to action #136130: test fails in iscsi_client due to salt 'host'/'nodename' confusion size:M added