QA (public) » openQA Project (public)

I added a MR for configuring 40 on OSD (https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1141 https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/772) and configured it on o3 workers via for i in $hosts; do echo $i && ssh root@$i "sed -i -e '/CACHELIMIT/a CRITICAL_LOAD_AVG_THRESHOLD = 40' /etc/openqa/workers.ini " ; done.

We found that most of our systems are below the load value of 40 but I have found worker-arm-1 which reaches into the range of 65. During that timeframe when the load was above, 2024-04-07 20:10Z until 2024-04-07 22:45Z, we found multiple test issues which could very well be related to a too high system load, among them mistyping, hanging keys, network communication problems and timeouts, e.g.:

• https://openqa.suse.de/tests/13960955#step/gnutls_base_check/27
• https://openqa.suse.de/tests/13962166#step/patch_sle/118
• https://openqa.suse.de/tests/13961947#step/first_boot/6
• https://openqa.suse.de/tests/13963467#step/zypper_in/83

https://github.com/os-autoinst/openQA/pull/5567 for a generic upstream default.

The change is effective and we got two alerts about it today. One would be sufficient so I created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1144.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1144 merged but can we consider something better than the state/term "broken" for states that are expected to be self-resolving? How about having the worker in state "Working" because it's not actually idle but picked up a job or is doing something and can not currently pick up other jobs but it's also not "broken" as in needs help from an admin?

Reworking the worker's states is probably out of scope for this ticket. Not sure whether "Working" would be the best idea; one had to check the code what other assumptions are associated with "Working". (In the worst case we need yet another state. Currently "Broken" is already abused even for graceful disconnects even causing occasional displaying bugs. So we should probably reconsider the design and perhaps add another explicit database column for the state. I just didn't do it back then because it was easier for the code to pass review without a database migration. EDIT: Looks like I've already investigated the problem with workers showing up as broken with the reason "graceful disconnect". The relevant ticket is #134924 and should probably be taken into account when changing the code handling the worker's state.)

I am currently looking at the situation at petrol were the CPU load alert fired earlier today. Right now all slots are actively working on jobs and the CPU load exceeds the configured load of 40 by a lot (it is around 76). I've just started another slot to see whether the threshold is effective at all and it is. The newly started slot did not pick-up a job and is shown as broken in the web UI with the expected values for the load and configured threshold. So the feature works. It is just that it might not be what we want. By just looking at the average load in the last 15 minutes we probably create the following situation:

1. The worker has just been started and no jobs are running. So the load is very low.
2. Then jobs are assigned on all worker slots because the load is very low.
3. Then the load goes up (overshooting the threshold by a lot). This doesn't change anything because all slots are already running jobs (which we don't interrupt) so no new jobs wouldn't run anyway.
4. If now just one job finishes early then the new feature probably makes a difference. We'd get one less occupied slot because due to the high load no new job would be picked up immediately. That is probably a rare case.
5. Probably the jobs finish around the same time. In a certain time window we'd still not pick-up any new jobs because it takes a while for the avg. load over the last 15 minutes to go below the threshold (especially because in 3. we overshot the threshold by a lot). This probably creates a big time window for almost all slots becoming idle again.
6. The cycle repeats at 1. and (besides 4.) we haven't gained anything.

We could change the feature to look at the average load of the last minute. That wouldn't change much until 5. where the window would be smaller so maybe the we'd then started to pick-up new jobs sooner instead of waiting quite long and then start new jobs on all slots at the same time. So the cycle would less likely repeat. That's probably still not smart enough.

EDIT: When I started writing this comment we were at step 3. It looks like now we are at step 5; so almost all worker slots are "broken" because the avg. load over the last 15 minutes is still over 40 (it is around 50). It looks like the remaining 2 jobs will have just enough time to finish until we are below the 40 again. Then we will start over with step 1.

So before we do any out-of-scope tweaks on how the worker status is handled (#158125#note-15) we should rather think on how to make this work at all.

Well, that's why we have the step of 40-60 between not picking up new jobs and alerting. As we have seen quite some issues with openQA tests in the 50-70 load range we should keep the load lower. I see two tasks:

1. Try with lower load limit, e.g. 20, in particular in PowerNV machines, i.e. mania, diesel, petrol
2. Investigate what increases the load so significantly. Is it the external video encoder?

We discussed it on multiple occassions today and came up with the following ideas after observing that e.g. petrol comes into the range of load values like load1 of 126 (!):

1. Introduce a random time back-off when restarting or looking for new jobs, e.g. [1m:15m] but we think that wouldn't help because initially too many jobs can be picked up at the same time already when the system load is still low. https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/Worker/WebUIConnection.pm#L349 already has a randomized back-off. We could tweak the OPENQA_WORKER_STATUS_MAX_INTERVAL variable
2. As our documentation states we should reserve 2 logical CPU cores per openQA worker instance. This means for petrol running PowerNV with 16 effective cores (SMP disabled) that 8 instances do not leave any room for other processes so we should reduce that. Will do that in #158910

1. Judging by the code changing OPENQA_WORKER_STATUS_MIN_INTERVAL and OPENQA_WORKER_STATUS_MAX_INTERVAL would in fact randomize the delay between the checks while the worker is in the broken state. While it wouldn't help with 1. it would at least help with 5./6. - although we'd probably still overshoot the threshold by quite a bit. Not sure whether that's worth it.
2. Ok, I suppose reducing the number of worker slots would make sense, indeed. Also in the light of #157636#note-7 - TLDR: We were probably even just using libtheora on petrol (and not an actually CPU intensive codec like VP9).

Uh. It seems weird to set a global default for this, as https://github.com/os-autoinst/openQA/commit/34ed70d80e5cbc90960fedeb7ac17006134049a5 did. Doesn't load average relate to CPU count? If the system has 1 CPU, a load average of 40 would be waaaaay too high. But if the system has 64 CPUs, a load average of 40 is totally fine.

Should this mechanism try to use a ratio of load average to CPU count (though determining that reliably can be tricky)? Or get rid of the universal default?

The load is a system load, not CPU load. So if your bottleneck is CPU then possibly the limit of 40 would not be useful otherwise I think it's better than nothing. Recently IO was much more of a bottleneck for us. But if you see actual problems then we can consider removing the default again

well, this threshold definitely throttled our aarch64 worker host, which runs 35 concurrent instances. About 1/3rd of them were doing nothing and showing as 'broken' in the admin interface; the logs showed the load was too high. I bumped the threshold to 60 in config and it's still hitting it, so now I'm trying 70.

In fairness, it's possible this is affecting test reliability, we do get flaky failures on that host, but I've never had time to see if they're related to system load or not (we've pretty much always got flakey failures on aarch64, whatever worker host HW we had at the time). I might try cutting it down to 20 workers for a bit and see if it makes a difference.