QA » openQA Project » openQA Infrastructure

I will look at this

I am a little bit struggling how to approach this. My assumption is, we want to prevent worker memory exhaustion to ensure stable worker (and thus reliable openqa-worker instances and tests inside).

I can easily prevent system crashes by ensuring there is a memory limit. All our workers are part of openqa-worker.slice, we can apply MemoryMax to this slice which will prevent openqa-workers to exhaust all the memory. In this case, systemd is very convenient and allow us to set percentage so a simple MemoryMax=90% should invoke oom killer inside the slice if it become hungry.

After this, we should solve the competition inside the slice. I will need to make some experiments with the MemoryMin.

Setting due date based on mean cycle time of SUSE QE Tools

jbaier demonstrated how the feature works and we can see it in effect in https://monitor.qa.suse.de/d/WDworker31/worker-dashboard-worker31?orgId=1&viewPanel=12054&from=1697445822192&to=1697447326787 as well where is a big dip of available memory and then the system automatically recovers from that. Given that the solution is already in our code base and effective we can conclude and do not even need a follow-up for the actual implementation which is already covered.