Try using tmpfs on openqaworker1 to use RAM more efficiently
openqaworker1 has 256GiB of RAM and currently hosts 20 workers which use around ~2GiB of RAM for the VM (depending on the test) and let's say 1GiB for the worker code each.
That means only ~60GiB of RAM are "actively" used during test runs, the remaining ~200GiB might be used as caches or buffers.
By using the NVMe storage devices (1T + 380G) as swap, it would be possible to use a tmpfs on
/var/lib/openqa to host the various isos and HDD images.
That way the kernel decides which data is written to disk (i.e. swap) at all, instead of flushing all data to a persistent filesystem regularly (triggered by QEMU). This might noticably increase test performance.
- Less wear on the SSDs -> increase of lifetime
- Better performance of tests
- Could possible increase worker count further
- Some downtime required
- Might not work as intended, or at all, so the change might have to be rolled back
- The cache is no longer persistent, so it would have to download necessary assets after each day. That shouldn't be much of an issue, but might be noticable.
- Priority changed from Normal to Low
- Target version set to future
There is also #30595 regarding tmpfs on workers. This is a nice idea in general. We also tweaked buffering parameters and by deep digging I think you would be able to find something as well in other older tickets.