action #106841
openTry using tmpfs on openqaworker1 to use RAM more efficiently
0%
Description
openqaworker1 has 256GiB of RAM and currently hosts 20 workers which use around ~2GiB of RAM for the VM (depending on the test) and let's say 1GiB for the worker code each.
That means only ~60GiB of RAM are "actively" used during test runs, the remaining ~200GiB might be used as caches or buffers.
By using the NVMe storage devices (1T + 380G) as swap, it would be possible to use a tmpfs on /var/lib/openqa
to host the various isos and HDD images.
That way the kernel decides which data is written to disk (i.e. swap) at all, instead of flushing all data to a persistent filesystem regularly (triggered by QEMU). This might noticably increase test performance.
Possible advantages¶
- Less wear on the SSDs -> increase of lifetime
- Better performance of tests
- Could possible increase worker count further
Risks¶
- Some downtime required
- Might not work as intended, or at all, so the change might have to be rolled back
- The cache is no longer persistent, so it would have to download necessary assets after each day. That shouldn't be much of an issue, but might be noticable.
Updated by okurz almost 3 years ago
- Related to action #30595: [ppc64le] Deploy bcache on tmpfs workers added
Updated by okurz almost 3 years ago
- Priority changed from Normal to Low
- Target version set to future
There is also #30595 regarding tmpfs on workers. This is a nice idea in general. We also tweaked buffering parameters and by deep digging I think you would be able to find something as well in other older tickets.