Project

General

Profile

action #106841

Try using tmpfs on openqaworker1 to use RAM more efficiently

Added by favogt 4 months ago. Updated 4 months ago.

Status:
New
Priority:
Low
Assignee:
-
Target version:
Start date:
2022-02-15
Due date:
% Done:

0%

Estimated time:

Description

openqaworker1 has 256GiB of RAM and currently hosts 20 workers which use around ~2GiB of RAM for the VM (depending on the test) and let's say 1GiB for the worker code each.
That means only ~60GiB of RAM are "actively" used during test runs, the remaining ~200GiB might be used as caches or buffers.

By using the NVMe storage devices (1T + 380G) as swap, it would be possible to use a tmpfs on /var/lib/openqa to host the various isos and HDD images.
That way the kernel decides which data is written to disk (i.e. swap) at all, instead of flushing all data to a persistent filesystem regularly (triggered by QEMU). This might noticably increase test performance.

Possible advantages

  • Less wear on the SSDs -> increase of lifetime
  • Better performance of tests
  • Could possible increase worker count further

Risks

  • Some downtime required
  • Might not work as intended, or at all, so the change might have to be rolled back
  • The cache is no longer persistent, so it would have to download necessary assets after each day. That shouldn't be much of an issue, but might be noticable.

Related issues

Related to openQA Infrastructure - action #30595: [ppc64le] Deploy bcache on tmpfs workersNew2018-01-22

History

#1 Updated by okurz 4 months ago

  • Related to action #30595: [ppc64le] Deploy bcache on tmpfs workers added

#2 Updated by okurz 4 months ago

  • Priority changed from Normal to Low
  • Target version set to future

There is also #30595 regarding tmpfs on workers. This is a nice idea in general. We also tweaked buffering parameters and by deep digging I think you would be able to find something as well in other older tickets.

Also available in: Atom PDF