Project

General

Profile

Actions

action #106841

open

Try using tmpfs on openqaworker1 to use RAM more efficiently

Added by favogt about 2 years ago. Updated about 1 year ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
Start date:
2022-02-15
Due date:
% Done:

0%

Estimated time:
Tags:

Description

openqaworker1 has 256GiB of RAM and currently hosts 20 workers which use around ~2GiB of RAM for the VM (depending on the test) and let's say 1GiB for the worker code each.
That means only ~60GiB of RAM are "actively" used during test runs, the remaining ~200GiB might be used as caches or buffers.

By using the NVMe storage devices (1T + 380G) as swap, it would be possible to use a tmpfs on /var/lib/openqa to host the various isos and HDD images.
That way the kernel decides which data is written to disk (i.e. swap) at all, instead of flushing all data to a persistent filesystem regularly (triggered by QEMU). This might noticably increase test performance.

Possible advantages

  • Less wear on the SSDs -> increase of lifetime
  • Better performance of tests
  • Could possible increase worker count further

Risks

  • Some downtime required
  • Might not work as intended, or at all, so the change might have to be rolled back
  • The cache is no longer persistent, so it would have to download necessary assets after each day. That shouldn't be much of an issue, but might be noticable.

Related issues 1 (0 open1 closed)

Related to openQA Infrastructure - action #30595: [ppc64le] Deploy bcache on tmpfs workersClosed2018-01-22

Actions
Actions #1

Updated by okurz about 2 years ago

  • Related to action #30595: [ppc64le] Deploy bcache on tmpfs workers added
Actions #2

Updated by okurz about 2 years ago

  • Priority changed from Normal to Low
  • Target version set to future

There is also #30595 regarding tmpfs on workers. This is a nice idea in general. We also tweaked buffering parameters and by deep digging I think you would be able to find something as well in other older tickets.

Actions #3

Updated by okurz about 1 year ago

  • Tags set to infra
Actions

Also available in: Atom PDF