Project

General

Profile

Actions

action #53912

open

Use available space for caching on each worker for increased performance

Added by okurz over 4 years ago. Updated almost 4 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2019-07-06
Due date:
% Done:

0%

Estimated time:

Description

I realized that by default we (still) limit our openQA worker cache size to the default on most if not all o3 and osd workers however much more space can be available. Probably we can save quite some downloading and increase performance (faster starting jobs and less load on webui host) if we increase the cache size depending on what is available on the workers.

Actions #1

Updated by okurz over 4 years ago

  • Status changed from In Progress to Feedback

I configured openqaworker4.o.o with 350 GB instead of the default 50 GB. Monitoring.

Actions #2

Updated by okurz over 4 years ago

filesystem ran full so I guess we need to leave some headroom. Configured for 300GB now and restarted about 300 incomplete jobs :(

Actions #3

Updated by okurz over 4 years ago

Discussed with nsinger. We had the idea for the openqa cacheservice to dynamically look for df instead of a fixed space usage, e.g. use a negative number to denote who much space should be left free instead of how much to occupy.

Actions #4

Updated by okurz over 4 years ago

  • Project changed from openQA Infrastructure to openQA Project
  • Category set to Feature requests
  • Status changed from Feedback to New
  • Assignee deleted (okurz)
Actions #5

Updated by coolo over 4 years ago

if cache would be on its own partition, it could fill it. I don't think that's generally the case

Actions #6

Updated by okurz almost 4 years ago

  • Priority changed from Normal to Low
  • Target version set to future
Actions #7

Updated by okurz almost 4 years ago

on aarch64.o.o I increased cache limit from default 50G to 140G as I observed at a time when 16 test jobs were running in parallel and the cache had been fully used to 50G and free disk space was 307G so that we can proably increase this limit. I selected 140G for now.

Actions

Also available in: Atom PDF