action #53912
open
Use available space for caching on each worker for increased performance
Added by okurz almost 6 years ago.
Updated about 5 years ago.
Category:
Feature requests
Description
I realized that by default we (still) limit our openQA worker cache size to the default on most if not all o3 and osd workers however much more space can be available. Probably we can save quite some downloading and increase performance (faster starting jobs and less load on webui host) if we increase the cache size depending on what is available on the workers.
- Status changed from In Progress to Feedback
I configured openqaworker4.o.o with 350 GB instead of the default 50 GB. Monitoring.
filesystem ran full so I guess we need to leave some headroom. Configured for 300GB now and restarted about 300 incomplete jobs :(
Discussed with nsinger. We had the idea for the openqa cacheservice to dynamically look for df instead of a fixed space usage, e.g. use a negative number to denote who much space should be left free instead of how much to occupy.
- Project changed from openQA Infrastructure (public) to openQA Project (public)
- Category set to Feature requests
- Status changed from Feedback to New
- Assignee deleted (
okurz)
if cache would be on its own partition, it could fill it. I don't think that's generally the case
- Priority changed from Normal to Low
- Target version set to future
on aarch64.o.o I increased cache limit from default 50G to 140G as I observed at a time when 16 test jobs were running in parallel and the cache had been fully used to 50G and free disk space was 307G so that we can proably increase this limit. I selected 140G for now.
Also available in: Atom
PDF