QA » openQA Project

Hmm.. I have couple of questions.
Why is the caching enabled only when share directory is missing? What about using different directory name which is possible now?
Is there a problem obtaining files also through NFS? (I imagine random I/O would drop once caching is used on workers). Why limit it to HTTP only? (Is it just because openQA can already do that?)
Are we space constrained on worker machine so we can't sync whole share directory to all workers?

During hackweek I read about this one https://github.com/chrislusf/seaweedfs which is an implementation of FaceBook`s Haystack ( www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf ). True, intended for smaller files, but in large numbers. Problem is that I think it is still experimental.

How about web seeding for bittorrent downloads between workers? Would help during the peak when new product is published IMO.

We don't need any fancy stuff and none of the problems bittorrent solves is something we have. We need to seperate downloading assets (and needles) from actually using it - to avoid IO bottle necks on the VM affecting the test.

And to give you a number: our workers have 300GB of NVME and our VM has 6TB of data. So no, we can't - and no, we want to sync all of it. And dropping NFS completely in favor of using a worker directory as sparse copy (do not call it cache - it's misleading) has the advantage of having unique filenames all over the stack. Whatever is /var/lib/openqa/share/X is CACHEDIR/X on the worker.

But if we mix /var/lib/openqa/share and CACHEDIR, we get in trouble - as the needle (mis-caching) just too prominently demonstrated. So I'm against any half-baked solution like copying off NFS.

Oh - and before that's get lost: I'm also against researching on experimental file systems at this point. We have a simple problem and we need a simple solution for it - ASAP.

To test locally, i'm using for rsync

gid = nobody
uid = nobody
read only = true
use chroot = true
transfer logging = true
log format = %h %o %f %l %b
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
slp refresh = 300
use slp = false
[openqa-tests]
path = /var/lib/openqa/share/tests
comment = OpenQA Test Distributions

Starting the server currently with rcrsyncd start

I've set up rsyncd on openqa.suse.de with a similiar config file as santi provided

rsync -av rsync://openqa.suse.de/tests . is working 

TODO: put it into a salt recipe (nsinger and me will work on this)

I finished up the PR, but my faith into the cache class is small. The locking looks broken IMO, because between the read_db and the write_db is a full download, so another worker will have overwritten the content in between. The locking needs to be around atomic operations like inserts and deletes.

And we need a size limit not an asset limit. For now we will survive - as our disks are hopefully large enough for 50 ISOs.

The cache.db needs to move to CACHEDIR. I tried to add it to the checklist, but failed

Since the cleanup of the cache is broken and workers run out of space I've added following cronjob to all workers:

0       4       *       *       *       if [ $(ls -l /var/lib/openqa/cache/* | wc -l) -gt 50 ]; then rcopenqa-worker stop && rm -rf /var/lib/openqa/cache/* && rcopenqa-worker start ;fi

Current commit broke more things on the worker so we had to apply a hotfix on them.
The file "/usr/share/openqa/lib/OpenQA/Worker/Cache.pm" (Line 58 for the time of writing) now contains:

$db_file = catdir($location, 'cache.db');

Instead of: $db_file = catdir($location, $db_file);

https://github.com/os-autoinst/openQA/pull/1279 has the hotfix

nicksinger wrote:

Current commit broke more things on the worker so we had to apply a hotfix on them.
The file "/usr/share/openqa/lib/OpenQA/Worker/Cache.pm" (Line 58 for the time of writing) now contains:

$db_file = catdir($location, 'cache.db');

Instead of: $db_file = catdir($location, $db_file);

Second hotfix for the caching: limit the number of assets.

The following gist has a new hotfix just to ensure that workers will have in cache directory only the elements that are present in the cache.db file

https://gist.github.com/foursixnine/623397e6f3231c3dc8a0c435d874e5df

please take a look into #17812 which is blocked by this issue.

https://openqa.suse.de/tests/905351 shows an issue which I consider serious: https://openqa.suse.de/tests/905351/modules/mediacheck/steps/1/src shows in lines 28 and following my workaround that I wanted to try but https://openqa.suse.de/tests/905351/file/autoinst-log.txt shows that the corresponding part was not executed and the code corresponds to the old source so the content on the worker was not updated properly at start of the test.

Sorry, I was wrong. I think an if just evaluated to false here.