QA (public) » openQA Project (public)

Some options I've found:

1. Implement local caching within os-autoinst as described in comments of #60272
   1. Manual work for us; especially handling concurrent accesses could be a bit of an effort
   2. We can make it behave exactly like we want it to and integrate it tightly into our existing logic for cloning a specific revision
   3. Use GIT_ALTERNATE_OBJECT_DIRECTORIES as mentioned on Reddit
      1. See official documentation
      2. I'll first have to look into this. Maybe using the environment variable directly allows for an implementation that is more transparent and with less effort on our side.
         1. No new objects will be written into the dirs specified via GIT_ALTERNATE_OBJECT_DIRECTORIES so the dir needed to be populated before which is likely not better than doing a checkout before and then invoking git worktree … or git clone --reference … --dissociate or git clone --share ….
   4. I also tried GIT_COMMON_DIR but couldn't make it work.
   5. Steam further ideas from the projects mentioned in next points
   6. Draft implementation: https://github.com/os-autoinst/os-autoinst/pull/2446
2. Use gitcache
   1. We just need to put the tool in path and configure it via some environment variable
   2. Unclear whether our existing logic for cloning a specific version will run into problems
   3. Very small project by a single author; not packaged yet
      1. The README is nevertheless extensive.
      2. The last update is from 2 month ago so it is still actively developed.
   4. Even contains rudimentary cleanup functionality
3. Create mirrors on a GitLab/Gitea instance and just clone from there
   1. It requires additional infrastructure on our side.
   2. One needed to create the required mirrors upfront.
   3. It could be made transparently like this: [url "https://git.oqa.prg2.suse.org/"] insteadOf = https://github.example.com/
   4. There will be a sync interval so AC2 would technically not be fulfilled.
4. REJECTED: Use git-cache
   1. Requires a server process but it is transparently
   2. Last update was 10 years ago
   3. Not packaged yet
5. REJECTED: Use goblet
   1. Last update was 3 years ago but at least an *un*official project by Google
   2. Not packaged yet
   3. A Go library; we needed to integrate it
6. REJECTED: Use git-cache
   1. Last update was 6 years ago
   2. No transparent usage
7. REJECTED: Use git-cache-http-server
   1. Last update was 4 years ago
   2. Written in Haxe
8. REJECTED: Use GHES repository cache
   1. GHES stands for GitHub Enterprise Server so this is not what we want.

The options are sorted in the order I encountered them during my search. I marked options as REJECTED that don't seem very promising for our use case or that can be considered unmaintained.

While my change helps with any kind of Git cloning done by os-autoinst the main intention was actually to help with cloning wheel repositories - at least the ticket title reads like that. So how does openQA come into play here?

Even if we consider tests/needles I'm not sure what you're asking for here. So far the web UI only has one checkout per test/needle repo anyway. Do you want os-autoinst to make use of that existing checkout (when it is synced via rsync on the worker host)? I think this would be possible although I'd have to check how to let the worker best specify the path of the rsynced repo.

mkittler wrote in #note-14:

While my change helps with any kind of Git cloning done by os-autoinst the main intention was actually to help with cloning wheel repositories - at least the ticket title reads like that. So how does openQA come into play here?

Yes, you are right. But as we use the same git clone approach in both cases there is a relation

Even if we consider tests/needles I'm not sure what you're asking for here. So far the web UI only has one checkout per test/needle repo anyway. Do you want os-autoinst to make use of that existing checkout (when it is synced via rsync on the worker host)? I think this would be possible although I'd have to check how to let the worker best specify the path of the rsynced repo.

Yes, something like that but as you stated it's not necessarily in scope for this ticket

I created a branch of the openSUSE test distribution that let's it depend on a wheel and uses it (just a simple use …; in isosize). Then I cloned an arbitrary job with the isosize module to test my changes: script/openqa-clone-job … --host http://localhost:9526 https://openqa.opensuse.org/tests/3883324 CASEDIR=https://github.com/Martchus/os-autoinst-distri-opensuse.git#wheels-test

With a few adjustments I have already pushed on https://github.com/os-autoinst/os-autoinst/pull/2446 it works now. Both, the my distribution fork and the wheel code are first cloned as bare repositories (only one time; on subsequent runs only the required branch/revision is fetched) and then the working tree is checked out from there.

All I had to do to configure this was putting GIT_CACHE_DIR = /hdd/openqa-devel/openqa/cache/git in workers.ini (right next to CACHEDIRECTORY = /hdd/openqa-devel/openqa/worker/cache but of course the configuration of the worker class is completely independent from the Git caching at this point¹).

Note that here the worker cache wouldn't be useful at all because the test code was on my own fork and the wheels code is not covered by the cache service anyway.