action #138029
closed
coordination #154777: [saga][epic] Shareable os-autoinst and test distribution plugins
coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributions
[research][timeboxed:10h] How to cache "wheel" repositories which are stored on github size:M
Added by okurz about 1 year ago.
Updated 11 months ago.
Category:
Feature requests
Description
Motivation¶
Whilst a wheels.yaml is in place a request to GitHub is made for each reference to a GitHub-hosted repo in that file every time a job is prepared. This can amount to several hundreds of attempts daily even with a single repo. This is why we should look into caching wheel repositories, i.e. tiny git repositories, which are commonly stored on github. So I suggest we research a bit what are current best practices how to handle that. Maybe there are generic solutions and we don't want to reinvent the wheel.
Acceptance criteria¶
- AC1: When there are no changes to a wheel repository we know how openQA tests on workers could access the content of wheels from a location within the openQA infrastructure without needing to reclone from github
- AC2: When there are changes to a wheel repository we know how openQA tests would be able to access to most recent code version from wheel repos
Suggestions¶
- Read #60272
- Conduct a web research how to "cache github repositories"
- Look into "git worktrees" for repositories and branches, see #60272
- Look into how git uses efficient hardlinks when cloning locally, e.g. from a cache directory to the pool of the openQA worker
- Look into the existing caching mechanisms used for needles and test distributions (rsync)
- Optional: Build a prototype
Out of scope¶
- Manage storage capacity long-term / clean-up service
- Copied from action #118903: Repositories for wheels should be cached added
- Tags deleted (
infra)
- Target version changed from Tools - Next to Ready
- Tags set to collaborative-session
Perhaps worth considering for our Thursday afternoon session - note that it hasn't been estimated yet
- Subject changed from [research][timeboxed:10h] How to cache "wheel" repositories which are stored on github to [research][timeboxed:10h] How to cache "wheel" repositories which are stored on github size:S
- Description updated (diff)
- Status changed from New to Workable
- Tags deleted (
collaborative-session)
- Description updated (diff)
@livdywan seems like something horribly went wrong when you updated the ticket as the whole description vanished now. I put back the original description but that means we might need to estimate again. I also removed the "collaborative-session" tag as IMHO this research task would be a very bad candidate for a collaborative session when the most important part is researching yourself.
- Subject changed from [research][timeboxed:10h] How to cache "wheel" repositories which are stored on github size:S to [research][timeboxed:10h] How to cache "wheel" repositories which are stored on github
let's plan for re-estimation
- Subject changed from [research][timeboxed:10h] How to cache "wheel" repositories which are stored on github to [research][timeboxed:10h] How to cache "wheel" repositories which are stored on github size:M
- Description updated (diff)
- Priority changed from Normal to High
- Status changed from Workable to In Progress
- Assignee set to mkittler
Some options I've found:
- Implement local caching within os-autoinst as described in comments of #60272
- Manual work for us; especially handling concurrent accesses could be a bit of an effort
- We can make it behave exactly like we want it to and integrate it tightly into our existing logic for cloning a specific revision
- Use
GIT_ALTERNATE_OBJECT_DIRECTORIES as mentioned on Reddit
- See official documentation
- I'll first have to look into this. Maybe using the environment variable directly allows for an implementation that is more transparent and with less effort on our side.
- No new objects will be written into the dirs specified via
GIT_ALTERNATE_OBJECT_DIRECTORIES so the dir needed to be populated before which is likely not better than doing a checkout before and then invoking git worktree … or git clone --reference … --dissociate or git clone --share ….
- I also tried
GIT_COMMON_DIR but couldn't make it work.
- Steam further ideas from the projects mentioned in next points
- Draft implementation: https://github.com/os-autoinst/os-autoinst/pull/2446
- Use gitcache
- We just need to put the tool in path and configure it via some environment variable
- Unclear whether our existing logic for cloning a specific version will run into problems
- Very small project by a single author; not packaged yet
- The README is nevertheless extensive.
- The last update is from 2 month ago so it is still actively developed.
- Even contains rudimentary cleanup functionality
- Create mirrors on a GitLab/Gitea instance and just clone from there
- It requires additional infrastructure on our side.
- One needed to create the required mirrors upfront.
- It could be made transparently like this:
[url "https://git.oqa.prg2.suse.org/"] insteadOf = https://github.example.com/
- There will be a sync interval so AC2 would technically not be fulfilled.
- REJECTED: Use git-cache
- Requires a server process but it is transparently
- Last update was 10 years ago
- Not packaged yet
- REJECTED: Use goblet
- Last update was 3 years ago but at least an *un*official project by Google
- Not packaged yet
- A Go library; we needed to integrate it
- REJECTED: Use git-cache
- Last update was 6 years ago
- No transparent usage
- REJECTED: Use git-cache-http-server
- Last update was 4 years ago
- Written in Haxe
- REJECTED: Use GHES repository cache
- GHES stands for GitHub Enterprise Server so this is not what we want.
The options are sorted in the order I encountered them during my search. I marked options as REJECTED that don't seem very promising for our use case or that can be considered unmaintained.
- Due date set to 2024-02-01
Setting due date based on mean cycle time of SUSE QE Tools
- Status changed from In Progress to Feedback
- Status changed from Feedback to In Progress
When we think about how to integrate with openQA I am considering some common use cases:
- Two jobs, one for os-autoinst-distri-openQA and another for os-autoinst-distri-opensuse
- Two jobs, one for os-autoinst-distri-opensuse "latest master" and another "arbitrary commit"
Your approach in https://github.com/os-autoinst/os-autoinst/pull/2446 looks resonable for os-autoinst however this does not answer questions how to integrate with openQA where we have the webUI that would need access to git repos as well as the caching services on the workers. So to stick to the time allocated in this timeboxed ticket I suggest to focus on integrating with openQA and not right now continue with https://github.com/os-autoinst/os-autoinst/pull/2446
While my change helps with any kind of Git cloning done by os-autoinst the main intention was actually to help with cloning wheel repositories - at least the ticket title reads like that. So how does openQA come into play here?
Even if we consider tests/needles I'm not sure what you're asking for here. So far the web UI only has one checkout per test/needle repo anyway. Do you want os-autoinst to make use of that existing checkout (when it is synced via rsync on the worker host)? I think this would be possible although I'd have to check how to let the worker best specify the path of the rsynced repo.
mkittler wrote in #note-14:
While my change helps with any kind of Git cloning done by os-autoinst the main intention was actually to help with cloning wheel repositories - at least the ticket title reads like that. So how does openQA come into play here?
Yes, you are right. But as we use the same git clone approach in both cases there is a relation
Even if we consider tests/needles I'm not sure what you're asking for here. So far the web UI only has one checkout per test/needle repo anyway. Do you want os-autoinst to make use of that existing checkout (when it is synced via rsync on the worker host)? I think this would be possible although I'd have to check how to let the worker best specify the path of the rsynced repo.
Yes, something like that but as you stated it's not necessarily in scope for this ticket
I created a branch of the openSUSE test distribution that let's it depend on a wheel and uses it (just a simple use …; in isosize). Then I cloned an arbitrary job with the isosize module to test my changes: script/openqa-clone-job … --host http://localhost:9526 https://openqa.opensuse.org/tests/3883324 CASEDIR=https://github.com/Martchus/os-autoinst-distri-opensuse.git#wheels-test
With a few adjustments I have already pushed on https://github.com/os-autoinst/os-autoinst/pull/2446 it works now. Both, the my distribution fork and the wheel code are first cloned as bare repositories (only one time; on subsequent runs only the required branch/revision is fetched) and then the working tree is checked out from there.
All I had to do to configure this was putting GIT_CACHE_DIR = /hdd/openqa-devel/openqa/cache/git in workers.ini (right next to CACHEDIRECTORY = /hdd/openqa-devel/openqa/worker/cache but of course the configuration of the worker class is completely independent from the Git caching at this point¹).
Note that here the worker cache wouldn't be useful at all because the test code was on my own fork and the wheels code is not covered by the cache service anyway.
- Status changed from In Progress to Feedback
- Copied to action #154156: [spike][timeboxed:10h] Cache test distributions from git on production size:S added
- Status changed from Feedback to Resolved
The PR has been merged and we have a follow-up on the backlog. So I'm considering this ticket resolved.
- Due date deleted (
2024-02-01)
Also available in: Atom
PDF