action #162125
closedcoordination #58184: [saga][epic][use case] full version control awareness within openQA
coordination #152847: [epic] version control awareness within openQA for test distributions
[timeboxed:10h][spike] Let openQA keep test distribution checkouts up to date without needing fetchneedles size:S
Description
Motivation¶
fetchneedles is a script provided within the openQA repo and we call it on o3+osd in a cron job every minute to keep test distribution checkouts updated but it's not well documented, can interfer with openQA internal git handling and (probably) still needs initial checkout of test distributions. Let's see what else would be necessary to use the new openQA internal support for checking out git test distributions if they don't exist yet.
Goals¶
- G1: A migration plan for existing test distribution on o3 exists so that
/var/lib/openqa/share/tests/*
would not need to be updated by fetchneedles, e.g. on o3 if checkouts already exist - G2: tests would still pass consistently
- G3: test details and source code views would still show content as expected
Suggestions¶
- Apply the approach from #156922 for other test distributions at best in a local reproduction environment but if you are careful or a daredevil you could do it in production :)
- Try where to update checkouts e.g. in an openQA minion job
- If you don't know the movie "Despicable me" then watch that first but don't count it as part of the 10h timebox :)
Out of scope¶
- Doing any kind of initial checkout if git working copies do not exist yet
Updated by okurz 6 months ago
- Copied from action #156922: Run os-autoinst-distri-openQA directly from git without anything related in o3:/var/lib/openqa/share/tests size:S added
Updated by okurz 5 months ago
- Subject changed from [timeboxed:10h][spike] Let openQA keep test distribution checkouts up to date without needing fetchneedles to [timeboxed:10h][spike] Let openQA keep test distribution checkouts up to date without needing fetchneedles size:S
- Description updated (diff)
- Status changed from New to Workable
Updated by livdywan 5 months ago
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/187 was just merged. Not sure what this is about.
Updated by ybonatakis 5 months ago
I did the following experiment:
- I clone a openqa-to-openqa job with git url in CASEDIR+NEEDLES_DIR. tests pulled the repos in /var/lib/openqa/share/tests/openqa and run successfully
- created a PR https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/187 and merged.
- then I restarted the cloned job in my local setup and it run against the new changes. However change was not in /var/lib/openqa/share/tests/openqa
Updated by ybonatakis 5 months ago
livdywan wrote in #note-12:
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/187 was just merged. Not sure what this is about.
I have to revert it.
Updated by ybonatakis 5 months ago
ybonatakis wrote in #note-14:
livdywan wrote in #note-12:
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/187 was just merged. Not sure what this is about.
I have to revert it.
Actually someone already did it https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/188
Updated by openqa_review 5 months ago
- Due date set to 2024-08-13
Setting due date based on mean cycle time of SUSE QE Tools
Updated by ybonatakis 5 months ago
- Status changed from In Progress to Feedback
I cant say I have any progress.
I start looking at the fetchneedles when I realised that this is not the objective of the ticket.
I used the registry.opensuse.org/devel/openqa/containers/openqa-single-instance
to run experiments.
Following the suggestion and try to understand how minion[0] works.
then I reviewed https://github.com/os-autoinst/openQA/commit/a38165bd7576d7ebba5497478636e176fb83daa1 but I dont understand how does it work and seems I cant figure it out how and what to expect inside the container. In the container the git_auto_clone
is enabled.
When I cloned a openqa2openqa job the /var/lib/openqa/share/tests/openqa got updated
[info] Ignoring invalid group {"id":"24"} when creating new job 1
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git ls-remote --symref https://github.com/os-autoinst/os-autoinst-distri-openQA HEAD
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git clone https://github.com/os-autoinst/os-autoinst-distri-openQA /var/lib/openqa/share/tests/openqa
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa remote get-url origin
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa branch --show-current
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git -C /var/lib/openqa/share/tests/openqa fetch origin master
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa reset --hard origin/master
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git ls-remote --symref https://github.com/os-autoinst/os-autoinst-needles-openQA HEAD
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git clone https://github.com/os-autoinst/os-autoinst-needles-openQA /var/lib/openqa/share/tests/openqa/needles
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa/needles remote get-url origin
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa/needles branch --show-current
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git -C /var/lib/openqa/share/tests/openqa/needles fetch origin master
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa/needles reset --hard origin/master
[info] cmd returned 0
Take a look at https://github.com/os-autoinst/openQA/commit/a38165bd7576d7ebba5497478636e176fb83daa1 too.
[0] https://metacpan.org/dist/Minion/view/lib/Minion/Guide.pod
Updated by tinita 5 months ago
It looks like we can just use the OpenQA::Task::Git::Clone minion job for this as well, with some changes.
https://github.com/os-autoinst/openQA/commit/a38165bd7576d7ebba5497478636e176fb83daa1
I made some minor adjustments, so that instead of passing
/path/to/share/tests/foo: git-url
/path/to/share/tests/foo/needles: git-url
you can leave the values empty (when no CASEDIR / NEEDLES_DIR is set):
/path/to/share/tests/foo: null
/path/to/share/tests/foo/needles: null
and then it will only do a git fetch
etc. if the clone already exists in the given path.
One thing to consider: If people are working with a git repo directly in /path/to/share/tests/foo
(or a symlink to their clone), then the minion job should probably not do anything in there. Most commands are harmless, except the git reset --hard
.
Question is, how to find out if the git clone is really just a clone or the actual developer git. Checking for a dirty status might not be enough.
Updated by tinita 5 months ago
https://github.com/os-autoinst/openQA/pull/5808 Proof of Concept: Support git_auto_clone for empty CASDIR/NEEDLES_DIR
To test:
openqa.ini:
[scm git]
git_auto_clone = yes
Clone https://github.com/os-autoinst/os-autoinst-distri-openQA into /path/to/openqa/share/tests/openqa
.
Start openqa daemons including gru.
Then run
openqa-clone-job --host http://localhost:9526 https://openqa.opensuse.org/tests/4360561 _TRIGGER_JOB_DONE_HOOK=0 SCHEDULE=tests/install/boot CASEDIR=
In the gru log you should see all executed git commands.
- Editing a file in the checkout results in the minion job not updating the repo because of a dirty status
- An empty CASEDIR/NEEDLES_DIR will auto clone, but explicitly specifying a local path will not
Todos
- Conflicts because of save_needle need to be handled
- Related: We regularly have problems with the index file when running save_needle: https://openqa.opensuse.org/minion/jobs?task=save_needle&state=&queue=¬e=
% git status
fatal: .git/index: index file smaller than expected
- save_needle code should be reviewed. Maybe code can be reused. OpenQA::Task::Needle::Save uses OpenQA::Git, OpenQA::Task::Git::Clone doesn't
- How to prevent updating a user's working git clone?
Since we currently haven't git_auto_clone
enabled on o3/osd, we can roll out the feature and enable it and can monitor.
Important: fetchneedles needs to be disabled then.
Updated by tinita 4 months ago
- Copied to action #164883: Use same minion guard for save_needle, delete_needles and git_clone size:S added
Updated by tinita 4 months ago
- Copied to action #164886: Use OpenQA::Git for all our git wrappers size:S added
Updated by tinita 4 months ago
- Copied to action #164889: Ensure git repos cloned by minions are cleaned up regularly size:S added
Updated by tinita 4 months ago
- Related to action #164895: o3 had corrupted needles git repo, lost uncommitted needles between 2024-07-31 and 2024-08-02 added
Updated by tinita 4 months ago
- Copied to action #164898: Replace fetchneedles with a minion job for the regular update of git repos size:M added