QA (public) &raquo; openQA Project (public)

Category:

Feature requests

Target version:

Ready

Start date:

2024-06-12

Due date:

% Done:

Estimated time:

Description

Motivation¶

fetchneedles is a script provided within the openQA repo and we call it on o3+osd in a cron job every minute to keep test distribution checkouts updated but it's not well documented, can interfer with openQA internal git handling and (probably) still needs initial checkout of test distributions. Let's see what else would be necessary to use the new openQA internal support for checking out git test distributions if they don't exist yet.

Goals¶

G1: A migration plan for existing test distribution on o3 exists so that /var/lib/openqa/share/tests/* would not need to be updated by fetchneedles, e.g. on o3 if checkouts already exist
G2: tests would still pass consistently
G3: test details and source code views would still show content as expected

Suggestions¶

Apply the approach from #156922 for other test distributions at best in a local reproduction environment but if you are careful or a daredevil you could do it in production :)
Try where to update checkouts e.g. in an openQA minion job
If you don't know the movie "Despicable me" then watch that first but don't count it as part of the 10h timebox :)

Out of scope¶

Doing any kind of initial checkout if git working copies do not exist yet

Related issues 6 (0 open — 6 closed)

Related to openQA Infrastructure (public) - action #164895: o3 had corrupted needles git repo, lost uncommitted needles between 2024-07-31 and 2024-08-02

Resolved

2024-08-02

Copied from openQA Project (public) - action #156922: Run os-autoinst-distri-openQA directly from git without anything related in o3:/var/lib/openqa/share/tests size:S

Resolved

dheidler

Copied to openQA Project (public) - action #164883: Use same minion guard for save_needle, delete_needles and git_clone size:S

Resolved

Copied to openQA Project (public) - action #164886: Use OpenQA::Git for all our git wrappers size:S

Resolved

robert.richardson

Copied to openQA Project (public) - action #164889: Ensure git repos cloned by minions are cleaned up regularly size:S

Resolved

Copied to openQA Project (public) - action #164898: Replace fetchneedles with a minion job for the regular update of git repos size:M

Resolved

Updated by okurz 6 months ago

Copied from action #156922: Run os-autoinst-distri-openQA directly from git without anything related in o3:/var/lib/openqa/share/tests size:S added

Actions

Updated by okurz 6 months ago

Status changed from New to Blocked
Assignee set to okurz

#161771

Actions

Updated by okurz 6 months ago

Status changed from Blocked to New
Assignee deleted (~~okurz~~)

Actions

Updated by okurz 6 months ago

Priority changed from Normal to High

Actions

Updated by livdywan 5 months ago

I feel like it's fair to say we didn't get to this at the workshop. Let's estimate it tomorrow.

Actions

Updated by okurz 5 months ago

Subject changed from [timeboxed:10h][spike] Let openQA keep test distribution checkouts up to date without needing fetchneedles to [timeboxed:10h][spike] Let openQA keep test distribution checkouts up to date without needing fetchneedles size:S
Description updated (diff)
Status changed from New to Workable

Actions

Updated by mkittler 5 months ago

Assignee set to mkittler

Actions

Updated by mkittler 5 months ago

Assignee deleted (~~mkittler~~)

Actions

Updated by livdywan 5 months ago

I feel like this might not be picked over other High tickets atm because it's a new feature? Maybe it should have lower priority? I'm asking in the team chat once more.

Actions

#10

Updated by ybonatakis 5 months ago

Assignee set to ybonatakis

Actions

#11

Updated by ybonatakis 5 months ago

Status changed from Workable to In Progress

Actions

#12

Updated by livdywan 5 months ago

https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/187 was just merged. Not sure what this is about.

Actions

#13

Updated by ybonatakis 5 months ago

I did the following experiment:

I clone a openqa-to-openqa job with git url in CASEDIR+NEEDLES_DIR. tests pulled the repos in /var/lib/openqa/share/tests/openqa and run successfully
created a PR https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/187 and merged.
then I restarted the cloned job in my local setup and it run against the new changes. However change was not in /var/lib/openqa/share/tests/openqa

Actions

#14

Updated by ybonatakis 5 months ago

livdywan wrote in #note-12:

https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/187 was just merged. Not sure what this is about.

I have to revert it.

Actions

#15

Updated by ybonatakis 5 months ago

ybonatakis wrote in #note-14:

livdywan wrote in #note-12:

https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/187 was just merged. Not sure what this is about.

I have to revert it.

Actually someone already did it https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/188

Actions

#16

Updated by openqa_review 5 months ago

Due date set to 2024-08-13

Setting due date based on mean cycle time of SUSE QE Tools

Actions

[0] https://metacpan.org/dist/Minion/view/lib/Minion/Guide.pod

#17

Updated by ybonatakis 5 months ago

Status changed from In Progress to Feedback

I cant say I have any progress.
I start looking at the fetchneedles when I realised that this is not the objective of the ticket.

I used the registry.opensuse.org/devel/openqa/containers/openqa-single-instance to run experiments.
Following the suggestion and try to understand how minion[0] works.
then I reviewed https://github.com/os-autoinst/openQA/commit/a38165bd7576d7ebba5497478636e176fb83daa1 but I dont understand how does it work and seems I cant figure it out how and what to expect inside the container. In the container the git_auto_clone is enabled.
When I cloned a openqa2openqa job the /var/lib/openqa/share/tests/openqa got updated

[info] Ignoring invalid group {"id":"24"} when creating new job 1
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git ls-remote --symref https://github.com/os-autoinst/os-autoinst-distri-openQA HEAD
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git clone https://github.com/os-autoinst/os-autoinst-distri-openQA /var/lib/openqa/share/tests/openqa
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa remote get-url origin
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa branch --show-current
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git -C /var/lib/openqa/share/tests/openqa fetch origin master
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa reset --hard origin/master
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git ls-remote --symref https://github.com/os-autoinst/os-autoinst-needles-openQA HEAD
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git clone https://github.com/os-autoinst/os-autoinst-needles-openQA /var/lib/openqa/share/tests/openqa/needles
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa/needles remote get-url origin
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa/needles branch --show-current
[info] cmd returned 0
[info] Running cmd: env GIT_SSH_COMMAND="ssh -oBatchMode=yes" git -C /var/lib/openqa/share/tests/openqa/needles fetch origin master
[info] cmd returned 0
[info] Running cmd: git -C /var/lib/openqa/share/tests/openqa/needles reset --hard origin/master
[info] cmd returned 0

Take a look at https://github.com/os-autoinst/openQA/commit/a38165bd7576d7ebba5497478636e176fb83daa1 too.

Actions

#18

Updated by tinita 5 months ago

Status changed from Feedback to In Progress
Assignee changed from ybonatakis to tinita

Actions

#19

Updated by tinita 5 months ago

It looks like we can just use the OpenQA::Task::Git::Clone minion job for this as well, with some changes.
https://github.com/os-autoinst/openQA/commit/a38165bd7576d7ebba5497478636e176fb83daa1
I made some minor adjustments, so that instead of passing

/path/to/share/tests/foo: git-url
/path/to/share/tests/foo/needles: git-url

you can leave the values empty (when no CASEDIR / NEEDLES_DIR is set):

/path/to/share/tests/foo: null
/path/to/share/tests/foo/needles: null

and then it will only do a git fetch etc. if the clone already exists in the given path.

One thing to consider: If people are working with a git repo directly in /path/to/share/tests/foo (or a symlink to their clone), then the minion job should probably not do anything in there. Most commands are harmless, except the git reset --hard.
Question is, how to find out if the git clone is really just a clone or the actual developer git. Checking for a dirty status might not be enough.

Actions

#20

Updated by tinita 5 months ago

https://github.com/os-autoinst/openQA/pull/5808 Proof of Concept: Support git_auto_clone for empty CASDIR/NEEDLES_DIR

To test:

openqa.ini:

[scm git]
git_auto_clone = yes

Clone https://github.com/os-autoinst/os-autoinst-distri-openQA into /path/to/openqa/share/tests/openqa.
Start openqa daemons including gru.
Then run

openqa-clone-job --host http://localhost:9526 https://openqa.opensuse.org/tests/4360561 _TRIGGER_JOB_DONE_HOOK=0 SCHEDULE=tests/install/boot CASEDIR=

In the gru log you should see all executed git commands.

Editing a file in the checkout results in the minion job not updating the repo because of a dirty status
An empty CASEDIR/NEEDLES_DIR will auto clone, but explicitly specifying a local path will not

Todos

Conflicts because of save_needle need to be handled
Related: We regularly have problems with the index file when running save_needle: https://openqa.opensuse.org/minion/jobs?task=save_needle&state=&queue=¬e=

% git status
fatal: .git/index: index file smaller than expected

save_needle code should be reviewed. Maybe code can be reused. OpenQA::Task::Needle::Save uses OpenQA::Git, OpenQA::Task::Git::Clone doesn't
How to prevent updating a user's working git clone?

Since we currently haven't git_auto_clone enabled on o3/osd, we can roll out the feature and enable it and can monitor.
Important: fetchneedles needs to be disabled then.

Actions

#21

Updated by tinita 5 months ago

We could already add the git gc and git diff-index to the current code.

Actions

#22

Updated by tinita 4 months ago

Copied to action #164883: Use same minion guard for save_needle, delete_needles and git_clone size:S added

Actions

#23

Updated by tinita 4 months ago

Copied to action #164886: Use OpenQA::Git for all our git wrappers size:S added

Actions

#24

Updated by tinita 4 months ago

Copied to action #164889: Ensure git repos cloned by minions are cleaned up regularly size:S added

Actions

#25

Updated by tinita 4 months ago

Related to action #164895: o3 had corrupted needles git repo, lost uncommitted needles between 2024-07-31 and 2024-08-02 added

Actions

#26

Updated by tinita 4 months ago

Copied to action #164898: Replace fetchneedles with a minion job for the regular update of git repos size:M added

Actions

#27

Updated by tinita 4 months ago

Status changed from In Progress to Feedback

I created #164898 as the main followup and 3 others as smaller preparation tickets that should be done before.
For the problem with the git index in the needle checkout see #164895

Actions

#28

Updated by tinita 4 months ago

Status changed from Feedback to Resolved

No comments, so I guess this is resolved

Actions