Project

General

Profile

Actions

action #168400

closed

coordination #58184: [saga][epic][use case] full version control awareness within openQA

coordination #152847: [epic] version control awareness within openQA for test distributions

Improve locking scope of git_clone tasks size:S

Added by tinita about 2 months ago. Updated 26 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2024-10-17
Due date:
% Done:

0%

Estimated time:

Description

Motivation

In #164898 we replaced the automatic git fetch cron (fetchneedles) with minion jobs.

There are three possible improvements.

Resolve symlinks before enqueuing

Compare the following two minion jobs:

args:
- /var/lib/openqa/share/tests/microos: ~
  /var/lib/openqa/share/tests/microos/products/microos/needles: ~
args:
- /var/lib/openqa/share/tests/opensuse: ~
  /var/lib/openqa/share/tests/opensuse/products/opensuse/needles: ~

microos is a symlink to opensuse. The path resolving happens inside the minion job before getting the lock, but we can avoid creating the job by resolving the symlinks before enqueuing. That way we would profit from the feature to not repeat the same minion job in less than one minute.

Avoid fetch when we can check locally for existing ref DONE

For the case when we get a certain git sha in CASEDIR, we currently blindly do a fetch, but we could avoid the fetch and first check locally if the ref is there, which might be likely (for shas only. For branches we must always fetch).
For example our investigation jobs usually specify a sha which is already present.
Avoiding git commands needing network is good.

# pseudo code
if ($ref !~ m/[^a-f0-9]/) {
    # sha
    git rev-parse  --verify $ref
    # if it's not there, fetch
}

Reduce scope of minion locks

This is more complicated and maybe too much effort for little benefit, but:

In some cases we don't need to call any git commands that could conflict with other commands, e.g. just looking up the remote url.
Or the case mentioned in the previous section would be another command that would not interfere with other git tasks.
Any command that does not change anything.
For those we don't need a lock.

Only at that point where we see we have to do a fetch, we have to get the lock.

This reduces the chances of other tasks having to retry because it couldn't get a lock.

It's just problematic because if a task has two paths to update, and you only realize for the second one that you need the lock, you would have to retry, but you don't want to repeat the commands for the first path.
The notes field could act as a kind of state which of the paths have already been done.

Acceptance criteria

  • AC1: Minion locks used for Git tasks take symlinking of case/needle directories into account.

Related issues 2 (0 open2 closed)

Copied from openQA Project (public) - action #164898: Replace fetchneedles with a minion job for the regular update of git repos size:MResolvedtinita

Actions
Copied to openQA Project (public) - action #169261: Reduce scope of minion locks size:SRejectedokurz2024-10-17

Actions
Actions #1

Updated by tinita about 2 months ago

  • Copied from action #164898: Replace fetchneedles with a minion job for the regular update of git repos size:M added
Actions #2

Updated by okurz about 2 months ago

  • Due date deleted (2024-10-18)
  • Target version set to Tools - Next
Actions #3

Updated by okurz about 2 months ago

  • Target version changed from Tools - Next to Ready
Actions #4

Updated by livdywan about 2 months ago

  • Subject changed from Improve locking scope of git_clone tasks to Improve locking scope of git_clone tasks size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #5

Updated by tinita about 2 months ago

A draft for the second item already exists: https://github.com/os-autoinst/openQA/pull/6017 Do not git fetch if requested sha is already present

Actions #6

Updated by tinita about 2 months ago

Re

AC1: Minion locks used for Git tasks take symlinking of case/needle directories into account.

The minion locks already take symlinking into account. This is about resolving symlinks even before the minion job is created. The args of the job should contain already resolved paths.

Actions #7

Updated by tinita about 2 months ago

Why is there only one AC for the first item in the list?
This ticket should probably have been split into three.

Actions #8

Updated by dheidler about 1 month ago

  • Status changed from Workable to In Progress
  • Assignee set to dheidler
Actions #9

Updated by tinita about 1 month ago ยท Edited

Ready: https://github.com/os-autoinst/openQA/pull/6017 Do not git fetch if requested sha is already present

edit: merged

Actions #10

Updated by openqa_review about 1 month ago

  • Due date set to 2024-11-13

Setting due date based on mean cycle time of SUSE QE Tools

Actions #11

Updated by tinita about 1 month ago

  • Description updated (diff)
Actions #12

Updated by livdywan about 1 month ago

Reduce scope of minion locks

Discussed it briefly. If you find this is getting too much it's also fine to move to a separate follow-up ticket.

Actions #13

Updated by dheidler about 1 month ago

  • Status changed from In Progress to Feedback
Actions #14

Updated by dheidler about 1 month ago

Actions #15

Updated by dheidler about 1 month ago

  • Status changed from Feedback to Resolved

Created https://progress.opensuse.org/issues/169261 for reduction of scope of minion locks.

Actions #16

Updated by okurz 26 days ago

  • Due date deleted (2024-11-13)
Actions

Also available in: Atom PDF