Project

General

Profile

Actions

action #169261

closed

coordination #58184: [saga][epic][use case] full version control awareness within openQA

coordination #152847: [epic] version control awareness within openQA for test distributions

Reduce scope of minion locks size:S

Added by dheidler about 1 month ago. Updated 4 days ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2024-10-17
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Followup to #168400. In #164898 we replaced the automatic git fetch cron (fetchneedles) with minion jobs.

There are three possible improvements.

  • Resolve symlinks before enqueuing => ALREADY DONE in #168400
  • Avoid fetch when we can check locally for existing ref DONE

That leaves missing:

Reduce scope of minion locks

This is more complicated and maybe too much effort for little benefit, but:

In some cases we don't need to call any git commands that could conflict with other commands, e.g. just looking up the remote url.
Or the case mentioned in the previous section would be another command that would not interfere with other git tasks.
Any command that does not change anything.
For those we don't need a lock.

Only at that point where we see we have to do a fetch, we have to get the lock.

This reduces the chances of other tasks having to retry because it couldn't get a lock.

It's just problematic because if a task has two paths to update, and you only realize for the second one that you need the lock, you would have to retry, but you don't want to repeat the commands for the first path.
The notes field could act as a kind of state which of the paths have already been done.

Acceptance criteria

  • AC1: Minion locks related to Git repositories are held only while they are actually needed so parallel execution of code in Minion jobs is possible more often (and e.g. tasks for saving needles are blocked less likely)

Suggestions

  • Abort if the code becomes unreasonably complicated (so it is not worth it)
  • Use the notes field to save state between calls (to not do a previous step a second time)

Related issues 1 (0 open1 closed)

Copied from openQA Project (public) - action #168400: Improve locking scope of git_clone tasks size:SResolveddheidler2024-10-17

Actions
Actions #1

Updated by dheidler about 1 month ago

  • Copied from action #168400: Improve locking scope of git_clone tasks size:S added
Actions #2

Updated by dheidler about 1 month ago

  • Description updated (diff)
Actions #3

Updated by okurz about 1 month ago

  • Description updated (diff)
  • Due date deleted (2024-11-13)
  • Target version set to Tools - Next
Actions #4

Updated by livdywan 27 days ago

  • Subject changed from Reduce scope of minion locks to Reduce scope of minion locks size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #5

Updated by okurz 12 days ago

  • Target version changed from Tools - Next to Ready
Actions #6

Updated by dheidler 4 days ago

Further thinking about this lets me believe that this is not that easy or feasible at all.
When there is no lock being used for read operations, there is no guarantee that there is noone else writing to the location that is being read at that point of time.

Actions #7

Updated by tinita 4 days ago

I think the read operations we have are fine even if there is a write operation. git should handle that, and I can't think of a set of git writing operations that would influence the results of our reading operations in a bad way.

But I still think the benefit of this is low compared to the effort. I would put this in the future queue.

Actions #8

Updated by okurz 4 days ago

  • Status changed from Workable to Rejected
  • Assignee set to okurz

fine

Actions

Also available in: Atom PDF