Project

General

Profile

Actions

action #154237

closed

coordination #154777: [saga][epic] Shareable os-autoinst and test distribution plugins

coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributions

[spike][timeboxed:10h] Ensure the worker cache doesn't duplicate git caching of test distributions on o3 size:S

Added by okurz 5 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Motivation

As part of #138029 support for git caching was included in os-autoinst covering test distributions as well as wheel repository. We can now look into caching test distributions from git fully enabled on a production openQA instance but also considering the worker cache service, e.g. circumvent the worker cache service or actually make use of it.

Acceptance criteria

  • AC1: There is no redundant syncing in the worker cache service, e.g. no cloning from github but the worker cache syncs test data that is never used

Suggestions

  • See what has been done in #154156
  • Consider the impact of the worker cache service, e.g. take a look into logs of affected openQA jobs and see how caching and git caching interact or not interact
  • One simple approach could be to simply not trigger syncing the worker test distribution directory caching in openQA jobs that have GIT_CACHE_DIR+CASEDIR+NEEDLESDIR includes a git URL set
  • Checkout the documentation of the worker cache to get started: https://open.qa/docs/#asset-caching
    • Consider adding something to the docs to make "worker cache" more discoverable
    • Add GIT_CACHE_DIR to the docs

Out of scope

  • Manage storage capacity long-term / clean-up

Related issues 2 (0 open2 closed)

Copied from openQA Project - action #154156: [spike][timeboxed:10h] Cache test distributions from git on production size:SResolvedmkittler

Actions
Copied to openQA Project - action #154240: Ensure cloning openQA jobs with GIT_CACHE_DIR works in usual use casesResolvedmkittler2024-01-25

Actions
Actions #1

Updated by okurz 5 months ago

  • Copied from action #154156: [spike][timeboxed:10h] Cache test distributions from git on production size:S added
Actions #2

Updated by okurz 5 months ago

  • Copied to action #154240: Ensure cloning openQA jobs with GIT_CACHE_DIR works in usual use cases added
Actions #3

Updated by okurz 4 months ago

  • Target version changed from Tools - Next to Ready
Actions #4

Updated by okurz 4 months ago

  • Priority changed from Normal to Low
Actions #5

Updated by livdywan 4 months ago

  • Subject changed from [spike][timeboxed:10h] Ensure the worker cache service is not calling redundant syncing for jobs using cached git cloned test and/or needle distributions to [spike][timeboxed:10h] Ensure the worker cache doesn't duplicate git caching of test distributions on o3 size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #6

Updated by mkittler 3 months ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions #7

Updated by mkittler 3 months ago

  • Status changed from In Progress to Feedback
Actions #8

Updated by okurz 3 months ago

  • Due date set to 2024-03-08
Actions #10

Updated by mkittler 3 months ago

  • Status changed from Feedback to Resolved

The follow-up PR https://github.com/os-autoinst/openQA/pull/5499 was merged yesterday and hasn't caused any big regressions anymore so far. So I consider this ticket resolved.

Actions #11

Updated by okurz 2 months ago

  • Due date deleted (2024-03-08)
Actions

Also available in: Atom PDF