Project

General

Profile

Actions

action #159171

closed

coordination #58184: [saga][epic][use case] full version control awareness within openQA

coordination #152847: [epic] version control awareness within openQA for test distributions

Create and maintain up to date version of test distri/needles for webui size:M

Added by dheidler 8 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2024-04-17
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Consider transient openQA instances like openQA-single-instance-container but also new instances where we do not want to rely on admins to manually put stuff into /var/lib/openqa/share/tests. For those we want to ensure that openQA automatically clones git repos specified as part of tests but also keeps those working copies up to date.

Acceptance criteria

  • AC1:
    • Given CASEDIR has a git repo and /var/lib/openqa/share/tests does not already have the corresponding distri checked out
    • When tests are created
    • Then openQA clones the test repo
  • AC2: Same as AC1 but for NEEDLE_DIR
  • AC3:
    • When new tests are created and repo already exists
    • Then the git repo is updated

Suggestions

The following describes something like a TOFU approach, where the first CASEDIR/NEEDLE_DIR for a DISTRI
is deemed to be the authoritative one in the future.

  • Job gets created
  • If job has CASEDIR / NEEDLE_DIR:
    • CASEDIR / NEEDLE_DIR is cloned or updated on the webui server
    • See HDD=1=http:// mechanism
    • Where to actually clone to? --> /var/lib/openqa/share/tests/DISTRI/products/DISTRI/needles
    • (- What to do with different forks? -> use https://git-scm.com/docs/git-worktree)
    • If there is not /var/lib/openqa/share/tests/DISTRI, clone CASEDIR to /var/lib/openqa/share/tests/DISTRI
    • If there is /var/lib/openqa/share/tests/DISTRI, only update it if git url is equal origin url in /var/lib/openqa/share/tests/DISTRI and branch in CASEDIR is equal checked out branch in /var/lib/openqa/share/tests/DISTRI
    • Analog with NEEDLE_DIR and /var/lib/openqa/share/tests/DISTRI/products/DISTRI/needles
  • Job is assigned

  • Also make sure that fetchneedles doesn't interfere.

  • The initial clone as well as updating should be handled in asynchronous tasks like ..._URL asset downloading


Related issues 7 (4 open3 closed)

Related to openQA Project (public) - action #157273: Run os-autoinst-distri-openQA directly from git without anything related in /var/lib/openqa/share/tests - Follow-up with ideas not tried out in the spike solution #157270 size:MBlockedokurz2024-04-17

Actions
Related to openQA Project (public) - action #164937: o3 logreport: warning: refname 'abc123...' is ambiguous size:SResolvedybonatakis2024-08-052024-08-31

Actions
Related to openQA Project (public) - action #165866: Jobs stuck waiting on background tasks git_clone size:MResolvedtinita2024-09-11

Actions
Blocks openQA Project (public) - action #159174: Decrease load on remote git servers by not cloning CASEDIR/NEEDLE_DIR from every workerNew2024-04-17

Actions
Copied to openQA Project (public) - action #160379: Support web UI checkouts of CASEDIR with a tag reference in a refspecNew

Actions
Copied to openQA Project (public) - action #160382: Support web UI checkouts of CASEDIR with a specific commit specified in a refspecNew

Actions
Copied to openQA Project (public) - action #161771: Create and maintain up to date version of test distri/needles for webui - take 2 size:MResolvedmkittler

Actions
Actions #1

Updated by dheidler 8 months ago

  • Blocks action #159174: Decrease load on remote git servers by not cloning CASEDIR/NEEDLE_DIR from every worker added
Actions #2

Updated by dheidler 8 months ago

  • Description updated (diff)
Actions #3

Updated by tinita 8 months ago

  • Target version set to Tools - Next
Actions #4

Updated by okurz 8 months ago

  • Parent task changed from #157273 to #152847
Actions #5

Updated by okurz 8 months ago

  • Related to action #157273: Run os-autoinst-distri-openQA directly from git without anything related in /var/lib/openqa/share/tests - Follow-up with ideas not tried out in the spike solution #157270 size:M added
Actions #6

Updated by okurz 8 months ago

  • Category set to Feature requests
  • Assignee set to okurz

will need to think about this

Actions #7

Updated by okurz 8 months ago

  • Assignee deleted (okurz)
  • Target version changed from Tools - Next to Ready

The initial clone as well as updating should be handled in asynchronous tasks like ..._URL asset downloading

Actions #8

Updated by jbaier_cz 8 months ago

  • Subject changed from Create and maintain up to date version of test distri/needles for webui to Create and maintain up to date version of test distri/needles for webui size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #9

Updated by okurz 8 months ago

  • Priority changed from Normal to High
Actions #10

Updated by dheidler 7 months ago

  • Status changed from Workable to In Progress
  • Assignee set to dheidler
Actions #11

Updated by openqa_review 7 months ago

  • Due date set to 2024-05-15

Setting due date based on mean cycle time of SUSE QE Tools

Actions #13

Updated by okurz 7 months ago

  • Description updated (diff)
Actions #14

Updated by okurz 7 months ago

  • Description updated (diff)
Actions #15

Updated by okurz 7 months ago

As we already about this today I suggest we discuss the approach in a bit broader context. I think that we actually need to re-estimate and split the ticket

Actions #16

Updated by dheidler 7 months ago

  • Status changed from In Progress to Workable

how to detect if a given string is a branch name, a tag name or a commit in a remote repository:

# git ls-remote --symref https://github.com/openSUSE/opi guoyunhe-patch-1
# -> \srefs/heads/ -> branch

# git ls-remote --symref https://github.com/openSUSE/opi v5.0.0
# -> \srefs/tags/ -> tag -> git fetch --tags --force

# else refspec is commit
# -> git fetch --all
Actions #17

Updated by dheidler 7 months ago

  • Status changed from Workable to In Progress

I will fix the tests for the PR that only covers branches for now.

Actions #18

Updated by livdywan 7 months ago

  • Copied to action #160379: Support web UI checkouts of CASEDIR with a tag reference in a refspec added
Actions #19

Updated by livdywan 7 months ago

  • Copied to action #160382: Support web UI checkouts of CASEDIR with a specific commit specified in a refspec added
Actions #20

Updated by livdywan 7 months ago

  • Description updated (diff)
Actions #21

Updated by okurz 7 months ago

  • Due date changed from 2024-05-15 to 2024-05-22

We discussed this yesterday and agreed to decrease the scope hence dheidler removed AC4 for now. With that bumping the due date a bit but I would still encourage better collaboration to bring this to completion soon.

Actions #22

Updated by dheidler 7 months ago

  • Status changed from In Progress to Feedback
Actions #23

Updated by livdywan 7 months ago

https://github.com/os-autoinst/openQA/pull/5622/files has some remarks on quoting of variables in messages e.g. "Please look at this '$url'" - note these only show in the "changes" view so I missed them at first. Should be good to merge once those are clarified

Actions #24

Updated by livdywan 7 months ago

  • Due date changed from 2024-05-22 to 2024-05-24

PR approved but test failures and, as of right now, codecov being investigated. Hence bumping the due date carefully.

Actions #25

Updated by livdywan 7 months ago

  • Due date changed from 2024-05-24 to 2024-05-31

livdywan wrote in #note-24:

PR approved but test failures and, as of right now, codecov being investigated. Hence bumping the due date carefully.

[13:50:56] t/full-stack.t ......... 33/?     # Unexpected Javascript console errors: [
    #   {
    #     level     => "SEVERE",
    #     message   => "http://localhost:9526/tests/Unable%20to%20read%20image:%20Can't%20open%20file%20%22/tmp/webui.worker-1.egZpq4BW/%22:%20No%20such%20file%20or%20directory%20at%20/home/squamata/project/lib/OpenQA/WebAPI/Controller/Running.pm%20line%20211. - Failed to load resource: the server responded with a status of 404 (Not Found)",
    #     source    => "network",
    #     timestamp => 1716558871922,
    #   },
    # ]
    # full result panel contents:
    # State: running, started less than a minute ago 
    #   
    # Scheduled product: job has not been created by posting an ISO
    # Assigned worker: 26686ca7dff7:1

    #   Failed test 'Expected result not found'
    #   at /home/squamata/project/t/lib/OpenQA/Test/FullstackUtils.pm line 76.

    #   Failed test 'job 8 is incomplete'
    #   at t/full-stack.t line 418.
    # 0

    #   Failed test 'reason for incomplete specified'
    #   at t/full-stack.t line 419.
    #                   'State: running, started less than a minute ago 
    #   
    # Scheduled product: job has not been created by posting an ISO
    # Assigned worker: 26686ca7dff7:1'
    #     doesn't match '(?^u:Failed to download.*non-existent.qcow2)'
[...]
#   Failed test 'Cache tests'
#   at t/full-stack.t line 442.
findElement: no such element: Unable to locate element: {"method":"css selector","selector":".embedded-logfile"}
Actions #26

Updated by dheidler 7 months ago

  • Status changed from Feedback to Resolved

PR got merged.

Actions #27

Updated by okurz 6 months ago

  • Copied to action #161771: Create and maintain up to date version of test distri/needles for webui - take 2 size:M added
Actions #28

Updated by okurz 6 months ago

  • Due date deleted (2024-05-31)
Actions #29

Updated by okurz 4 months ago

  • Related to action #164937: o3 logreport: warning: refname 'abc123...' is ambiguous size:S added
Actions #30

Updated by tinita 3 months ago

  • Related to action #165866: Jobs stuck waiting on background tasks git_clone size:M added
Actions

Also available in: Atom PDF