Project

General

Profile

Actions

coordination #58184

open

[saga][epic][use case] full version control awareness within openQA

Added by okurz almost 5 years ago. Updated 3 days ago.

Status:
Blocked
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2019-08-01
Due date:
2024-10-18 (Due in 5 days)
% Done:

87%

Estimated time:
(Total: 110.00 h)

Description

Motivation

This is linked to Use case 4 and motivated by a discussion by the QA tools team in the weekly meeting 2019-10-15. What we should have are for example user forks and branches, fully versioned test schedules and configuration settings

User story

As a test case contributor during test case development I want to run tests on production instances with all necessary changes recorded in version control before merging to master so that my change will have minimal unexpected impact (test regressions) on existing tests

Further user stories (from https://confluence.suse.com/pages/viewpage.action?pageId=365527173)

  1. I want to start a job based on a modified test in production (In production tests can behave differently, for example because of the heavier load) -> see openqa-clone-job + CASEDIR
  2. I want to edit needles and test if they work before proposing changes
  3. I want to compare the results of a certain job group between two of my branches
  4. I want to schedule a test 100 times without it showing up in the group overview -> see statistical-investigation
  5. I want to trigger multiple cloned jobs for each pull-request (Sometimes you want to trigger VR for different jobs against the same PR. it would be nice to do that in one command line)
  6. I want to trigger the relevant tests automatically by creating a PR

Implications and suggestions

  • The usual test contributor workflows should be supported and made easier by making openQA fully aware of tests triggered for development purposes without negatively impacting existing validation tests

    • Potential impact on asset management
    • No pollution of validation test reports by development tests
  • If there are new/modified needles involved, the existing workflow cannot handle that. The current practice is:

    • Test your changes (and possibly needle changes) locally and create PR(s)
    • Edit needles online and save them (then they will be committed to master). Requires admin rights
  • DONE: Cloning cancelled or incomplete jobs currently does not work as openqa-clone-custom-git-refspec requires the vars.json file from a completed job with this file uploaded -> https://github.com/os-autoinst/openQA/pull/3170

  • Replace "fetchneedles" by inherent git support

  • Provide support for github pull request validation

  • DONE: Extend openqa-clone-custom-git-refspec to accept list of source tests to clone -> https://github.com/os-autoinst/openQA/pull/2577

  • DONE: openqa-clone-custom-git-refspec: Output in markdown format for easy copy/pasting into git commit messages and github PR comments -> https://github.com/os-autoinst/openQA/pull/2577

  • openqa-clone-custom-git-refspec: Provide link to /tests/overview page for the custom build when multiple tests have been cloned

  • Make the trigger source of test jobs apparent, e.g. the source git repositories

  • #14818#note-18 : "Tim got a ticket from Ray that the docker test failed and wants openQA to reproduce the issue and pause at the beginning of the docker test. Afterwards he wants openQA to make a disk snapshot and step through the test execution to find out where the problem is. After he found out, he reloads the snapshot to tweak the execution. During this process, openQA records his steps and allows to add needles."


Subtasks 74 (8 open66 closed)

coordination #48641: [epic] Trigger openQA tests in pull requests of any product github pull requestResolvedokurz2020-11-15

Actions
coordination #77698: [epic] synchronous qemu based system level test in pull request CI runs, e.g. standalone isotovideo or openQA testsResolvedokurz2020-11-15

Actions
action #77905: CI pipeline proof-of-concept running isotovideoResolvedokurz2020-11-15

Actions
action #159399: Cover native os-autoinst in CI checks in readme/documentation size:SResolvedmkittler2024-04-22

Actions
action #87695: Full openQA test development, maintenance and administration from browser without the need of a local terminal size:MResolvedtinita

Actions
coordination #124466: [epic] Put open points from okurz's hackweek 22 project into proper ticketsResolvedmkittler2023-02-14

Actions
action #124502: [spike][timeboxed:20h] complete test definition from yaml schedule in git checked out test distributionResolvedmkittler2023-02-14

Actions
action #125720: [spike][timeboxed:20h] Add monitoring-support into openqa-cliResolvedmkittler

Actions
action #125723: Provide a ready-to-use container image or GitHub action repository to trigger/monitor openQA jobs as CI checks size:MResolvedmkittler2023-03-09

Actions
action #126950: [openQA-in-openQA] openQA tests in pull requests to github.com/os-autoinst/os-autoinst-distri-openQA/ size:MResolvedjbaier_cz2023-03-30

Actions
action #127949: [spike][timeboxed:20h] Research native GitHub for running openQA tests as CI checks size:MResolvedmkittler2023-04-19

Actions
action #128360: Supporting fork based development model size:MResolvedmkittler

Actions
action #129730: Adapt http://open.qa/docs/#_running_openqa_jobs_as_ci_checks for the use of github pull_request_target size:MResolvedmkittler

Actions
coordination #130850: [epic] Use openqa-clone-custom-git-refspec to parse github description+comments and trigger openQA tests as part of CIResolvedokurz2023-06-15

Actions
action #130934: Trigger openQA tests mentioned in github description as part of CI size:MResolvedmkittler2023-06-15

Actions
action #130940: Trigger openQA tests mentioned in github comments as part of automatic testing as well - trusted group "tests-maintainer" only size:MResolvedmkittler2023-06-15

Actions
action #130943: Test parameterization for github description/comments mentioned openQA job clones as part of CI size:SResolvedmkittler2023-06-15

Actions
action #138203: [openQA-in-openQA] CI jobs show error but don't fail the CI job as they should *and* openqa_install+publish missing size:MResolvedjbaier_cz2023-10-18

Actions
action #150992: [timeboxed][spike solution:20h] openQA tests in pull requests to github.com/os-autoinst/os-autoinst-distri-opensuse/ size:MResolvedmkittler

Actions
action #152170: Run openQA tests in pull requests to github.com/os-autoinst/os-autoinst-distri-opensuse/ size:MResolvedokurz

Actions
action #152939: Find "last build" of a product over API size:MResolvedtinita2023-12-27

Actions
action #161399: Use Tumbleweed again for openqa-single-instance container size:SResolvedjbaier_cz

Actions
action #162632: Ensure openqa-clone-custom-git-refspec works fine with git-only test distributions like os-autoinst-distri-openQA size:MResolvedybonatakis2024-06-20

Actions
action #164210: Run a basic openQA test in osado is triggeded blindly even if nothing is publishedResolvedokurz2024-07-19

Actions
action #164811: Attempt to run job with non-existant git branch specified in CASEDIR will not fail with clear error size:SResolvedmkittler2024-08-01

Actions
coordination #45302: [epic] smarter fetchneedles (was: fetchneedles should ensure we are always on a branch (and try to self-repair))Resolvedokurz2019-12-20

Actions
openQA Infrastructure - action #61221: osd: unable to save needles, minion fails with "fatal: Unable to create '/var/lib/openqa/.../needles/.git/index.lock'"Resolvedokurz2019-12-20

Actions
action #63133: SLE needles git repo looses upstream branch configuration, needles are saved but fail to be pushedResolvedokurz2020-02-05

Actions
action #54965: Cannot inspect the source code of the tests from my forkResolvedokurz2019-08-01

Actions
action #71758: [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folderResolvedXiaojing_liu

Actions
coordination #80372: [epic] Cleanup vars.json as initial information container between openQA worker and isotovideoResolvedokurz2021-02-08

Actions
coordination #67723: [epic] Remote openQA worker fails to run tests from openqa-clone-custom-git-refspecResolvedokurz2021-03-18

Actions
action #90290: Relative paths for CASEDIR and others as default to be not bound to specific workersResolvedokurz2021-03-18

Actions
action #90293: Optional relative paths for CASEDIR and others to be not bound to specific workersResolvedXiaojing_liu2021-03-18

Actions
action #90302: Remote openQA worker fails to run tests from openqa-clone-custom-git-refspec due to differing pathsResolvedXiaojing_liu2021-03-18

Actions
action #88482: Two absolute paths concatenated to form a default needle dir when PRODUCT_DIR/needles doesn't existResolvedXiaojing_liu2021-02-08

Actions
coordination #88561: [epic] Extend needle version control handlingWorkable2021-02-12

Actions
action #88564: text field for git commit details in needle editorResolvedmkittler2021-02-12

Actions
action #162035: Support development of "Use needles from correct ref of NEEDLES_DIR" https://github.com/os-autoinst/openQA/pull/5175 size:MWorkablelivdywan2024-06-10

Actions
action #92311: Complete test definition from a single yaml schedule file in local test distribution folderResolvedmkittler

Actions
coordination #92314: [epic] Cleanup vars.json as initial information container between openQA worker and isotovideo - further cleanupResolvedokurz2021-01-04

Actions
action #81703: The values of 'ISO' and 'HDD' includes absolute path in vars.jsonResolvedXiaojing_liu2021-01-04

Actions
coordination #94750: [epic] Support needles from git-cloned or separate repo if casedir points to a git-repoResolvedokurz2021-06-25

Actions
coordination #33745: [epic] Improve handling of external Git repositories (for needles)Resolvedmkittler2024-06-20

Actions
action #162647: Ensure the developer mode and its feature to reload newly created needles work when NEEDLES_DIR is a Git sourceResolvedmkittler2024-06-20

Actions
action #94735: needles not found in `needles` subdirectory when CASEDIR is a git repositoryResolvedmkittler2021-06-25

Actions
action #97112: Support relative needle directories together with tests checked out from gitResolvedokurz

Actions
action #164577: [timeboxed:6h][research] Understand the current behaviour of openQA test distributions including needlesResolvedmkittler2024-07-29

Actions
coordination #152847: [epic] version control awareness within openQA for test distributionsBlockedokurz2023-12-152024-10-18

Actions
action #152681: openqa-investigate creates wrong CASEDIR setting for some tests size:MResolvedybonatakis2023-12-15

Actions
action #154156: [spike][timeboxed:10h] Cache test distributions from git on production size:SResolvedmkittler

Actions
action #154783: [spike][timeboxed:10h] Run os-autoinst-distri-example directly from git and ensure candidate needles show up on the web UI size:SResolvedmkittler

Actions
action #156922: Run os-autoinst-distri-openQA directly from git without anything related in o3:/var/lib/openqa/share/testsBlockedokurz

Actions
action #157270: [spike solution][timeboxed:20h] Run os-autoinst-distri-openQA directly from git without anything related in /var/lib/openqa/share/testsRejectedokurz2024-03-14

Actions
action #159171: Create and maintain up to date version of test distri/needles for webui size:MResolveddheidler2024-04-17

Actions
action #161771: Create and maintain up to date version of test distri/needles for webui - take 2 size:MResolvedmkittler

Actions
action #162077: Create and maintain up to date version of test distri/needles for webui - enabled by default size:SResolvedmkittler

Actions
action #162125: [timeboxed:10h][spike] Let openQA keep test distribution checkouts up to date without needing fetchneedles size:SResolvedtinita2024-06-12

Actions
action #164883: Use same minion guard for save_needle, delete_needles and git_clone size:SResolvedtinita

Actions
action #164886: Use OpenQA::Git for all our git wrappers size:SResolvedrobert.richardson

Actions
action #164889: Ensure git repos cloned by minions are cleaned up regularly size:SResolved

Actions
action #164898: Replace fetchneedles with a minion job size:MIn Progresstinita2024-10-18

Actions
action #165066: Ensure local changes to git repos cloned by git_auto_clone are left alone size:SResolveddheidler2024-08-08

Actions
action #167386: Handle too many warnings "Local checkout at … but requesting to clone from" size:SResolveddheidler2024-09-25

Actions
action #167971: test detail pages for unfinished jobs on git-cloned test distributions point to invalid source URLResolvedtinita2024-10-08

Actions
action #168013: Only make one api call in openqa-advanced-retrigger-jobsFeedbackmkittler2024-10-09

Actions
coordination #154780: [epic] openQA scenario definitions fully in gitResolvedokurz2023-06-01

Actions
action #132335: In openqa-in-openqa use scenario definitions instead of job group templates size:MResolvedosukup2023-06-01

Actions
action #155218: [spike][timeboxed:30h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse size:MResolvedtinita

Actions
action #157774: Empty scenario definitions causing Use of uninitialized value $testsuite_name in hash element in lib/OpenQA/Schema/Result/JobGroups.pm size:SResolvedlivdywan2024-03-22

Actions
action #160206: Minimum working example for job templates YAML import/include/inherit in kernelqa-openqa-yaml size:SResolvedtinita

Actions
action #160209: Reduce duplication in machine definitions in https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml#L8, e.g. with re-usable machine "defaults" size:SResolveddheidler

Actions
coordination #159573: [epic] Marketing/training/documentation about new version control featuresBlockedokurz2024-03-20

Actions
action #157612: Bump openQA version from 4.6.x to 5.x to be consistent with semverBlockedokurz2024-03-20

Actions

Related issues 15 (6 open9 closed)

Related to QA - coordination #49502: [qe-core] Automatic validation test on github PRsBlockedszarate2019-03-20

Actions
Related to openQA Project - action #57452: Automatic summary of failuresRejected2019-09-27

Actions
Related to openQA Project - action #10192: Improve source code windowWorkable2016-01-12

Actions
Related to openQA Project - action #34549: implement a way to retire needles or "Keep copies of needles while running tests"Rejectedokurz2018-04-09

Actions
Related to openQA Project - action #25572: [tools][needles]needles pushing can interfer with "fetchneedles"Resolvedokurz2017-09-26

Actions
Related to openQA Project - action #59184: Research about testing with a custom git refResolvedtinita2019-11-07

Actions
Related to openQA Project - action #58304: A personal activity view for developers Resolvedlivdywan2019-10-172020-10-17

Actions
Related to openQA Project - action #62600: Improve error output when calling openqa-clone-custom-git-refspec with wrong args, not just exit codeResolvedokurz2020-01-23

Actions
Related to openQA Project - action #66071: TEST is overridden in parent job when doing `openqa-clone-custom-git-refspec`Resolvedokurz2020-04-26

Actions
Related to openQA Tests - coordination #15132: [saga][epic] Better structure of test plans in main.pmBlockedokurz2018-11-20

Actions
Related to openQA Tests - action #77071: [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuseWorkable

Actions
Related to openQA Tests - action #113528: [qe-core] test fails in bootloader_zkvm - performance degradation in the s390 network is causing serial console to be unreliable (and killing jobs slowly)Resolvedszarate2022-07-132022-07-18

Actions
Related to openQA Project - action #138416: Unify GitHub Actions for QA Projects size:MResolvedybonatakis2023-10-24

Actions
Related to openQA Project - coordination #154777: [saga][epic] Shareable os-autoinst and test distribution pluginsNew2021-01-08

Actions
Related to openQA Project - coordination #162539: [saga][epic] future ideas version for version control features within openQANew2019-09-11

Actions
Actions

Also available in: Atom PDF