Project

General

Profile

Actions

coordination #58184

open

[saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches, fully versioned test schedules and configuration settings

Added by okurz about 4 years ago. Updated 1 day ago.

Status:
Blocked
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2018-03-23
Due date:
% Done:

56%

Estimated time:
(Total: 110.00 h)
Difficulty:

Description

Motivation

This is linked to Use case 4 and motivated by a discussion by the QA tools team in the weekly meeting 2019-10-15

User story

As a test case contributor during test case development I want to run tests on production instances with all necessary changes recorded in version control before merging to master so that my change will have minimal unexpected impact (test regressions) on existing tests

Further user stories (from https://confluence.suse.com/pages/viewpage.action?pageId=365527173)

  1. I want to start a job based on a modified test in production (In production tests can behave differently, for example because of the heavier load) -> see openqa-clone-job + CASEDIR
  2. I want to edit needles and test if they work before proposing changes
  3. I want to compare the results of a certain job group between two of my branches
  4. I want to schedule a test 100 times without it showing up in the group overview -> see statistical-investigation
  5. I want to trigger multiple cloned jobs for each pull-request (Sometimes you want to trigger VR for different jobs against the same PR. it would be nice to do that in one command line)
  6. I want to trigger the relevant tests automatically by creating a PR

Implications and suggestions

  • The usual test contributor workflows should be supported and made easier by making openQA fully aware of tests triggered for development purposes without negatively impacting existing validation tests

    • Potential impact on asset management
    • No pollution of validation test reports by development tests
  • If there are new/modified needles involved, the existing workflow cannot handle that. The current practice is:

    • Test your changes (and possibly needle changes) locally and create PR(s)
    • Edit needles online and save them (then they will be committed to master). Requires admin rights
  • DONE: Cloning cancelled or incomplete jobs currently does not work as openqa-clone-custom-git-refspec requires the vars.json file from a completed job with this file uploaded -> https://github.com/os-autoinst/openQA/pull/3170

  • Replace "fetchneedles" by inherent git support

  • Provide support for github pull request validation

  • DONE: Extend openqa-clone-custom-git-refspec to accept list of source tests to clone -> https://github.com/os-autoinst/openQA/pull/2577

  • DONE: openqa-clone-custom-git-refspec: Output in markdown format for easy copy/pasting into git commit messages and github PR comments -> https://github.com/os-autoinst/openQA/pull/2577

  • openqa-clone-custom-git-refspec: Provide link to /tests/overview page for the custom build when multiple tests have been cloned

  • Make the trigger source of test jobs apparent, e.g. the source git repositories

  • #14818#note-18 : "Tim got a ticket from Ray that the docker test failed and wants openQA to reproduce the issue and pause at the beginning of the docker test. Afterwards he wants openQA to make a disk snapshot and step through the test execution to find out where the problem is. After he found out, he reloads the snapshot to tweak the execution. During this process, openQA records his steps and allows to add needles."


Subtasks 67 (34 open33 closed)

coordination #48641: [epic] Trigger openQA tests in pull requests of any product github pull requestBlockedokurz2020-11-15

Actions
coordination #77698: [epic] synchronous qemu based system level test in pull request CI runs, e.g. standalone isotovideo or openQA testsNew2020-11-15

Actions
action #77905: CI pipeline proof-of-concept running isotovideoResolvedokurz2020-11-15

Actions
action #86063: [epic] Add possibility to trigger openQA API calls, e.g. single "jobs", without the need of the client / over the webUI / with curlBlockedokurz2021-01-13

Actions
action #87698: openQA jobs can be triggered with single curl callsResolvedkraih2021-01-13

Actions
action #90788: openQA jobs with arbitrary parameters can be triggered over the webUI for authenticated users with right permissions (operator+)Workable

Actions
action #87695: Full openQA test development, maintenance and administration from browser without the need of a local terminal size:MWorkable

Actions
coordination #124466: [epic] Put open points from okurz's hackweek 22 project into proper ticketsResolvedmkittler2023-02-14

Actions
action #124502: [spike][timeboxed:20h] complete test definition from yaml schedule in git checked out test distributionResolvedmkittler2023-02-14

Actions
action #125720: [spike][timeboxed:20h] Add monitoring-support into openqa-cliResolvedmkittler

Actions
action #125723: Provide a ready-to-use container image or GitHub action repository to trigger/monitor openQA jobs as CI checks size:MResolvedmkittler2023-03-09

Actions
action #126950: [openQA-in-openQA] openQA tests in pull requests to github.com/os-autoinst/os-autoinst-distri-openQA/ size:MResolvedjbaier_cz2023-03-30

Actions
action #127949: [spike][timeboxed:20h] Research native GitHub for running openQA tests as CI checks size:MResolvedmkittler2023-04-19

Actions
action #128360: Supporting fork based development model size:MResolvedmkittler

Actions
action #129730: Adapt http://open.qa/docs/#_running_openqa_jobs_as_ci_checks for the use of github pull_request_target size:MResolvedmkittler

Actions
coordination #130850: [epic] Use openqa-clone-custom-git-refspec to parse github description+comments and trigger openQA tests as part of CIBlockedokurz2023-06-15

Actions
action #130934: Trigger openQA tests mentioned in github description as part of CI size:MWorkable2023-06-15

Actions
action #130940: Trigger openQA tests mentioned in github comments as part of CI as wellNew2023-06-15

Actions
action #130943: Test parameterization for github description/comments mentioned openQA job clones as part of CINew2023-06-15

Actions
action #138203: [openQA-in-openQA] CI jobs show error but don't fail the CI job as they should *and* openqa_install+publish missing size:MFeedbackjbaier_cz2023-10-18

Actions
action #150992: [timeboxed][spike solution:20h] openQA tests in pull requests to github.com/os-autoinst/os-autoinst-distri-opensuse/ size:MResolvedmkittler

Actions
action #152170: Run openQA tests in pull requests to github.com/os-autoinst/os-autoinst-distri-opensuse/ size:MWorkable

Actions
coordination #45302: [epic] smarter fetchneedles (was: fetchneedles should ensure we are always on a branch (and try to self-repair))Blockedokurz2019-12-20

Actions
openQA Infrastructure - action #61221: osd: unable to save needles, minion fails with "fatal: Unable to create '/var/lib/openqa/.../needles/.git/index.lock'"New2019-12-20

Actions
action #63133: SLE needles git repo looses upstream branch configuration, needles are saved but fail to be pushedResolvedokurz2020-02-05

Actions
action #54965: Cannot inspect the source code of the tests from my forkResolvedokurz2019-08-01

Actions
action #60272: Make fetching custom git repos (e.g. needles) more efficientNew2019-11-26

Actions
action #71758: [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folderResolvedXiaojing_liu

Actions
coordination #80372: [epic] Cleanup vars.json as initial information container between openQA worker and isotovideoResolvedokurz2021-02-08

Actions
coordination #67723: [epic] Remote openQA worker fails to run tests from openqa-clone-custom-git-refspecResolvedokurz2021-03-18

Actions
action #90290: Relative paths for CASEDIR and others as default to be not bound to specific workersResolvedokurz2021-03-18

Actions
action #90293: Optional relative paths for CASEDIR and others to be not bound to specific workersResolvedXiaojing_liu2021-03-18

Actions
action #90302: Remote openQA worker fails to run tests from openqa-clone-custom-git-refspec due to differing pathsResolvedXiaojing_liu2021-03-18

Actions
action #88482: Two absolute paths concatenated to form a default needle dir when PRODUCT_DIR/needles doesn't existResolvedXiaojing_liu2021-02-08

Actions
coordination #88561: [epic] Extend needle version control handlingWorkable2021-02-12

Actions
action #88564: text field for git commit details in needle editorResolvedmkittler2021-02-12

Actions
action #91905: webui fails to display needles when using a git repository as CASEDIRNew2021-04-28

Actions
action #92311: Complete test definition from a single yaml schedule file in local test distribution folderResolvedmkittler

Actions
coordination #92314: [epic] Cleanup vars.json as initial information container between openQA worker and isotovideo - further cleanupBlockedokurz2021-01-04

Actions
action #81703: The values of 'ISO' and 'HDD' includes absolute path in vars.jsonResolvedXiaojing_liu2021-01-04

Actions
coordination #94750: [epic] Support needles from git-cloned or separate repo if casedir points to a git-repoNew2018-03-23

Actions
action #33745: Improve handling external Git repositories (for needles)New2018-03-23

Actions
action #56789: New needles from git repository not working with openqa-clone-custom-git-refspecNew2019-09-11

Actions
action #94735: needles not found in `needles` subdirectory when CASEDIR is a git repositoryResolvedmkittler2021-06-25

Actions
action #97112: Support relative needle directories together with tests checked out from gitNew

Actions
coordination #106922: [epic][sporadic] openqa_from_git fails in dashboard due to ensure_unlocked_desktop not expecting password entry screen in case of locked desktop auto_review:"match=desktop-runner,screenlock timed out.*":retryBlockedokurz2022-02-16

Actions
coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributionsBlockedokurz2021-01-08

Actions
action #81899: Move code from isotovideo to a module size:MResolvedlivdywan2021-01-08

Actions
action #108530: os-autoinst wheels: x11_start_program from os-autoinst-distri-openQA dynamically loaded from another git repo size:MResolvedlivdywan2022-03-17

Actions
openQA Infrastructure - action #114914: Container devel/openqa/containers/isotovideo:qemu-x86 is outdatedRejectedokurz2022-03-17

Actions
action #115001: Add documentation how to use/develop wheels visible on open.qa/docsNew2022-08-04

Actions
action #115004: Use github.com/os-autoinst/os-autoinst-wheel-launcher in os-autoinst-distri-opensuse as wellNew2022-09-16

Actions
action #115013: os-autoinst-plugin as a wheel helper toolNew2022-08-04

Actions
action #115016: Conduct a workshop to introduce wheels and discuss how to go forwardNew2022-08-04

Actions
action #115019: os-autoinst/wheel-boot which can be used to test os-autoinst/wheel-launcher size:MWorkable2022-08-04

Actions
action #115022: Complete unit test coverage (with coverage analysis) in os-autoinst/wheel-launcherResolvedmkittler2022-08-04

Actions
coordination #117097: Evaluate GitHub template repositories for wheelsNew2022-09-23

Actions
coordination #117673: [epic][tools] sporadic "Unable to clone Git repository" for wheelsNew2022-10-13

Actions
action #118633: Re-try on cloning of wheel repositories size:MResolvedlivdywan2022-10-13

Actions
action #118882: Make available wheels more discoverable and share knowledge size:SWorkable2022-10-13

Actions
action #118903: Repositories for wheels should be cachedNew2022-10-13

Actions
action #130369: [spike][timeboxed:20h] Reduce duplication of openQA-in-openQA tests in os-autoinst-distri-opensuse and os-autoinst-distri-openQA size:SResolvedtinita2023-06-01

Actions
action #132335: In openqa-in-openqa use scenario definitions instead of job group templates size:MResolvedosukup2023-06-01

Actions
action #133301: quick-fix for openQA-in-openQA lockscreen race conditionResolvedokurz2022-08-04

Actions
action #134390: Log proactively where wheel components are located to ease debugging size:MResolvedlivdywan2023-06-01

Actions
action #138029: [research][timeboxed:10h] How to cache "wheel" repositories which are stored on github size:SWorkable

Actions
action #138032: Find out most/least used testapi functions to decide about where to extend/cleanup size:MWorkable

Actions

Related issues 13 (5 open8 closed)

Related to QA - coordination #49502: [qe-core] Automatic validation test on github PRsBlockedszarate2019-03-20

Actions
Related to openQA Project - action #57452: Automatic summary of failuresRejected2019-09-27

Actions
Related to openQA Project - action #10192: Improve source code windowWorkable2016-01-12

Actions
Related to openQA Project - action #34549: implement a way to retire needles or "Keep copies of needles while running tests"Rejectedokurz2018-04-09

Actions
Related to openQA Project - action #25572: [tools][needles]needles pushing can interfer with "fetchneedles"Resolvedokurz2017-09-26

Actions
Related to openQA Project - action #59184: Research about testing with a custom git refResolvedtinita2019-11-07

Actions
Related to openQA Project - action #58304: A personal activity view for developers Resolvedlivdywan2019-10-172020-10-17

Actions
Related to openQA Project - action #62600: Improve error output when calling openqa-clone-custom-git-refspec with wrong args, not just exit codeResolvedokurz2020-01-23

Actions
Related to openQA Project - action #66071: TEST is overridden in parent job when doing `openqa-clone-custom-git-refspec`Resolvedokurz2020-04-26

Actions
Related to openQA Tests - coordination #15132: [saga][epic] Better structure of test plans in main.pmBlockedokurz2018-11-20

Actions
Related to openQA Tests - action #77071: [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuseWorkable

Actions
Related to openQA Tests - action #113528: [qe-core] test fails in bootloader_zkvm - performance degradation in the s390 network is causing serial console to be unreliable (and killing jobs slowly)Resolvedszarate2022-07-132022-07-18

Actions
Related to openQA Project - action #138416: [qe-core] Unify GitHub Actions for QA Projects size:MIn Progressjosegomezr2023-10-242023-12-14

Actions
Actions #1

Updated by okurz about 4 years ago

Actions #2

Updated by okurz about 4 years ago

Actions #3

Updated by okurz about 4 years ago

  • Description updated (diff)
Actions #4

Updated by okurz about 4 years ago

Actions #5

Updated by okurz about 4 years ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version set to Current Sprint

blocked by subtasks

Actions #6

Updated by okurz about 4 years ago

  • Related to action #34549: implement a way to retire needles or "Keep copies of needles while running tests" added
Actions #7

Updated by okurz about 4 years ago

  • Blocks action #25572: [tools][needles]needles pushing can interfer with "fetchneedles" added
Actions #8

Updated by okurz about 4 years ago

  • Blocks deleted (action #25572: [tools][needles]needles pushing can interfer with "fetchneedles")
Actions #9

Updated by okurz about 4 years ago

  • Related to action #25572: [tools][needles]needles pushing can interfer with "fetchneedles" added
Actions #10

Updated by okurz about 4 years ago

Brain-storming between mkittler and okurz 2019-11-19: We propose to start with a system level test, e.g. copy/extension of full stack test that has no machines, products, test suites nor job templates setup in database and reads everything from a git repo. The starting point would be e.g. isos post distri=https://github.com/os-autoinst/os-autoinst-distri-openQA which then, e.g. in the usual isos post controller method, handles:

  1. checkout on webUI host
  2. evaluate job templates from git repo, e.g. by convention from ./schedule.yml
  3. trigger jobs
  4. map needle + source code
  5. test done: Needles need to be referenced from git, needle editor need to save to master by default and the custom git repo if used

EDIT: 2020-01-25: Allowing job templates from git repo as well introduces another source where settings come from and can be confusing when trying to find out what changed in jobs and where the change comes from. For this I provided suggestions in #19720 e.g. to mark the source of settings in the database table. So this shouldn't stop us to add a test distri VCS as another source of settings

Actions #11

Updated by okurz about 4 years ago

  • Related to action #59184: Research about testing with a custom git ref added
Actions #12

Updated by okurz about 4 years ago

  • Related to action #58304: A personal activity view for developers added
Actions #13

Updated by okurz about 4 years ago

  • Description updated (diff)
  • Status changed from Blocked to Workable
  • Assignee deleted (okurz)

Updated with feedback from #59184 , incorporated the use cases and suggestions and problems into the description. Setting back to "Workable" for anyone to further specify subtickets which cover all suggestions and user stories.

Actions #14

Updated by okurz about 4 years ago

  • Description updated (diff)
Actions #15

Updated by okurz almost 4 years ago

  • Description updated (diff)
Actions #16

Updated by okurz almost 4 years ago

  • Related to action #62600: Improve error output when calling openqa-clone-custom-git-refspec with wrong args, not just exit code added
Actions #17

Updated by livdywan almost 4 years ago

  • Assignee set to coolo
Actions #18

Updated by mkittler almost 4 years ago

I see two overall challenges and I'd like to point out their current state and possible improvements:

  1. Improve the way we get custom/versioned tests into openQA.
    1. Currently one can override CASEDIR and NEEDLES_DIR to achieve that (e.g. via openqa-clone-custom-git-refspec). The way the git cloning is done by os-autoinst is not very efficient but it works.
    2. We likely want a more CI-like approach. It is conceivable to make openQA's job scheduling version-aware, e.g. allowing to override DISTRI with a Git URL. The referenced distri would contain test suites, job templates and other scheduling definitions we use for producing jobs (e.g. as a YAML file). Preferably test code and needles are contained by the same repository¹. See https://github.com/os-autoinst/openQA/pull/2706 for a draft.
    3. Point 1.2. leads to thinking about scalability, e.g. one might want to run os-autoinst and maybe even a dedicated openQA instance within a Kubernetes cluster. It isn't clear how test results would be transferred from such test runs into common web UI.
  2. Displaying custom/versioned tests within the openQA web UI.
    1. The job module code view is not version aware. I suppose making it a link to a Git repository would be a small change and helpful regardless how we continue with our design for 1..
    2. All places where needle candidates are displayed are not version aware. In contrast to 2.1. this is a rather challenging problem. It is not clear what the best solution is and how the it would interfere with our design for 1..
    3. The dashboard is not version aware so custom tests are not filtered out. It has been discussed what filtering policies we want and likely a view per user would make sense. Just supporting this filtering on job level would be easy. However, we likely want custom job groups and scheduling tables as well.
  3. Cleanup of custom/versioned tests results, assets and needles.
    1. Currently custom tests results and assets are cleaned up following the global cleanup policies which mainly depend on the global group the job belongs to. If we decide to implement custom/user-specific groups the cleanup algorithm would naturally consider these as separate (and therefore separately configurable) groups. However, at least the asset cleanup page (which shows e.g. which groups an asset is accounted for) would likely not scale when having lots of custom groups within an instance and therefore needed to be adjusted.
    2. As mentioned in 2.2 custom needles are challenging. They also require extra work regarding cleanup.

¹ It is generally questionable whether we want to support splitted test and needle repositories for running versioned/custom tests.

Actions #19

Updated by okurz almost 4 years ago

mkittler wrote:

I see two overall challenges […]

the above notes are based on a discussion that mkittler and me had so I am in line with him :) So I am not objecting, just adding information:

The way the git cloning is done by os-autoinst is not very efficient but it works.

  • cons: Not network-efficient to reclone the same or similar repos over and over again from external upstream; not easily possible to display test/needles from webui as only the worker knew the state of code at the time of test execution
  • pros: automatic cleanup of temporary test distribution checkouts

It isn't clear how test results would be transferred from [kubernetes] test runs into common web UI.

Our idea so far was to support an "asynchronous update", e.g. whenever a worker is started on a pool dir with existing test data walks over dir content and publishes to webui, webui decises if there are any updates necessary as the test data might be already complete on the webui host.

Actions #20

Updated by mkittler almost 4 years ago

Our idea so far was to support an "asynchronous update" [...]

Right, I wanted to add a comment about that for the "offline worker" user case but couldn't find the ticket. However, it is actually related here as well as it might improve the CI/scalability point. You could run os-autoinst somewhere (e.g. isolated within a container) and upload the results later to any web UI when this is wanted. I'd also like to note that the required modifications to the worker and web UI shouldn't be hard to implement. Decoupling test execution from exporting test results to a web UI also seems nice from the perspective of the overall software architecture. It would also help with the problem of incomplete jobs without logs.

Actions #21

Updated by okurz almost 4 years ago

  • Due date set to 2020-02-05

due to changes in a related task

Actions #22

Updated by okurz almost 4 years ago

Discussed in tools-team meeting 2020-02-18: I personally tried to phrase two different paths to follow - which do not need to conflict:

  1. Scaled openQA for user centric tests: More motivated by #58304 . Few, big openQA instances that support existing product-centric testing as well as derived user-centric testing sharing assets, e.g. "As QA SLE engineer I want to run tests against build X of last version of SLE in development against my own tests git branch to find out if a potential bug fix can work or to investigate test failures or try a fix".
  2. Integrated openQA tests within existing CI systems: More motivated by #48641 . Numerous independant openQA instances spawned dynamically within the scope of already existing CI systems, e.g. gitlab CI, travis CI, spawning openQA instances (or linking to existing ones internally).

We agreed that 1. is the overall more challenging but also more important path to follow. This can mean: Have everything optionally linked to a user, e.g. add a user column to every database table, all tests+needles per user. Personally I suggest to exactly try that out in a scratch refactoring, e.g. add a user column to every database table + tests + needles and show how each influenced component can handle the user. While doing little steps as improvement in this direction we should still follow this overall vision.

For the approach to segment "interesting areas" one could follow gitlab/github/travisci/circleci and such and have URLs like /$repo/$user . For openQA that could mean that all existing URLs would be accessible instead or as well below that prefix, e.g. /$repo/$user/tests . As we already have a big list of top-level routes that can prevent certain "$repo" names. The alternative being to break the compatibility of webui routes at least, another to add a prefix, e.g. "/u/$repo/$user", which isn't very clear or a more explicit but longer part.

Actions #23

Updated by okurz over 3 years ago

  • Status changed from Workable to Blocked
  • Assignee changed from coolo to okurz

We could still flesh out this epic more but we have identified some subtasks and related features to solve first.

Important part for finding out specific requirements: "[spike][timeboxed:20h] complete test definition from yaml schedule in git checked out test distribution"

Actions #24

Updated by okurz over 3 years ago

  • Related to action #66071: TEST is overridden in parent job when doing `openqa-clone-custom-git-refspec` added
Actions #25

Updated by okurz over 3 years ago

Actions #26

Updated by okurz over 3 years ago

  • Description updated (diff)
Actions #27

Updated by asmorodskyi about 3 years ago

  • Related to action #71809: Enable multi-machine jobs trigger without "isos post" added
Actions #28

Updated by szarate about 3 years ago

  • Tracker changed from action to coordination
  • Status changed from Blocked to New
Actions #30

Updated by okurz about 3 years ago

  • Target version changed from Current Sprint to Ready
Actions #31

Updated by okurz about 3 years ago

  • Status changed from New to Blocked
Actions #32

Updated by okurz about 3 years ago

  • Subject changed from [epic][use case] full version control awareness within openQA, e.g. user forks and branches to [saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches
Actions #33

Updated by okurz about 3 years ago

  • Related to action #77071: [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse added
Actions #34

Updated by okurz about 3 years ago

  • Subject changed from [saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches to [saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches, fully versioned test schedules and configuration settings
Actions #35

Updated by okurz over 2 years ago

  • Target version changed from Ready to future
Actions #36

Updated by okurz almost 2 years ago

  • Related to deleted (action #71809: Enable multi-machine jobs trigger without "isos post")
Actions #37

Updated by szarate over 1 year ago

  • Related to action #113528: [qe-core] test fails in bootloader_zkvm - performance degradation in the s390 network is causing serial console to be unreliable (and killing jobs slowly) added
Actions #38

Updated by okurz about 1 year ago

  • Related to coordination #108527: [epic] os-autoinst wheels for scalable code reuse of helper functions and segmented test distributions added
Actions #39

Updated by okurz 10 months ago

  • Target version changed from future to Ready

Discussed in daily SUSE QE Tools meeting 2023-02-14. We will put this saga on the backlog again and should review the currently pending tasks. Likely we should move some features out of this saga into a "Future ideas" one.

Actions #40

Updated by okurz 17 days ago

  • Related to action #138416: [qe-core] Unify GitHub Actions for QA Projects size:M added
Actions

Also available in: Atom PDF