action #58184

[epic][use case] full version control awareness within openQA, e.g. user forks and branches

Added by okurz 4 months ago. Updated about 17 hours ago.

Status:WorkableStart date:23/03/2018
Priority:NormalDue date:
Assignee:coolo% Done:

0%

Category:Feature requests
Target version:Current Sprint
Difficulty:
Duration:

Description

Motivation

This is linked to Use case 4 and motivated by a discussion by the QA tools team in the weekly meeting 2019-10-15

User story

As a test case contributor during test case development I want to run tests on production instances with all necessary changes recorded in version control before merging to master so that my change will have minimal unexpected impact (test regressions) on existing tests

Further user stories (from https://confluence.suse.com/pages/viewpage.action?pageId=365527173)

  1. I want to start a job based on a modified test in production (In production tests can behave differently, for example because of the heavier load) -> see openqa-clone-job + CASEDIR
  2. I want to edit needles and test if they work before proposing changes
  3. I want to compare the results of a certain job group between two of my branches
  4. I want to schedule a test 100 times without it showing up in the group overview -> see statistical-investigation
  5. I want to trigger multiple cloned jobs for each pull-request (Sometimes you want to trigger VR for different jobs against the same PR. it would be nice to do that in one command line)
  6. I want to trigger the relevant tests automatically by creating a PR

Implications and suggestions

  • The usual test contributor workflows should be supported and made easier by making openQA fully aware of tests triggered for development purposes without negatively impacting existing validation tests

    • Potential impact on asset management
    • No pollution of validation test reports by development tests
  • If there are new/modified needles involved, the existing workflow cannot handle that. The current practice is:

    • Test your changes (and possibly needle changes) locally and create PR(s)
    • Edit needles online and save them (then they will be committed to master). Requires admin rights
  • Cloning cancelled or incomplete jobs currently does not work as openqa-clone-custom-git-refspec requires the vars.json file from a completed job with this file uploaded

  • Replace "fetchneedles" by inherent git support

  • Provide support for github pull request validation

  • DONE: Extend openqa-clone-custom-git-refspec to accept list of source tests to clone -> https://github.com/os-autoinst/openQA/pull/2577

  • DONE: openqa-clone-custom-git-refspec: Output in markdown format for easy copy/pasting into git commit messages and github PR comments -> https://github.com/os-autoinst/openQA/pull/2577

  • openqa-clone-custom-git-refspec: Provide link to /tests/overview page for the custom build when multiple tests have been cloned

  • Make the trigger source of test jobs apparent, e.g. the source git repositories

  • #14818#note-18 : "Tim got a ticket from Ray that the docker test failed and wants openQA to reproduce the issue and pause at the beginning of the docker test. Afterwards he wants openQA to make a disk snapshot and step through the test execution to find out where the problem is. After he found out, he reloads the snapshot to tweak the execution. During this process, openQA records his steps and allows to add needles."


Subtasks

action #48641: Trigger openQA tests in pull requests of any product gith...New

action #33745: Improve handling external Git repositories (for needles)New

action #56789: New needles from git repository not working with openqa-c...New

action #45302: [epic] smarter fetchneedles (was: fetchneedles should ens...New

action #63133: SLE needles git repo looses upstream branch configuration...Feedbackokurz

action #54965: Cannot inspect the source code of the tests from my forkIn Progressokurz

action #60272: Make fetching custom git repos (e.g. needles) more efficientNew


Related issues

Related to QA - action #49502: Automatic validation test on github PRs New 20/03/2019
Related to openQA Project - action #57452: Automatic summary of failures Rejected 27/09/2019
Related to openQA Project - action #10192: Improve source code window New 12/01/2016
Related to openQA Project - action #34549: implement a way to retire needles or "Keep copies of need... Rejected 09/04/2018
Related to openQA Project - action #25572: [tools][needles]needles pushing can interfer with "fetchn... Resolved 26/09/2017
Related to openQA Project - action #59184: Research about testing with a custom git ref Resolved 07/11/2019
Related to openQA Project - action #58304: A personal activity view for developers New 17/10/2019
Related to openQA Project - action #62600: Improve error output when calling openqa-clone-custom-git... Workable 23/01/2020

History

#1 Updated by okurz 4 months ago

  • Related to action #49502: Automatic validation test on github PRs added

#2 Updated by okurz 4 months ago

#3 Updated by okurz 4 months ago

  • Description updated (diff)

#4 Updated by okurz 4 months ago

#5 Updated by okurz 4 months ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version set to Current Sprint

blocked by subtasks

#6 Updated by okurz 4 months ago

  • Related to action #34549: implement a way to retire needles or "Keep copies of needles while running tests" added

#7 Updated by okurz 4 months ago

  • Blocks action #25572: [tools][needles]needles pushing can interfer with "fetchneedles" added

#8 Updated by okurz 3 months ago

  • Blocks deleted (action #25572: [tools][needles]needles pushing can interfer with "fetchneedles")

#9 Updated by okurz 3 months ago

  • Related to action #25572: [tools][needles]needles pushing can interfer with "fetchneedles" added

#10 Updated by okurz 3 months ago

Brain-storming between mkittler and okurz 2019-11-19: We propose to start with a system level test, e.g. copy/extension of full stack test that has no machines, products, test suites nor job templates setup in database and reads everything from a git repo. The starting point would be e.g. isos post distri=https://github.com/os-autoinst/os-autoinst-distri-openQA which then, e.g. in the usual isos post controller method, handles:

  1. checkout on webUI host
  2. evaluate job templates from git repo, e.g. by convention from ./schedule.yml
  3. trigger jobs
  4. map needle + source code
  5. test done: Needles need to be referenced from git, needle editor need to save to master by default and the custom git repo if used

EDIT: 2020-01-25: Allowing job templates from git repo as well introduces another source where settings come from and can be confusing when trying to find out what changed in jobs and where the change comes from. For this I provided suggestions in #19720 e.g. to mark the source of settings in the database table. So this shouldn't stop us to add a test distri VCS as another source of settings

#11 Updated by okurz 3 months ago

  • Related to action #59184: Research about testing with a custom git ref added

#12 Updated by okurz 3 months ago

  • Related to action #58304: A personal activity view for developers added

#13 Updated by okurz 2 months ago

  • Description updated (diff)
  • Status changed from Blocked to Workable
  • Assignee deleted (okurz)

Updated with feedback from #59184 , incorporated the use cases and suggestions and problems into the description. Setting back to "Workable" for anyone to further specify subtickets which cover all suggestions and user stories.

#14 Updated by okurz 2 months ago

  • Description updated (diff)

#15 Updated by okurz 2 months ago

  • Description updated (diff)

#16 Updated by okurz 26 days ago

  • Related to action #62600: Improve error output when calling openqa-clone-custom-git-refspec with wrong args, not just exit code added

#17 Updated by cdywan 22 days ago

  • Assignee set to coolo

#18 Updated by mkittler 19 days ago

I see two overall challenges and I'd like to point out their current state and possible improvements:

  1. Improve the way we get custom/versioned tests into openQA.
    1. Currently one can override CASEDIR and NEEDLES_DIR to achieve that (e.g. via openqa-clone-custom-git-refspec). The way the git cloning is done by os-autoinst is not very efficient but it works.
    2. We likely want a more CI-like approach. It is conceivable to make openQA's job scheduling version-aware, e.g. allowing to override DISTRI with a Git URL. The referenced distri would contain test suites, job templates and other scheduling definitions we use for producing jobs (e.g. as a YAML file). Preferably test code and needles are contained by the same repository¹. See https://github.com/os-autoinst/openQA/pull/2706 for a draft.
    3. Point 1.2. leads to thinking about scalability, e.g. one might want to run os-autoinst and maybe even a dedicated openQA instance within a Kubernetes cluster. It isn't clear how test results would be transferred from such test runs into common web UI.
  2. Displaying custom/versioned tests within the openQA web UI.
    1. The job module code view is not version aware. I suppose making it a link to a Git repository would be a small change and helpful regardless how we continue with our design for 1..
    2. All places where needle candidates are displayed are not version aware. In contrast to 2.1. this is a rather challenging problem. It is not clear what the best solution is and how the it would interfere with our design for 1..
    3. The dashboard is not version aware so custom tests are not filtered out. It has been discussed what filtering policies we want and likely a view per user would make sense. Just supporting this filtering on job level would be easy. However, we likely want custom job groups and scheduling tables as well.
  3. Cleanup of custom/versioned tests results, assets and needles.
    1. Currently custom tests results and assets are cleaned up following the global cleanup policies which mainly depend on the global group the job belongs to. If we decide to implement custom/user-specific groups the cleanup algorithm would naturally consider these as separate (and therefore separately configurable) groups. However, at least the asset cleanup page (which shows e.g. which groups an asset is accounted for) would likely not scale when having lots of custom groups within an instance and therefore needed to be adjusted.
    2. As mentioned in 2.2 custom needles are challenging. They also require extra work regarding cleanup.

¹ It is generally questionable whether we want to support splitted test and needle repositories for running versioned/custom tests.

#19 Updated by okurz 19 days ago

mkittler wrote:

I see two overall challenges […]

the above notes are based on a discussion that mkittler and me had so I am in line with him :) So I am not objecting, just adding information:

The way the git cloning is done by os-autoinst is not very efficient but it works.

  • cons: Not network-efficient to reclone the same or similar repos over and over again from external upstream; not easily possible to display test/needles from webui as only the worker knew the state of code at the time of test execution
  • pros: automatic cleanup of temporary test distribution checkouts

It isn't clear how test results would be transferred from [kubernetes] test runs into common web UI.

Our idea so far was to support an "asynchronous update", e.g. whenever a worker is started on a pool dir with existing test data walks over dir content and publishes to webui, webui decises if there are any updates necessary as the test data might be already complete on the webui host.

#20 Updated by mkittler 19 days ago

Our idea so far was to support an "asynchronous update" [...]

Right, I wanted to add a comment about that for the "offline worker" user case but couldn't find the ticket. However, it is actually related here as well as it might improve the CI/scalability point. You could run os-autoinst somewhere (e.g. isolated within a container) and upload the results later to any web UI when this is wanted. I'd also like to note that the required modifications to the worker and web UI shouldn't be hard to implement. Decoupling test execution from exporting test results to a web UI also seems nice from the perspective of the overall software architecture. It would also help with the problem of incomplete jobs without logs.

#21 Updated by okurz 14 days ago

  • Due date set to 05/02/2020

due to changes in a related task

#22 Updated by okurz about 17 hours ago

Discussed in tools-team meeting 2020-02-18: I personally tried to phrase two different paths to follow - which do not need to conflict:

  1. Scaled openQA for user centric tests: More motivated by #58304 . Few, big openQA instances that support existing product-centric testing as well as derived user-centric testing sharing assets, e.g. "As QA SLE engineer I want to run tests against build X of last version of SLE in development against my own tests git branch to find out if a potential bug fix can work or to investigate test failures or try a fix".
  2. Integrated openQA tests within existing CI systems: More motivated by #48641 . Numerous independant openQA instances spawned dynamically within the scope of already existing CI systems, e.g. gitlab CI, travis CI, spawning openQA instances (or linking to existing ones internally).

We agreed that 1. is the overall more challenging but also more important path to follow. This can mean: Have everything optionally linked to a user, e.g. add a user column to every database table, all tests+needles per user. Personally I suggest to exactly try that out in a scratch refactoring, e.g. add a user column to every database table + tests + needles and show how each influenced component can handle the user. While doing little steps as improvement in this direction we should still follow this overall vision.

Also available in: Atom PDF