Project

General

Profile

action #58184

[epic][use case] full version control awareness within openQA, e.g. user forks and branches

Added by okurz 8 months ago. Updated 2 months ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2018-03-23
Due date:
% Done:

33%

Estimated time:
(Total: 0.00 h)
Difficulty:
Duration:

Description

Motivation

This is linked to Use case 4 and motivated by a discussion by the QA tools team in the weekly meeting 2019-10-15

User story

As a test case contributor during test case development I want to run tests on production instances with all necessary changes recorded in version control before merging to master so that my change will have minimal unexpected impact (test regressions) on existing tests

Further user stories (from https://confluence.suse.com/pages/viewpage.action?pageId=365527173)

  1. I want to start a job based on a modified test in production (In production tests can behave differently, for example because of the heavier load) -> see openqa-clone-job + CASEDIR
  2. I want to edit needles and test if they work before proposing changes
  3. I want to compare the results of a certain job group between two of my branches
  4. I want to schedule a test 100 times without it showing up in the group overview -> see statistical-investigation
  5. I want to trigger multiple cloned jobs for each pull-request (Sometimes you want to trigger VR for different jobs against the same PR. it would be nice to do that in one command line)
  6. I want to trigger the relevant tests automatically by creating a PR

Implications and suggestions

  • The usual test contributor workflows should be supported and made easier by making openQA fully aware of tests triggered for development purposes without negatively impacting existing validation tests

    • Potential impact on asset management
    • No pollution of validation test reports by development tests
  • If there are new/modified needles involved, the existing workflow cannot handle that. The current practice is:

    • Test your changes (and possibly needle changes) locally and create PR(s)
    • Edit needles online and save them (then they will be committed to master). Requires admin rights
  • Cloning cancelled or incomplete jobs currently does not work as openqa-clone-custom-git-refspec requires the vars.json file from a completed job with this file uploaded

  • Replace "fetchneedles" by inherent git support

  • Provide support for github pull request validation

  • DONE: Extend openqa-clone-custom-git-refspec to accept list of source tests to clone -> https://github.com/os-autoinst/openQA/pull/2577

  • DONE: openqa-clone-custom-git-refspec: Output in markdown format for easy copy/pasting into git commit messages and github PR comments -> https://github.com/os-autoinst/openQA/pull/2577

  • openqa-clone-custom-git-refspec: Provide link to /tests/overview page for the custom build when multiple tests have been cloned

  • Make the trigger source of test jobs apparent, e.g. the source git repositories

  • #14818#note-18 : "Tim got a ticket from Ray that the docker test failed and wants openQA to reproduce the issue and pause at the beginning of the docker test. Afterwards he wants openQA to make a disk snapshot and step through the test execution to find out where the problem is. After he found out, he reloads the snapshot to tweak the execution. During this process, openQA records his steps and allows to add needles."


Subtasks

action #48641: Trigger openQA tests in pull requests of any product github pull requestNew

action #33745: Improve handling external Git repositories (for needles)New

action #56789: New needles from git repository not working with openqa-clone-custom-git-refspecNew

action #45302: [epic] smarter fetchneedles (was: fetchneedles should ensure we are always on a branch (and try to self-repair))New

action #63133: SLE needles git repo looses upstream branch configuration, needles are saved but fail to be pushedResolvedokurz

action #54965: Cannot inspect the source code of the tests from my forkResolvedokurz

action #60272: Make fetching custom git repos (e.g. needles) more efficientNew


Related issues

Related to QA - action #49502: Automatic validation test on github PRsNew2019-03-20

Related to openQA Project - action #57452: Automatic summary of failuresRejected2019-09-27

Related to openQA Project - action #10192: Improve source code windowWorkable2016-01-12

Related to openQA Project - action #34549: implement a way to retire needles or "Keep copies of needles while running tests"Rejected2018-04-09

Related to openQA Project - action #25572: [tools][needles]needles pushing can interfer with "fetchneedles"Resolved2017-09-26

Related to openQA Project - action #59184: Research about testing with a custom git refResolved2019-11-07

Related to openQA Project - action #58304: A personal activity view for developers New2019-10-17

Related to openQA Project - action #62600: Improve error output when calling openqa-clone-custom-git-refspec with wrong args, not just exit codeResolved2020-01-23

Related to openQA Project - action #66071: TEST is overridden in parent job when doing `openqa-clone-custom-git-refspec`New2020-04-26

Related to openQA Tests - action #15132: [saga][epic] Better structure of test plans in main.pmBlocked2018-11-202020-11-27

History

#1 Updated by okurz 8 months ago

  • Related to action #49502: Automatic validation test on github PRs added

#2 Updated by okurz 8 months ago

#3 Updated by okurz 8 months ago

  • Description updated (diff)

#4 Updated by okurz 8 months ago

#5 Updated by okurz 7 months ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version set to Current Sprint

blocked by subtasks

#6 Updated by okurz 7 months ago

  • Related to action #34549: implement a way to retire needles or "Keep copies of needles while running tests" added

#7 Updated by okurz 7 months ago

  • Blocks action #25572: [tools][needles]needles pushing can interfer with "fetchneedles" added

#8 Updated by okurz 7 months ago

  • Blocks deleted (action #25572: [tools][needles]needles pushing can interfer with "fetchneedles")

#9 Updated by okurz 7 months ago

  • Related to action #25572: [tools][needles]needles pushing can interfer with "fetchneedles" added

#10 Updated by okurz 7 months ago

Brain-storming between mkittler and okurz 2019-11-19: We propose to start with a system level test, e.g. copy/extension of full stack test that has no machines, products, test suites nor job templates setup in database and reads everything from a git repo. The starting point would be e.g. isos post distri=https://github.com/os-autoinst/os-autoinst-distri-openQA which then, e.g. in the usual isos post controller method, handles:

  1. checkout on webUI host
  2. evaluate job templates from git repo, e.g. by convention from ./schedule.yml
  3. trigger jobs
  4. map needle + source code
  5. test done: Needles need to be referenced from git, needle editor need to save to master by default and the custom git repo if used

EDIT: 2020-01-25: Allowing job templates from git repo as well introduces another source where settings come from and can be confusing when trying to find out what changed in jobs and where the change comes from. For this I provided suggestions in #19720 e.g. to mark the source of settings in the database table. So this shouldn't stop us to add a test distri VCS as another source of settings

#11 Updated by okurz 6 months ago

  • Related to action #59184: Research about testing with a custom git ref added

#12 Updated by okurz 6 months ago

  • Related to action #58304: A personal activity view for developers added

#13 Updated by okurz 6 months ago

  • Description updated (diff)
  • Status changed from Blocked to Workable
  • Assignee deleted (okurz)

Updated with feedback from #59184 , incorporated the use cases and suggestions and problems into the description. Setting back to "Workable" for anyone to further specify subtickets which cover all suggestions and user stories.

#14 Updated by okurz 6 months ago

  • Description updated (diff)

#15 Updated by okurz 6 months ago

  • Description updated (diff)

#16 Updated by okurz 4 months ago

  • Related to action #62600: Improve error output when calling openqa-clone-custom-git-refspec with wrong args, not just exit code added

#17 Updated by cdywan 4 months ago

  • Assignee set to coolo

#18 Updated by mkittler 4 months ago

I see two overall challenges and I'd like to point out their current state and possible improvements:

  1. Improve the way we get custom/versioned tests into openQA.
    1. Currently one can override CASEDIR and NEEDLES_DIR to achieve that (e.g. via openqa-clone-custom-git-refspec). The way the git cloning is done by os-autoinst is not very efficient but it works.
    2. We likely want a more CI-like approach. It is conceivable to make openQA's job scheduling version-aware, e.g. allowing to override DISTRI with a Git URL. The referenced distri would contain test suites, job templates and other scheduling definitions we use for producing jobs (e.g. as a YAML file). Preferably test code and needles are contained by the same repository¹. See https://github.com/os-autoinst/openQA/pull/2706 for a draft.
    3. Point 1.2. leads to thinking about scalability, e.g. one might want to run os-autoinst and maybe even a dedicated openQA instance within a Kubernetes cluster. It isn't clear how test results would be transferred from such test runs into common web UI.
  2. Displaying custom/versioned tests within the openQA web UI.
    1. The job module code view is not version aware. I suppose making it a link to a Git repository would be a small change and helpful regardless how we continue with our design for 1..
    2. All places where needle candidates are displayed are not version aware. In contrast to 2.1. this is a rather challenging problem. It is not clear what the best solution is and how the it would interfere with our design for 1..
    3. The dashboard is not version aware so custom tests are not filtered out. It has been discussed what filtering policies we want and likely a view per user would make sense. Just supporting this filtering on job level would be easy. However, we likely want custom job groups and scheduling tables as well.
  3. Cleanup of custom/versioned tests results, assets and needles.
    1. Currently custom tests results and assets are cleaned up following the global cleanup policies which mainly depend on the global group the job belongs to. If we decide to implement custom/user-specific groups the cleanup algorithm would naturally consider these as separate (and therefore separately configurable) groups. However, at least the asset cleanup page (which shows e.g. which groups an asset is accounted for) would likely not scale when having lots of custom groups within an instance and therefore needed to be adjusted.
    2. As mentioned in 2.2 custom needles are challenging. They also require extra work regarding cleanup.

¹ It is generally questionable whether we want to support splitted test and needle repositories for running versioned/custom tests.

#19 Updated by okurz 4 months ago

mkittler wrote:

I see two overall challenges […]

the above notes are based on a discussion that mkittler and me had so I am in line with him :) So I am not objecting, just adding information:

The way the git cloning is done by os-autoinst is not very efficient but it works.

  • cons: Not network-efficient to reclone the same or similar repos over and over again from external upstream; not easily possible to display test/needles from webui as only the worker knew the state of code at the time of test execution
  • pros: automatic cleanup of temporary test distribution checkouts

It isn't clear how test results would be transferred from [kubernetes] test runs into common web UI.

Our idea so far was to support an "asynchronous update", e.g. whenever a worker is started on a pool dir with existing test data walks over dir content and publishes to webui, webui decises if there are any updates necessary as the test data might be already complete on the webui host.

#20 Updated by mkittler 4 months ago

Our idea so far was to support an "asynchronous update" [...]

Right, I wanted to add a comment about that for the "offline worker" user case but couldn't find the ticket. However, it is actually related here as well as it might improve the CI/scalability point. You could run os-autoinst somewhere (e.g. isolated within a container) and upload the results later to any web UI when this is wanted. I'd also like to note that the required modifications to the worker and web UI shouldn't be hard to implement. Decoupling test execution from exporting test results to a web UI also seems nice from the perspective of the overall software architecture. It would also help with the problem of incomplete jobs without logs.

#21 Updated by okurz 4 months ago

  • Due date set to 2020-02-05

due to changes in a related task

#22 Updated by okurz 3 months ago

Discussed in tools-team meeting 2020-02-18: I personally tried to phrase two different paths to follow - which do not need to conflict:

  1. Scaled openQA for user centric tests: More motivated by #58304 . Few, big openQA instances that support existing product-centric testing as well as derived user-centric testing sharing assets, e.g. "As QA SLE engineer I want to run tests against build X of last version of SLE in development against my own tests git branch to find out if a potential bug fix can work or to investigate test failures or try a fix".
  2. Integrated openQA tests within existing CI systems: More motivated by #48641 . Numerous independant openQA instances spawned dynamically within the scope of already existing CI systems, e.g. gitlab CI, travis CI, spawning openQA instances (or linking to existing ones internally).

We agreed that 1. is the overall more challenging but also more important path to follow. This can mean: Have everything optionally linked to a user, e.g. add a user column to every database table, all tests+needles per user. Personally I suggest to exactly try that out in a scratch refactoring, e.g. add a user column to every database table + tests + needles and show how each influenced component can handle the user. While doing little steps as improvement in this direction we should still follow this overall vision.

For the approach to segment "interesting areas" one could follow gitlab/github/travisci/circleci and such and have URLs like /$repo/$user . For openQA that could mean that all existing URLs would be accessible instead or as well below that prefix, e.g. /$repo/$user/tests . As we already have a big list of top-level routes that can prevent certain "$repo" names. The alternative being to break the compatibility of webui routes at least, another to add a prefix, e.g. "/u/$repo/$user", which isn't very clear or a more explicit but longer part.

#23 Updated by okurz 2 months ago

  • Status changed from Workable to Blocked
  • Assignee changed from coolo to okurz

We could still flesh out this epic more but we have identified some subtasks and related features to solve first.

Important part for finding out specific requirements: "[spike][timeboxed:20h] complete test definition from yaml schedule in git checked out test distribution"

#24 Updated by okurz about 1 month ago

  • Related to action #66071: TEST is overridden in parent job when doing `openqa-clone-custom-git-refspec` added

#25 Updated by okurz 21 days ago

  • Related to action #15132: [saga][epic] Better structure of test plans in main.pm added

Also available in: Atom PDF