QA » openQA Project

Brain-storming between mkittler and okurz 2019-11-19: We propose to start with a system level test, e.g. copy/extension of full stack test that has no machines, products, test suites nor job templates setup in database and reads everything from a git repo. The starting point would be e.g. isos post distri=https://github.com/os-autoinst/os-autoinst-distri-openQA which then, e.g. in the usual isos post controller method, handles:

1. checkout on webUI host
2. evaluate job templates from git repo, e.g. by convention from ./schedule.yml
3. trigger jobs
4. map needle + source code
5. test done: Needles need to be referenced from git, needle editor need to save to master by default and the custom git repo if used

EDIT: 2020-01-25: Allowing job templates from git repo as well introduces another source where settings come from and can be confusing when trying to find out what changed in jobs and where the change comes from. For this I provided suggestions in #19720 e.g. to mark the source of settings in the database table. So this shouldn't stop us to add a test distri VCS as another source of settings

I see two overall challenges and I'd like to point out their current state and possible improvements:

1. Improve the way we get custom/versioned tests into openQA.
   1. Currently one can override CASEDIR and NEEDLES_DIR to achieve that (e.g. via openqa-clone-custom-git-refspec). The way the git cloning is done by os-autoinst is not very efficient but it works.
   2. We likely want a more CI-like approach. It is conceivable to make openQA's job scheduling version-aware, e.g. allowing to override DISTRI with a Git URL. The referenced distri would contain test suites, job templates and other scheduling definitions we use for producing jobs (e.g. as a YAML file). Preferably test code and needles are contained by the same repository¹. See https://github.com/os-autoinst/openQA/pull/2706 for a draft.
   3. Point 1.2. leads to thinking about scalability, e.g. one might want to run os-autoinst and maybe even a dedicated openQA instance within a Kubernetes cluster. It isn't clear how test results would be transferred from such test runs into common web UI.
2. Displaying custom/versioned tests within the openQA web UI.
   1. The job module code view is not version aware. I suppose making it a link to a Git repository would be a small change and helpful regardless how we continue with our design for 1..
   2. All places where needle candidates are displayed are not version aware. In contrast to 2.1. this is a rather challenging problem. It is not clear what the best solution is and how the it would interfere with our design for 1..
   3. The dashboard is not version aware so custom tests are not filtered out. It has been discussed what filtering policies we want and likely a view per user would make sense. Just supporting this filtering on job level would be easy. However, we likely want custom job groups and scheduling tables as well.
3. Cleanup of custom/versioned tests results, assets and needles.
   1. Currently custom tests results and assets are cleaned up following the global cleanup policies which mainly depend on the global group the job belongs to. If we decide to implement custom/user-specific groups the cleanup algorithm would naturally consider these as separate (and therefore separately configurable) groups. However, at least the asset cleanup page (which shows e.g. which groups an asset is accounted for) would likely not scale when having lots of custom groups within an instance and therefore needed to be adjusted.
   2. As mentioned in 2.2 custom needles are challenging. They also require extra work regarding cleanup.

¹ It is generally questionable whether we want to support splitted test and needle repositories for running versioned/custom tests.

mkittler wrote:

I see two overall challenges […]

the above notes are based on a discussion that mkittler and me had so I am in line with him :) So I am not objecting, just adding information:

The way the git cloning is done by os-autoinst is not very efficient but it works.

• cons: Not network-efficient to reclone the same or similar repos over and over again from external upstream; not easily possible to display test/needles from webui as only the worker knew the state of code at the time of test execution
• pros: automatic cleanup of temporary test distribution checkouts

It isn't clear how test results would be transferred from [kubernetes] test runs into common web UI.

Our idea so far was to support an "asynchronous update", e.g. whenever a worker is started on a pool dir with existing test data walks over dir content and publishes to webui, webui decises if there are any updates necessary as the test data might be already complete on the webui host.

Our idea so far was to support an "asynchronous update" [...]

Right, I wanted to add a comment about that for the "offline worker" user case but couldn't find the ticket. However, it is actually related here as well as it might improve the CI/scalability point. You could run os-autoinst somewhere (e.g. isolated within a container) and upload the results later to any web UI when this is wanted. I'd also like to note that the required modifications to the worker and web UI shouldn't be hard to implement. Decoupling test execution from exporting test results to a web UI also seems nice from the perspective of the overall software architecture. It would also help with the problem of incomplete jobs without logs.

Discussed in tools-team meeting 2020-02-18: I personally tried to phrase two different paths to follow - which do not need to conflict:

1. Scaled openQA for user centric tests: More motivated by #58304 . Few, big openQA instances that support existing product-centric testing as well as derived user-centric testing sharing assets, e.g. "As QA SLE engineer I want to run tests against build X of last version of SLE in development against my own tests git branch to find out if a potential bug fix can work or to investigate test failures or try a fix".
2. Integrated openQA tests within existing CI systems: More motivated by #48641 . Numerous independant openQA instances spawned dynamically within the scope of already existing CI systems, e.g. gitlab CI, travis CI, spawning openQA instances (or linking to existing ones internally).

We agreed that 1. is the overall more challenging but also more important path to follow. This can mean: Have everything optionally linked to a user, e.g. add a user column to every database table, all tests+needles per user. Personally I suggest to exactly try that out in a scratch refactoring, e.g. add a user column to every database table + tests + needles and show how each influenced component can handle the user. While doing little steps as improvement in this direction we should still follow this overall vision.

For the approach to segment "interesting areas" one could follow gitlab/github/travisci/circleci and such and have URLs like /$repo/$user . For openQA that could mean that all existing URLs would be accessible instead or as well below that prefix, e.g. /$repo/$user/tests . As we already have a big list of top-level routes that can prevent certain "$repo" names. The alternative being to break the compatibility of webui routes at least, another to add a prefix, e.g. "/u/$repo/$user", which isn't very clear or a more explicit but longer part.

See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html