QA » openQA Project

Motivation¶

• Make it easier to come up with test scenarios for verification runs
• Make it harder to miss verification runs of test scenarios that are important for the changed code

Acceptance criteria¶

• AC1: After creating or updating a PR one receives feedback in form of one or more openQA test results, e.g. in form of a failed check in GitHub's UI.
• AC2: The openQA tests mentioned in AC1 execute the changed code so as much of the changed code as possible is run.
• AC3: The openQA tests mentioned in AC1 generally pass. In case of a failure the likeliness that the failure relates to the changes must be very high and a passing run (without the changes from the PR) must be available for comparison.
• AC4: Not too many openQA jobs are triggered, e.g. changing code in a utility module that is used everywhere does not trigger all conceivable test scenarios. Redundant test runs are avoided.

Further remarks¶

• openQA has support for running test jobs as CI checks, see https://open.qa/docs/#_running_openqa_jobs_as_ci_checks but the crux here is to decide which test scenarios to run considering the diff of the PR.
• Supplying the required assets will be challenging as well. This already meant not using the webhook-based approach in os-autoinst-distri-openQA and using just openqa-cli schedule --monitor … to be able to scrape o3 for assets before the actual scheduling (https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/.github/workflows/openqa.yml).
• Considering OSD-specific test scenarios will very likely be wanted. This is certainly not a must have (especially from the beginning) but we should definitely take it into account.