QA

Add Rodion and Matthias(stand-in Oliver as PO) in the loop.

@all, feel free give suitable flag to category

We already make some steps in #37958. In short: for the next sprint we are going to move staging tests to the new scheduling mechanism so no unintended changes to the schedule affect staging. The next step would be to use that list of the module to provide some warning in case PR modifies any of test modules executed in staging, so we can request VR for it.

Idea which we have in parallel is some kind of intermediate state for the master branch, for which we will run actual openQA job, and only if that succeeds - move it to the production.

Does it cover motivation of this ticket?

riafarov wrote:

We already make some steps in #37958. In short: for the next sprint we are going to move staging tests to the new scheduling mechanism so no unintended changes to the schedule affect staging. The next step would be to use that list of the module to provide some warning in case PR modifies any of test modules executed in staging, so we can request VR for it.

Idea which we have in parallel is some kind of intermediate state for the master branch, for which we will run actual openQA job, and only if that succeeds - move it to the production.

Does it cover motivation of this ticket?

IMO #37958 is part of this ticket motivation as well as #47441 "[functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github",

After read the original ideas in #44327 and several discussion in emails with coolo and Oliver, I distinguish the self-test(or validation) into 2 stages as the fully-automation requirements, but feel free correct me:

1)Before commit:
-locally pre-checking(testing)to help QA developer for maximum quality of the modified codes before commit, tuning their codes with some standard(such as perl-tidy), then they don't need fix/rebase time to time after the committing,
-we also can put in this stage the "automatic analyze the modified codes for which modules in openQA tests, and automatic generate the according tests, and help QA developer automatic trigger(such as "openqa-clone-custom-git-refspec $GITHUB_PR_URL $OPENQA_TEST_URL"), or just suggest the according tests should be validated before commit,
-roughly idea is this can be made by github hooks to such as pre-commit, and the test status can be automatic generated to such as prepare-commit-msg. For this, reviewers will feel easier.

2)On pull request: we still need bots and integrate to github,
-to analyze the modified codes for the affected modules, automatic assign/notify reviewers(the modified code's owner/maintainer for example)
-to read the test status, compare and schedule according test modules, trigger the additional tests, waiting finish and update status to PR

It looks to me, #37958 #47441 are much more in the above stage 2) and can be extended? Currently I don't have much idea how the 2) be made, coolo has pointed to me 2 links in my description and "the hard part is actually deciding what to schedule". Here need you all's intelligence.

therefore, before any implementation it need an overall consideration to a integrated design.

Add Sunny and Yifan as well, in case guys from them are also interested on this interesting automation project.

cachen wrote:

riafarov wrote:

We already make some steps in #37958. In short: for the next sprint we are going to move staging tests to the new scheduling mechanism so no unintended changes to the schedule affect staging. The next step would be to use that list of the module to provide some warning in case PR modifies any of test modules executed in staging, so we can request VR for it.

Idea which we have in parallel is some kind of intermediate state for the master branch, for which we will run actual openQA job, and only if that succeeds - move it to the production.

Does it cover motivation of this ticket?

IMO #37958 is part of this ticket motivation as well as #47441 "[functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github",

After read the original ideas in #44327 and several discussion in emails with coolo and Oliver, I distinguish the self-test(or validation) into 2 stages as the fully-automation requirements, but feel free correct me:

1)Before commit:
-locally pre-checking(testing)to help QA developer for maximum quality of the modified codes before commit, tuning their codes with some standard(such as perl-tidy), then they don't need fix/rebase time to time after the committing,
-we also can put in this stage the "automatic analyze the modified codes for which modules in openQA tests, and automatic generate the according tests, and help QA developer automatic trigger(such as "openqa-clone-custom-git-refspec $GITHUB_PR_URL $OPENQA_TEST_URL"), or just suggest the according tests should be validated before commit,
-roughly idea is this can be made by github hooks to such as pre-commit, and the test status can be automatic generated to such as prepare-commit-msg. For this, reviewers will feel easier.

2)On pull request: we still need bots and integrate to github,
-to analyze the modified codes for the affected modules, automatic assign/notify reviewers(the modified code's owner/maintainer for example)
-to read the test status, compare and schedule according test modules, trigger the additional tests, waiting finish and update status to PR

It looks to me, #37958 #47441 are much more in the above stage 2) and can be extended? Currently I don't have much idea how the 2) be made, coolo has pointed to me 2 links in my description and "the hard part is actually deciding what to schedule". Here need you all's intelligence.
I would say checking everything before even committing is an overkill, so yes, it's more focused on 2), but all CI checks we do with travis can be triggered manually in local setup. So it doesn't matter if we have working solution, it will be possible to trigger locally (will require CI setup similar to travis though). But yeah, it's good to know that we share same concerns, so can address it ;)

One note: If I understand different ideas right then you would appreciate automatic openQA verification runs for every PR in github.com/os-autoinst/os-autoinst-distri-opensuse/ . But I only see tests possible that run on a public instance for that public repo. That means we can not run SLE tests on openqa.suse.de as access is not possible for all contributors. I would not enable automatic SLE tests which are not available for all. Maybe we can even run all SLE Maintenance tests in the open just want to keep "pre-public-Beta" builds of new versions in development internal. I know that technically this is not a problem at all. This is really about politics, marketing and a proper process. We could ask responsibles within SUSE how to make it public and after the "political approval" we can do that. #52004 is the corresponding ticket for that.

How about the code changes on projects "openQA / os-autoinst / os-autoinst-needles-sle"? Will this poo also cover the "Automatic validation" on their code/needle changes?

llzhao wrote:

How about the code changes on projects "openQA / os-autoinst / os-autoinst-needles-sle"? Will this poo also cover the "Automatic validation" on their code/needle changes?

Well, first, this ticket can be seen as more generic than just openQA as it is in the general "QA" context. Also there is no one currently even planning to implement this change as described here so it is undefined if the mentioned projects would be covered or not.

However I can state what I find feasible and useful. os-autoinst and openQA already have a good test coverage within their projects so I consider it "done" already. What can be improved a lot is test coverage for "exotic backends", e.g. the backends that are used for s390x, powerVM, bare-metal. This is something I am expecting from the power users that rely on these backends as this is Out-of-Scope for SUSE QE Tools.

For os-autoinst-needles-sle there are already some tiny static CI checks but not more. As long as needle changes are automatically submitted to the master branch by the use of the webUI of openQA I see no use in extending the existing CI checks as the minority of users that create merge requests would be punished for more problems that others have introduced working around the CI. The only feasible option I see to improve the validation of SLE needle changes would be to have all automatic commits from the webUI to go into a different branch, then create automatic merge requests with more checks and tests and review before accepting into the master branch or another "stable" branch which would be used for some or all products on osd.