action #49502

Automatic validation test on github PRs

Added by cachen 10 months ago. Updated 10 months ago.

Status:NewStart date:20/03/2019
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Duration:

Description

Purpose:
-To reduce the unexpected disturbance by other's code change especially on those sharing lib/module
-To make QA developers and code reviewers life much easier

What we have:
-QSF team had an implementation which can manually trigger the expected validation test on OSD, description is in http://open.qa/docs/#_triggering_tests_based_on_an_any_remote_git_refspec_or_open_github_pull_request

What are missing:
-ideally to make it fully automated, it need a review bot integrate to github can automatic monitor the PR status -> trigger/schedule validation tests on OSD or Staging area -> update status to github

The points from coolo:
-easiest to do is in python
-https://developer.github.com/v3/repos/statuses/ is the github api to set a status report and there are python apis for it
-https://github.com/openSUSE/obs-tools/tree/master/pull_request_package is a ruby bot that obs team maintains to set a status in their PRs if the package builds
-so you would poll the pull requests if the status is set - and if it isn't, schedule a set of openQA tests the way okurz mentioned, wait for them to finish and update the github status

What we need:
-contributor who is interested on python and github bot for automation :)


Related issues

Related to openQA Project - action #58184: [epic][use case] full version control awareness within op... Workable 23/03/2018

History

#1 Updated by cachen 10 months ago

Add Rodion and Matthias(stand-in Oliver as PO) in the loop.

@all, feel free give suitable flag to category

#2 Updated by riafarov 10 months ago

We already make some steps in #37958. In short: for the next sprint we are going to move staging tests to the new scheduling mechanism so no unintended changes to the schedule affect staging. The next step would be to use that list of the module to provide some warning in case PR modifies any of test modules executed in staging, so we can request VR for it.

Idea which we have in parallel is some kind of intermediate state for the master branch, for which we will run actual openQA job, and only if that succeeds - move it to the production.

Does it cover motivation of this ticket?

#3 Updated by cachen 10 months ago

riafarov wrote:

We already make some steps in #37958. In short: for the next sprint we are going to move staging tests to the new scheduling mechanism so no unintended changes to the schedule affect staging. The next step would be to use that list of the module to provide some warning in case PR modifies any of test modules executed in staging, so we can request VR for it.


Idea which we have in parallel is some kind of intermediate state for the master branch, for which we will run actual openQA job, and only if that succeeds - move it to the production.


Does it cover motivation of this ticket?

IMO #37958 is part of this ticket motivation as well as #47441 "[functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github",

After read the original ideas in #44327 and several discussion in emails with coolo and Oliver, I distinguish the self-test(or validation) into 2 stages as the fully-automation requirements, but feel free correct me:

1)Before commit:
-locally pre-checking(testing)to help QA developer for maximum quality of the modified codes before commit, tuning their codes with some standard(such as perl-tidy), then they don't need fix/rebase time to time after the committing,
-we also can put in this stage the "automatic analyze the modified codes for which modules in openQA tests, and automatic generate the according tests, and help QA developer automatic trigger(such as "openqa-clone-custom-git-refspec $GITHUB_PR_URL $OPENQA_TEST_URL"), or just suggest the according tests should be validated before commit,
-roughly idea is this can be made by github hooks to such as pre-commit, and the test status can be automatic generated to such as prepare-commit-msg. For this, reviewers will feel easier.

2)On pull request: we still need bots and integrate to github,
-to analyze the modified codes for the affected modules, automatic assign/notify reviewers(the modified code's owner/maintainer for example)
-to read the test status, compare and schedule according test modules, trigger the additional tests, waiting finish and update status to PR

It looks to me, #37958 #47441 are much more in the above stage 2) and can be extended? Currently I don't have much idea how the 2) be made, coolo has pointed to me 2 links in my description and "the hard part is actually deciding what to schedule". Here need you all's intelligence.

#4 Updated by cachen 10 months ago

therefore, before any implementation it need an overall consideration to a integrated design.

Add Sunny and Yifan as well, in case guys from them are also interested on this interesting automation project.

#5 Updated by riafarov 10 months ago

cachen wrote:

riafarov wrote:

We already make some steps in #37958. In short: for the next sprint we are going to move staging tests to the new scheduling mechanism so no unintended changes to the schedule affect staging. The next step would be to use that list of the module to provide some warning in case PR modifies any of test modules executed in staging, so we can request VR for it.


Idea which we have in parallel is some kind of intermediate state for the master branch, for which we will run actual openQA job, and only if that succeeds - move it to the production.


Does it cover motivation of this ticket?


IMO #37958 is part of this ticket motivation as well as #47441 "[functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github",


After read the original ideas in #44327 and several discussion in emails with coolo and Oliver, I distinguish the self-test(or validation) into 2 stages as the fully-automation requirements, but feel free correct me:


1)Before commit:

-locally pre-checking(testing)to help QA developer for maximum quality of the modified codes before commit, tuning their codes with some standard(such as perl-tidy), then they don't need fix/rebase time to time after the committing,

-we also can put in this stage the "automatic analyze the modified codes for which modules in openQA tests, and automatic generate the according tests, and help QA developer automatic trigger(such as "openqa-clone-custom-git-refspec $GITHUB_PR_URL $OPENQA_TEST_URL"), or just suggest the according tests should be validated before commit,

-roughly idea is this can be made by github hooks to such as pre-commit, and the test status can be automatic generated to such as prepare-commit-msg. For this, reviewers will feel easier.


2)On pull request: we still need bots and integrate to github,

-to analyze the modified codes for the affected modules, automatic assign/notify reviewers(the modified code's owner/maintainer for example)

-to read the test status, compare and schedule according test modules, trigger the additional tests, waiting finish and update status to PR


It looks to me, #37958 #47441 are much more in the above stage 2) and can be extended? Currently I don't have much idea how the 2) be made, coolo has pointed to me 2 links in my description and "the hard part is actually deciding what to schedule". Here need you all's intelligence.

I would say checking everything before even committing is an overkill, so yes, it's more focused on 2), but all CI checks we do with travis can be triggered manually in local setup. So it doesn't matter if we have working solution, it will be possible to trigger locally (will require CI setup similar to travis though). But yeah, it's good to know that we share same concerns, so can address it ;)

#6 Updated by okurz 3 months ago

  • Related to action #58184: [epic][use case] full version control awareness within openQA, e.g. user forks and branches added

Also available in: Atom PDF