Project

General

Profile

coordination #94105

Updated by okurz over 2 years ago

## User stories 
 * **US1:** As test writer introducing regressions in test code I want to be informed on my pull request in case my pull request is unambiguously identified as culprit for failing openQA tests so that I can quickly provide a regression fix 
 * **US2:** As a test writer I want to be informed on my pull request in case a failing openQA test is identified to be a clear test regression with multiple git commits as candidates that likely introduced a regression so that I can crosscheck my PR if it could have caused the failure 
 * **US3:** As a QE squad PO I would like to receive a ticket in the openQA tests issue tracker in case a failing openQA test is identified to be a clear test regression so that we can plan on fixing that issue 
 * **US4:** As an openQA infrastructure admin I would like to receive a ticket in the openQA infrastructure issue tracker in case a failing openQA test is identified to be a clear infrastructure regression so that we can plan on fixing that issue 
 * **US5:** As a maintenance coordination engineer creating maintenance (release) requests I want to receive a notification in case a failing openQA test is identified to be a clear product regression, i.e. if submitted changes trigger the problem, to be able to fix my submission 
 * **US6:** As an openQA test reviewer I want openqa-investigate to automatically create auto-review tickets that handle restarting tests that fail for the same reason so that I do not need to retrigger manually while test maintainers have time to fix the sporadic issue 

 ## Suggestions 
 * Incorporate https://progress.opensuse.org/projects/openqav3/wiki/#Categorization-scheme into an automatic decision tree with specific actions 
 * Consider the different state results from combination of openqa-investigate jobs ("X" meaning failed, "V" passed, "O" other failure; `2^4=16` possible combination of results): 
  * S0: retry X, last_good_test X, last_good_build X, last_good_test+build X -> infrastructure issue => report ticket in progress.opensuse.org/projects/openqa-infrastructure/issues/ with e.g. "Urgent" priority 
  * S1: retry X, last_good_test X, last_good_build X, last_good_test+build V -> sporadic issue => see S8-15 S9-15 
  * S2: retry X, last_good_test X, last_good_build V, last_good_test+build X -> sporadic issue => see S8-15 S9-15 
  * S3: retry X, last_good_test X, last_good_build V, last_good_test+build V -> reproducible product issue => if QAM test write comment on IBS/OBS or smelt, for non-QAM report product bug 
  * S4: retry X, last_good_test V, last_good_build X, last_good_test+build X -> sporadic issue => see S8-15 S9-15 
  * S5: retry X, last_good_test V, last_good_build X, last_good_test+build V -> reproducible test regression => bisect git log, inform on pull request, report ticket in progress.opensuse.org/projects/openqatests/issues/ 
  * S6: retry X, last_good_test V, last_good_build V, last_good_test+build X -> sporadic issue => see S8-15 S9-15 
  * S7: retry X, last_good_test V, last_good_build V, last_good_test+build V -> sporadic issue => see S8-15 S9-15 
  * S8-15: retry V (all 8 combinations) -> sporadic issue => automatically create auto-review tickets that handle restarting tests and retrigger original -> first step #94105 


 * Intermediate steps from weekly 2021-07-16: 
  * we can always start with just writing yet another comment on openQA jobs 

 * Simplified approach: If last_good+build fails, a product regression is unlikely, state a comment for that 
 * If last_good_test+build fails then likely to be infrastructure issue

Back