action #15132

[epic] Better structure of test plans in main.pm

Added by okurz about 3 years ago. Updated about 1 month ago.

Status:BlockedStart date:20/11/2018
Priority:NormalDue date:27/11/2020
Assignee:okurz% Done:

88%

Category:Enhancement to existing testsEstimated time:13.00 hours
Target version:QA - future
Difficulty:
Duration: 529

Description

motivation

Currently the structure of the test plan in main.pm file is using simple if/else and variables which are mainly boolean to define the test plans for different scenarios with the help of some grouping functions. While there are problems advantages of this approach are that we are using simple tools and there is nearly no duplication of test module calls.

problems

  • P1: The test plan for individual scenarios is hard to follow and it is especially hard for new contributors of individual test modules to find a good place where a test module should be stored
  • P2: It is hard to find out under which scenarios a certain test module is called
  • P3: There is no easy way to add a module based on conditions but only based on existing scenarios which cover these conditions
  • P4: There is no explicit place where to define the variables. The variables are mainly defined by name and use only.
  • P5: Too much business logic is in the database settings which is not versioned in sync with the test code
    • P5.1: The templates files are old and therefore confusing to some. As they are flattened database dumps they are not useful for manual editing

suggestions

  • S1: Better use basetest::is_applicable from os-autoinst in test modules with multiple inheritance and based on mixin patterns for modules
  • S2: Use proper inheritance for variable definitions that influence the test plan, e.g. with perl hashes mapping to JSON objects. ('ENCRYPT_ACTIVATE_EXISTING' is a good example as it should basically be below the top level folder switch 'ENCRYPT' so that we can use it like $vars{ENCRYPT}{ACTIVATE_EXISTING})
  • S3: Define the flow of scenarios in a more concise way top-down with appropriate language
    • S3.1: Use dot language or similar tools to describe the test plan as graphs and then use some automagic to encode that into the more verbose real test plan
    • S3.2: Come up with a language proposal based on YAML

Subtasks

action #44030: [functional][y] Extend the use of scheduling test parametersRejectedriafarov

action #44252: [functional][u] Make more tests booting from HDD image "e...Resolveddheidler

openQA Project - action #44360: [epic] Parameterize test suites within job groupsBlockedokurz

openQA Project - action #46667: Define version-able and human readable format for job sch...Resolvedcdywan

openQA Project - action #46670: Create import for format defined in #46667Resolvedcdywan

openQA Project - action #46673: Extend tables for job scheduling by another table for par...Resolvedcdywan

openQA Project - action #50675: Commit changes to scheduling YAML to Git repositoryNew

openQA Project - action #54143: Support parameters in {load,dump}_templatesResolvedcdywan

openQA Project - action #54146: Expose saving YAML and opt-in migration in the editorResolvedcdywan

openQA Project - action #54149: Diffs for better error handling in the YAML editorResolvedcdywan

openQA Project - action #54179: Re-use YAML betweens different groupsNew

openQA Project - action #55454: The job group YAML schedule should support multiple scena...Resolvedcdywan

openQA Project - action #55730: [epic] Move parameters from test suites into job groupsBlockedokurz

openQA Project - action #56540: convert staging job groups to YAMLResolvedcdywan

openQA Project - action #57845: Switch more job groups to YAML job templatesResolvedokurz

openQA Project - action #58652: Write a training file about how to use YAML in job groupResolvedXiaojing_liu

openQA Project - action #60329: Use more parameterized job templates for test suites only...In Progresstinita

openQA Project - action #60782: descriptions for parameterized job templates independant ...Resolvedtinita

openQA Project - action #60014: Show YAML editor for unmigrated groupsResolvedokurz

openQA Project - action #60020: Provide a script to migrate job groups to YAML automaticallyResolvedokurz

openQA Project - action #60017: Document process and potential issues with migration to YAMLResolvedcdywan

action #44420: [functional][y][timeboxed:6h] proof-of-concept of declara...Resolvedriafarov

action #53159: [functional][y] Implement import of the test_data from an...ResolvedJERiveraMoya

action #54839: [functional][y] Use new yaml structure for test suites to...Resolvedriafarov

action #57536: [functional][y] implement autodeployment of the job group...Resolvedybonatakis


Related issues

Related to openQA Tests - action #13040: [functional][epic][u]more flexible console test selection... Rejected 05/08/2016
Related to openQA Tests - action #31972: [opensuse][functional][epic][userspace][u][medium]Harmoni... Workable 19/02/2018 09/10/2018
Related to openQA Project - action #36994: Dynamic test flow definition + override Resolved 08/06/2018
Related to openQA Tests - action #37958: [functional][y][epic] self-tests in os-autoinst-distri-op... Blocked 25/02/2019 24/03/2020
Related to openQA Tests - action #18956: [sle][virtualization][x11regressions] unused tests 'tests... In Progress 04/05/2017
Related to openQA Tests - action #42440: [qam] default "kde" test incompletes sometimes because it... Resolved 13/10/2018

History

#1 Updated by okurz about 3 years ago

  • Related to action #13040: [functional][epic][u]more flexible console test selection by addons, testsuite variable, etc. (was: Run zypper lifecycle more selectively) added

#2 Updated by okurz about 3 years ago

  • Target version set to Milestone 5

#3 Updated by okurz about 3 years ago

  • Target version changed from Milestone 5 to Milestone 6

Considering amount of tickets in M5 I guess M6 might be a good plan.

#4 Updated by okurz about 3 years ago

  • Target version changed from Milestone 6 to Milestone 7

#5 Updated by okurz about 3 years ago

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2468 is adding the "load_testdir" function which can help a bit.

#6 Updated by okurz almost 3 years ago

  • Target version deleted (Milestone 7)

#7 Updated by okurz about 2 years ago

  • Subject changed from Better structure of test plans in main.pm to [functional]Better structure of test plans in main.pm
  • Status changed from New to In Progress
  • Assignee set to okurz
  • Target version set to future

#8 Updated by okurz about 2 years ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

merged, ready to be picked up by others for next steps.

#9 Updated by okurz over 1 year ago

  • Target version changed from future to future

#10 Updated by okurz over 1 year ago

  • Related to action #31972: [opensuse][functional][epic][userspace][u][medium]Harmonize openSUSE/SLE test module schedule added

#11 Updated by okurz over 1 year ago

  • Related to action #36994: Dynamic test flow definition + override added

#12 Updated by okurz over 1 year ago

  • Related to action #37958: [functional][y][epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule added

#13 Updated by okurz over 1 year ago

One outcome of a discussion by coolo and me: Would be good if the existing tables of testsuites and the actual test plan (main.pm) reside together. Currently testsuites on o3/osd grew a lot, e.g. huge bunch of LTP and migration tests

#14 Updated by riafarov over 1 year ago

A bit more details from the discussion. First of all, we all acknowledge the problem with main.pm and we predicted that it will become un-maintainable, which I would say already happened.
Just to write brainstorming outcomes, which was more than just what was mentioned by Oliver.

Possible solutions:
- Storing plain schedules for each test suite with possibility to inherit them. For migration part, we can evaluate schedule for each existing test suite, for each medium and each machine
* - redundancy
* - more efforts for maintenance, however with simple steps
* + transparency
* + granularity, individual test suite reflects business logic
- Storing test suites using serialization language with code injections and dependencies (yml + pre/macro processors)
* - may get as complex as main.pm
* - generalization which may lead to executing tests with no business value for particular setups
* + audit and versions control for no cost using git
* + no/less redundancy
- Parameterized test suites:
* - conditions may get as complex as we currently have affecting transparency
* - generalization which may lead to executing tests with no business value for particular setups
* + no/less redundancy
* + localized complexity for conditions (even with complex conditions, one test suite one affect conditions for another one)
- Forking main.pm per area/job group
* + cheapest solution
* - will only postpone current issue and introduce new challenges

Steps to mitigate consequences of un-maintainable main.pm:
* split existing test-suites to smaller ones
* reuse common images, where snapshots are supported
* reduce test coverage on bare metal and revisit business values of those by evaluating risks of package/component not to work, when tested in the different environment (e.g. firefox on ipmi)

#15 Updated by okurz over 1 year ago

  • Description updated (diff)

#16 Updated by okurz over 1 year ago

  • Related to action #18956: [sle][virtualization][x11regressions] unused tests 'tests/virtualization' -> load_virtualization_tests() in products/sle/main.pm is never called added

#17 Updated by okurz over 1 year ago

  • Subject changed from [functional]Better structure of test plans in main.pm to [functional][u][y] Better structure of test plans in main.pm

#18 Updated by okurz over 1 year ago

  • Related to action #42440: [qam] default "kde" test incompletes sometimes because it takes too long added

#19 Updated by okurz over 1 year ago

Ideas from workshop meeting 2018-11-26:

  • "Copy-pasting" happens either in "main.pm" or openQA test suites to evade the pain, we have to cover the complexity somewhere
  • test suites should prescribe the workflow, main.pm "was meant to" provide a parameterized run
  • test suite inheritance for code reuse?
  • code reuse between work flows can be improved, e.g. have a list of settings in test suites with parameterized test suite names, e.g. switch_keyboard_%DESKTOP% with setting DESKTOP=[textmode,gnome] and START_AFTER_TEST=%DESKTOP% -> how would the specific scenario be mapped in the job groups then? Best opinion so far: Have test suites as well as job groups defined within the test distribution
  • dynamic test suite definition by test code?

#20 Updated by mgriessmeier about 1 year ago

  • Subject changed from [functional][u][y] Better structure of test plans in main.pm to [functional][u][y][epic] Better structure of test plans in main.pm

#21 Updated by okurz about 1 year ago

  • Status changed from Workable to Blocked
  • Assignee set to okurz

we have enough subtasks defined for now, tracking the epic, blocked by subtasks.

#22 Updated by okurz about 1 year ago

  • Due date set to 12/02/2019

due to changes in a related task

#23 Updated by okurz about 1 year ago

  • Due date changed from 12/02/2019 to 04/06/2019

due to changes in a related task

#24 Updated by okurz 9 months ago

  • Assignee changed from okurz to riafarov

Move to new QSF-y PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-y. Feel free to reassign to me or someone else in this case. Thanks.

#25 Updated by riafarov 5 months ago

  • Due date changed from 05/11/2019 to 22/10/2019

due to changes in a related task

#26 Updated by riafarov 5 months ago

  • Due date changed from 22/10/2019 to 05/11/2019

due to changes in a related task

#27 Updated by okurz 4 months ago

  • Due date changed from 05/11/2019 to 19/11/2019

due to changes in a related task

#28 Updated by riafarov 3 months ago

  • Subject changed from [functional][u][y][epic] Better structure of test plans in main.pm to [epic] Better structure of test plans in main.pm
  • Assignee changed from riafarov to okurz

I believe everything is done for QSF part, so assigning to okurz to track the progress inside of tool team. Thanks for the work!

#29 Updated by okurz 3 months ago

sure, thx. All ticket relations are still current so I can just keep he status as is.

#30 Updated by okurz 3 months ago

  • Due date changed from 19/11/2019 to 17/12/2019

due to changes in a related task

#31 Updated by okurz 3 months ago

  • Due date changed from 17/12/2019 to 19/11/2019

due to changes in a related task

Also available in: Atom PDF