coordination #15132
open
[saga][epic] Better structure of test plans in main.pm
Added by okurz about 8 years ago.
Updated over 3 years ago.
Category:
Enhancement to existing tests
Estimated time:
(Total: 13.00 h)
Description
motivation¶
Currently the structure of the test plan in main.pm file is using simple if/else and variables which are mainly boolean to define the test plans for different scenarios with the help of some grouping functions. While there are problems advantages of this approach are that we are using simple tools and there is nearly no duplication of test module calls.
problems¶
- P1: The test plan for individual scenarios is hard to follow and it is especially hard for new contributors of individual test modules to find a good place where a test module should be stored
- P2: It is hard to find out under which scenarios a certain test module is called
- P3: There is no easy way to add a module based on conditions but only based on existing scenarios which cover these conditions
- P4: There is no explicit place where to define the variables. The variables are mainly defined by name and use only.
- P5: Too much business logic is in the database settings which is not versioned in sync with the test code
- P5.1: The
templates files are old and therefore confusing to some. As they are flattened database dumps they are not useful for manual editing
suggestions¶
- S1: Better use
basetest::is_applicable from os-autoinst in test modules with multiple inheritance and based on mixin patterns for modules
- S2: Use proper inheritance for variable definitions that influence the test plan, e.g. with perl hashes mapping to JSON objects. ('ENCRYPT_ACTIVATE_EXISTING' is a good example as it should basically be below the top level folder switch 'ENCRYPT' so that we can use it like
$vars{ENCRYPT}{ACTIVATE_EXISTING})
- S3: Define the flow of scenarios in a more concise way top-down with appropriate language
- S3.1: Use dot language or similar tools to describe the test plan as graphs and then use some automagic to encode that into the more verbose real test plan
- S3.2: Come up with a language proposal based on YAML
- Related to coordination #13040: [functional][epic][u]more flexible console test selection by addons, testsuite variable, etc. (was: Run zypper lifecycle more selectively) added
- Target version set to Milestone 5
- Target version changed from Milestone 5 to Milestone 6
Considering amount of tickets in M5 I guess M6 might be a good plan.
- Target version changed from Milestone 6 to Milestone 7
- Target version deleted (
Milestone 7)
- Subject changed from Better structure of test plans in main.pm to [functional]Better structure of test plans in main.pm
- Status changed from New to In Progress
- Assignee set to okurz
- Target version set to future
- Status changed from In Progress to Workable
- Assignee deleted (
okurz)
merged, ready to be picked up by others for next steps.
- Target version changed from future to future
- Related to coordination #31972: [qe-core][opensuse][functional][epic][userspace][medium]Harmonize openSUSE/SLE test module schedule added
- Related to action #36994: Dynamic test flow definition + override added
- Related to coordination #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule added
One outcome of a discussion by coolo and me: Would be good if the existing tables of testsuites and the actual test plan (main.pm) reside together. Currently testsuites on o3/osd grew a lot, e.g. huge bunch of LTP and migration tests
A bit more details from the discussion. First of all, we all acknowledge the problem with main.pm and we predicted that it will become un-maintainable, which I would say already happened.
Just to write brainstorming outcomes, which was more than just what was mentioned by Oliver.
Possible solutions:
- Storing plain schedules for each test suite with possibility to inherit them. For migration part, we can evaluate schedule for each existing test suite, for each medium and each machine
- - redundancy
- - more efforts for maintenance, however with simple steps
- + transparency
- + granularity, individual test suite reflects business logic
- Storing test suites using serialization language with code injections and dependencies (yml + pre/macro processors)
- - may get as complex as main.pm
- - generalization which may lead to executing tests with no business value for particular setups
- + audit and versions control for no cost using git
- + no/less redundancy
- Parameterized test suites:
- - conditions may get as complex as we currently have affecting transparency
- - generalization which may lead to executing tests with no business value for particular setups
- + no/less redundancy
- + localized complexity for conditions (even with complex conditions, one test suite one affect conditions for another one)
- Forking main.pm per area/job group
- + cheapest solution
- - will only postpone current issue and introduce new challenges
Steps to mitigate consequences of un-maintainable main.pm:
- split existing test-suites to smaller ones
- reuse common images, where snapshots are supported
- reduce test coverage on bare metal and revisit business values of those by evaluating risks of package/component not to work, when tested in the different environment (e.g. firefox on ipmi)
- Description updated (diff)
- Subject changed from [functional]Better structure of test plans in main.pm to [functional][u][y] Better structure of test plans in main.pm
- Related to action #42440: [qam] default "kde" test incompletes sometimes because it takes too long added
Ideas from workshop meeting 2018-11-26:
- "Copy-pasting" happens either in "main.pm" or openQA test suites to evade the pain, we have to cover the complexity somewhere
- test suites should prescribe the workflow, main.pm "was meant to" provide a parameterized run
- test suite inheritance for code reuse?
- code reuse between work flows can be improved, e.g. have a list of settings in test suites with parameterized test suite names, e.g.
switch_keyboard_%DESKTOP% with setting DESKTOP=[textmode,gnome] and START_AFTER_TEST=%DESKTOP% -> how would the specific scenario be mapped in the job groups then? Best opinion so far: Have test suites as well as job groups defined within the test distribution
- dynamic test suite definition by test code?
- Subject changed from [functional][u][y] Better structure of test plans in main.pm to [functional][u][y][epic] Better structure of test plans in main.pm
- Status changed from Workable to Blocked
- Assignee set to okurz
we have enough subtasks defined for now, tracking the epic, blocked by subtasks.
- Due date set to 2019-02-12
due to changes in a related task
- Due date changed from 2019-02-12 to 2019-06-04
due to changes in a related task
- Assignee changed from okurz to riafarov
Move to new QSF-y PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-y. Feel free to reassign to me or someone else in this case. Thanks.
- Due date changed from 2019-11-05 to 2019-10-22
due to changes in a related task
- Due date changed from 2019-10-22 to 2019-11-05
due to changes in a related task
- Due date changed from 2019-11-05 to 2019-11-19
due to changes in a related task
- Subject changed from [functional][u][y][epic] Better structure of test plans in main.pm to [epic] Better structure of test plans in main.pm
- Assignee changed from riafarov to okurz
I believe everything is done for QSF part, so assigning to okurz to track the progress inside of tool team. Thanks for the work!
sure, thx. All ticket relations are still current so I can just keep he status as is.
- Due date changed from 2019-11-19 to 2019-12-17
due to changes in a related task
- Due date changed from 2019-12-17 to 2019-11-19
due to changes in a related task
- Subject changed from [epic] Better structure of test plans in main.pm to [epic][saga] Better structure of test plans in main.pm
- Subject changed from [epic][saga] Better structure of test plans in main.pm to [saga][epic] Better structure of test plans in main.pm
- Related to coordination #58184: [saga][epic][use case] full version control awareness within openQA added
- Tracker changed from action to coordination
- Status changed from Blocked to New
- Status changed from New to Blocked
Also available in: Atom
PDF