Project

General

Profile

Actions
Copy link

coordination #15132

open

[saga][epic] Better structure of test plans in main.pm

Added by okurz about 8 years ago. Updated over 3 years ago.

Status:
Blocked
Priority:
Low
Assignee:
okurz
Category:
Enhancement to existing tests
Target version:
QA (public) - future
Start date:
2018-11-20
Due date:
% Done:

86%

Estimated time:
(Total: 13.00 h)
Difficulty:

Description

motivation

Currently the structure of the test plan in main.pm file is using simple if/else and variables which are mainly boolean to define the test plans for different scenarios with the help of some grouping functions. While there are problems advantages of this approach are that we are using simple tools and there is nearly no duplication of test module calls.

problems

  • P1: The test plan for individual scenarios is hard to follow and it is especially hard for new contributors of individual test modules to find a good place where a test module should be stored
  • P2: It is hard to find out under which scenarios a certain test module is called
  • P3: There is no easy way to add a module based on conditions but only based on existing scenarios which cover these conditions
  • P4: There is no explicit place where to define the variables. The variables are mainly defined by name and use only.
  • P5: Too much business logic is in the database settings which is not versioned in sync with the test code
    • P5.1: The templates files are old and therefore confusing to some. As they are flattened database dumps they are not useful for manual editing

suggestions

  • S1: Better use basetest::is_applicable from os-autoinst in test modules with multiple inheritance and based on mixin patterns for modules
  • S2: Use proper inheritance for variable definitions that influence the test plan, e.g. with perl hashes mapping to JSON objects. ('ENCRYPT_ACTIVATE_EXISTING' is a good example as it should basically be below the top level folder switch 'ENCRYPT' so that we can use it like $vars{ENCRYPT}{ACTIVATE_EXISTING})
  • S3: Define the flow of scenarios in a more concise way top-down with appropriate language
    • S3.1: Use dot language or similar tools to describe the test plan as graphs and then use some automagic to encode that into the more verbose real test plan
    • S3.2: Come up with a language proposal based on YAML

Subtasks 29 (4 open — 25 closed)

action #44030: [functional][y] Extend the use of scheduling test parametersRejectedriafarov2018-11-20

Actions
action #44252: [functional][u] Make more tests booting from HDD image "extratests" following the model of https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6232Resolveddheidler2018-11-22

Actions
openQA Project (public) - coordination #44360: [epic] Parameterize test suites within job groupsBlockedokurz2019-01-25

Actions
openQA Project (public) - action #46667: Define version-able and human readable format for job scheduling-related tablesResolvedlivdywan2019-01-25

Actions
openQA Project (public) - action #46670: Create import for format defined in #46667Resolvedlivdywan2019-01-25

Actions
openQA Project (public) - action #46673: Extend tables for job scheduling by another table for parametersResolvedlivdywan2019-01-25

Actions
openQA Project (public) - action #50675: Commit changes to scheduling YAML to Git repositoryNew2019-04-24

Actions
openQA Project (public) - action #54143: Support parameters in {load,dump}_templatesResolvedlivdywan2019-07-11

Actions
openQA Project (public) - action #54146: Expose saving YAML and opt-in migration in the editorResolvedlivdywan2019-07-11

Actions
openQA Project (public) - action #54149: Diffs for better error handling in the YAML editorResolvedlivdywan2019-07-11

Actions
openQA Project (public) - action #54179: Re-use YAML betweens different groupsNew2019-07-12

Actions
openQA Project (public) - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingsResolvedlivdywan2019-08-13

Actions
openQA Project (public) - coordination #55730: [epic] Move parameters from test suites into job groupsResolvedokurz2019-09-06

Actions
openQA Project (public) - action #56540: convert staging job groups to YAMLResolvedlivdywan2019-09-06

Actions
openQA Project (public) - action #57845: Switch more job groups to YAML job templatesResolvedokurz2019-10-09

Actions
openQA Project (public) - action #58652: Write a training file about how to use YAML in job groupResolvedXiaojing_liu2019-10-24

Actions
openQA Project (public) - action #60329: Use more parameterized job templates for test suites only used onceResolvedtinita2019-11-27

Actions
openQA Project (public) - action #60782: descriptions for parameterized job templates independant of test suite descriptionsResolvedtinita2019-12-06

Actions
action #64967: job templates are duplicated as job template in job groups as well as test suitesResolvedtinita2020-03-29

Actions
openQA Project (public) - action #60014: Show YAML editor for unmigrated groupsResolvedokurz2019-11-19

Actions
openQA Project (public) - action #60020: Provide a script to migrate job groups to YAML automaticallyResolvedokurz2019-11-19

Actions
openQA Project (public) - action #60017: Document process and potential issues with migration to YAMLResolvedlivdywan2019-11-19

Actions
openQA Project (public) - coordination #66727: [epic] Define structure to define test suites not in openQA databaseResolvedokurz2020-05-13

Actions
openQA Project (public) - action #66781: hidden keys in yaml job templatesResolvedlivdywan2020-05-13

Actions
action #44420: [functional][y][timeboxed:6h] proof-of-concept of declarative test schedule definition, e.g. in YAML file(s)Resolvedriafarov2018-11-28

Actions
action #53159: [functional][y] Implement import of the test_data from another yaml fileResolvedJERiveraMoya2019-06-17

Actions
action #54839: [functional][y] Use new yaml structure for test suites to define different schedules for different envResolvedriafarov2019-07-30

Actions
action #57536: [functional][y] implement autodeployment of the job group settings to osdResolvedybonatakis2019-09-30

Actions
openQA Project (public) - action #66865: Support !inherit tag in YAML job templatesWorkable2020-05-14

Actions

Related issues 6 (2 open — 4 closed)

Related to openQA Tests (public) - coordination #13040: [functional][epic][u]more flexible console test selection by addons, testsuite variable, etc. (was: Run zypper lifecycle more selectively)Rejected2016-08-05

Actions
Related to openQA Tests (public) - coordination #31972: [qe-core][opensuse][functional][epic][userspace][medium]Harmonize openSUSE/SLE test module scheduleWorkableszarate2018-02-19

Actions
Related to openQA Project (public) - action #36994: Dynamic test flow definition + overrideResolvedokurz2018-06-08

Actions
Related to openQA Project (public) - coordination #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test scheduleResolvedokurz2019-02-252020-05-19

Actions
Related to openQA Tests (public) - action #42440: [qam] default "kde" test incompletes sometimes because it takes too longResolvedokurz2018-10-13

Actions
Related to openQA Project (public) - coordination #58184: [saga][epic][use case] full version control awareness within openQABlockedokurz2019-08-01

Actions
Actions
Copy link
 #1

Updated by okurz about 8 years ago

  • Related to coordination #13040: [functional][epic][u]more flexible console test selection by addons, testsuite variable, etc. (was: Run zypper lifecycle more selectively) added
Actions
Copy link
 #2

Updated by okurz about 8 years ago

  • Target version set to Milestone 5
Actions
Copy link
 #3

Updated by okurz almost 8 years ago

  • Target version changed from Milestone 5 to Milestone 6

Considering amount of tickets in M5 I guess M6 might be a good plan.

Actions
Copy link
 #4

Updated by okurz almost 8 years ago

  • Target version changed from Milestone 6 to Milestone 7
Actions
Copy link
 #5

Updated by okurz almost 8 years ago

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2468 is adding the "load_testdir" function which can help a bit.

Actions
Copy link
 #6

Updated by okurz over 7 years ago

  • Target version deleted (Milestone 7)
Actions
Copy link
 #7

Updated by okurz almost 7 years ago

  • Subject changed from Better structure of test plans in main.pm to [functional]Better structure of test plans in main.pm
  • Status changed from New to In Progress
  • Assignee set to okurz
  • Target version set to future
Actions
Copy link
 #8

Updated by okurz almost 7 years ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

merged, ready to be picked up by others for next steps.

Actions
Copy link
 #9

Updated by okurz over 6 years ago

  • Target version changed from future to future
Actions
Copy link
 #10

Updated by okurz over 6 years ago

  • Related to coordination #31972: [qe-core][opensuse][functional][epic][userspace][medium]Harmonize openSUSE/SLE test module schedule added
Actions
Copy link
 #11

Updated by okurz over 6 years ago

  • Related to action #36994: Dynamic test flow definition + override added
Actions
Copy link
 #12

Updated by okurz over 6 years ago

  • Related to coordination #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule added
Actions
Copy link
 #13

Updated by okurz over 6 years ago

One outcome of a discussion by coolo and me: Would be good if the existing tables of testsuites and the actual test plan (main.pm) reside together. Currently testsuites on o3/osd grew a lot, e.g. huge bunch of LTP and migration tests

Actions
Copy link
 #14

Updated by riafarov over 6 years ago

A bit more details from the discussion. First of all, we all acknowledge the problem with main.pm and we predicted that it will become un-maintainable, which I would say already happened.
Just to write brainstorming outcomes, which was more than just what was mentioned by Oliver.

Possible solutions:

  • Storing plain schedules for each test suite with possibility to inherit them. For migration part, we can evaluate schedule for each existing test suite, for each medium and each machine
    • - redundancy
    • - more efforts for maintenance, however with simple steps
    • + transparency
    • + granularity, individual test suite reflects business logic
  • Storing test suites using serialization language with code injections and dependencies (yml + pre/macro processors)
    • - may get as complex as main.pm
    • - generalization which may lead to executing tests with no business value for particular setups
    • + audit and versions control for no cost using git
    • + no/less redundancy
  • Parameterized test suites:
    • - conditions may get as complex as we currently have affecting transparency
    • - generalization which may lead to executing tests with no business value for particular setups
    • + no/less redundancy
    • + localized complexity for conditions (even with complex conditions, one test suite one affect conditions for another one)
  • Forking main.pm per area/job group
    • + cheapest solution
    • - will only postpone current issue and introduce new challenges

Steps to mitigate consequences of un-maintainable main.pm:

  • split existing test-suites to smaller ones
  • reuse common images, where snapshots are supported
  • reduce test coverage on bare metal and revisit business values of those by evaluating risks of package/component not to work, when tested in the different environment (e.g. firefox on ipmi)
Actions
Copy link
 #15

Updated by okurz over 6 years ago

  • Description updated (diff)
Actions
Copy link
 #17

Updated by okurz about 6 years ago

  • Subject changed from [functional]Better structure of test plans in main.pm to [functional][u][y] Better structure of test plans in main.pm
Actions
Copy link
 #18

Updated by okurz about 6 years ago

  • Related to action #42440: [qam] default "kde" test incompletes sometimes because it takes too long added
Actions
Copy link
 #19

Updated by okurz about 6 years ago

Ideas from workshop meeting 2018-11-26:

  • "Copy-pasting" happens either in "main.pm" or openQA test suites to evade the pain, we have to cover the complexity somewhere
  • test suites should prescribe the workflow, main.pm "was meant to" provide a parameterized run
  • test suite inheritance for code reuse?
  • code reuse between work flows can be improved, e.g. have a list of settings in test suites with parameterized test suite names, e.g. switch_keyboard_%DESKTOP% with setting DESKTOP=[textmode,gnome] and START_AFTER_TEST=%DESKTOP% -> how would the specific scenario be mapped in the job groups then? Best opinion so far: Have test suites as well as job groups defined within the test distribution
  • dynamic test suite definition by test code?
Actions
Copy link
 #20

Updated by mgriessmeier about 6 years ago

  • Subject changed from [functional][u][y] Better structure of test plans in main.pm to [functional][u][y][epic] Better structure of test plans in main.pm
Actions
Copy link
 #21

Updated by okurz about 6 years ago

  • Status changed from Workable to Blocked
  • Assignee set to okurz

we have enough subtasks defined for now, tracking the epic, blocked by subtasks.

Actions
Copy link
 #22

Updated by okurz almost 6 years ago

  • Due date set to 2019-02-12

due to changes in a related task

Actions
Copy link
 #23

Updated by okurz almost 6 years ago

  • Due date changed from 2019-02-12 to 2019-06-04

due to changes in a related task

Actions
Copy link
 #24

Updated by okurz over 5 years ago

  • Assignee changed from okurz to riafarov

Move to new QSF-y PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-y. Feel free to reassign to me or someone else in this case. Thanks.

Actions
Copy link
 #25

Updated by riafarov about 5 years ago

  • Due date changed from 2019-11-05 to 2019-10-22

due to changes in a related task

Actions
Copy link
 #26

Updated by riafarov about 5 years ago

  • Due date changed from 2019-10-22 to 2019-11-05

due to changes in a related task

Actions
Copy link
 #27

Updated by okurz about 5 years ago

  • Due date changed from 2019-11-05 to 2019-11-19

due to changes in a related task

Actions
Copy link
 #28

Updated by riafarov about 5 years ago

  • Subject changed from [functional][u][y][epic] Better structure of test plans in main.pm to [epic] Better structure of test plans in main.pm
  • Assignee changed from riafarov to okurz

I believe everything is done for QSF part, so assigning to okurz to track the progress inside of tool team. Thanks for the work!

Actions
Copy link
 #29

Updated by okurz about 5 years ago

sure, thx. All ticket relations are still current so I can just keep he status as is.

Actions
Copy link
 #30

Updated by okurz about 5 years ago

  • Due date changed from 2019-11-19 to 2019-12-17

due to changes in a related task

Actions
Copy link
 #31

Updated by okurz about 5 years ago

  • Due date changed from 2019-12-17 to 2019-11-19

due to changes in a related task

Actions
Copy link
 #32

Updated by okurz almost 5 years ago

  • Subject changed from [epic] Better structure of test plans in main.pm to [epic][saga] Better structure of test plans in main.pm
Actions
Copy link
 #33

Updated by okurz almost 5 years ago

  • Subject changed from [epic][saga] Better structure of test plans in main.pm to [saga][epic] Better structure of test plans in main.pm
Actions
Copy link
 #34

Updated by okurz over 4 years ago

  • Related to coordination #58184: [saga][epic][use case] full version control awareness within openQA added
Actions
Copy link
 #35

Updated by szarate about 4 years ago

  • Tracker changed from action to coordination
  • Status changed from Blocked to New
Actions
Copy link
 #37

Updated by okurz about 4 years ago

  • Status changed from New to Blocked
Actions
Copy link

Also available in: Atom PDF