Project

General

Profile

action #15132

[saga][epic] Better structure of test plans in main.pm

Added by okurz over 3 years ago. Updated 26 days ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Target version:
Start date:
2018-11-20
Due date:
2020-11-27
% Done:

97%

Estimated time:
(Total: 13.00 h)
Difficulty:
Duration: 529

Description

motivation

Currently the structure of the test plan in main.pm file is using simple if/else and variables which are mainly boolean to define the test plans for different scenarios with the help of some grouping functions. While there are problems advantages of this approach are that we are using simple tools and there is nearly no duplication of test module calls.

problems

  • P1: The test plan for individual scenarios is hard to follow and it is especially hard for new contributors of individual test modules to find a good place where a test module should be stored
  • P2: It is hard to find out under which scenarios a certain test module is called
  • P3: There is no easy way to add a module based on conditions but only based on existing scenarios which cover these conditions
  • P4: There is no explicit place where to define the variables. The variables are mainly defined by name and use only.
  • P5: Too much business logic is in the database settings which is not versioned in sync with the test code
    • P5.1: The templates files are old and therefore confusing to some. As they are flattened database dumps they are not useful for manual editing

suggestions

  • S1: Better use basetest::is_applicable from os-autoinst in test modules with multiple inheritance and based on mixin patterns for modules
  • S2: Use proper inheritance for variable definitions that influence the test plan, e.g. with perl hashes mapping to JSON objects. ('ENCRYPT_ACTIVATE_EXISTING' is a good example as it should basically be below the top level folder switch 'ENCRYPT' so that we can use it like $vars{ENCRYPT}{ACTIVATE_EXISTING})
  • S3: Define the flow of scenarios in a more concise way top-down with appropriate language
    • S3.1: Use dot language or similar tools to describe the test plan as graphs and then use some automagic to encode that into the more verbose real test plan
    • S3.2: Come up with a language proposal based on YAML

Subtasks

action #44030: [functional][y] Extend the use of scheduling test parametersRejectedriafarov

action #44252: [functional][u] Make more tests booting from HDD image "extratests" following the model of https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6232Resolveddheidler

openQA Project - action #44360: [epic] Parameterize test suites within job groupsBlockedokurz

openQA Project - action #46667: Define version-able and human readable format for job scheduling-related tablesResolvedcdywan

openQA Project - action #46670: Create import for format defined in #46667Resolvedcdywan

openQA Project - action #46673: Extend tables for job scheduling by another table for parametersResolvedcdywan

openQA Project - action #50675: Commit changes to scheduling YAML to Git repositoryNew

openQA Project - action #54143: Support parameters in {load,dump}_templatesResolvedcdywan

openQA Project - action #54146: Expose saving YAML and opt-in migration in the editorResolvedcdywan

openQA Project - action #54149: Diffs for better error handling in the YAML editorResolvedcdywan

openQA Project - action #54179: Re-use YAML betweens different groupsNew

openQA Project - action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingsResolvedcdywan

openQA Project - action #55730: [epic] Move parameters from test suites into job groupsResolvedokurz

openQA Project - action #56540: convert staging job groups to YAMLResolvedcdywan

openQA Project - action #57845: Switch more job groups to YAML job templatesResolvedokurz

openQA Project - action #58652: Write a training file about how to use YAML in job groupResolvedXiaojing_liu

openQA Project - action #60329: Use more parameterized job templates for test suites only used onceResolvedtinita

openQA Project - action #60782: descriptions for parameterized job templates independant of test suite descriptionsResolvedtinita

action #64967: job templates are duplicated as job template in job groups as well as test suitesResolvedtinita

openQA Project - action #60014: Show YAML editor for unmigrated groupsResolvedokurz

openQA Project - action #60020: Provide a script to migrate job groups to YAML automaticallyResolvedokurz

openQA Project - action #60017: Document process and potential issues with migration to YAMLResolvedcdywan

openQA Project - action #66727: [epic] Define structure to define test suites not in openQA databaseBlockedokurz

openQA Project - action #66781: hidden keys in yaml job templatesFeedbackcdywan

action #44420: [functional][y][timeboxed:6h] proof-of-concept of declarative test schedule definition, e.g. in YAML file(s)Resolvedriafarov

action #53159: [functional][y] Implement import of the test_data from another yaml fileResolvedJERiveraMoya

action #54839: [functional][y] Use new yaml structure for test suites to define different schedules for different envResolvedriafarov

action #57536: [functional][y] implement autodeployment of the job group settings to osdResolvedybonatakis


Related issues

Related to openQA Tests - action #13040: [functional][epic][u]more flexible console test selection by addons, testsuite variable, etc. (was: Run zypper lifecycle more selectively)Rejected2016-08-05

Related to openQA Tests - action #31972: [opensuse][functional][epic][userspace][u][medium]Harmonize openSUSE/SLE test module scheduleWorkable2018-02-192018-10-09

Related to openQA Project - action #36994: Dynamic test flow definition + overrideResolved2018-06-08

Related to openQA Project - action #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test scheduleNew2019-02-252020-05-19

Related to openQA Tests - action #18956: [sle][virtualization][x11regressions] unused tests 'tests/virtualization' -> load_virtualization_tests() in products/sle/main.pm is never calledIn Progress2017-05-04

Related to openQA Tests - action #42440: [qam] default "kde" test incompletes sometimes because it takes too longResolved2018-10-13

Related to openQA Project - action #58184: [epic][use case] full version control awareness within openQA, e.g. user forks and branchesBlocked2018-03-23

History

#1 Updated by okurz over 3 years ago

  • Related to action #13040: [functional][epic][u]more flexible console test selection by addons, testsuite variable, etc. (was: Run zypper lifecycle more selectively) added

#2 Updated by okurz over 3 years ago

  • Target version set to Milestone 5

#3 Updated by okurz over 3 years ago

  • Target version changed from Milestone 5 to Milestone 6

Considering amount of tickets in M5 I guess M6 might be a good plan.

#4 Updated by okurz over 3 years ago

  • Target version changed from Milestone 6 to Milestone 7

#5 Updated by okurz over 3 years ago

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2468 is adding the "load_testdir" function which can help a bit.

#6 Updated by okurz about 3 years ago

  • Target version deleted (Milestone 7)

#7 Updated by okurz over 2 years ago

  • Subject changed from Better structure of test plans in main.pm to [functional]Better structure of test plans in main.pm
  • Status changed from New to In Progress
  • Assignee set to okurz
  • Target version set to future

#8 Updated by okurz over 2 years ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

merged, ready to be picked up by others for next steps.

#9 Updated by okurz almost 2 years ago

  • Target version changed from future to future

#10 Updated by okurz almost 2 years ago

  • Related to action #31972: [opensuse][functional][epic][userspace][u][medium]Harmonize openSUSE/SLE test module schedule added

#11 Updated by okurz almost 2 years ago

  • Related to action #36994: Dynamic test flow definition + override added

#12 Updated by okurz almost 2 years ago

  • Related to action #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule added

#13 Updated by okurz over 1 year ago

One outcome of a discussion by coolo and me: Would be good if the existing tables of testsuites and the actual test plan (main.pm) reside together. Currently testsuites on o3/osd grew a lot, e.g. huge bunch of LTP and migration tests

#14 Updated by riafarov over 1 year ago

A bit more details from the discussion. First of all, we all acknowledge the problem with main.pm and we predicted that it will become un-maintainable, which I would say already happened.
Just to write brainstorming outcomes, which was more than just what was mentioned by Oliver.

Possible solutions:

  • Storing plain schedules for each test suite with possibility to inherit them. For migration part, we can evaluate schedule for each existing test suite, for each medium and each machine
    • - redundancy
    • - more efforts for maintenance, however with simple steps
    • + transparency
    • + granularity, individual test suite reflects business logic
  • Storing test suites using serialization language with code injections and dependencies (yml + pre/macro processors)
    • - may get as complex as main.pm
    • - generalization which may lead to executing tests with no business value for particular setups
    • + audit and versions control for no cost using git
    • + no/less redundancy
  • Parameterized test suites:
    • - conditions may get as complex as we currently have affecting transparency
    • - generalization which may lead to executing tests with no business value for particular setups
    • + no/less redundancy
    • + localized complexity for conditions (even with complex conditions, one test suite one affect conditions for another one)
  • Forking main.pm per area/job group
    • + cheapest solution
    • - will only postpone current issue and introduce new challenges

Steps to mitigate consequences of un-maintainable main.pm:

  • split existing test-suites to smaller ones
  • reuse common images, where snapshots are supported
  • reduce test coverage on bare metal and revisit business values of those by evaluating risks of package/component not to work, when tested in the different environment (e.g. firefox on ipmi)

#15 Updated by okurz over 1 year ago

  • Description updated (diff)

#16 Updated by okurz over 1 year ago

  • Related to action #18956: [sle][virtualization][x11regressions] unused tests 'tests/virtualization' -> load_virtualization_tests() in products/sle/main.pm is never called added

#17 Updated by okurz over 1 year ago

  • Subject changed from [functional]Better structure of test plans in main.pm to [functional][u][y] Better structure of test plans in main.pm

#18 Updated by okurz over 1 year ago

  • Related to action #42440: [qam] default "kde" test incompletes sometimes because it takes too long added

#19 Updated by okurz over 1 year ago

Ideas from workshop meeting 2018-11-26:

  • "Copy-pasting" happens either in "main.pm" or openQA test suites to evade the pain, we have to cover the complexity somewhere
  • test suites should prescribe the workflow, main.pm "was meant to" provide a parameterized run
  • test suite inheritance for code reuse?
  • code reuse between work flows can be improved, e.g. have a list of settings in test suites with parameterized test suite names, e.g. switch_keyboard_%DESKTOP% with setting DESKTOP=[textmode,gnome] and START_AFTER_TEST=%DESKTOP% -> how would the specific scenario be mapped in the job groups then? Best opinion so far: Have test suites as well as job groups defined within the test distribution
  • dynamic test suite definition by test code?

#20 Updated by mgriessmeier over 1 year ago

  • Subject changed from [functional][u][y] Better structure of test plans in main.pm to [functional][u][y][epic] Better structure of test plans in main.pm

#21 Updated by okurz over 1 year ago

  • Status changed from Workable to Blocked
  • Assignee set to okurz

we have enough subtasks defined for now, tracking the epic, blocked by subtasks.

#22 Updated by okurz over 1 year ago

  • Due date set to 2019-02-12

due to changes in a related task

#23 Updated by okurz over 1 year ago

  • Due date changed from 2019-02-12 to 2019-06-04

due to changes in a related task

#24 Updated by okurz about 1 year ago

  • Assignee changed from okurz to riafarov

Move to new QSF-y PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-y. Feel free to reassign to me or someone else in this case. Thanks.

#25 Updated by riafarov 8 months ago

  • Due date changed from 2019-11-05 to 2019-10-22

due to changes in a related task

#26 Updated by riafarov 8 months ago

  • Due date changed from 2019-10-22 to 2019-11-05

due to changes in a related task

#27 Updated by okurz 7 months ago

  • Due date changed from 2019-11-05 to 2019-11-19

due to changes in a related task

#28 Updated by riafarov 7 months ago

  • Subject changed from [functional][u][y][epic] Better structure of test plans in main.pm to [epic] Better structure of test plans in main.pm
  • Assignee changed from riafarov to okurz

I believe everything is done for QSF part, so assigning to okurz to track the progress inside of tool team. Thanks for the work!

#29 Updated by okurz 7 months ago

sure, thx. All ticket relations are still current so I can just keep he status as is.

#30 Updated by okurz 7 months ago

  • Due date changed from 2019-11-19 to 2019-12-17

due to changes in a related task

#31 Updated by okurz 7 months ago

  • Due date changed from 2019-12-17 to 2019-11-19

due to changes in a related task

#32 Updated by okurz 3 months ago

  • Subject changed from [epic] Better structure of test plans in main.pm to [epic][saga] Better structure of test plans in main.pm

#33 Updated by okurz 3 months ago

  • Subject changed from [epic][saga] Better structure of test plans in main.pm to [saga][epic] Better structure of test plans in main.pm

#34 Updated by okurz 26 days ago

  • Related to action #58184: [epic][use case] full version control awareness within openQA, e.g. user forks and branches added

Also available in: Atom PDF