action #44360

openQA Tests - action #15132: [saga][epic] Better structure of test plans in main.pm

[epic] Parameterize test suites within job groups

Added by coolo over 1 year ago. Updated 6 days ago.

Status:BlockedStart date:25/01/2019
Priority:NormalDue date:27/11/2020
Assignee:okurz% Done:

84%

Category:Feature requests
Target version:Current Sprint
Difficulty:hard
Duration: 481

Description

We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters.

This has several implicitions - hence the epic.

  • we need to provide a variable setting in the job group interface
  • we need to summarize the test suite differently in the review interfaces, possibly by requiring the user to give the paramterized test suite another name
  • we need to extend the DB interface of job groups
  • we need to provide a new API for this

Subtasks

action #46667: Define version-able and human readable format for job sch...Resolvedcdywan

action #46670: Create import for format defined in #46667Resolvedcdywan

action #46673: Extend tables for job scheduling by another table for par...Resolvedcdywan

action #50675: Commit changes to scheduling YAML to Git repositoryNew

action #54143: Support parameters in {load,dump}_templatesResolvedcdywan

action #54146: Expose saving YAML and opt-in migration in the editorResolvedcdywan

action #54149: Diffs for better error handling in the YAML editorResolvedcdywan

action #54179: Re-use YAML betweens different groupsNew

action #55454: The job group YAML schedule should support multiple scena...Resolvedcdywan

action #55730: [epic] Move parameters from test suites into job groupsBlockedokurz

action #56540: convert staging job groups to YAMLResolvedcdywan

action #57845: Switch more job groups to YAML job templatesResolvedokurz

action #58652: Write a training file about how to use YAML in job groupResolvedXiaojing_liu

action #60329: Use more parameterized job templates for test suites only...Resolvedtinita

action #60782: descriptions for parameterized job templates independant ...Resolvedtinita

openQA Tests - action #64967: job templates are duplicated as job template in job group...Newtinita

action #60014: Show YAML editor for unmigrated groupsResolvedokurz

action #60020: Provide a script to migrate job groups to YAML automaticallyResolvedokurz

action #60017: Document process and potential issues with migration to YAMLResolvedcdywan


Related issues

Blocks openQA Project - action #45413: [tools][functional][y] Support markdown in the test suite... Blocked 19/12/2018

History

#1 Updated by okurz over 1 year ago

  • Parent task set to #15132

#2 Updated by mkittler over 1 year ago

we need to extend the DB interface of job groups

Or, more specifically, of the job templates table.


In the last meeting we also discussed that having a config file in some Git repo could be preferable compared to saving the config in the database. So I suppose we save this idea for later?

#3 Updated by okurz over 1 year ago

mkittler wrote:

So I suppose we save this idea for later?

I think that was the idea, yes.

Btw, I realized that with the "inter-machine dependencies" we reference machines within testsuites even though they are not defined there. Maybe this could be streamlined along with this epic here as I see it related.

#4 Updated by JERiveraMoya about 1 year ago

  • Blocked by action #45413: [tools][functional][y] Support markdown in the test suites description added

#5 Updated by JERiveraMoya about 1 year ago

  • Blocked by deleted (action #45413: [tools][functional][y] Support markdown in the test suites description)

#6 Updated by JERiveraMoya about 1 year ago

  • Blocks action #45413: [tools][functional][y] Support markdown in the test suites description added

#7 Updated by mkittler about 1 year ago

In the last meeting the idea to have the data in a version-able and human readable document (e.g. YAML file) was picked up again.

#8 Updated by cdywan 10 months ago

  • Due date set to 18/06/2019

due to changes in a related task

#9 Updated by okurz 10 months ago

  • Category changed from 122 to Feature requests

#10 Updated by sebchlad 8 months ago

"We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters."

The best example to try this out, which I can think of, would be hpc/mpi testing.

For SLE15 we might be supporting 5+ MPI implementations.
Testing of MPIs requires the multimachine set-up; currently I define following test suits for each MPI flavor: master, slaves*2, supportserver.
That means 5 MPIs * 4 test suits = 20 entries using the same test code beneath.
The matter is actually worse as the real bugs were found with different CPU count.
Currently I maintain 2 different CPU counts; QEMUCPUS=1 and QEMUCPUS=2.
So effectively = 20 test suits * 2.

We could consider parameterizing mpi_flavour to begin with.

@Oli: thx for pointing this poo to me.

#11 Updated by okurz 7 months ago

  • Due date changed from 18/06/2019 to 06/09/2019

due to changes in a related task

#12 Updated by okurz 6 months ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from Ready to Current Sprint

Discussed in QA tools meeting 2019-10-08, waiting for colleagues mainly from "QA SLE Migration" and "QA SLE Virtualization" to be able to follow on with migration, roughly mid of October. Blocked by subtask #55730

#13 Updated by okurz 6 months ago

  • Due date changed from 18/06/2019 to 22/10/2019

due to changes in a related task

#14 Updated by okurz 5 months ago

#15 Updated by okurz 5 months ago

  • Status changed from Blocked to Feedback

Going further with a deprecation notice: https://github.com/os-autoinst/openQA/pull/2518

#16 Updated by okurz 5 months ago

  • Due date changed from 19/11/2019 to 29/02/2020

due to changes in a related task

#17 Updated by okurz 5 months ago

  • Status changed from Feedback to Blocked

PR merged and also #60014 done. Rest is again in subtickets.

#18 Updated by okurz 4 months ago

  • Due date changed from 29/02/2020 to 27/11/2020

due to changes in a related task

Also available in: Atom PDF