action #44360

openQA Tests - action #15132: [epic] Better structure of test plans in main.pm

[epic] Parameterize test suites within job groups

Added by coolo 12 months ago. Updated 37 minutes ago.

Status:BlockedStart date:25/01/2019
Priority:NormalDue date:29/02/2020
Assignee:okurz% Done:

81%

Category:Feature requests
Target version:Current Sprint
Difficulty:hard
Duration: 286

Description

We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters.

This has several implicitions - hence the epic.

  • we need to provide a variable setting in the job group interface
  • we need to summarize the test suite differently in the review interfaces, possibly by requiring the user to give the paramterized test suite another name
  • we need to extend the DB interface of job groups
  • we need to provide a new API for this

Subtasks

action #46667: Define version-able and human readable format for job sch...Resolvedcdywan

action #46670: Create import for format defined in #46667Resolvedcdywan

action #46673: Extend tables for job scheduling by another table for par...Resolvedcdywan

action #50675: Commit changes to scheduling YAML to Git repositoryNew

action #54143: Support parameters in {load,dump}_templatesResolvedcdywan

action #54146: Expose saving YAML and opt-in migration in the editorResolvedcdywan

action #54149: Diffs for better error handling in the YAML editorResolvedcdywan

action #54179: Re-use YAML betweens different groupsNew

action #55454: The job group YAML schedule should support multiple scena...Resolvedcdywan

action #55730: [epic] Move parameters from test suites into job groupsNew

action #56540: convert staging job groups to YAMLResolvedcdywan

action #57845: Switch more job groups to YAML job templatesResolvedokurz

action #58652: Write a training file about how to use YAML in job groupResolvedXiaojing_liu

action #60014: Show YAML editor for unmigrated groupsResolvedokurz

action #60020: Provide a script to migrate job groups to YAML automaticallyFeedbackokurz

action #60017: Document process and potential issues with migration to YAMLResolvedcdywan


Related issues

Blocks openQA Project - action #45413: [tools][functional][y] Support markdown in the test suite... Blocked 19/12/2018

History

#1 Updated by okurz 12 months ago

  • Parent task set to #15132

#2 Updated by mkittler 12 months ago

we need to extend the DB interface of job groups

Or, more specifically, of the job templates table.


In the last meeting we also discussed that having a config file in some Git repo could be preferable compared to saving the config in the database. So I suppose we save this idea for later?

#3 Updated by okurz 12 months ago

mkittler wrote:

So I suppose we save this idea for later?

I think that was the idea, yes.

Btw, I realized that with the "inter-machine dependencies" we reference machines within testsuites even though they are not defined there. Maybe this could be streamlined along with this epic here as I see it related.

#4 Updated by JERiveraMoya 10 months ago

  • Blocked by action #45413: [tools][functional][y] Support markdown in the test suites description added

#5 Updated by JERiveraMoya 10 months ago

  • Blocked by deleted (action #45413: [tools][functional][y] Support markdown in the test suites description)

#6 Updated by JERiveraMoya 10 months ago

  • Blocks action #45413: [tools][functional][y] Support markdown in the test suites description added

#7 Updated by mkittler 10 months ago

In the last meeting the idea to have the data in a version-able and human readable document (e.g. YAML file) was picked up again.

#8 Updated by cdywan 6 months ago

  • Due date set to 18/06/2019

due to changes in a related task

#9 Updated by okurz 5 months ago

  • Category changed from 122 to Feature requests

#10 Updated by sebchlad 3 months ago

"We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters."

The best example to try this out, which I can think of, would be hpc/mpi testing.

For SLE15 we might be supporting 5+ MPI implementations.
Testing of MPIs requires the multimachine set-up; currently I define following test suits for each MPI flavor: master, slaves*2, supportserver.
That means 5 MPIs * 4 test suits = 20 entries using the same test code beneath.
The matter is actually worse as the real bugs were found with different CPU count.
Currently I maintain 2 different CPU counts; QEMUCPUS=1 and QEMUCPUS=2.
So effectively = 20 test suits * 2.

We could consider parameterizing mpi_flavour to begin with.

@Oli: thx for pointing this poo to me.

#11 Updated by okurz 3 months ago

  • Due date changed from 18/06/2019 to 06/09/2019

due to changes in a related task

#12 Updated by okurz about 1 month ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from Ready to Current Sprint

Discussed in QA tools meeting 2019-10-08, waiting for colleagues mainly from "QA SLE Migration" and "QA SLE Virtualization" to be able to follow on with migration, roughly mid of October. Blocked by subtask #55730

#13 Updated by okurz about 1 month ago

  • Due date changed from 18/06/2019 to 22/10/2019

due to changes in a related task

#14 Updated by okurz 2 days ago

#15 Updated by okurz 2 days ago

  • Status changed from Blocked to Feedback

Going further with a deprecation notice: https://github.com/os-autoinst/openQA/pull/2518

#16 Updated by okurz 2 days ago

  • Due date changed from 19/11/2019 to 29/02/2020

due to changes in a related task

#17 Updated by okurz 2 days ago

  • Status changed from Feedback to Blocked

PR merged and also #60014 done. Rest is again in subtickets.

Also available in: Atom PDF