Project

General

Profile

action #44360

openQA Tests - action #15132: [saga][epic] Better structure of test plans in main.pm

[epic] Parameterize test suites within job groups

Added by coolo over 1 year ago. Updated 22 days ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2019-01-25
Due date:
2020-11-27
% Done:

86%

Estimated time:
(Total: 0.00 h)
Difficulty:
hard
Duration: 481

Description

We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters.

This has several implicitions - hence the epic.

  • we need to provide a variable setting in the job group interface
  • we need to summarize the test suite differently in the review interfaces, possibly by requiring the user to give the paramterized test suite another name
  • we need to extend the DB interface of job groups
  • we need to provide a new API for this

Subtasks

action #46667: Define version-able and human readable format for job scheduling-related tablesResolvedcdywan

action #46670: Create import for format defined in #46667Resolvedcdywan

action #46673: Extend tables for job scheduling by another table for parametersResolvedcdywan

action #50675: Commit changes to scheduling YAML to Git repositoryNew

action #54143: Support parameters in {load,dump}_templatesResolvedcdywan

action #54146: Expose saving YAML and opt-in migration in the editorResolvedcdywan

action #54149: Diffs for better error handling in the YAML editorResolvedcdywan

action #54179: Re-use YAML betweens different groupsNew

action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingsResolvedcdywan

action #55730: [epic] Move parameters from test suites into job groupsResolvedokurz

action #56540: convert staging job groups to YAMLResolvedcdywan

action #57845: Switch more job groups to YAML job templatesResolvedokurz

action #58652: Write a training file about how to use YAML in job groupResolvedXiaojing_liu

action #60329: Use more parameterized job templates for test suites only used onceResolvedtinita

action #60782: descriptions for parameterized job templates independant of test suite descriptionsResolvedtinita

openQA Tests - action #64967: job templates are duplicated as job template in job groups as well as test suitesResolvedtinita

action #60014: Show YAML editor for unmigrated groupsResolvedokurz

action #60020: Provide a script to migrate job groups to YAML automaticallyResolvedokurz

action #60017: Document process and potential issues with migration to YAMLResolvedcdywan

action #66727: [epic] Define structure to define test suites not in openQA databaseBlockedokurz

action #66781: hidden keys in yaml job templatesResolvedcdywan


Related issues

Blocks openQA Project - action #45413: [tools][functional][y] Support markdown in the test suites descriptionBlocked2018-12-19

History

#1 Updated by okurz over 1 year ago

  • Parent task set to #15132

#2 Updated by mkittler over 1 year ago

we need to extend the DB interface of job groups

Or, more specifically, of the job templates table.


In the last meeting we also discussed that having a config file in some Git repo could be preferable compared to saving the config in the database. So I suppose we save this idea for later?

#3 Updated by okurz over 1 year ago

mkittler wrote:

So I suppose we save this idea for later?

I think that was the idea, yes.

Btw, I realized that with the "inter-machine dependencies" we reference machines within testsuites even though they are not defined there. Maybe this could be streamlined along with this epic here as I see it related.

#4 Updated by JERiveraMoya over 1 year ago

  • Blocked by action #45413: [tools][functional][y] Support markdown in the test suites description added

#5 Updated by JERiveraMoya over 1 year ago

  • Blocked by deleted (action #45413: [tools][functional][y] Support markdown in the test suites description)

#6 Updated by JERiveraMoya over 1 year ago

  • Blocks action #45413: [tools][functional][y] Support markdown in the test suites description added

#7 Updated by mkittler over 1 year ago

In the last meeting the idea to have the data in a version-able and human readable document (e.g. YAML file) was picked up again.

#8 Updated by cdywan about 1 year ago

  • Due date set to 2019-06-18

due to changes in a related task

#9 Updated by okurz about 1 year ago

  • Category changed from 122 to Feature requests

#10 Updated by sebchlad 11 months ago

"We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters."

The best example to try this out, which I can think of, would be hpc/mpi testing.

For SLE15 we might be supporting 5+ MPI implementations.
Testing of MPIs requires the multimachine set-up; currently I define following test suits for each MPI flavor: master, slaves*2, supportserver.
That means 5 MPIs * 4 test suits = 20 entries using the same test code beneath.
The matter is actually worse as the real bugs were found with different CPU count.
Currently I maintain 2 different CPU counts; QEMUCPUS=1 and QEMUCPUS=2.
So effectively = 20 test suits * 2.

We could consider parameterizing mpi_flavour to begin with.

@Oli: thx for pointing this poo to me.

#11 Updated by okurz 10 months ago

  • Due date changed from 2019-06-18 to 2019-09-06

due to changes in a related task

#12 Updated by okurz 9 months ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from Ready to Current Sprint

Discussed in QA tools meeting 2019-10-08, waiting for colleagues mainly from "QA SLE Migration" and "QA SLE Virtualization" to be able to follow on with migration, roughly mid of October. Blocked by subtask #55730

#13 Updated by okurz 9 months ago

  • Due date changed from 2019-06-18 to 2019-10-22

due to changes in a related task

#14 Updated by okurz 8 months ago

#15 Updated by okurz 8 months ago

  • Status changed from Blocked to Feedback

Going further with a deprecation notice: https://github.com/os-autoinst/openQA/pull/2518

#16 Updated by okurz 8 months ago

  • Due date changed from 2019-11-19 to 2020-02-29

due to changes in a related task

#17 Updated by okurz 8 months ago

  • Status changed from Feedback to Blocked

PR merged and also #60014 done. Rest is again in subtickets.

#18 Updated by okurz 7 months ago

  • Due date changed from 2020-02-29 to 2020-11-27

due to changes in a related task

Also available in: Atom PDF