Project

General

Profile

Actions

coordination #44360

open

openQA Tests - coordination #15132: [saga][epic] Better structure of test plans in main.pm

[epic] Parameterize test suites within job groups

Added by coolo over 5 years ago. Updated over 2 years ago.

Status:
Blocked
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2019-01-25
Due date:
% Done:

86%

Estimated time:
(Total: 0.00 h)

Description

We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters.

This has several implicitions - hence the epic.

  • we need to provide a variable setting in the job group interface
  • we need to summarize the test suite differently in the review interfaces, possibly by requiring the user to give the paramterized test suite another name
  • we need to extend the DB interface of job groups
  • we need to provide a new API for this

Subtasks 21 (2 open19 closed)

action #46667: Define version-able and human readable format for job scheduling-related tablesResolvedlivdywan2019-01-25

Actions
action #46670: Create import for format defined in #46667Resolvedlivdywan2019-01-25

Actions
action #46673: Extend tables for job scheduling by another table for parametersResolvedlivdywan2019-01-25

Actions
action #50675: Commit changes to scheduling YAML to Git repositoryNew2019-04-24

Actions
action #54143: Support parameters in {load,dump}_templatesResolvedlivdywan2019-07-11

Actions
action #54146: Expose saving YAML and opt-in migration in the editorResolvedlivdywan2019-07-11

Actions
action #54149: Diffs for better error handling in the YAML editorResolvedlivdywan2019-07-11

Actions
action #54179: Re-use YAML betweens different groupsNew2019-07-12

Actions
action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingsResolvedlivdywan2019-08-13

Actions
coordination #55730: [epic] Move parameters from test suites into job groupsResolvedokurz2019-09-06

Actions
action #56540: convert staging job groups to YAMLResolvedlivdywan2019-09-06

Actions
action #57845: Switch more job groups to YAML job templatesResolvedokurz2019-10-09

Actions
action #58652: Write a training file about how to use YAML in job groupResolvedXiaojing_liu2019-10-24

Actions
action #60329: Use more parameterized job templates for test suites only used onceResolvedtinita2019-11-27

Actions
action #60782: descriptions for parameterized job templates independant of test suite descriptionsResolvedtinita2019-12-06

Actions
openQA Tests - action #64967: job templates are duplicated as job template in job groups as well as test suitesResolvedtinita2020-03-29

Actions
action #60014: Show YAML editor for unmigrated groupsResolvedokurz2019-11-19

Actions
action #60020: Provide a script to migrate job groups to YAML automaticallyResolvedokurz2019-11-19

Actions
action #60017: Document process and potential issues with migration to YAMLResolvedlivdywan2019-11-19

Actions
coordination #66727: [epic] Define structure to define test suites not in openQA databaseResolvedokurz2020-05-13

Actions
action #66781: hidden keys in yaml job templatesResolvedlivdywan2020-05-13

Actions

Related issues 1 (0 open1 closed)

Related to openQA Project - action #45413: Support markdown in the test suites descriptionRejectedokurz2018-12-19

Actions
Actions #1

Updated by okurz over 5 years ago

  • Parent task set to #15132
Actions #2

Updated by mkittler about 5 years ago

we need to extend the DB interface of job groups

Or, more specifically, of the job templates table.


In the last meeting we also discussed that having a config file in some Git repo could be preferable compared to saving the config in the database. So I suppose we save this idea for later?

Actions #3

Updated by okurz about 5 years ago

mkittler wrote:

So I suppose we save this idea for later?

I think that was the idea, yes.

Btw, I realized that with the "inter-machine dependencies" we reference machines within testsuites even though they are not defined there. Maybe this could be streamlined along with this epic here as I see it related.

Actions #4

Updated by JERiveraMoya about 5 years ago

  • Blocked by action #45413: Support markdown in the test suites description added
Actions #5

Updated by JERiveraMoya about 5 years ago

  • Blocked by deleted (action #45413: Support markdown in the test suites description)
Actions #6

Updated by JERiveraMoya about 5 years ago

  • Blocks action #45413: Support markdown in the test suites description added
Actions #7

Updated by mkittler about 5 years ago

In the last meeting the idea to have the data in a version-able and human readable document (e.g. YAML file) was picked up again.

Actions #8

Updated by livdywan almost 5 years ago

  • Due date set to 2019-06-18

due to changes in a related task

Actions #9

Updated by okurz over 4 years ago

  • Category changed from 122 to Feature requests
Actions #10

Updated by sebchlad over 4 years ago

"We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters."

The best example to try this out, which I can think of, would be hpc/mpi testing.

For SLE15 we might be supporting 5+ MPI implementations.
Testing of MPIs requires the multimachine set-up; currently I define following test suits for each MPI flavor: master, slaves*2, supportserver.
That means 5 MPIs * 4 test suits = 20 entries using the same test code beneath.
The matter is actually worse as the real bugs were found with different CPU count.
Currently I maintain 2 different CPU counts; QEMUCPUS=1 and QEMUCPUS=2.
So effectively = 20 test suits * 2.

We could consider parameterizing mpi_flavour to begin with.

@Oli: thx for pointing this poo to me.

Actions #11

Updated by okurz over 4 years ago

  • Due date changed from 2019-06-18 to 2019-09-06

due to changes in a related task

Actions #12

Updated by okurz over 4 years ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from Ready to Current Sprint

Discussed in QA tools meeting 2019-10-08, waiting for colleagues mainly from "QA SLE Migration" and "QA SLE Virtualization" to be able to follow on with migration, roughly mid of October. Blocked by subtask #55730

Actions #13

Updated by okurz over 4 years ago

  • Due date changed from 2019-06-18 to 2019-10-22

due to changes in a related task

Actions #14

Updated by okurz over 4 years ago

Actions #15

Updated by okurz over 4 years ago

  • Status changed from Blocked to Feedback

Going further with a deprecation notice: https://github.com/os-autoinst/openQA/pull/2518

Actions #16

Updated by okurz over 4 years ago

  • Due date changed from 2019-11-19 to 2020-02-29

due to changes in a related task

Actions #17

Updated by okurz over 4 years ago

  • Status changed from Feedback to Blocked

PR merged and also #60014 done. Rest is again in subtickets.

Actions #18

Updated by okurz about 4 years ago

  • Due date changed from 2020-02-29 to 2020-11-27

due to changes in a related task

Actions #19

Updated by szarate over 3 years ago

  • Tracker changed from action to coordination
  • Status changed from Blocked to New
  • Difficulty deleted (hard)
Actions #21

Updated by okurz over 3 years ago

  • Target version changed from Current Sprint to Ready
Actions #22

Updated by okurz over 3 years ago

  • Blocks deleted (action #45413: Support markdown in the test suites description)
Actions #23

Updated by okurz over 3 years ago

  • Related to action #45413: Support markdown in the test suites description added
Actions #24

Updated by okurz over 3 years ago

  • Status changed from New to Blocked
Actions #25

Updated by okurz over 2 years ago

  • Target version changed from Ready to future

saga not in current backlog as well as the only remaining subtask, moving to "future" for now again.

Actions

Also available in: Atom PDF