Project

General

Profile

coordination #44360

openQA Tests - coordination #15132: [saga][epic] Better structure of test plans in main.pm

[epic] Parameterize test suites within job groups

Added by coolo over 2 years ago. Updated about 1 month ago.

Status:
Blocked
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2019-01-25
Due date:
% Done:

86%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters.

This has several implicitions - hence the epic.

  • we need to provide a variable setting in the job group interface
  • we need to summarize the test suite differently in the review interfaces, possibly by requiring the user to give the paramterized test suite another name
  • we need to extend the DB interface of job groups
  • we need to provide a new API for this

Subtasks

action #46667: Define version-able and human readable format for job scheduling-related tablesResolvedcdywan

action #46670: Create import for format defined in #46667Resolvedcdywan

action #46673: Extend tables for job scheduling by another table for parametersResolvedcdywan

action #50675: Commit changes to scheduling YAML to Git repositoryNew

action #54143: Support parameters in {load,dump}_templatesResolvedcdywan

action #54146: Expose saving YAML and opt-in migration in the editorResolvedcdywan

action #54149: Diffs for better error handling in the YAML editorResolvedcdywan

action #54179: Re-use YAML betweens different groupsNew

action #55454: The job group YAML schedule should support multiple scenarios with different variables, not just settingsResolvedcdywan

coordination #55730: [epic] Move parameters from test suites into job groupsResolvedokurz

action #56540: convert staging job groups to YAMLResolvedcdywan

action #57845: Switch more job groups to YAML job templatesResolvedokurz

action #58652: Write a training file about how to use YAML in job groupResolvedXiaojing_liu

action #60329: Use more parameterized job templates for test suites only used onceResolvedtinita

action #60782: descriptions for parameterized job templates independant of test suite descriptionsResolvedtinita

openQA Tests - action #64967: job templates are duplicated as job template in job groups as well as test suitesResolvedtinita

action #60014: Show YAML editor for unmigrated groupsResolvedokurz

action #60020: Provide a script to migrate job groups to YAML automaticallyResolvedokurz

action #60017: Document process and potential issues with migration to YAMLResolvedcdywan

coordination #66727: [epic] Define structure to define test suites not in openQA databaseResolvedokurz

action #66781: hidden keys in yaml job templatesResolvedcdywan


Related issues

Related to openQA Project - action #45413: Support markdown in the test suites descriptionRejected2018-12-19

History

#1 Updated by okurz over 2 years ago

  • Parent task set to #15132

#2 Updated by mkittler over 2 years ago

we need to extend the DB interface of job groups

Or, more specifically, of the job templates table.


In the last meeting we also discussed that having a config file in some Git repo could be preferable compared to saving the config in the database. So I suppose we save this idea for later?

#3 Updated by okurz over 2 years ago

mkittler wrote:

So I suppose we save this idea for later?

I think that was the idea, yes.

Btw, I realized that with the "inter-machine dependencies" we reference machines within testsuites even though they are not defined there. Maybe this could be streamlined along with this epic here as I see it related.

#4 Updated by JERiveraMoya over 2 years ago

  • Blocked by action #45413: Support markdown in the test suites description added

#5 Updated by JERiveraMoya over 2 years ago

  • Blocked by deleted (action #45413: Support markdown in the test suites description)

#6 Updated by JERiveraMoya over 2 years ago

  • Blocks action #45413: Support markdown in the test suites description added

#7 Updated by mkittler over 2 years ago

In the last meeting the idea to have the data in a version-able and human readable document (e.g. YAML file) was picked up again.

#8 Updated by cdywan about 2 years ago

  • Due date set to 2019-06-18

due to changes in a related task

#9 Updated by okurz about 2 years ago

  • Category changed from 122 to Feature requests

#10 Updated by sebchlad almost 2 years ago

"We discussed how we can reduce the number of test suites and one solution is to move
the workflow parameters to job groups.

So instead of install-kde, install-gnome and install-text, we schedule
install 3 times with 3 different DESKTOP parameters."

The best example to try this out, which I can think of, would be hpc/mpi testing.

For SLE15 we might be supporting 5+ MPI implementations.
Testing of MPIs requires the multimachine set-up; currently I define following test suits for each MPI flavor: master, slaves*2, supportserver.
That means 5 MPIs * 4 test suits = 20 entries using the same test code beneath.
The matter is actually worse as the real bugs were found with different CPU count.
Currently I maintain 2 different CPU counts; QEMUCPUS=1 and QEMUCPUS=2.
So effectively = 20 test suits * 2.

We could consider parameterizing mpi_flavour to begin with.

@Oli: thx for pointing this poo to me.

#11 Updated by okurz almost 2 years ago

  • Due date changed from 2019-06-18 to 2019-09-06

due to changes in a related task

#12 Updated by okurz almost 2 years ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from Ready to Current Sprint

Discussed in QA tools meeting 2019-10-08, waiting for colleagues mainly from "QA SLE Migration" and "QA SLE Virtualization" to be able to follow on with migration, roughly mid of October. Blocked by subtask #55730

#13 Updated by okurz almost 2 years ago

  • Due date changed from 2019-06-18 to 2019-10-22

due to changes in a related task

#15 Updated by okurz over 1 year ago

  • Status changed from Blocked to Feedback

Going further with a deprecation notice: https://github.com/os-autoinst/openQA/pull/2518

#16 Updated by okurz over 1 year ago

  • Due date changed from 2019-11-19 to 2020-02-29

due to changes in a related task

#17 Updated by okurz over 1 year ago

  • Status changed from Feedback to Blocked

PR merged and also #60014 done. Rest is again in subtickets.

#18 Updated by okurz over 1 year ago

  • Due date changed from 2020-02-29 to 2020-11-27

due to changes in a related task

#19 Updated by szarate 10 months ago

  • Tracker changed from action to coordination
  • Status changed from Blocked to New
  • Difficulty deleted (hard)

#21 Updated by okurz 9 months ago

  • Target version changed from Current Sprint to Ready

#22 Updated by okurz 9 months ago

  • Blocks deleted (action #45413: Support markdown in the test suites description)

#23 Updated by okurz 9 months ago

  • Related to action #45413: Support markdown in the test suites description added

#24 Updated by okurz 9 months ago

  • Status changed from New to Blocked

#25 Updated by okurz about 1 month ago

  • Target version changed from Ready to future

saga not in current backlog as well as the only remaining subtask, moving to "future" for now again.

Also available in: Atom PDF