Project

General

Profile

action #55454

openQA Tests - action #15132: [saga][epic] Better structure of test plans in main.pm

action #44360: [epic] Parameterize test suites within job groups

The job group YAML schedule should support multiple scenarios with different variables, not just settings

Added by rbranco 11 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2019-08-13
Due date:
% Done:

0%

Estimated time:
Difficulty:
Duration:

Description

scenarios:
  ppc64le:
    - sles4sap_scc_gnome_saptune_v2:
      settings:
      MR_TEST: overrides
    - sles4sap_scc_gnome_saptune_v2:
      settings:
      MR_TEST: sapconf

History

#1 Updated by okurz 11 months ago

  • Subject changed from The job group description YAML should support multiple jobs with different variables, not just settings. to The job group YAML schedule should support multiple scenarios with different variables, not just settings
  • Category set to Feature requests

#2 Updated by okurz 11 months ago

  • Parent task set to #44360

#3 Updated by cdywan 11 months ago

  • Assignee set to cdywan

#4 Updated by okurz 11 months ago

I can imagine the following different ways how the YAML format could look like as examples and what the consequences would be:

  • Option 1: Override the automatic scenario name with a variable "name"
scenarios:
  ppc64le:
    - gnome
    - gnome:
      - name: gnome-overrides
      - settings:
        MR_TEST: overrides
    - gnome:
      - name: gnome-sapconf
      - settings:
        MR_TEST: sapconf

which should end up in three scenarios, "gnome", "gnome-overrides" and "gnome-sapconf".

  • Option 2: Define testsuites in-place with the name
scenarios:
  ppc64le:
    - gnome
    - gnome-overrides:
      - settings:
        DESKTOP: gnome
        MR_TEST: overrides
    - gnome-sapconf:
      - settings:
        DESKTOP: gnome
        MR_TEST: sapconf

this should do the same as Option 1, three scenarios "gnome", "gnome-overrides" and "gnome-sapconf" where only the first maps to an existing test suite, the others are defined in-place. There would be no error-handling in this case about testsuites not being mapped. As an alternative, Option 2b, one can make that explicit, e.g. with - testsuite: none as an additional parameter. I like this option because it allows to migrate to a complete test plan definition managed in plain text files that we can eventually store next to the test modules and the test plan in YAML format we already have there.

  • Option 3: Auto-generated scenario names when ambiguous
scenarios:
  ppc64le:
    - gnome
    - gnome:
      - settings:
        MR_TEST: overrides
    - gnome:
      - settings:
        MR_TEST: sapconf

ending up in auto-generated names with the testsuite component replaced by "gnome", "gnome-overrides", "gnome-sapconf".

All three options are compatible and could be all allowed in parallel.

#5 Updated by coolo 11 months ago

OK, just for context: the user story of rbranco are 28 test suites with 13 settings, where only 1 is different between the various test suites. The perfect use case for what we did this :)

I kind of like the idea of potentially moving away from predefined test suites at all, BUT that would require in this use case either duplicating 13 settings in every section (and then there would be basically no distinction between test suite settings and parameters anymore) or some kind of inheritance.

Having the name of the job as key in yaml sounds to me the most natural way to express this, but then we need a testsuite setting. Aka option 2+

scenarios:
  ppc64le:
    - gnome # follows test suite gnome blindly
    - gnome-overrides:
      - testsuite: gnome # overwrites gnome with a different name
      - settings:
        MR_TEST: overrides
    - gnome: # follows test suite gnome too, but will be named gnome_2 (and the first gnome will be gnome_1)
      - settings:
        MR_TEST: sapconf

#6 Updated by cdywan 11 months ago

coolo wrote:

I kind of like the idea of potentially moving away from predefined test suites at all, BUT that would require in this use case either duplicating 13 settings in every section (and then there would be basically no distinction between test suite settings and parameters anymore) or some kind of inheritance.

There is #54179 which proposes sharing the YAML between groups.

Actually I find the idea of inheritance from existing test suites ie. option 2b a little more intuitive and straightforward without adding another layer. I'd probably stay away from generating names or even specifying names entirely and just re-use test suites where they are a good basis.
Although test-suite: none sounds a bit hackish... it might be cleaner to just use unique names instead.

I've been pondering if we should have a visualization of the actual scenarios/ dynamic test suites that would make it very clear what's used and what's generated. That would address accidentally creating new test suites when it wasn't intended as well as aliases behaving different to what one was expecting.
Although so far I'm not aware that anyone is asking for this so far. Seeing the build might be good enough.

#7 Updated by coolo 11 months ago

That sounds like a misunderstanding. We for sure don't want to generate test suites from here. Only name jobs

#8 Updated by okurz 11 months ago

I think coolo means "scenario" when he says "job", as defined on http://open.qa/docs/#concepts . He will probably say that I am wrong because he can't admit that he's wrong ;)

#9 Updated by cdywan 11 months ago

I didn't (consciously) suggest to generate test suites. But I'll try and clarify a bit.

We can generate names, or specify names, and I think both is easy enough in a technical sense. test_suite_HASHED_SETTINGS or some such.

But when you save the YAML, the job templates are updated. These are (currently) a unique combination of a test suite, a product and a machine - a scenario. So at the least we need a new column with the (generated) scenario name. This is where the job eventually comes from. We don't create jobs based on YAML.

#10 Updated by coolo 10 months ago

  • Target version set to Current Sprint

we are actually working on it, so move to CS

#11 Updated by okurz 10 months ago

As discussed in the weekly jangouts call, we actually agreed on option 2+. The likely necessary tasks to continue are:

  • Add a column for the test suite name that is defined within the YAML document to the job templates table
  • Add parsing for that key into the job templates parser
  • Read out the test suite name from the job templates table when generating jobs

#12 Updated by cdywan 10 months ago

  • Status changed from New to Resolved

#13 Updated by okurz 10 months ago

  • Status changed from Resolved to Feedback

Feedback on the feature:

  • The left column in the YAML editor does not seem to document the feature
  • The schema mentions the new parameter test suite but does not explain that the job template name is generated from the section header of scenarios when the new parameter is used (and does not need to match to an test suite)

#14 Updated by tinita 10 months ago

Just wanted to mention that the examples probably have mistakes.

The following YAML

  - settings:
    MR_TEST: overrides

is aequivalent to

  - settings: null
    MR_TEST: overrides

Probably it was meant to be this?

  - settings:
      MR_TEST: overrides

#15 Updated by cdywan 10 months ago

okurz wrote:

Feedback on the feature:

  • The left column in the YAML editor does not seem to document the feature
  • The schema mentions the new parameter test suite but does not explain that the job template name is generated from the section header of scenarios when the new parameter is used (and does not need to match to an test suite)

https://github.com/os-autoinst/openQA/pull/2323

#16 Updated by cdywan 10 months ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF