QA » openQA Project

I can imagine the following different ways how the YAML format could look like as examples and what the consequences would be:

• Option 1: Override the automatic scenario name with a variable "name"

scenarios:
  ppc64le:
    - gnome
    - gnome:
      - name: gnome-overrides
      - settings:
        MR_TEST: overrides
    - gnome:
      - name: gnome-sapconf
      - settings:
        MR_TEST: sapconf

which should end up in three scenarios, "gnome", "gnome-overrides" and "gnome-sapconf".

• Option 2: Define testsuites in-place with the name

scenarios:
  ppc64le:
    - gnome
    - gnome-overrides:
      - settings:
        DESKTOP: gnome
        MR_TEST: overrides
    - gnome-sapconf:
      - settings:
        DESKTOP: gnome
        MR_TEST: sapconf

this should do the same as Option 1, three scenarios "gnome", "gnome-overrides" and "gnome-sapconf" where only the first maps to an existing test suite, the others are defined in-place. There would be no error-handling in this case about testsuites not being mapped. As an alternative, Option 2b, one can make that explicit, e.g. with - testsuite: none as an additional parameter. I like this option because it allows to migrate to a complete test plan definition managed in plain text files that we can eventually store next to the test modules and the test plan in YAML format we already have there.

• Option 3: Auto-generated scenario names when ambiguous

scenarios:
  ppc64le:
    - gnome
    - gnome:
      - settings:
        MR_TEST: overrides
    - gnome:
      - settings:
        MR_TEST: sapconf

ending up in auto-generated names with the testsuite component replaced by "gnome", "gnome-overrides", "gnome-sapconf".

All three options are compatible and could be all allowed in parallel.

OK, just for context: the user story of rbranco are 28 test suites with 13 settings, where only 1 is different between the various test suites. The perfect use case for what we did this :)

I kind of like the idea of potentially moving away from predefined test suites at all, BUT that would require in this use case either duplicating 13 settings in every section (and then there would be basically no distinction between test suite settings and parameters anymore) or some kind of inheritance.

Having the name of the job as key in yaml sounds to me the most natural way to express this, but then we need a testsuite setting. Aka option 2+

scenarios:
  ppc64le:
    - gnome # follows test suite gnome blindly
    - gnome-overrides:
      - testsuite: gnome # overwrites gnome with a different name
      - settings:
        MR_TEST: overrides
    - gnome: # follows test suite gnome too, but will be named gnome_2 (and the first gnome will be gnome_1)
      - settings:
        MR_TEST: sapconf

coolo wrote:

I kind of like the idea of potentially moving away from predefined test suites at all, BUT that would require in this use case either duplicating 13 settings in every section (and then there would be basically no distinction between test suite settings and parameters anymore) or some kind of inheritance.

There is #54179 which proposes sharing the YAML between groups.

Actually I find the idea of inheritance from existing test suites ie. option 2b a little more intuitive and straightforward without adding another layer. I'd probably stay away from generating names or even specifying names entirely and just re-use test suites where they are a good basis.
Although test-suite: none sounds a bit hackish... it might be cleaner to just use unique names instead.

I've been pondering if we should have a visualization of the actual scenarios/ dynamic test suites that would make it very clear what's used and what's generated. That would address accidentally creating new test suites when it wasn't intended as well as aliases behaving different to what one was expecting.
Although so far I'm not aware that anyone is asking for this so far. Seeing the build might be good enough.

That sounds like a misunderstanding. We for sure don't want to generate test suites from here. Only name jobs

I think coolo means "scenario" when he says "job", as defined on http://open.qa/docs/#concepts . He will probably say that I am wrong because he can't admit that he's wrong ;)

I didn't (consciously) suggest to generate test suites. But I'll try and clarify a bit.

We can generate names, or specify names, and I think both is easy enough in a technical sense. test_suite_HASHED_SETTINGS or some such.

But when you save the YAML, the job templates are updated. These are (currently) a unique combination of a test suite, a product and a machine - a scenario. So at the least we need a new column with the (generated) scenario name. This is where the job eventually comes from. We don't create jobs based on YAML.

As discussed in the weekly jangouts call, we actually agreed on option 2+. The likely necessary tasks to continue are:

• Add a column for the test suite name that is defined within the YAML document to the job templates table
• Add parsing for that key into the job templates parser
• Read out the test suite name from the job templates table when generating jobs

Just wanted to mention that the examples probably have mistakes.

The following YAML

  - settings:
    MR_TEST: overrides

is aequivalent to

  - settings: null
    MR_TEST: overrides

Probably it was meant to be this?

  - settings:
      MR_TEST: overrides

okurz wrote:

Feedback on the feature:

• The left column in the YAML editor does not seem to document the feature
• The schema mentions the new parameter test suite but does not explain that the job template name is generated from the section header of scenarios when the new parameter is used (and does not need to match to an test suite)

https://github.com/os-autoinst/openQA/pull/2323