Project

General

Profile

Actions

action #56546

closed

list-merge feature in YAML job group templates

Added by okurz over 4 years ago. Updated over 4 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
-
Start date:
2019-09-06
Due date:
% Done:

0%

Estimated time:

Description

Motivation

https://openqa.suse.de/admin/job_templates/112 is a good example of "many scenarios same but not all" so a good candidate where we would like to have a "list-merge" feature from YAML e.g. so that we can write:

scenarios:
  ppc64le:
    sle-15-SP2-disabled-Installer-DVD-ppc64le: *tests
  s390x:
    sle-15-SP2-disabled-Installer-DVD-s390x: &tests
    - qa_userspace_apparmor-profiles
    - qa_userspace_bzip2
    - qa_userspace_cpio
  x86_64:
    sle-15-SP2-disabled-Installer-DVD-x86_64:
    - <<: *tests
    - qa_userspace_coreutils
    - qa_userspace_findutils
Actions #1

Updated by tinita over 4 years ago

Unfortunately this is not the way merge keys '<<' are working. They are only designed for merging mappings.

But we could handle it in our code. Taking your example:

scenarios:
  ppc64le:
    sle-15-SP2-disabled-Installer-DVD-ppc64le: &tests
    - qa_userspace_apparmor-profiles
    - qa_userspace_bzip2
    - qa_userspace_cpio
  s390x:
    sle-15-SP2-disabled-Installer-DVD-s390x: *tests
  x86_64:
    sle-15-SP2-disabled-Installer-DVD-x86_64:
    - *tests
    - qa_userspace_coreutils
    - qa_userspace_findutils

Then the code which reads the YAML can expand *tests if it is a list.
I guess this would make the JSON Schema a bit more complicated.

Another possibility would be to add our own merge-list tag:

scenarios:
  ppc64le:
    sle-15-SP2-disabled-Installer-DVD-ppc64le: &tests
    - qa_userspace_apparmor-profiles
    - qa_userspace_bzip2
    - qa_userspace_cpio
  s390x:
    sle-15-SP2-disabled-Installer-DVD-s390x: *tests
  x86_64:
    sle-15-SP2-disabled-Installer-DVD-x86_64: !merge-lists
    - *tests
    - - qa_userspace_coreutils
      - qa_userspace_findutils

But that's slighty more unreadable.
Alternatively, using flow sequence style:

scenarios:
  ppc64le:
    sle-15-SP2-disabled-Installer-DVD-ppc64le: &tests
    - qa_userspace_apparmor-profiles
    - qa_userspace_bzip2
    - qa_userspace_cpio
  s390x:
    sle-15-SP2-disabled-Installer-DVD-s390x: *tests
  x86_64:
    sle-15-SP2-disabled-Installer-DVD-x86_64: !merge-lists [
      *tests,
      [qa_userspace_coreutils, qa_userspace_findutils]
     ]

Still, harder to understand.

And adding our custom tag makes us less flexible. This wouldn't be read correctly by a python script using pyyaml, unless we also add this tag to the pyyaml loader.

Actions #2

Updated by tinita over 4 years ago

Yet another alternative:

scenarios:
  ppc64le:
    sle-15-SP2-disabled-Installer-DVD-ppc64le: *tests
  s390x:
    sle-15-SP2-disabled-Installer-DVD-s390x: &tests
    - qa_userspace_apparmor-profiles
    - qa_userspace_bzip2
    - qa_userspace_cpio
  x86_64:
    sle-15-SP2-disabled-Installer-DVD-x86_64:
    - _merge_: *tests
    - qa_userspace_coreutils
    - qa_userspace_findutils

This could then be handled by our code. It would have the advantage to not interfere with the official !!merge tag <<

Actions #3

Updated by okurz over 4 years ago

What I would like to have is what is described on https://blog.daemonl.com/2016/02/yaml.html in the section "Extend <<"

Actions #4

Updated by tinita over 4 years ago

Like discussed in chat, this could work:

product-arch1: &common
  testsuite1:
  testsuite2:
    settings:
      foo: bar
  testsuite3:
product-arch2:
  <<: *common
  suite-x:
  suite-y:
    settings:
      foo: baz
  suite-z:

And this is how it would look like if the scenario defining the anchor has additional testsuites as well:

product-arch1:
  <<: &common
    testsuite1:
    testsuite2:
      settings:
        foo: bar
    testsuite3:
  suite-a:
  suite-b:
product-arch2:
  <<: *common
  suite-x:
  suite-y:
    settings:
      foo: baz
  suite-z:
Actions #5

Updated by coolo over 4 years ago

For reference: I'm voting against non standard yaml extensions 'in our code'. I think any kind of pre-processing/merging/whatnot should be done by a client working on git level.
I accept the user story (and would even agree it's pretty high on the list), but I reject the feature request.

Actions #6

Updated by tinita over 4 years ago

coolo wrote:

For reference: I'm voting against non standard yaml extensions 'in our code'.

I agree.
One question is if we see the merge-keys as a standard. I would say it is, just that so far there was no perl module that could handle it.

But as far as I can see we cannot use mappings here anyway because it's possible to have the same testsuite several times, with different settings.

Actions #7

Updated by livdywan over 4 years ago

coolo wrote:

For reference: I'm voting against non standard yaml extensions 'in our code'. I think any kind of pre-processing/merging/whatnot should be done by a client working on git level.
I accept the user story (and would even agree it's pretty high on the list), but I reject the feature request.

I kind of agree. I'd prefer the YAML document always be valid against the schema and readable with other parsers. Otherwise the story of consuming YAML in scripts is going to be a lot weaker.

Actions #8

Updated by coolo over 4 years ago

  • Status changed from New to Rejected

I object to openQA suggesting this use. Validating this will be painful - so I repeat: changes like this should be done in a remote git workflow where we can run validations on change requests. This git workflow can then upload 'simplifed yaml' to openQA. This also has the adventage that we can apply non-yaml expansions, templating solutions, ...

Actions

Also available in: Atom PDF