QA (public)

@okurz Thank you so much for the quick replying.
The reason why I set it "High" priority is QE-SAP has a JIRA ticket aims to enhance this "packages list" during our current scrum sprint.
After out JIRA ticket was done, then followed by adding more test cases using this packages list.
Also, QE-SAP team plans to using this method (packages white list) for triggering almost all our test cases.
We do not want to copy -> paste similar packages list to almost all sap*.yml files.

YAML standard dont have any include or import directive .. --> so it will be some custom changes to PyYAML parsing ( outside scope of bot-ng...)

osukup wrote:

YAML standard dont have any include or import directive .. --> so it will be some custom changes to PyYAML parsing ( outside scope of bot-ng...)

Thanks for the support and info.
There is an example in https://github.com/os-autoinst/os-autoinst-distri-opensuse. It supports "<<: !include $path" in yaml files:

# cd .../tests/sle
# find .| xargs grep -s '<<: !include'

Then we can find lots of examples.

llzhao wrote:

@okurz Thank you so much for the quick replying.
The reason why I set it "High" priority is QE-SAP has a JIRA ticket aims to enhance this "packages list" during our current scrum sprint.

Ok, fine. But we won't be able to do that within the usual SLO reaction time of one month for "High" so I suggest either you plan to do something like that within your team or reduce the prio to "Low"

okurz wrote:

llzhao wrote:

@okurz Thank you so much for the quick replying.
The reason why I set it "High" priority is QE-SAP has a JIRA ticket aims to enhance this "packages list" during our current scrum sprint.

Ok, fine. But we won't be able to do that within the usual SLO reaction time of one month for "High" so I suggest either you plan to do something like that within your team or reduce the prio to "Low"

Thanks for the info, we would like to increase the Priority to "urgent: <1 week" according to "https://progress.opensuse.org/projects/qa/wiki/Tools"
Please help to edit the "Priority" as I can not edit it (I do not know why).

I was asking for a concrete example of the intended usage.

It is clear how the !include tag is working in os-autoinst-distri-opensuse in combination with the YAML << merge key.
The merge key is able to merge several mappings (aka dictionaries, hashes, objects, hash maps) into one mapping.
https://yaml.org/type/merge.html

However, you mentioned that you would like to reuse the packages in the link to the sap* yaml files.
What I can see there is:

      packages:
        - aaa_base
        - ...

This (the list of packages) is a YAML sequence (aka list, array).
The merge key in YAML is not able to merge (or rather concatenate) sequences.

That's why we really need a concrete example of how your YAML would look like. It is a feature request, and before anyone works on it, it should be clear what is wanted.
But from what I can see without having an example, I think what you want is not possible with the merge key feature.

It would require a custom tag, e.g. something like !concatenate, that would concatenate two or more sequences.

And additionally, it would still require the !include tag. So that would mean implementing two custom YAML tags.
Especially the !include tag is quite some work, because it has to provide some configuration, and in its default config be implemented safely (no absolute file paths, prevent cyclic inclusion). I did implement that for the YAML::PP module in perl, which we use in os-autoinst-distri-opensuse, so I can at least imagine what kind of work would be required here.

And just for completeness, in case you really want to merge mappings, then there is still the issue that we will have to figure out if it would be possible to use << and !include in combination.

Like I wrote in the chat in PyYAML it would currently not be possible to do something like this:

<<: !include file.yaml

The reason is how the << merge key tag is implemented. On the right hand side it always expects something to merge. It doesn't consider tags there, so even if the !include tag would return a mapping that would be mergeable, PyYAML will reject it without even attempting to resolve the !include tag.

In qem-bot we are using ruamel.yaml, which was originally forked from PyYAML. I would have to check the sources whether this restriction in the implementation is still there or not. Even if it worked, it would mean we would never be able to switch back to PyYAML (for whatever reason).

I would like to fix PyYAML upstream to allow this, but I don't know when I will have time for this.

So, please first clarify what you would need, and how important this is, and then we have to figure out what to do.

llzhao wrote:

[…]
Thanks for the info, we would like to increase the Priority to "urgent: <1 week" according to "https://progress.opensuse.org/projects/qa/wiki/Tools"
Please help to edit the "Priority" as I can not edit it (I do not know why).

sorry, we will not be able to treat this as "Urgent". In general we should not treat any feature request as "Urgent" in comparison to bug fixes and such. Even "High" is an exception. This ticket is in "future" meaning that it is not currently planned to work on this.

But first, please see tinita's comments.

@okurz, Got it. Thanks for the updates and explanation.

@tinita,
Thanks for the explanation and updates. I am not expert on Python and Perl but I can understand what you said in general.
What we want to solve is avoiding copy->paste those system packages everywhere (It is important for us as we plan to apply similar
package whitelist for other SAP test cases' openQA trigger in the future):

        - aaa_base
        - bash
        - ...
        - vim

For example we already copied-> pasted to files:
https://gitlab.suse.de/qa-maintenance/metadata/-/blob/master/bot-ng/sap15sp4.yml (sap15)
...
https://gitlab.suse.de/qa-maintenance/metadata/-/blob/master/bot-ng/sap12sp4.yml (sap12)

Any method to solve this issue is OK for us.

Btw, I checked os-autoinst-distri-opensuse and it looks like this way (or similar method is OK too):

./test_data/yast/raid/raid5_prep_boot.yaml:<<: !include test_data/yast/raid/raid_disks_prep_boot.yaml
./test_data/yast/raid/raid6_prep_boot.yaml:<<: !include test_data/yast/raid/raid_disks_prep_boot.yaml

If you really want the list of packages, then you cannot use the merge key, like I said.
It merges two mappings.

But if you really want to just have that same list in all the files, that can be done with the include tag, yes.
I'm not sure why you are quoting the merge key.

So:

# packages.yaml:
- package1
- package2

# sap15sp1.yml
incidents:
  FLAVOR:
    Server-DVD-...:
      ...
      packages: !include packages.yaml

So it would be necessary to implement an include tag.
Alternatively one could add some custom syntax and load it in the bot-ng code when the YAML loader has loaded it into the python data structure.

@tinita, thanks for the updates, your example is what I expected. Thanks.