action #59148
closedSupport YAML merge keys in Job Group Editor
0%
Description
The Job group editor is using YAML::XS, which does not support YAML merge keys, which allow you to merge several mappings into one:
https://yaml.org/type/merge.html
E.g.
---
defaults: &defaults
a: 1
b: 2
settings:
<<: *defaults
b: 22
c: 3
There is currently only one module in perl supporting that: YAML::PP https://metacpan.org/pod/YAML::PP.
It allows to use libyaml as a parser backend via YAML::PP::LibYAML https://metacpan.org/pod/YAML::PP::LibYAML, so we can rely on the battle tested parsing of libyaml while using the features of the pure perl loader.
The only disadvantage is that YAML::PP is still pretty new and not yet "battle tested" like YAML::XS.
The advantage is, that YAML::PP adheres to the YAML 1.2 spec (regarding the schema, e.g. TRUE, True, true are booleans), while YAML::XS did only implement a small subset of YAML 1.1 types.
So it will be compatible to other YAML 1.2 processors.
The change in the code itself would be easy:
Instead of
my $data = YAML::XS::Load(...);
-->
my $yp = YAML::PP->new( schema => [qw/ + Merge /] );
my $data = $yp->load_string(...);
Checklist
- Add YAML::PP (and poosibly YAML::PP::LibYAML) deps and submit them to Factory/Leap
- Change code to use the new module
- Add tests
- Possibly check all existing templates if they can be parsed with the new module
- Add documentation on how to use merge keys
- Add a gerneral description for YAML 1.2 (e.g. which strings are special)
Updated by coolo over 4 years ago
- Target version set to future
As long as we have enough test coverage, an unstable api shouldn't hurt - mojolicious suprises us from time to time as well :)
But for now I don't want to invest further into complex yaml features.
Updated by tinita over 4 years ago
- Related to action #62603: Anchor/alias for hash map in job template settings in Job Groups YAML produces error added
Updated by tinita over 4 years ago
- Checklist item changed from to [ ] Add YAML::PP (and poosibly YAML::PP::LibYAML) deps and submit them to Factory/Leap, [ ] Change code to use the new module, [ ] Add tests
- Description updated (diff)
Updated by tinita over 4 years ago
- Checklist item changed from [ ] Add YAML::PP (and poosibly YAML::PP::LibYAML) deps and submit them to Factory/Leap, [ ] Change code to use the new module, [ ] Add tests to [ ] Add YAML::PP (and poosibly YAML::PP::LibYAML) deps and submit them to Factory/Leap, [ ] Change code to use the new module, [ ] Add tests, [ ] Possibly check all existing templates if they can be parsed with the new module
Updated by tinita about 4 years ago
- Checklist item changed from to [x] Possibly check all existing templates if they can be parsed with the new module
Updated by tinita about 4 years ago
Checked templates with:
% lwp-request -m GET https://openqa.opensuse.org/api/v1/job_templates_scheduling >templates.yaml
% perl -wE'
use YAML::PP;
use YAML::PP::LibYAML;
my $file = "templates.yaml";
my $data = YAML::PP::LoadFile($file);
for my $key (sort keys %$data) {
say $key;
my $yaml = $data->{ $key };
my $template = YAML::PP::Load($yaml);
$template = YAML::PP::LibYAML::Load($yaml);
}'
# same with openqa.suse.de
% lwp-request -m GET https://openqa.suse.de/api/v1/job_templates_scheduling >templates.yaml
% ...
Updated by coolgw about 4 years ago
I like this feature, i think we even can start discard test suite if this support.
We just need create a template within yaml, and many other case can use template with some specific setting.
Hope this can be supported in near future.
Updated by tinita about 4 years ago
- Status changed from New to In Progress
- Assignee set to tinita
- Target version changed from future to Current Sprint
Updated by tinita about 4 years ago
YAML::PP and YAML::LibYAML::API are already in openQA:devel.
(I found a bug in YAML::LibYAML::API though and released version 0.008 today).
I submitted YAML::PP::LibYAML today.
I changed the code to use YAML::PP. As soon as the *LibYAML* packages are in the repo, I can replace it by YAML::PP::LibYAML.
I also added a test for merge keys.
Updated by tinita about 4 years ago
- Checklist item changed from to [x] Change code to use the new module
Updated by tinita about 4 years ago
- Checklist item changed from to [x] Add tests
Updated by tinita about 4 years ago
- Checklist item changed from [ ] Add YAML::PP (and poosibly YAML::PP::LibYAML) deps and submit them to Factory/Leap, [x] Change code to use the new module, [x] Add tests, [x] Possibly check all existing templates if they can be parsed with the new module to [ ] Add YAML::PP (and poosibly YAML::PP::LibYAML) deps and submit them to Factory/Leap, [x] Change code to use the new module, [x] Add tests, [x] Possibly check all existing templates if they can be parsed with the new module, [ ] Add documentation on how to use merge keys, [ ] Add a gerneral description for YAML 1.2 (e.g. which strings are special)
Updated by tinita about 4 years ago
- Checklist item changed from to [x] Add documentation on how to use merge keys
Updated by tinita about 4 years ago
- Checklist item changed from to [x] Add a gerneral description for YAML 1.2 (e.g. which strings are special)
Updated by tinita about 4 years ago
Created a draft PR: https://github.com/os-autoinst/openQA/pull/2751
Now waiting for the libyaml packages in OBS
Updated by tinita about 4 years ago
As stated in the PR, I think we won't see a performance impact by using YAML::PP, since the files are only loaded in the Jobtemplate editor.
Updated by tinita about 4 years ago
- Checklist item changed from [ ] Add YAML::PP (and poosibly YAML::PP::LibYAML) deps and submit them to Factory/Leap, [x] Change code to use the new module, [x] Add tests, [x] Possibly check all existing templates if they can be parsed with the new module, [x] Add documentation on how to use merge keys, [x] Add a gerneral description for YAML 1.2 (e.g. which strings are special) to [x] Add YAML::PP (and poosibly YAML::PP::LibYAML) deps and submit them to Factory/Leap, [x] Change code to use the new module, [x] Add tests, [x] Possibly check all existing templates if they can be parsed with the new module, [x] Add documentation on how to use merge keys, [x] Add a gerneral description for YAML 1.2 (e.g. which strings are special)
- Status changed from In Progress to Feedback
Updated by tinita about 4 years ago
I sent information to opensuse-factory and openqa@suse.de
https://lists.opensuse.org/opensuse-factory/2020-02/msg00532.html