action #71251
closed
Forbid duplicate mapping keys in Job Templates YAML files
Added by tinita over 3 years ago.
Updated over 3 years ago.
Category:
Feature requests
Description
The YAML spec says YAML Loaders should forbid duplicate mapping keys.
Until now, YAML::Tiny is the only Perl module implementing that. Modues like YAML::XS, YAML.pm, YAML::Syck and also several implementations in other languages like PyYAML just ignore it and override the previous key.
But automatically detecting duplicate keys can be useful if the YAML mapping is so long it doesn't fit in your editor.
YAML::PP 0.026 (which was released yesterday) adds an option duplicate_keys. The default is allowing duplicates for now.
The plan is to set the default to forbid in 0.027.
So the Todos here are:
- Add
duplicate_keys to YAML::PP ✔️
- Wait until YAML::PP 0.026 is in our devel:openQA repo ✔️
- Test all existing job templates with
duplicate_keys => 0 ✔️
- Test YAML files in os-autoinst-distri-opensuse ✔️
- Fix broken templates, if there are any ✔️
- Set
YAML::PP >= 0.26 and duplicate_keys => 0 in our openQA code ✔️
- When 0.027 gets released, we can remove the
duplicate_keys => 0
Tested all templates.
There is one template with a duplicate key in o3 and four in osd.
Created a MR for one of the osd templates. The others don't seem to be maintained in git, so I will fix them in the webui.
- Description updated (diff)
- Description updated (diff)
Code used for testing:
% ./script/openqa-cli api --osd job_templates_scheduling --json >osd-templates.json
% perl -wE'
use JSON::XS;
use YAML::PP;
use Data::Dumper;
my ($f) = @ARGV;
my $yp = YAML::PP->new( duplicate_keys => 0 );
my $json = do { open my $fh, "<", $f or die $!; local $/; <$fh> };
my $data = decode_json($json);
for my $key (sort keys %$data) {
say "===== $key";
my $yaml = $data->{ $key };
my $t = eval { $yp->load_string($yaml); };
if ($@) {
say "ERROR($key): $@\n$yaml";
}
}
' osd-templates.json
All templates have been fixed now.
- Status changed from In Progress to Feedback
- Status changed from Feedback to Resolved
As you reported that all existing templates on o3 and osd should be fixed and the change is now deployed to o3 and osd as well I consider this "Resolved".
Also available in: Atom
PDF