action #71251
closedForbid duplicate mapping keys in Job Templates YAML files
Description
The YAML spec says YAML Loaders should forbid duplicate mapping keys.
Until now, YAML::Tiny is the only Perl module implementing that. Modues like YAML::XS, YAML.pm, YAML::Syck and also several implementations in other languages like PyYAML just ignore it and override the previous key.
But automatically detecting duplicate keys can be useful if the YAML mapping is so long it doesn't fit in your editor.
Updated by tinita over 4 years ago
YAML::PP 0.026 (which was released yesterday) adds an option duplicate_keys
. The default is allowing duplicates for now.
The plan is to set the default to forbid in 0.027.
So the Todos here are:
- Add
duplicate_keys
to YAML::PP ✔️ - Wait until YAML::PP 0.026 is in our devel:openQA repo ✔️
- Test all existing job templates with
duplicate_keys => 0
✔️ - Test YAML files in os-autoinst-distri-opensuse ✔️
- Fix broken templates, if there are any ✔️
- Set
YAML::PP >= 0.26
andduplicate_keys => 0
in our openQA code ✔️ - When 0.027 gets released, we can remove the
duplicate_keys => 0
Updated by tinita over 4 years ago
Tested all templates.
There is one template with a duplicate key in o3 and four in osd.
Created a MR for one of the osd templates. The others don't seem to be maintained in git, so I will fix them in the webui.
Updated by tinita over 4 years ago
Fixed https://openqa.opensuse.org/admin/job_templates/66 and the other 3 osd templates.
Updated by tinita over 4 years ago
Code used for testing:
% ./script/openqa-cli api --osd job_templates_scheduling --json >osd-templates.json
% perl -wE'
use JSON::XS;
use YAML::PP;
use Data::Dumper;
my ($f) = @ARGV;
my $yp = YAML::PP->new( duplicate_keys => 0 );
my $json = do { open my $fh, "<", $f or die $!; local $/; <$fh> };
my $data = decode_json($json);
for my $key (sort keys %$data) {
say "===== $key";
my $yaml = $data->{ $key };
my $t = eval { $yp->load_string($yaml); };
if ($@) {
say "ERROR($key): $@\n$yaml";
}
}
' osd-templates.json
Updated by tinita over 4 years ago
- Status changed from In Progress to Feedback
PR https://github.com/os-autoinst/openQA/pull/3390 was merged.
Updated by okurz over 4 years ago
- Status changed from Feedback to Resolved
As you reported that all existing templates on o3 and osd should be fixed and the change is now deployed to o3 and osd as well I consider this "Resolved".