Project

General

Profile

Actions

action #71251

closed

Forbid duplicate mapping keys in Job Templates YAML files

Added by tinita over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-09-11
Due date:
% Done:

0%

Estimated time:

Description

The YAML spec says YAML Loaders should forbid duplicate mapping keys.

Until now, YAML::Tiny is the only Perl module implementing that. Modues like YAML::XS, YAML.pm, YAML::Syck and also several implementations in other languages like PyYAML just ignore it and override the previous key.

But automatically detecting duplicate keys can be useful if the YAML mapping is so long it doesn't fit in your editor.

Actions #1

Updated by tinita over 4 years ago

YAML::PP 0.026 (which was released yesterday) adds an option duplicate_keys. The default is allowing duplicates for now.
The plan is to set the default to forbid in 0.027.

So the Todos here are:

  1. Add duplicate_keys to YAML::PP ✔️
  2. Wait until YAML::PP 0.026 is in our devel:openQA repo ✔️
  3. Test all existing job templates with duplicate_keys => 0 ✔️
  4. Test YAML files in os-autoinst-distri-opensuse ✔️
  5. Fix broken templates, if there are any ✔️
  6. Set YAML::PP >= 0.26 and duplicate_keys => 0 in our openQA code ✔️
  7. When 0.027 gets released, we can remove the duplicate_keys => 0
Actions #2

Updated by tinita over 4 years ago

Tested all templates.
There is one template with a duplicate key in o3 and four in osd.

Created a MR for one of the osd templates. The others don't seem to be maintained in git, so I will fix them in the webui.

Actions #3

Updated by tinita over 4 years ago

  • Description updated (diff)
Actions #4

Updated by tinita over 4 years ago

  • Description updated (diff)
Actions #5

Updated by tinita over 4 years ago

Fixed https://openqa.opensuse.org/admin/job_templates/66 and the other 3 osd templates.

Actions #6

Updated by tinita over 4 years ago

Code used for testing:

% ./script/openqa-cli api --osd job_templates_scheduling --json >osd-templates.json
% perl -wE'
use JSON::XS;
use YAML::PP;
use Data::Dumper;
my ($f) = @ARGV;
my $yp = YAML::PP->new( duplicate_keys => 0 );
my $json = do { open my $fh, "<", $f or die $!; local $/; <$fh> };
my $data = decode_json($json);
for my $key (sort keys %$data) {
  say "===== $key";
  my $yaml = $data->{ $key };
  my $t = eval { $yp->load_string($yaml); };
  if ($@) {
    say "ERROR($key): $@\n$yaml";
  }
}
' osd-templates.json
Actions #7

Updated by tinita over 4 years ago

All templates have been fixed now.

Actions #8

Updated by tinita over 4 years ago

  • Status changed from In Progress to Feedback
Actions #9

Updated by okurz over 4 years ago

  • Status changed from Feedback to Resolved

As you reported that all existing templates on o3 and osd should be fixed and the change is now deployed to o3 and osd as well I consider this "Resolved".

Actions

Also available in: Atom PDF