Project

General

Profile

action #71251

Forbid duplicate mapping keys in Job Templates YAML files

Added by tinita 4 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Low
Assignee:
tinita
Category:
Feature requests
Target version:
Ready
Start date:
2020-09-11
Due date:
% Done:

0%

Estimated time:
Difficulty:
easy

Description

The YAML spec says YAML Loaders should forbid duplicate mapping keys.

Until now, YAML::Tiny is the only Perl module implementing that. Modues like YAML::XS, YAML.pm, YAML::Syck and also several implementations in other languages like PyYAML just ignore it and override the previous key.

But automatically detecting duplicate keys can be useful if the YAML mapping is so long it doesn't fit in your editor.

History

#1 Updated by tinita 4 months ago

YAML::PP 0.026 (which was released yesterday) adds an option duplicate_keys. The default is allowing duplicates for now.
The plan is to set the default to forbid in 0.027.

So the Todos here are:

  1. Add duplicate_keys to YAML::PP ✔️
  2. Wait until YAML::PP 0.026 is in our devel:openQA repo ✔️
  3. Test all existing job templates with duplicate_keys => 0 ✔️
  4. Test YAML files in os-autoinst-distri-opensuse ✔️
  5. Fix broken templates, if there are any ✔️
  6. Set YAML::PP >= 0.26 and duplicate_keys => 0 in our openQA code ✔️
  7. When 0.027 gets released, we can remove the duplicate_keys => 0

#2 Updated by tinita 4 months ago

Tested all templates.
There is one template with a duplicate key in o3 and four in osd.

Created a MR for one of the osd templates. The others don't seem to be maintained in git, so I will fix them in the webui.

#3 Updated by tinita 4 months ago

  • Description updated (diff)

#4 Updated by tinita 4 months ago

  • Description updated (diff)

#5 Updated by tinita 4 months ago

Fixed https://openqa.opensuse.org/admin/job_templates/66 and the other 3 osd templates.

#6 Updated by tinita 4 months ago

Code used for testing:

% ./script/openqa-cli api --osd job_templates_scheduling --json >osd-templates.json
% perl -wE'
use JSON::XS;
use YAML::PP;
use Data::Dumper;
my ($f) = @ARGV;
my $yp = YAML::PP->new( duplicate_keys => 0 );
my $json = do { open my $fh, "<", $f or die $!; local $/; <$fh> };
my $data = decode_json($json);
for my $key (sort keys %$data) {
  say "===== $key";
  my $yaml = $data->{ $key };
  my $t = eval { $yp->load_string($yaml); };
  if ($@) {
    say "ERROR($key): $@\n$yaml";
  }
}
' osd-templates.json

#7 Updated by tinita 4 months ago

All templates have been fixed now.

#8 Updated by tinita 4 months ago

  • Status changed from In Progress to Feedback

#9 Updated by okurz 4 months ago

  • Status changed from Feedback to Resolved

As you reported that all existing templates on o3 and osd should be fixed and the change is now deployed to o3 and osd as well I consider this "Resolved".

Also available in: Atom PDF