Project

General

Profile

Actions

action #97418

closed

Pipeline in salt-states-openqa can fail occasionally size:M

Added by mkittler about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2021-08-23
Due date:
% Done:

0%

Estimated time:

Description

Observation

The pipeline test-monitor in salt-states-openqa failed today. It isn't clear what the problem is, the log is quite extensive: https://gitlab.suse.de/mkittler/salt-states-openqa/-/jobs/540688

This job ran as part of https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/562 but re-triggering the job helped. So it is likely not related to the SR but another sporadically happening issue.

Seems like something in /etc/salt/minion, wrong YAML syntax or likely salt states with the same ID from different files:

.[ERROR   ] Error parsing configuration file: /etc/salt/minion - while constructing a mapping
9868  in "/etc/salt/minion", line 936, column 1
9869found conflicting ID 'disable_grains'
9870  in "/etc/salt/minion", line 942, column 1
9871[ERROR   ] Error parsing configuration file: /etc/salt/minion - while constructing a mapping
[...]
9890    raise ConstructorError(
9891yaml.constructor.ConstructorError: while constructing a mapping
9892  in "/etc/salt/minion", line 936, column 1
9893found conflicting ID 'disable_grains'
9894  in "/etc/salt/minion", line 942, column 1

Maybe a consequence of the other error?

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1778, in _load_module
    module_init(self.opts)
  File "/usr/lib/python3.8/site-packages/salt/modules/boto3_elasticsearch.py", line 87, in __init__
    __utils__["boto3.assign_funcs"](__name__, "es")
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1283, in __getitem__
    func = super().__getitem__(item)
  File "/usr/lib/python3.8/site-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'boto3.assign_funcs'

Suggestions

Actions #1

Updated by okurz about 3 years ago

  • Target version set to Ready
Actions #2

Updated by okurz about 3 years ago

  • Subject changed from Pipeline in salt-states-openqa can fail occasionally to Pipeline in salt-states-openqa can fail occasionally size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #3

Updated by livdywan almost 3 years ago

  • Description updated (diff)
Actions #4

Updated by livdywan almost 3 years ago

  • Description updated (diff)
Actions #5

Updated by mkittler almost 3 years ago

  • Status changed from Workable to Feedback
  • Assignee set to mkittler

I suppose the error message means that disable_grains occurs multiple times in /etc/salt/minion (so these occurrences are conflicting with each other). We're appending this setting in salt-states-openqa/salt/minion.sls using https://docs.saltproject.io/en/latest/ref/states/all/salt.states.file.html#salt.states.file.append. So it is possible that the section has been appended multiple times because the file has been edited manually (and therefore had a different "end"). I have never seen this kind of failure again and a re-try helped. I conclude this was a one-time problem.

We could still try to improve, e.g. it would make sense to process /etc/salt/minion as a YAML document instead of just trying to append some text. However, I haven't found a salt state for managing YAML (like https://docs.saltproject.io/en/latest/ref/states/all/salt.states.ini_manage.html for INI files). So I suppose it isn't worth it.

Actions #6

Updated by okurz almost 3 years ago

You could try to use https://docs.saltproject.io/en/latest/ref/states/all/salt.states.file.html#salt.states.file.serialize with "merge_if_exists: True". If you consider this not worth the effort then leave it as is

Actions #7

Updated by mkittler almost 3 years ago

I gave it a shot and it works but has also downsides, see https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/611.

Actions #8

Updated by mkittler almost 3 years ago

  • Status changed from Feedback to Resolved

The SR has been merged. Note that the comments in those files are currently still present and it looks like the SR didn't worked. However, I've observed the same when testing in a container before and it is normal that the file is not touched at all if there are no changes to be made anyways. So the comments will only be gone after the next actual change.

Actions

Also available in: Atom PDF