action #97418
closedPipeline in salt-states-openqa can fail occasionally size:M
0%
Description
Observation¶
The pipeline test-monitor in salt-states-openqa failed today. It isn't clear what the problem is, the log is quite extensive: https://gitlab.suse.de/mkittler/salt-states-openqa/-/jobs/540688
This job ran as part of https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/562 but re-triggering the job helped. So it is likely not related to the SR but another sporadically happening issue.
Seems like something in /etc/salt/minion, wrong YAML syntax or likely salt states with the same ID from different files:
.[ERROR ] Error parsing configuration file: /etc/salt/minion - while constructing a mapping
9868 in "/etc/salt/minion", line 936, column 1
9869found conflicting ID 'disable_grains'
9870 in "/etc/salt/minion", line 942, column 1
9871[ERROR ] Error parsing configuration file: /etc/salt/minion - while constructing a mapping
[...]
9890 raise ConstructorError(
9891yaml.constructor.ConstructorError: while constructing a mapping
9892 in "/etc/salt/minion", line 936, column 1
9893found conflicting ID 'disable_grains'
9894 in "/etc/salt/minion", line 942, column 1
Maybe a consequence of the other error?
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1778, in _load_module
module_init(self.opts)
File "/usr/lib/python3.8/site-packages/salt/modules/boto3_elasticsearch.py", line 87, in __init__
__utils__["boto3.assign_funcs"](__name__, "es")
File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1283, in __getitem__
func = super().__getitem__(item)
File "/usr/lib/python3.8/site-packages/salt/utils/lazy.py", line 108, in __getitem__
raise KeyError(key)
KeyError: 'boto3.assign_funcs'
Suggestions¶
- Stare at https://gitlab.suse.de/mkittler/salt-states-openqa/-/jobs/540688 for at least 10 minutes trying to find the actual error cause before giving up
- Investigate what could cause the YAML/id issues
- Confirm if this is really flaky (re-trigger a couple more times)
Updated by okurz over 2 years ago
- Subject changed from Pipeline in salt-states-openqa can fail occasionally to Pipeline in salt-states-openqa can fail occasionally size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by mkittler over 2 years ago
- Status changed from Workable to Feedback
- Assignee set to mkittler
I suppose the error message means that disable_grains occurs multiple times in /etc/salt/minion (so these occurrences are conflicting with each other). We're appending this setting in salt-states-openqa/salt/minion.sls using https://docs.saltproject.io/en/latest/ref/states/all/salt.states.file.html#salt.states.file.append. So it is possible that the section has been appended multiple times because the file has been edited manually (and therefore had a different "end"). I have never seen this kind of failure again and a re-try helped. I conclude this was a one-time problem.
We could still try to improve, e.g. it would make sense to process /etc/salt/minion as a YAML document instead of just trying to append some text. However, I haven't found a salt state for managing YAML (like https://docs.saltproject.io/en/latest/ref/states/all/salt.states.ini_manage.html for INI files). So I suppose it isn't worth it.
Updated by okurz over 2 years ago
You could try to use https://docs.saltproject.io/en/latest/ref/states/all/salt.states.file.html#salt.states.file.serialize with "merge_if_exists: True". If you consider this not worth the effort then leave it as is
Updated by mkittler over 2 years ago
I gave it a shot and it works but has also downsides, see https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/611.
Updated by mkittler over 2 years ago
- Status changed from Feedback to Resolved
The SR has been merged. Note that the comments in those files are currently still present and it looks like the SR didn't worked. However, I've observed the same when testing in a container before and it is normal that the file is not touched at all if there are no changes to be made anyways. So the comments will only be gone after the next actual change.