action #120921
[alert] Salt states fail to compile with "Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable" size:M
Start date:
2022-11-24
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Observation¶
All workers failed to apply salt states, e.g.:
…
worker2.oqa.suse.de:
Data failed to compile:
----------
Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 37, in top-level template code
TypeError: argument of type 'NoneType' is not iterable
; line 37
---
[...]
# Those hosts are identified by WORKER_CLASS value from pillar defined in "multihostclass" variable
# Only one class is supported and its name have to be unique - will match also the strings that contain it!
{% set multihostclass = 'tap' %}
{% set multihostworkers = [] %}
{% for host in pillar['workerconf'] %}
{% if 'global' in pillar['workerconf'][host] %} <======================
{% if multihostclass in pillar['workerconf'][host]['global']['WORKER_CLASS'] | default('undefined') %}
{% do multihostworkers.append(host) %}
{% endif %}
{% endif %}
# The class can be defined in both places (global X numbered workers) at the same time
[...]
---
openqaworker14.qa.suse.cz:
Data failed to compile:
----------
Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 37, in top-level template code
TypeError: argument of type 'NoneType' is not iterable
; line 37
---
[...]
# Those hosts are identified by WORKER_CLASS value from pillar defined in "multihostclass" variable
# Only one class is supported and its name have to be unique - will match also the strings that contain it!
{% set multihostclass = 'tap' %}
{% set multihostworkers = [] %}
{% for host in pillar['workerconf'] %}
{% if 'global' in pillar['workerconf'][host] %} <======================
{% if multihostclass in pillar['workerconf'][host]['global']['WORKER_CLASS'] | default('undefined') %}
{% do multihostworkers.append(host) %}
{% endif %}
{% endif %}
# The class can be defined in both places (global X numbered workers) at the same time
[...]
---
…
(see https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1255094/raw for full logs)
Additional context¶
The last change in that template was done for #120163. Then we ran into problems that were likely caused by the salt mine not being populated. However, this is now a different error and if the line the logs point to is correct than it now even fails before the modifications from #120163. So I've been creating this new ticket (instead of reopening #120163).
History
#1
Updated by mkittler 2 months ago
Updated by - Related to action #120163: Use salt grains instead of manually specifying IPs in "bridge_ip" size:M added
Updated by #3
Updated by cdywan 2 months ago
Updated by - Subject changed from [alert] Salt states fail to compile with "Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable" to [alert] Salt states fail to compile with "Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable" size:M
- Description updated (diff)
- Status changed from New to Workable
#4
Updated by mkittler 2 months ago
- Tags deleted (
alert) - Description updated (diff)
- Status changed from Workable to In Progress
- Assignee set to mkittler
- Target version deleted (
Ready)
The problem persists after retrying the pipeline and also shows when invoking state.apply manually on OSD (and I ran into "The function "state.highstate" is running as PID …" again). Maybe https://gitlab.suse.de/openqa/salt-pillars-openqa/-/commit/9f6245d521c5ba4145a485f802b64f09f9998dd2 or some other pillar-related MR caused the problem. I'll try to revert it "locally" on OSD and see if it changes anything.
#5
Updated by mkittler 2 months ago
It is that MR, indeed:
martchus@openqa:/srv/pillar> sudo git revert 9f6245d521c5ba4145a485f802b64f09f9998dd2 [revert-9f6245d521c5ba4145a485f802b64f09f9998dd2 f89df66] Revert "Add IPMI contact details for w16/17/18" 1 file changed, 12 deletions(-) martchus@openqa:/srv/pillar> sudo salt '*' saltutil.refresh_pillar … martchus@openqa:/srv/pillar> sudo salt -l error --state-output=changes \* state.apply … [everything looks good, including workers]
I'll create a MR to fix the syntax.
#7
Updated by mkittler 2 months ago
- Related to deleted (action #120163: Use salt grains instead of manually specifying IPs in "bridge_ip" size:M)