Project

General

Profile

action #120921

[alert] Salt states fail to compile with "Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable" size:M

Added by mkittler 2 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2022-11-24
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

All workers failed to apply salt states, e.g.:

…
worker2.oqa.suse.de:
    Data failed to compile:
----------
    Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 37, in top-level template code
TypeError: argument of type 'NoneType' is not iterable

; line 37

---
[...]
# Those hosts are identified by WORKER_CLASS value from pillar defined in "multihostclass" variable
# Only one class is supported and its name have to be unique - will match also the strings that contain it!
{% set multihostclass = 'tap' %}
{% set multihostworkers = [] %}
{% for host in pillar['workerconf'] %}
{%   if 'global' in pillar['workerconf'][host] %}    <======================
{%     if multihostclass in pillar['workerconf'][host]['global']['WORKER_CLASS'] | default('undefined') %}
{%       do multihostworkers.append(host) %}
{%     endif %}
{%   endif %}
# The class can be defined in both places (global X numbered workers) at the same time
[...]
---
openqaworker14.qa.suse.cz:
    Data failed to compile:
----------
    Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 37, in top-level template code
TypeError: argument of type 'NoneType' is not iterable

; line 37

---
[...]
# Those hosts are identified by WORKER_CLASS value from pillar defined in "multihostclass" variable
# Only one class is supported and its name have to be unique - will match also the strings that contain it!
{% set multihostclass = 'tap' %}
{% set multihostworkers = [] %}
{% for host in pillar['workerconf'] %}
{%   if 'global' in pillar['workerconf'][host] %}    <======================
{%     if multihostclass in pillar['workerconf'][host]['global']['WORKER_CLASS'] | default('undefined') %}
{%       do multihostworkers.append(host) %}
{%     endif %}
{%   endif %}
# The class can be defined in both places (global X numbered workers) at the same time
[...]
---
…

(see https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1255094/raw for full logs)

Additional context

The last change in that template was done for #120163. Then we ran into problems that were likely caused by the salt mine not being populated. However, this is now a different error and if the line the logs point to is correct than it now even fails before the modifications from #120163. So I've been creating this new ticket (instead of reopening #120163).

History

#1 Updated by mkittler 2 months ago

  • Related to action #120163: Use salt grains instead of manually specifying IPs in "bridge_ip" size:M added

#2 Updated by okurz 2 months ago

  • Target version set to Ready

for alert reaction tickets like those you can add to "Ready" directly yourself

#3 Updated by cdywan 2 months ago

  • Subject changed from [alert] Salt states fail to compile with "Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable" to [alert] Salt states fail to compile with "Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: argument of type 'NoneType' is not iterable" size:M
  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by mkittler 2 months ago

  • Tags deleted (alert)
  • Description updated (diff)
  • Status changed from Workable to In Progress
  • Assignee set to mkittler
  • Target version deleted (Ready)

The problem persists after retrying the pipeline and also shows when invoking state.apply manually on OSD (and I ran into "The function "state.highstate" is running as PID …" again). Maybe https://gitlab.suse.de/openqa/salt-pillars-openqa/-/commit/9f6245d521c5ba4145a485f802b64f09f9998dd2 or some other pillar-related MR caused the problem. I'll try to revert it "locally" on OSD and see if it changes anything.

#5 Updated by mkittler 2 months ago

It is that MR, indeed:

martchus@openqa:/srv/pillar> sudo git revert 9f6245d521c5ba4145a485f802b64f09f9998dd2
[revert-9f6245d521c5ba4145a485f802b64f09f9998dd2 f89df66] Revert "Add IPMI contact details for w16/17/18"
 1 file changed, 12 deletions(-)
martchus@openqa:/srv/pillar> sudo salt '*' saltutil.refresh_pillar
…
martchus@openqa:/srv/pillar> sudo salt -l error --state-output=changes \* state.apply
…
[everything looks good, including workers]

I'll create a MR to fix the syntax.

#6 Updated by mkittler 2 months ago

  • Status changed from In Progress to Feedback

#7 Updated by mkittler 2 months ago

  • Related to deleted (action #120163: Use salt grains instead of manually specifying IPs in "bridge_ip" size:M)

#8 Updated by mkittler 2 months ago

  • Tags set to alert
  • Target version set to Ready

#9 Updated by mkittler 2 months ago

  • Status changed from Feedback to Resolved

The SR has been merged. I've already checked it in production before so I'm confident it works. Also haven't seen any pipeline failures anymore.

Also available in: Atom PDF