action #132818
closedsalt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M
0%
Description
Observation¶
From https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1693089#L10505
----------
ID: /var/lib/openqa
Function: mount.mounted
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/salt/state.py", line 2401, in call
ret = self.states[cdata["full"]](
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1234, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1249, in _run_as
return _func_or_method(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1282, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/states/mount.py", line 774, in mounted
out = __salt__["mount.set_fstab"](
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1234, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1249, in _run_as
return _func_or_method(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/salt/modules/mount.py", line 877, in set_fstab
raise CommandExecutionError('Bad config file "{}"'.format(config))
salt.exceptions.CommandExecutionError: Bad config file "/etc/fstab"
Started: 10:13:09.444309
Duration: 35.476 ms
Changes:
Acceptance criteria¶
- AC1: No errors concerning file mounts in salt CI pipelines
Steps to reproduce¶
- Can be reproduced within the CI jobs, should be crosschecked how it's reproducible manually but it seems changes on production workers apply just fine
- Crosscheck in a container environment same as CI does
- DuckDuckGo for the error message
Updated by livdywan 9 months ago
- Subject changed from salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" to salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by livdywan 9 months ago
okurz wrote:
- Can be reproduced within the CI jobs, should be crosschecked how it's reproducible manually but it seems changes on production workers apply just fine
Indeed it can be reproduced in CI. I just saw it in another MR with unrelated changes.
Updated by livdywan 9 months ago
- Status changed from Workable to In Progress
- Assignee set to livdywan
Most likely the issue is coming from openqa/nvme_store/init.sls where /var/lib/openqa: is being mounted.
The way to verify this locally seems to be:
podman run --rm -it -v $PWD:/srv/salt -v $PWD/../salt-pillars:/srv/pillar registry.opensuse.org/home/okurz/container/containers/tumbleweed:salt-minion-git-core
echo -e "noservices: True\nroles: worker" >> /etc/salt/grains && salt-call -l debug --local state.apply
It's hard to tell if this works since I'm getting 16 failures and unsure if the one I'm looking for is or would be in there.
Updated by livdywan 9 months ago
- Status changed from In Progress to Workable
- Assignee deleted (
livdywan)
It's hard to tell if this works since I'm getting 16 failures and unsure if the one I'm looking for is or would be in there.
Using mock=True actually got rid of the errors. However I'm still not seeing the fstab errors. I tried adding openqa_store: True to the grains and even removing the conditionals in openqa/nvme_store/init.sls with no success.
echo -e "noservices: True\nopenqa_store: True\nroles: worker" >> /etc/salt/grains && salt-call -l debug --local state.apply mock=True 2>&1 | grep -C 5 nvm
DuckDuckGo for the error message
I didn't really find anything useful researching the error message. So no guess on a fix so far.
Updated by okurz 9 months ago
- Status changed from Feedback to Resolved
merged, https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1714235 shows that tests pass now