Project

General

Profile

Actions

action #132818

closed

salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M

Added by okurz 11 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
Start date:
2023-07-16
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

From https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1693089#L10505

----------
          ID: /var/lib/openqa
    Function: mount.mounted
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.11/site-packages/salt/state.py", line 2401, in call
                  ret = self.states[cdata["full"]](
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1234, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1249, in _run_as
                  return _func_or_method(*args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1282, in wrapper
                  return f(*args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/states/mount.py", line 774, in mounted
                  out = __salt__["mount.set_fstab"](
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1234, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1249, in _run_as
                  return _func_or_method(*args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/modules/mount.py", line 877, in set_fstab
                  raise CommandExecutionError('Bad config file "{}"'.format(config))
              salt.exceptions.CommandExecutionError: Bad config file "/etc/fstab"
     Started: 10:13:09.444309
    Duration: 35.476 ms
     Changes:   

Acceptance criteria

  • AC1: No errors concerning file mounts in salt CI pipelines

Steps to reproduce

  • Can be reproduced within the CI jobs, should be crosschecked how it's reproducible manually but it seems changes on production workers apply just fine
  • Crosscheck in a container environment same as CI does
  • DuckDuckGo for the error message
Actions #1

Updated by nicksinger 11 months ago

  • Assignee set to nicksinger
Actions #2

Updated by livdywan 11 months ago

  • Subject changed from salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" to salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #3

Updated by okurz 11 months ago

  • Assignee deleted (nicksinger)
Actions #4

Updated by mkittler 11 months ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions #5

Updated by mkittler 11 months ago

  • Status changed from In Progress to Workable
  • Assignee deleted (mkittler)
Actions #6

Updated by livdywan 11 months ago

okurz wrote:

  • Can be reproduced within the CI jobs, should be crosschecked how it's reproducible manually but it seems changes on production workers apply just fine

Indeed it can be reproduced in CI. I just saw it in another MR with unrelated changes.

Actions #7

Updated by livdywan 11 months ago

  • Status changed from Workable to In Progress
  • Assignee set to livdywan

Most likely the issue is coming from openqa/nvme_store/init.sls where /var/lib/openqa: is being mounted.

The way to verify this locally seems to be:

podman run --rm -it -v $PWD:/srv/salt -v $PWD/../salt-pillars:/srv/pillar registry.opensuse.org/home/okurz/container/containers/tumbleweed:salt-minion-git-core
echo -e "noservices: True\nroles: worker" >> /etc/salt/grains && salt-call -l debug --local state.apply

It's hard to tell if this works since I'm getting 16 failures and unsure if the one I'm looking for is or would be in there.

Actions #8

Updated by livdywan 11 months ago

  • Status changed from In Progress to Workable
  • Assignee deleted (livdywan)

It's hard to tell if this works since I'm getting 16 failures and unsure if the one I'm looking for is or would be in there.

Using mock=True actually got rid of the errors. However I'm still not seeing the fstab errors. I tried adding openqa_store: True to the grains and even removing the conditionals in openqa/nvme_store/init.sls with no success.

echo -e "noservices: True\nopenqa_store: True\nroles: worker" >> /etc/salt/grains && salt-call -l debug --local state.apply mock=True 2>&1 | grep -C 5 nvm

DuckDuckGo for the error message

I didn't really find anything useful researching the error message. So no guess on a fix so far.

Actions #9

Updated by okurz 11 months ago

  • Status changed from Workable to Feedback
  • Assignee set to okurz
Actions #10

Updated by okurz 11 months ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF