action #132818: salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M - openQA Infrastructure (public) - openSUSE Project Management Tool

Actions

Copy link

action #132818

closed

salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:

Resolved

Priority:

Urgent

Assignee:

okurz

Category:

Target version:

openQA Project (public) - Ready

Start date:

2023-07-16

Due date:

% Done:

Estimated time:

Tags:

salt, infra

Description

Observation¶

From https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1693089#L10505

----------
          ID: /var/lib/openqa
    Function: mount.mounted
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.11/site-packages/salt/state.py", line 2401, in call
                  ret = self.states[cdata["full"]](
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1234, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1249, in _run_as
                  return _func_or_method(*args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1282, in wrapper
                  return f(*args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/states/mount.py", line 774, in mounted
                  out = __salt__["mount.set_fstab"](
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1234, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/loader/lazy.py", line 1249, in _run_as
                  return _func_or_method(*args, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/lib/python3.11/site-packages/salt/modules/mount.py", line 877, in set_fstab
                  raise CommandExecutionError('Bad config file "{}"'.format(config))
              salt.exceptions.CommandExecutionError: Bad config file "/etc/fstab"
     Started: 10:13:09.444309
    Duration: 35.476 ms
     Changes:

Acceptance criteria¶

AC1: No errors concerning file mounts in salt CI pipelines

Steps to reproduce¶

Can be reproduced within the CI jobs, should be crosschecked how it's reproducible manually but it seems changes on production workers apply just fine
Crosscheck in a container environment same as CI does
DuckDuckGo for the error message

Actions

Copy link

Updated by nicksinger over 1 year ago

Assignee set to nicksinger

Actions

Copy link

Updated by livdywan over 1 year ago

Subject changed from salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" to salt state for worker in CI test does not apply anymore, "ID: /var/lib/openqa, Function: mount.mounted" size:M
Description updated (diff)
Status changed from New to Workable

Actions

Copy link

Updated by okurz over 1 year ago

Assignee deleted (~~nicksinger~~)

Actions

Copy link

Updated by mkittler over 1 year ago

Status changed from Workable to In Progress
Assignee set to mkittler

Actions

Copy link

Updated by mkittler over 1 year ago

Status changed from In Progress to Workable
Assignee deleted (~~mkittler~~)

Actions

Copy link

Updated by livdywan over 1 year ago

okurz wrote:

Can be reproduced within the CI jobs, should be crosschecked how it's reproducible manually but it seems changes on production workers apply just fine

Indeed it can be reproduced in CI. I just saw it in another MR with unrelated changes.

Actions

Copy link

Updated by livdywan over 1 year ago

Status changed from Workable to In Progress
Assignee set to livdywan

Most likely the issue is coming from openqa/nvme_store/init.sls where /var/lib/openqa: is being mounted.

The way to verify this locally seems to be:

podman run --rm -it -v $PWD:/srv/salt -v $PWD/../salt-pillars:/srv/pillar registry.opensuse.org/home/okurz/container/containers/tumbleweed:salt-minion-git-core
echo -e "noservices: True\nroles: worker" >> /etc/salt/grains && salt-call -l debug --local state.apply

It's hard to tell if this works since I'm getting 16 failures and unsure if the one I'm looking for is or would be in there.

Actions

Copy link

Updated by livdywan over 1 year ago

Status changed from In Progress to Workable
Assignee deleted (~~livdywan~~)

It's hard to tell if this works since I'm getting 16 failures and unsure if the one I'm looking for is or would be in there.

Using mock=True actually got rid of the errors. However I'm still not seeing the fstab errors. I tried adding openqa_store: True to the grains and even removing the conditionals in openqa/nvme_store/init.sls with no success.

echo -e "noservices: True\nopenqa_store: True\nroles: worker" >> /etc/salt/grains && salt-call -l debug --local state.apply mock=True 2>&1 | grep -C 5 nvm