action #174985
Updated by livdywan about 1 month ago
## Observation job: job1: https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/3609342 job2: https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/3609258 ## Further details **job** error: **job1** have only one error `ERROR: Minions returned with non-zero exit code`, but it seems that is not related to the line where it is printed. Also [this job](https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/3443316#L430) have the same error, but different position in the log. **job2** is more explicit: ``` monitor.qe.nue2.suse.org: The minion function caused an exception: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1912, in _thread_return function_name, function_args, executors, opts, data File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1870, in _execute_job_function return_data = self.executors[fname](opts, data, func, args, kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__ return self.loader.run(run_func, *args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1234, in run return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs) File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run return callable(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in _run_as ret = _func_or_method(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__ return self.loader.run(run_func, *args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1234, in run return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs) File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run return callable(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in _run_as ret = _func_or_method(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 1161, in highstate initial_pillar=_get_initial_pillar(opts), File "/usr/lib/python3.6/site-packages/salt/state.py", line 4953, in __init__ initial_pillar=initial_pillar, File "/usr/lib/python3.6/site-packages/salt/state.py", line 774, in __init__ self.opts["pillar"] = self._gather_pillar() File "/usr/lib/python3.6/site-packages/salt/state.py", line 877, in _gather_pillar return pillar.compile_pillar() File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 360, in compile_pillar dictkey="pillar", File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 112, in wrap lambda: getattr(self.obj, key)(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync return future_cell[0].result() File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "<string>", line 4, in raise_exc_info File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run yielded = self.gen.throw(*exc_info) File "/usr/lib/python3.6/site-packages/salt/channel/client.py", line 172, in crypted_transfer_decode_dictentry timeout=timeout, File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run value = future.result() File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "<string>", line 4, in raise_exc_info File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run yielded = self.gen.throw(*exc_info) File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 920, in send ret = yield self.message_client.send(load, timeout=timeout) File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run value = future.result() File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "<string>", line 4, in raise_exc_info File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run yielded = self.gen.throw(*exc_info) File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 630, in send recv = yield future File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run value = future.result() File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "<string>", line 4, in raise_exc_info salt.exceptions.SaltReqTimeoutError: Message timed out ``` ## Suggestions * Confirm if this is related to any recent work or known errors * look into system load of monitor https://stats.openqa-monitor.qa.suse.de/d/GDmonitor/dashboard-for-monitor?from=now-1h&to=now&timezone=browser&var-datasource=000000001&refresh=1m * Possibly bump either VM parameters or salt req timeout (check that gitlab job timeout matches salt)?