QA » openQA Project » openQA Infrastructure

Trying to reproduce with

for i in {1..100}; do echo "TRY: $i" && sudo salt -l error --state-output=changes '*arm*3*' state.apply,test.ping openqa.ntp, || break; done

which does not reproduce it. Neither does

for i in {1..100}; do echo "TRY: $i" && sudo salt -l error --state-output=changes '*arm*3*' state.apply,state.apply,test.ping salt.fix,salt.minion, || break; done

but

 for i in {1..100}; do echo "TRY: $i" && sudo salt -l error --state-output=changes '*arm*3*' test.ping,state.apply,test.ping || break; done

immediately shows a problem.

on the machine in /var/log/salt/minion :

2019-10-31 16:38:42,580 [salt.minion      :1899][WARNING ][43915] The minion function caused an exception: expected str, bytes or os.PathLike object, not list
2019-10-31 16:38:44,213 [salt.minion      :1899][WARNING ][43920] The minion function caused an exception: expected str, bytes or os.PathLike object, not list
2019-10-31 16:38:45,760 [salt.minion      :1899][WARNING ][43925] The minion function caused an exception: expected str, bytes or os.PathLike object, not list
2019-10-31 16:38:47,411 [salt.minion      :1899][WARNING ][43930] The minion function caused an exception: expected str, bytes or os.PathLike object, not list
2019-10-31 16:38:49,003 [salt.minion      :1899][WARNING ][43935] The minion function caused an exception: expected str, bytes or os.PathLike object, not list
2019-10-31 16:39:06,981 [salt.state       :1011][ERROR   ][43946] Error encountered during module reload. Modules were not reloaded.

I attached to the minion process(es) on arm3 with strace and could see that while the connection from master to minion times out the salt minion is (simply) still busy applying the state. Calling salt with parameter -t 180 for a longer timeout should help here.

https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/217