action #95140
closed
regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M
Added by okurz over 3 years ago.
Updated over 3 years ago.
Description
Observation¶
https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/483455#L484 looks wrong:
ssh $TARGET "salt --hide-timeout \* state.highstate queue=True"
…
QA-Power8-4-kvm.qa.suse.de:
Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list
Retrieve the state data from the salt master for this minion and execute it
only the machine QA-Power8-4-kvm.qa.suse.de reported the problem, all other machines were just fine, e.g. just stated success or something.
Expected result¶
- All salt nodes within the OSD infrastructure are able to apply a consistent and clean state again (as in before)
Further details¶
At least the version of salt-minion is the same on all nodes. Other nodes are fine and also QA-Power8-4-kvm.qa.suse.de was fine still some days ago:
sudo salt \* cmd.run 'salt-minion --version'
storage.qa.suse.de:
salt-minion 3002.2
openqaworker2.suse.de:
salt-minion 3002.2
QA-Power8-5-kvm.qa.suse.de:
salt-minion 3002.2
openqaworker9.suse.de:
salt-minion 3002.2
malbec.arch.suse.de:
salt-minion 3002.2
QA-Power8-4-kvm.qa.suse.de:
salt-minion 3002.2
openqaworker8.suse.de:
salt-minion 3002.2
grenache-1.qa.suse.de:
salt-minion 3002.2
backup.qa.suse.de:
salt-minion 3002.2
openqa.suse.de:
salt-minion 3002.2
powerqaworker-qam-1.qa.suse.de:
salt-minion 3002.2
openqaworker10.suse.de:
salt-minion 3002.2
openqaworker6.suse.de:
salt-minion 3002.2
openqaworker3.suse.de:
salt-minion 3002.2
openqaworker13.suse.de:
salt-minion 3002.2
openqaworker5.suse.de:
salt-minion 3002.2
openqaworker-arm-1.suse.de:
salt-minion 3002.2
openqa-monitor.qa.suse.de:
salt-minion 3002.2
openqaworker-arm-3.suse.de:
salt-minion 3002.2
openqaworker-arm-2.suse.de:
salt-minion 3002.2
- Description updated (diff)
- Subject changed from salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list"
- Description updated (diff)
- Subject changed from regression: salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list"
- Subject changed from regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M
- Status changed from New to Workable
- Status changed from Workable to In Progress
- Assignee set to okurz
I tried sudo salt 'QA-Power8-4-kvm*' state.apply queue=True
from OSD and got no response in reasonable time. I found on the machine that at least a mount call was seemingly stuck in the process table. I triggered a reboot and afterwards the mentioned command finished within 47s. The machine was up for 17 days in before. I should check if automatic rebooting works.
EDIT: automatic reboot on the host is enabled and should be fine. Likely there were simply no updates for the host needing a reboot during that time and we had exceptional network problems. I guess it would have been a sane choice to trigger a reboot for all machines after we ensured that network is fine again :)
sudo salt \* state.apply queue=True
returned just fine now. Will trigger some failed pipelines, e.g. salt states and pillars.
- Status changed from In Progress to Resolved
- Related to action #133469: [alert] Salt states don't apply sometimes on individual workers size:M added
- Related to action #158023: salt-states-openqa pipeline invalid arguments to state.highstate on monitor.qe.nue2.suse.org added
Also available in: Atom
PDF