action #95140
closedregression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M
0%
Description
Observation¶
https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/483455#L484 looks wrong:
ssh $TARGET "salt --hide-timeout \* state.highstate queue=True"
…
QA-Power8-4-kvm.qa.suse.de:
Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list
Retrieve the state data from the salt master for this minion and execute it
only the machine QA-Power8-4-kvm.qa.suse.de reported the problem, all other machines were just fine, e.g. just stated success or something.
Expected result¶
- All salt nodes within the OSD infrastructure are able to apply a consistent and clean state again (as in before)
Further details¶
At least the version of salt-minion is the same on all nodes. Other nodes are fine and also QA-Power8-4-kvm.qa.suse.de was fine still some days ago:
sudo salt \* cmd.run 'salt-minion --version'
storage.qa.suse.de:
salt-minion 3002.2
openqaworker2.suse.de:
salt-minion 3002.2
QA-Power8-5-kvm.qa.suse.de:
salt-minion 3002.2
openqaworker9.suse.de:
salt-minion 3002.2
malbec.arch.suse.de:
salt-minion 3002.2
QA-Power8-4-kvm.qa.suse.de:
salt-minion 3002.2
openqaworker8.suse.de:
salt-minion 3002.2
grenache-1.qa.suse.de:
salt-minion 3002.2
backup.qa.suse.de:
salt-minion 3002.2
openqa.suse.de:
salt-minion 3002.2
powerqaworker-qam-1.qa.suse.de:
salt-minion 3002.2
openqaworker10.suse.de:
salt-minion 3002.2
openqaworker6.suse.de:
salt-minion 3002.2
openqaworker3.suse.de:
salt-minion 3002.2
openqaworker13.suse.de:
salt-minion 3002.2
openqaworker5.suse.de:
salt-minion 3002.2
openqaworker-arm-1.suse.de:
salt-minion 3002.2
openqa-monitor.qa.suse.de:
salt-minion 3002.2
openqaworker-arm-3.suse.de:
salt-minion 3002.2
openqaworker-arm-2.suse.de:
salt-minion 3002.2
Updated by okurz over 3 years ago
- Subject changed from salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list"
- Description updated (diff)
Updated by livdywan over 3 years ago
- Subject changed from regression: salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list"
Updated by livdywan over 3 years ago
- Subject changed from regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M
- Status changed from New to Workable
Updated by okurz over 3 years ago
- Status changed from Workable to In Progress
- Assignee set to okurz
I tried sudo salt 'QA-Power8-4-kvm*' state.apply queue=True
from OSD and got no response in reasonable time. I found on the machine that at least a mount call was seemingly stuck in the process table. I triggered a reboot and afterwards the mentioned command finished within 47s. The machine was up for 17 days in before. I should check if automatic rebooting works.
EDIT: automatic reboot on the host is enabled and should be fine. Likely there were simply no updates for the host needing a reboot during that time and we had exceptional network problems. I guess it would have been a sane choice to trigger a reboot for all machines after we ensured that network is fine again :)
sudo salt \* state.apply queue=True
returned just fine now. Will trigger some failed pipelines, e.g. salt states and pillars.
Updated by okurz over 3 years ago
- Status changed from In Progress to Resolved
Updated by okurz about 1 year ago
- Related to action #133469: [alert] Salt states don't apply sometimes on individual workers size:M added
Updated by okurz 7 months ago
- Related to action #158023: salt-states-openqa pipeline invalid arguments to state.highstate on monitor.qe.nue2.suse.org added