Project

General

Profile

Actions

action #95140

closed

regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M

Added by okurz over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2021-07-06
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/483455#L484 looks wrong:

ssh $TARGET "salt --hide-timeout \* state.highstate queue=True"
…
QA-Power8-4-kvm.qa.suse.de:
    Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list

        Retrieve the state data from the salt master for this minion and execute it

only the machine QA-Power8-4-kvm.qa.suse.de reported the problem, all other machines were just fine, e.g. just stated success or something.

Expected result

  • All salt nodes within the OSD infrastructure are able to apply a consistent and clean state again (as in before)

Further details

At least the version of salt-minion is the same on all nodes. Other nodes are fine and also QA-Power8-4-kvm.qa.suse.de was fine still some days ago:

sudo salt \* cmd.run 'salt-minion --version'
storage.qa.suse.de:
    salt-minion 3002.2
openqaworker2.suse.de:
    salt-minion 3002.2
QA-Power8-5-kvm.qa.suse.de:
    salt-minion 3002.2
openqaworker9.suse.de:
    salt-minion 3002.2
malbec.arch.suse.de:
    salt-minion 3002.2
QA-Power8-4-kvm.qa.suse.de:
    salt-minion 3002.2
openqaworker8.suse.de:
    salt-minion 3002.2
grenache-1.qa.suse.de:
    salt-minion 3002.2
backup.qa.suse.de:
    salt-minion 3002.2
openqa.suse.de:
    salt-minion 3002.2
powerqaworker-qam-1.qa.suse.de:
    salt-minion 3002.2
openqaworker10.suse.de:
    salt-minion 3002.2
openqaworker6.suse.de:
    salt-minion 3002.2
openqaworker3.suse.de:
    salt-minion 3002.2
openqaworker13.suse.de:
    salt-minion 3002.2
openqaworker5.suse.de:
    salt-minion 3002.2
openqaworker-arm-1.suse.de:
    salt-minion 3002.2
openqa-monitor.qa.suse.de:
    salt-minion 3002.2
openqaworker-arm-3.suse.de:
    salt-minion 3002.2
openqaworker-arm-2.suse.de:
    salt-minion 3002.2

Related issues 2 (0 open2 closed)

Related to openQA Infrastructure - action #133469: [alert] Salt states don't apply sometimes on individual workers size:MResolvednicksinger2023-07-27

Actions
Related to openQA Infrastructure - action #158023: salt-states-openqa pipeline invalid arguments to state.highstate on monitor.qe.nue2.suse.orgResolvedokurz2024-03-26

Actions
Actions #1

Updated by okurz over 3 years ago

  • Description updated (diff)
Actions #2

Updated by okurz over 3 years ago

  • Subject changed from salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list"
  • Description updated (diff)
Actions #3

Updated by livdywan over 3 years ago

  • Subject changed from regression: salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list"
Actions #4

Updated by livdywan over 3 years ago

  • Subject changed from regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M
  • Status changed from New to Workable
Actions #5

Updated by okurz over 3 years ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz

I tried sudo salt 'QA-Power8-4-kvm*' state.apply queue=True from OSD and got no response in reasonable time. I found on the machine that at least a mount call was seemingly stuck in the process table. I triggered a reboot and afterwards the mentioned command finished within 47s. The machine was up for 17 days in before. I should check if automatic rebooting works.

EDIT: automatic reboot on the host is enabled and should be fine. Likely there were simply no updates for the host needing a reboot during that time and we had exceptional network problems. I guess it would have been a sane choice to trigger a reboot for all machines after we ensured that network is fine again :)

sudo salt \* state.apply queue=True returned just fine now. Will trigger some failed pipelines, e.g. salt states and pillars.

Actions #7

Updated by okurz about 1 year ago

  • Related to action #133469: [alert] Salt states don't apply sometimes on individual workers size:M added
Actions #8

Updated by okurz 7 months ago

  • Related to action #158023: salt-states-openqa pipeline invalid arguments to state.highstate on monitor.qe.nue2.suse.org added
Actions

Also available in: Atom PDF