action #95140: regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M - openQA Infrastructure - openSUSE Project Management Tool

Actions

Copy link

action #95140

closed

regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M

Added by okurz almost 3 years ago. Updated almost 3 years ago.

Status:

Resolved

Priority:

High

Assignee:

okurz

Category:

Target version:

openQA Project - Ready

Start date:

2021-07-06

Due date:

% Done:

Estimated time:

Description

Observation¶

https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/483455#L484 looks wrong:

ssh $TARGET "salt --hide-timeout \* state.highstate queue=True"
…
QA-Power8-4-kvm.qa.suse.de:
    Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list

        Retrieve the state data from the salt master for this minion and execute it

only the machine QA-Power8-4-kvm.qa.suse.de reported the problem, all other machines were just fine, e.g. just stated success or something.

Expected result¶

All salt nodes within the OSD infrastructure are able to apply a consistent and clean state again (as in before)

Further details¶

At least the version of salt-minion is the same on all nodes. Other nodes are fine and also QA-Power8-4-kvm.qa.suse.de was fine still some days ago:

sudo salt \* cmd.run 'salt-minion --version'
storage.qa.suse.de:
    salt-minion 3002.2
openqaworker2.suse.de:
    salt-minion 3002.2
QA-Power8-5-kvm.qa.suse.de:
    salt-minion 3002.2
openqaworker9.suse.de:
    salt-minion 3002.2
malbec.arch.suse.de:
    salt-minion 3002.2
QA-Power8-4-kvm.qa.suse.de:
    salt-minion 3002.2
openqaworker8.suse.de:
    salt-minion 3002.2
grenache-1.qa.suse.de:
    salt-minion 3002.2
backup.qa.suse.de:
    salt-minion 3002.2
openqa.suse.de:
    salt-minion 3002.2
powerqaworker-qam-1.qa.suse.de:
    salt-minion 3002.2
openqaworker10.suse.de:
    salt-minion 3002.2
openqaworker6.suse.de:
    salt-minion 3002.2
openqaworker3.suse.de:
    salt-minion 3002.2
openqaworker13.suse.de:
    salt-minion 3002.2
openqaworker5.suse.de:
    salt-minion 3002.2
openqaworker-arm-1.suse.de:
    salt-minion 3002.2
openqa-monitor.qa.suse.de:
    salt-minion 3002.2
openqaworker-arm-3.suse.de:
    salt-minion 3002.2
openqaworker-arm-2.suse.de:
    salt-minion 3002.2

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by okurz almost 3 years ago

Description updated (diff)

Actions

Copy link

Updated by okurz almost 3 years ago

Subject changed from salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list"
Description updated (diff)

Actions

Copy link

Updated by livdywan almost 3 years ago

Subject changed from regression: salt state fails one selected workers with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list"

Actions

Copy link

Updated by livdywan almost 3 years ago

Subject changed from regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" to regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M
Status changed from New to Workable

Actions

Copy link

Updated by okurz almost 3 years ago

Status changed from Workable to In Progress
Assignee set to okurz

I tried sudo salt 'QA-Power8-4-kvm*' state.apply queue=True from OSD and got no response in reasonable time. I found on the machine that at least a mount call was seemingly stuck in the process table. I triggered a reboot and afterwards the mentioned command finished within 47s. The machine was up for 17 days in before. I should check if automatic rebooting works.

EDIT: automatic reboot on the host is enabled and should be fine. Likely there were simply no updates for the host needing a reboot during that time and we had exceptional network problems. I guess it would have been a sane choice to trigger a reboot for all machines after we ensured that network is fine again :)

sudo salt \* state.apply queue=True returned just fine now. Will trigger some failed pipelines, e.g. salt states and pillars.

Actions

Copy link