Project

General

Profile

Actions

action #133469

closed

[alert] Salt states don't apply sometimes on individual workers size:M

Added by mkittler 12 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2023-07-27
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

This lead to https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1716103/raw failing with:

ERROR: Minions returned with non-zero exit code

openqaworker-arm-3.suse.de:
    Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list
…

Trying to apply the states manually leads to the following error (which persisted after multiple attempts and waiting a few minutes between them):

martchus@openqa:~> sudo salt -C 'openqaworker-arm-3.suse.de' -l error --state-output=changes state.apply
openqaworker-arm-3.suse.de:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 5343 and was started at 2023, Jul 24 21:09:36.760016 with jid 20230724210936760016
----------
    The function "state.highstate" is running as PID 93318 and was started at 2023, Jul 27 14:59:24.641229 with jid 20230727145924641229
ERROR: Minions returned with non-zero exit code

Acceptance criteria

  • AC1: It is known what causes the problem
  • AC2: The problem is fixed or worked around as far as possible

Suggestions

  • We have seen this problem before on other workers (see #95140) so it would make sense to research more.
  • Research if the problem is discussed upstream
    • If nothing can be found ask directly in e.g. chats or the suse internal mailinglist salt@suse.de

Related issues 5 (1 open4 closed)

Related to openQA Infrastructure - action #95140: regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:MResolvedokurz2021-07-06

Actions
Related to openQA Infrastructure - action #133793: salt-pillars-openqa failing to apply within 2h and it is not clear which minion(s) are missing size:MResolvedokurz2023-08-04

Actions
Related to openQA Project - action #135134: [tools] GitlabCI salt-pillars-openqa deploy failed on baremetal-support.qa.suse.deResolvednicksinger2023-09-04

Actions
Related to openQA Infrastructure - action #134819: Errors in salt minion and master log on osdNew2023-08-30

Actions
Related to openQA Infrastructure - action #137744: [alert] gitlab pipeline fails in deploy step for salt-pillars-openqa: Data failed to compileRejectedokurz2023-10-11

Actions
Actions

Also available in: Atom PDF