Project

General

Profile

Actions

action #133469

closed

[alert] Salt states don't apply sometimes on individual workers size:M

Added by mkittler 11 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2023-07-27
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

This lead to https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1716103/raw failing with:

ERROR: Minions returned with non-zero exit code

openqaworker-arm-3.suse.de:
    Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list
…

Trying to apply the states manually leads to the following error (which persisted after multiple attempts and waiting a few minutes between them):

martchus@openqa:~> sudo salt -C 'openqaworker-arm-3.suse.de' -l error --state-output=changes state.apply
openqaworker-arm-3.suse.de:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 5343 and was started at 2023, Jul 24 21:09:36.760016 with jid 20230724210936760016
----------
    The function "state.highstate" is running as PID 93318 and was started at 2023, Jul 27 14:59:24.641229 with jid 20230727145924641229
ERROR: Minions returned with non-zero exit code

Acceptance criteria

  • AC1: It is known what causes the problem
  • AC2: The problem is fixed or worked around as far as possible

Suggestions

  • We have seen this problem before on other workers (see #95140) so it would make sense to research more.
  • Research if the problem is discussed upstream
    • If nothing can be found ask directly in e.g. chats or the suse internal mailinglist salt@suse.de

Related issues 5 (1 open4 closed)

Related to openQA Infrastructure - action #95140: regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:MResolvedokurz2021-07-06

Actions
Related to openQA Infrastructure - action #133793: salt-pillars-openqa failing to apply within 2h and it is not clear which minion(s) are missing size:MResolvedokurz2023-08-04

Actions
Related to openQA Project - action #135134: [tools] GitlabCI salt-pillars-openqa deploy failed on baremetal-support.qa.suse.deResolvednicksinger2023-09-04

Actions
Related to openQA Infrastructure - action #134819: Errors in salt minion and master log on osdNew2023-08-30

Actions
Related to openQA Infrastructure - action #137744: [alert] gitlab pipeline fails in deploy step for salt-pillars-openqa: Data failed to compileRejectedokurz2023-10-11

Actions
Actions #1

Updated by okurz 11 months ago

We have seen this before but not in recent weeks. I don't know if this is a sporadic error. Maybe salt minion logs will tell more details

Actions #2

Updated by okurz 11 months ago

  • Tags changed from alert to alert, infra
Actions #3

Updated by okurz 11 months ago

  • Related to action #95140: regression: salt state fails on one worker with "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" size:M added
Actions #4

Updated by mkittler 11 months ago

  • Subject changed from [alert] Salt states don't apply on openqaworker-arm-3 to [alert] Salt states don't apply sometimes on individual workers size:M
  • Description updated (diff)
  • Status changed from New to Workable

The issue has fixed itself, currently only the following state is failing:

          ID: /var/lib/openqa/share
    Function: mount.mounted
      Result: False
     Comment: The following requisites were not found:
                                 require:
                                     sls: static_nfs_hostname
     Started: 13:18:53.824692
    Duration: 0.04 ms

So this is indeed just like #95140.

Actions #5

Updated by okurz 11 months ago

latest occurence https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1734165/raw on worker35.oqa.prg2.suse.org.

Actions #6

Updated by okurz 11 months ago

  • Related to action #133793: salt-pillars-openqa failing to apply within 2h and it is not clear which minion(s) are missing size:M added
Actions #7

Updated by nicksinger 10 months ago

  • Assignee set to nicksinger
Actions #8

Updated by nicksinger 10 months ago

  • Status changed from Workable to Feedback

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/964 to introduce some more insights on what wrong parameters are passed

Actions #10

Updated by okurz 10 months ago

  • Due date set to 2023-09-14
Actions #11

Updated by okurz 9 months ago

  • Related to action #135134: [tools] GitlabCI salt-pillars-openqa deploy failed on baremetal-support.qa.suse.de added
Actions #12

Updated by nicksinger 9 months ago

  • Priority changed from High to Normal

Checked after the weekend and saw nothing pointing to issues (we also didn't see any pipeline issues so nothing unexpected). This is how it looks atm:

openqa:~ # salt --state-output=changes --state-verbose=false '*' cmd.run 'salt-minion --version && journalctl --since="1 week ago" -u salt-minion | grep "args:" | grep highstate'
backup-qam.qe.nue2.suse.org:
    salt-minion 3004
storage.oqa.suse.de:
    salt-minion 3004
    Aug 31 22:45:07 storage salt-minion[31462]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:29:50 storage salt-minion[32601]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 storage salt-minion[3520]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:02 storage salt-minion[4386]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:36:47 storage salt-minion[10946]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:33:42 storage salt-minion[14438]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:27 storage salt-minion[22517]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:52 storage salt-minion[29599]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:33 storage salt-minion[1077]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:43 storage salt-minion[10950]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 storage salt-minion[17112]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 storage salt-minion[3847]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 storage salt-minion[4263]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker36.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:36 worker36 salt-minion[50861]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:57 worker36 salt-minion[55448]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:40 worker36 salt-minion[57933]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:49 worker36 salt-minion[68480]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker36 salt-minion[76789]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker36 salt-minion[35486]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker36 salt-minion[35789]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker30.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:36 worker30 salt-minion[51618]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:59 worker30 salt-minion[56239]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:41 worker30 salt-minion[58742]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:50 worker30 salt-minion[69982]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker30 salt-minion[76696]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker30 salt-minion[45192]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker30 salt-minion[45399]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker35.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:36 worker35 salt-minion[48992]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:57 worker35 salt-minion[53582]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:39 worker35 salt-minion[56069]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:47 worker35 salt-minion[64173]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker35 salt-minion[69744]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker35 salt-minion[15017]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker35 salt-minion[15816]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker39.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:35 worker39 salt-minion[63508]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:58 worker39 salt-minion[68132]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:39 worker39 salt-minion[70616]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:50 worker39 salt-minion[82868]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker39 salt-minion[93740]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker39 salt-minion[95321]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker39 salt-minion[95636]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker33.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:35 worker33 salt-minion[58622]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:57 worker33 salt-minion[63208]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:39 worker33 salt-minion[65692]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:49 worker33 salt-minion[77119]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker33 salt-minion[83474]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker33 salt-minion[40105]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker33 salt-minion[40418]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker29.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:36 worker29 salt-minion[51998]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:59 worker29 salt-minion[56644]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:41 worker29 salt-minion[59149]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:51 worker29 salt-minion[71191]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker29 salt-minion[82745]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker29 salt-minion[56550]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker29 salt-minion[57406]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker34.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:35 worker34 salt-minion[23553]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:57 worker34 salt-minion[28156]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:40 worker34 salt-minion[30642]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:48 worker34 salt-minion[46338]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker34 salt-minion[58515]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker34 salt-minion[28408]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker34 salt-minion[28782]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker31.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:36 worker31 salt-minion[131071]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:59 worker31 salt-minion[5906]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:41 worker31 salt-minion[8406]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:49 worker31 salt-minion[21355]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker31 salt-minion[32369]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker31 salt-minion[32691]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker31 salt-minion[33085]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqaworker16.qa.suse.cz:
    salt-minion 3004
    Sep 04 11:22:38 openqaworker16 salt-minion[62989]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:01 openqaworker16 salt-minion[1214]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:43 openqaworker16 salt-minion[3200]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:54 openqaworker16 salt-minion[10693]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 openqaworker16 salt-minion[19219]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 openqaworker16 salt-minion[63122]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 openqaworker16 salt-minion[63751]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker37.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:36 worker37 salt-minion[41794]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:57 worker37 salt-minion[46398]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:39 worker37 salt-minion[48887]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:47 worker37 salt-minion[63519]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker37 salt-minion[73962]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker37 salt-minion[65720]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker37 salt-minion[66016]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqaworker17.qa.suse.cz:
    salt-minion 3004
    Sep 04 11:22:38 openqaworker17 salt-minion[49431]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:01 openqaworker17 salt-minion[52664]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:43 openqaworker17 salt-minion[54367]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:52 openqaworker17 salt-minion[60774]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 openqaworker17 salt-minion[1452]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 openqaworker17 salt-minion[59504]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 openqaworker17 salt-minion[61433]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker32.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:36 worker32 salt-minion[52877]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:58 worker32 salt-minion[57510]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:41 worker32 salt-minion[60017]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:50 worker32 salt-minion[71812]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker32 salt-minion[83148]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker32 salt-minion[53497]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker32 salt-minion[53629]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker38.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:35 worker38 salt-minion[22796]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:58 worker38 salt-minion[27384]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:40 worker38 salt-minion[29869]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:50 worker38 salt-minion[46462]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker38 salt-minion[57763]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker38 salt-minion[24556]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker38 salt-minion[25509]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqaworker18.qa.suse.cz:
    salt-minion 3004
    Sep 04 11:22:36 openqaworker18 salt-minion[51778]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:59 openqaworker18 salt-minion[55035]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:40 openqaworker18 salt-minion[56735]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:50 openqaworker18 salt-minion[64260]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 openqaworker18 salt-minion[6658]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 openqaworker18 salt-minion[49333]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 openqaworker18 salt-minion[49840]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqaworker14.qa.suse.cz:
    salt-minion 3004
    Sep 04 11:22:43 openqaworker14 salt-minion[77412]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:09 openqaworker14 salt-minion[80571]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:50 openqaworker14 salt-minion[875]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:00 openqaworker14 salt-minion[6841]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 openqaworker14 salt-minion[12424]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 openqaworker14 salt-minion[53987]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 openqaworker14 salt-minion[54054]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker40.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:35 worker40 salt-minion[68369]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:57 worker40 salt-minion[72951]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:40 worker40 salt-minion[75435]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:50 worker40 salt-minion[86024]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker40 salt-minion[94446]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker40 salt-minion[63649]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker40 salt-minion[63787]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
qamasternue.qa.suse.de:
    salt-minion 3004
    Sep 01 10:30:05 qamaster salt-minion[13259]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 qamaster salt-minion[16751]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:17 qamaster salt-minion[17492]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:08 qamaster salt-minion[26238]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:33:59 qamaster salt-minion[31763]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:41 qamaster salt-minion[25262]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:11 qamaster salt-minion[32674]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:48 qamaster salt-minion[4352]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:20 qamaster salt-minion[14736]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:38 qamaster salt-minion[21213]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:06 qamaster salt-minion[11984]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 qamaster salt-minion[12414]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker9.oqa.suse.de:
    salt-minion 3004
    Sep 04 11:22:40 worker9 salt-minion[45236]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:03 worker9 salt-minion[49423]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:48 worker9 salt-minion[51703]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:58 worker9 salt-minion[60986]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker9 salt-minion[4955]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker9 salt-minion[61365]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker9 salt-minion[61713]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker8.oqa.suse.de:
    salt-minion 3004
    Sep 04 11:22:39 worker8 salt-minion[2263]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:05 worker8 salt-minion[7015]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:47 worker8 salt-minion[9531]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:59 worker8 salt-minion[16969]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker8 salt-minion[22772]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker8 salt-minion[10731]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker8 salt-minion[11140]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker-arm1.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:31 worker-arm1 salt-minion[19337]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:56 worker-arm1 salt-minion[24068]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:38 worker-arm1 salt-minion[26648]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:51 worker-arm1 salt-minion[47318]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker-arm1 salt-minion[71442]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker-arm1 salt-minion[52922]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker-arm1 salt-minion[53207]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker-arm2.oqa.prg2.suse.org:
    salt-minion 3004
    Sep 04 11:22:35 worker-arm2 salt-minion[16114]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:57 worker-arm2 salt-minion[20923]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:39 worker-arm2 salt-minion[23475]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:50 worker-arm2 salt-minion[41931]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 worker-arm2 salt-minion[67743]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker-arm2 salt-minion[39359]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker-arm2 salt-minion[39588]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqa-monitor.qa.suse.de:
    salt-minion 3004
    Sep 04 11:25:08 monitor salt-minion[13598]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:51:09 monitor salt-minion[20683]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:48 monitor salt-minion[24558]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:49:27 monitor salt-minion[1701]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:58 monitor salt-minion[7430]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 monitor salt-minion[17638]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:11 monitor salt-minion[17822]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
qesapworker-prg6.qa.suse.cz:
    salt-minion 3004
    Sep 04 11:22:38 qesapworker-prg6 salt-minion[23250]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:02 qesapworker-prg6 salt-minion[26535]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:47 qesapworker-prg6 salt-minion[28259]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:58 qesapworker-prg6 salt-minion[32552]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 qesapworker-prg6 salt-minion[35020]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 qesapworker-prg6 salt-minion[55362]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 qesapworker-prg6 salt-minion[55430]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
sapworker3.qe.nue2.suse.org:
    salt-minion 3004
    Aug 31 11:21:50 sapworker3 salt-minion[21770]: [WARNING ] state.highstate args: [{'queue': True}]
    Aug 31 11:22:28 sapworker3 salt-minion[22676]: [WARNING ] state.highstate args: [{'queue': True}]
    Aug 31 13:22:56 sapworker3 salt-minion[36312]: [WARNING ] state.highstate args: [{'queue': True}]
    Aug 31 16:10:02 sapworker3 salt-minion[69251]: [WARNING ] state.highstate args: [{'queue': True}]
    Aug 31 20:58:44 sapworker3 salt-minion[97406]: [WARNING ] state.highstate args: [{'queue': True}]
    Aug 31 22:41:58 sapworker3 salt-minion[105964]: [WARNING ] state.highstate args: [{'queue': True}]
    Aug 31 22:45:13 sapworker3 salt-minion[107116]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:29:52 sapworker3 salt-minion[121943]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 sapworker3 salt-minion[123595]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:06 sapworker3 salt-minion[123856]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:36:52 sapworker3 salt-minion[16268]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:33:44 sapworker3 salt-minion[43467]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:37 sapworker3 salt-minion[29435]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:58 sapworker3 salt-minion[32707]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:39 sapworker3 salt-minion[34423]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:47 sapworker3 salt-minion[41492]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 sapworker3 salt-minion[47688]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 sapworker3 salt-minion[86277]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 sapworker3 salt-minion[86416]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
qesapworker-prg5.qa.suse.cz:
    salt-minion 3004
    Sep 04 11:22:40 qesapworker-prg5 salt-minion[23127]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:03 qesapworker-prg5 salt-minion[26406]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:43 qesapworker-prg5 salt-minion[28129]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:55 qesapworker-prg5 salt-minion[32397]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 qesapworker-prg5 salt-minion[34875]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 qesapworker-prg5 salt-minion[55161]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 qesapworker-prg5 salt-minion[55249]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
qesapworker-prg4.qa.suse.cz:
    salt-minion 3004
    Sep 04 11:22:40 qesapworker-prg4 salt-minion[24716]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:02 qesapworker-prg4 salt-minion[28085]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:48 qesapworker-prg4 salt-minion[29859]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:56 qesapworker-prg4 salt-minion[34238]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 qesapworker-prg4 salt-minion[36757]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 qesapworker-prg4 salt-minion[52971]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 qesapworker-prg4 salt-minion[53040]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
sapworker1.qe.nue2.suse.org:
    salt-minion 3004
    Sep 04 11:22:29 sapworker1 salt-minion[41078]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:57 sapworker1 salt-minion[44274]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:38 sapworker1 salt-minion[45943]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:48 sapworker1 salt-minion[53580]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 sapworker1 salt-minion[61446]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 sapworker1 salt-minion[4376]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 sapworker1 salt-minion[4530]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
qesapworker-prg7.qa.suse.cz:
    salt-minion 3004
    Sep 04 11:22:40 qesapworker-prg7 salt-minion[23130]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:01 qesapworker-prg7 salt-minion[26409]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:43 qesapworker-prg7 salt-minion[28137]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:56 qesapworker-prg7 salt-minion[32417]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 qesapworker-prg7 salt-minion[34884]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 qesapworker-prg7 salt-minion[50940]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 qesapworker-prg7 salt-minion[51008]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
sapworker2.qe.nue2.suse.org:
    salt-minion 3004
    Sep 04 11:22:35 sapworker2 salt-minion[40033]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:59 sapworker2 salt-minion[43320]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:38 sapworker2 salt-minion[45037]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:46 sapworker2 salt-minion[53566]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:37 sapworker2 salt-minion[59321]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 sapworker2 salt-minion[113827]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 sapworker2 salt-minion[114030]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker5.oqa.suse.de:
    salt-minion 3004
    Sep 04 11:22:43 worker5 salt-minion[37172]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:07 worker5 salt-minion[1174]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:49 worker5 salt-minion[3841]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:04 worker5 salt-minion[12267]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:40 worker5 salt-minion[20388]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker5 salt-minion[1865]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker5 salt-minion[2314]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker2.oqa.suse.de:
    salt-minion 3004
    Sep 04 11:22:43 worker2 salt-minion[9879]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:09 worker2 salt-minion[14579]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:49 worker2 salt-minion[17122]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:58 worker2 salt-minion[23943]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:40 worker2 salt-minion[30443]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker2 salt-minion[11760]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker2 salt-minion[12227]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
malbec.arch.suse.de:
    salt-minion 3004
    Sep 01 10:30:25 malbec salt-minion[135586]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 malbec salt-minion[137954]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:37 malbec salt-minion[138394]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:23 malbec salt-minion[162447]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:34:18 malbec salt-minion[43902]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:23:04 malbec salt-minion[92177]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:27 malbec salt-minion[97182]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:10 malbec salt-minion[99723]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:18 malbec salt-minion[106429]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:41 malbec salt-minion[110374]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 malbec salt-minion[139948]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 malbec salt-minion[140319]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqa-piworker.qa.suse.de:
    salt-minion 3005.1
    Sep 04 11:23:12 openqa-piworker salt-minion[25164]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:34 openqa-piworker salt-minion[31947]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:19 openqa-piworker salt-minion[3235]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:25 openqa-piworker salt-minion[12935]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:38 openqa-piworker salt-minion[18985]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:07 openqa-piworker salt-minion[3626]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:11 openqa-piworker salt-minion[3865]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
QA-Power8-5-kvm.qa.suse.de:
    salt-minion 3004
    Sep 01 10:30:07 QA-Power8-5-kvm salt-minion[10004]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 QA-Power8-5-kvm salt-minion[11580]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:23 QA-Power8-5-kvm salt-minion[11847]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:08 QA-Power8-5-kvm salt-minion[34288]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:34:00 QA-Power8-5-kvm salt-minion[24051]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:47 QA-Power8-5-kvm salt-minion[58837]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:11 QA-Power8-5-kvm salt-minion[61956]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:55 QA-Power8-5-kvm salt-minion[65722]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:02 QA-Power8-5-kvm salt-minion[70674]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:43 QA-Power8-5-kvm salt-minion[75737]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 QA-Power8-5-kvm salt-minion[100291]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 QA-Power8-5-kvm salt-minion[100378]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
powerqaworker-qam-1.qa.suse.de:
    salt-minion 3004
    Sep 01 10:30:03 powerqaworker-qam-1 salt-minion[97398]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 powerqaworker-qam-1 salt-minion[99687]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:16 powerqaworker-qam-1 salt-minion[100171]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:04 powerqaworker-qam-1 salt-minion[124185]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:33:56 powerqaworker-qam-1 salt-minion[36838]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:44 powerqaworker-qam-1 salt-minion[124116]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:08 powerqaworker-qam-1 salt-minion[128809]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:49 powerqaworker-qam-1 salt-minion[1051]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:56 powerqaworker-qam-1 salt-minion[8351]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:42 powerqaworker-qam-1 salt-minion[12740]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 powerqaworker-qam-1 salt-minion[44524]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 powerqaworker-qam-1 salt-minion[44689]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
QA-Power8-4-kvm.qa.suse.de:
    salt-minion 3004
    Sep 04 11:22:59 QA-Power8-4-kvm salt-minion[3063]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:23 QA-Power8-4-kvm salt-minion[6074]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:06 QA-Power8-4-kvm salt-minion[7676]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:15 QA-Power8-4-kvm salt-minion[12500]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:42 QA-Power8-4-kvm salt-minion[15438]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 QA-Power8-4-kvm salt-minion[33153]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 QA-Power8-4-kvm salt-minion[33244]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqaw5-xen.qa.suse.de:
    salt-minion 3004
    Sep 01 10:29:57 openqaw5-xen salt-minion[24632]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 openqaw5-xen salt-minion[27213]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:12 openqaw5-xen salt-minion[27793]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:00 openqaw5-xen salt-minion[3592]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:33:47 openqaw5-xen salt-minion[27447]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:35 openqaw5-xen salt-minion[10317]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:49:58 openqaw5-xen salt-minion[15846]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:41 openqaw5-xen salt-minion[18826]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:52 openqaw5-xen salt-minion[26674]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:44 openqaw5-xen salt-minion[327]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:11 openqaw5-xen salt-minion[18990]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 openqaw5-xen salt-minion[19198]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker13.oqa.suse.de:
    salt-minion 3004
    Sep 04 11:23:10 worker13 salt-minion[1430]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:35 worker13 salt-minion[5810]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:17 worker13 salt-minion[8125]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:35 worker13 salt-minion[15408]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:40 worker13 salt-minion[21846]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker13 salt-minion[7943]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker13 salt-minion[8494]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker10.oqa.suse.de:
    salt-minion 3004
    Sep 04 11:23:16 worker10 salt-minion[15350]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:37 worker10 salt-minion[19394]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:19 worker10 salt-minion[21608]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:32 worker10 salt-minion[28126]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:40 worker10 salt-minion[915]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker10 salt-minion[1519]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker10 salt-minion[2431]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqaworker1.qe.nue2.suse.org:
    salt-minion 3004
    Sep 04 11:22:42 openqaworker1 salt-minion[14688]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:04 openqaworker1 salt-minion[18180]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:47 openqaworker1 salt-minion[19995]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:02 openqaworker1 salt-minion[26987]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:42 openqaworker1 salt-minion[2587]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 openqaworker1 salt-minion[10494]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 openqaworker1 salt-minion[10656]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqaworker-arm-2.suse.de:
    salt-minion 3004
    Sep 04 11:24:41 openqaworker-arm-2 salt-minion[74623]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:51:59 openqaworker-arm-2 salt-minion[77001]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:06:43 openqaworker-arm-2 salt-minion[78745]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:47:56 openqaworker-arm-2 salt-minion[81990]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:41 openqaworker-arm-2 salt-minion[89235]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:07 openqaworker-arm-2 salt-minion[75599]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:12 openqaworker-arm-2 salt-minion[76027]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqaworker-arm-3.suse.de:
    salt-minion 3004
    Sep 04 11:24:44 openqaworker-arm-3 salt-minion[83917]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:52:11 openqaworker-arm-3 salt-minion[86275]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:06:48 openqaworker-arm-3 salt-minion[88867]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:48:04 openqaworker-arm-3 salt-minion[92075]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:40 openqaworker-arm-3 salt-minion[1788]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:08 openqaworker-arm-3 salt-minion[89546]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:13 openqaworker-arm-3 salt-minion[89997]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
worker3.oqa.suse.de:
    salt-minion 3004
    Sep 04 11:22:39 worker3 salt-minion[27045]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:02 worker3 salt-minion[31342]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:44 worker3 salt-minion[1331]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:45:54 worker3 salt-minion[8047]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:42 worker3 salt-minion[13609]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:05 worker3 salt-minion[30123]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 worker3 salt-minion[30493]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
openqa.suse.de:
    salt-minion 3004
    Sep 01 10:35:42 openqa salt-minion[7849]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:18 openqa salt-minion[29174]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:48:55 openqa salt-minion[22618]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 11:22:45 openqa salt-minion[5709]: [WARNING ] state.highstate args: []
    Sep 01 11:36:09 openqa salt-minion[25342]: [WARNING ] state.highstate args: []
    Sep 01 11:44:18 openqa salt-minion[32485]: [WARNING ] state.highstate args: []
    Sep 01 13:44:16 openqa salt-minion[26273]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:38:02 openqa salt-minion[24077]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:59 openqa salt-minion[8313]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:33 openqa salt-minion[5949]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:15 openqa salt-minion[24436]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:51:08 openqa salt-minion[15702]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:10:40 openqa salt-minion[21998]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:10 openqa salt-minion[3460]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 openqa salt-minion[10506]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
backup.qa.suse.de:
    salt-minion 3004
    Sep 01 10:30:12 backup-vm salt-minion[6950]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 backup-vm salt-minion[9517]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:25 backup-vm salt-minion[10085]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:40 backup-vm salt-minion[8493]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:34:05 backup-vm salt-minion[28833]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:48 backup-vm salt-minion[4248]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:14 backup-vm salt-minion[9711]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:02 backup-vm salt-minion[12643]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:48:11 backup-vm salt-minion[20040]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:11:19 backup-vm salt-minion[24887]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:27 backup-vm salt-minion[597]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:11 backup-vm salt-minion[809]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
schort-server.qa.suse.de:
    salt-minion 3004
    Sep 01 10:30:20 schort-server salt-minion[15347]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 schort-server salt-minion[17902]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:30 schort-server salt-minion[18464]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:28 schort-server salt-minion[16845]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:34:25 schort-server salt-minion[4626]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:53 schort-server salt-minion[9054]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:23 schort-server salt-minion[14496]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:59 schort-server salt-minion[17409]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:49:32 schort-server salt-minion[24726]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:11:05 schort-server salt-minion[29653]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:24 schort-server salt-minion[3955]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:11 schort-server salt-minion[4136]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
tumblesle.qa.suse.de:
    salt-minion 3004
    Sep 01 10:30:20 tumblesle salt-minion[11730]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:18 tumblesle salt-minion[14287]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:31 tumblesle salt-minion[14850]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:20 tumblesle salt-minion[13138]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:34:12 tumblesle salt-minion[770]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:57 tumblesle salt-minion[5682]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:23 tumblesle salt-minion[11113]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:05:03 tumblesle salt-minion[14021]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:48:58 tumblesle salt-minion[21310]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:11:34 tumblesle salt-minion[26227]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:38 tumblesle salt-minion[1897]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:11 tumblesle salt-minion[2086]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
baremetal-support.qa.suse.de:
    salt-minion 3004
    Sep 01 13:38:38 baremetal-support salt-minion[2443]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:35:37 baremetal-support salt-minion[30861]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:28:03 baremetal-support salt-minion[26530]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:55:33 baremetal-support salt-minion[476]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:47 baremetal-support salt-minion[4441]: [WARNING ] state.highstate args: []
    Sep 04 12:05:00 baremetal-support salt-minion[4508]: [WARNING ] state.highstate args: []
    Sep 04 12:05:04 baremetal-support salt-minion[4518]: [WARNING ] state.highstate args: []
    Sep 04 12:05:25 baremetal-support salt-minion[4599]: [WARNING ] state.highstate args: []
    Sep 04 12:06:55 baremetal-support salt-minion[4949]: [WARNING ] state.highstate args: []
    Sep 04 12:10:12 baremetal-support salt-minion[4063]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:17:26 baremetal-support salt-minion[5984]: [WARNING ] state.highstate args: []
    Sep 04 12:37:14 baremetal-support salt-minion[10325]: [WARNING ] state.highstate args: []
    Sep 04 12:47:08 baremetal-support salt-minion[14975]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:11:04 baremetal-support salt-minion[21037]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:24 baremetal-support salt-minion[3722]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:10 baremetal-support salt-minion[3950]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']
jenkins.qa.suse.de:
    salt-minion 3004
    Sep 01 10:30:22 jenkins salt-minion[14093]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 10:41:17 jenkins salt-minion[17112]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 01 10:45:52 jenkins salt-minion[17763]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 13:37:41 jenkins salt-minion[25463]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 01 18:34:21 jenkins salt-minion[25869]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:22:58 jenkins salt-minion[31635]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 11:50:21 jenkins salt-minion[5681]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:04:59 jenkins salt-minion[9394]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 12:46:23 jenkins salt-minion[18364]: [WARNING ] state.highstate args: [{'queue': True}]
    Sep 04 13:11:42 jenkins salt-minion[25043]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:52:46 jenkins salt-minion[9928]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=yesterday -u salt-minion | grep "args:" | grep highstate']
    Sep 04 16:53:11 jenkins salt-minion[10163]: [WARNING ] cmd.run args: ['salt-minion --version && journalctl --since=1w -u salt-minion | grep "args:" | grep highstate']

as discussed in the daily today I will keep checking again in the future but as we don't see direct impact currently we reduce the priority now.

Actions #13

Updated by okurz 9 months ago

  • Related to action #134819: Errors in salt minion and master log on osd added
Actions #14

Updated by livdywan 9 months ago

as discussed in the daily today I will keep checking again in the future but as we don't see direct impact currently we reduce the priority now.

I'm wondering if this isn't affecting #134282#note-33 afterall as I was going through Alvaro's notes on debugging workers with faulty mm setups. worker.packages seems to specify all of the packages yet libcap-progs ended up missing.

It seems as though currently all workers have it, probably 1) because it was manually installed in the meantime 2) and/or salt states were applied eventually.

Actions #15

Updated by livdywan 9 months ago

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1806628/raw reveals this at the top:

    Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 44, in top-level template code
TypeError: 'NoneType' object is not iterable
Actions #16

Updated by nicksinger 9 months ago

livdywan wrote in #note-15:

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1806628/raw reveals this at the top:

    Rendering SLS 'base:openqa.openvswitch' failed: Jinja error: 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 44, in top-level template code
TypeError: 'NoneType' object is not iterable

This is not related to the issue at hand here and rather describes a wrong definition in our workersconf.sls. However, I was wondering too if a broken template somewhere could cause this. But as long as we don't see "Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list" we have to assume it is something else.

Actions #17

Updated by okurz 9 months ago

  • Due date deleted (2023-09-14)
  • Status changed from Feedback to Resolved

We should have more information available now if the issue reappears but we did not see it even over the past days so resolving.

Actions #18

Updated by jbaier_cz 8 months ago

  • Related to action #137744: [alert] gitlab pipeline fails in deploy step for salt-pillars-openqa: Data failed to compile added
Actions

Also available in: Atom PDF