Project

General

Profile

Actions
Copy link

action #134819

open

Errors in salt minion and master log on osd

Added by tinita 8 months ago. Updated 8 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
QA - future
Start date:
2023-08-30
Due date:
% Done:

0%

Estimated time:

Description

Observation

# /var/log/salt/master
2023-08-29 12:33:18,360 [salt.utils.event :914 ][ERROR   ][16216] Event iteration failed with exception: 'user'
2023-08-29 13:04:15,459 [salt.utils.event :914 ][ERROR   ][16216] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-29 13:04:41,586 [salt.utils.event :914 ][ERROR   ][16213] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-29 13:11:02,777 [salt.utils.event :914 ][ERROR   ][16213] Event iteration failed with exception: 'user'
# goes on until today ever hour like this:
2023-08-30 10:08:24,870 [salt.utils.event :914 ][ERROR   ][16224] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:08:26,378 [salt.utils.event :914 ][ERROR   ][16224] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:08:30,671 [salt.utils.event :914 ][ERROR   ][16215] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:08:31,068 [salt.utils.event :914 ][ERROR   ][16215] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:08:31,196 [salt.utils.event :914 ][ERROR   ][16223] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:08:32,336 [salt.utils.event :914 ][ERROR   ][16216] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:08:59,066 [salt.utils.event :914 ][ERROR   ][16224] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:00,068 [salt.utils.event :914 ][ERROR   ][16216] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:07,593 [salt.utils.event :914 ][ERROR   ][16213] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:14,522 [salt.utils.event :914 ][ERROR   ][16224] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:14,610 [salt.utils.event :914 ][ERROR   ][16213] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:16,156 [salt.utils.event :914 ][ERROR   ][16223] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:17,975 [salt.utils.event :914 ][ERROR   ][16213] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:18,018 [salt.utils.event :914 ][ERROR   ][16223] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:18,378 [salt.utils.event :914 ][ERROR   ][16216] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:18,739 [salt.utils.event :914 ][ERROR   ][16224] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:18,833 [salt.utils.event :914 ][ERROR   ][16224] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:09:46,341 [salt.utils.event :914 ][ERROR   ][16223] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-08-30 10:11:48,757 [salt.utils.event :914 ][ERROR   ][16215] Event iteration failed with exception: 'user'

Checked back to 2023-08-06, this error started 2023-08-29

# /var/log/salt/minion
2023-08-29 12:31:44,145 [salt.state       :327 ][ERROR   ][31768] Pillar openqa.oqa.prg2.suse.org.key does not exist
2023-08-29 12:31:44,148 [salt.state       :327 ][ERROR   ][31768] Pillar openqa.oqa.prg2.suse.org.crt does not exist
# goes on until today every hour
2023-08-30 10:11:18,745 [salt.state       :327 ][ERROR   ][1625] Pillar openqa.oqa.prg2.suse.org.key does not exist
2023-08-30 10:11:18,749 [salt.state       :327 ][ERROR   ][1625] Pillar openqa.oqa.prg2.suse.org.crt does not exist

Related issues 3 (0 open — 3 closed)

Related to openQA Infrastructure - action #134522: [alert] Certificate renewal on monitor.qa.suse.de might not be working causing alerts size:MResolvednicksinger2023-08-232023-09-07

Actions
Related to QA - action #133457: salt-states-openqa gitlab CI pipeline aborted with error after 2h of execution size:MResolvedokurz

Actions
Related to openQA Infrastructure - action #133469: [alert] Salt states don't apply sometimes on individual workers size:MResolvednicksinger2023-07-27

Actions
Actions
Copy link
 #1

Updated by livdywan 8 months ago

  • Related to action #134522: [alert] Certificate renewal on monitor.qa.suse.de might not be working causing alerts size:M added
Actions
Copy link
 #2

Updated by okurz 8 months ago

  • Related to action #133457: salt-states-openqa gitlab CI pipeline aborted with error after 2h of execution size:M added
Actions
Copy link
 #3

Updated by okurz 8 months ago

  • Target version set to future

first #133457

Actions
Copy link
 #4

Updated by okurz 8 months ago

  • Status changed from New to Resolved
  • Assignee set to nicksinger
  • Target version changed from future to Ready

I assume nicksinger fixed that certificate apache related issues meanwhile

Actions
Copy link
 #5

Updated by okurz 8 months ago

  • Status changed from Resolved to New
  • Assignee deleted (nicksinger)
  • Priority changed from Normal to Low
  • Target version changed from Ready to future

tinita still saw issues in the logs today though no critical impact

Actions
Copy link
 #6

Updated by tinita 8 months ago 

# /var/log/salt/master
2023-09-04 12:46:44,748 [salt.utils.event :908 ][ERROR   ][2004] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-09-05 09:09:05,078 [tornado.application:640 ][ERROR   ][23489] Exception in callback None
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 889, in start
    fd_obj, handler_func = self._handlers[fd]
KeyError: 18
2023-09-05 10:12:01,744 [salt.loaded.int.returner.local_cache:150 ][ERROR   ][2005] An extra return was detected from minion openqa-piworker.qa.suse.de, please verify the minion, this could be a replay attack
2023-09-05 10:12:45,558 [salt.loaded.int.returner.local_cache:150 ][ERROR   ][2006] An extra return was detected from minion openqa-piworker.qa.suse.de, please verify the minion, this could be a replay attack
2023-09-05 11:43:13,199 [salt.utils.event :908 ][ERROR   ][1973] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:13,245 [salt.utils.event :908 ][ERROR   ][1972] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:13,255 [salt.utils.event :908 ][ERROR   ][1974] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:13,271 [salt.utils.event :908 ][ERROR   ][1977] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:13,277 [salt.utils.event :908 ][ERROR   ][1977] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:13,302 [salt.utils.event :908 ][ERROR   ][1974] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:13,306 [salt.utils.event :908 ][ERROR   ][1974] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:13,377 [salt.utils.event :908 ][ERROR   ][1973] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:14,022 [salt.utils.event :908 ][ERROR   ][1975] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:43:14,047 [salt.utils.event :908 ][ERROR   ][1975] Event iteration failed with exception: 'str' object has no attribute 'items'
2023-09-05 11:54:19,582 [salt.utils.parsers:1113][WARNING ][1272] Master received a SIGTERM. Exiting.

# /var/log/salt/minion
2023-09-05 10:18:36,821 [salt.minion      :1919][WARNING ][15366] saltutil.find_job args: ['20230905081403575217']
2023-09-05 10:18:46,854 [salt.minion      :1919][WARNING ][15856] saltutil.find_job args: ['20230905081403575217']
2023-09-05 10:18:56,992 [salt.minion      :1919][WARNING ][16311] saltutil.find_job args: ['20230905081403575217']
2023-09-05 10:19:07,357 [salt.minion      :1919][WARNING ][16671] saltutil.find_job args: ['20230905081403575217']
2023-09-05 10:19:17,598 [salt.minion      :1919][WARNING ][16902] saltutil.find_job args: ['20230905081403575217']
2023-09-05 10:19:27,731 [salt.minion      :1919][WARNING ][17549] saltutil.find_job args: ['20230905081403575217']
2023-09-05 10:19:37,866 [salt.minion      :1919][WARNING ][18799] saltutil.find_job args: ['20230905081403575217']
2023-09-05 11:54:12,527 [salt.utils.parsers:1113][WARNING ][1697] Minion received a SIGTERM. Exiting.
2023-09-05 11:59:36,792 [salt.minion      :1919][WARNING ][5309] test.ping args: []
2023-09-05 13:03:01,966 [salt.loaded.int.module.cmdmod:880 ][ERROR   ][21566] Command '/usr/bin/patch' failed with return code: 1
2023-09-05 13:03:01,968 [salt.loaded.int.module.cmdmod:882 ][ERROR   ][21566] stdout: patching file /tmp/__salt.tmp.puryqpm8 (read from /usr/lib/python3.6/site-packages/salt/minion.py)
Reversed (or previously applied) patch detected!  Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file /tmp/__salt.tmp.ob3oyyvb
2023-09-05 13:03:01,968 [salt.loaded.int.module.cmdmod:886 ][ERROR   ][21566] retcode: 1
Actions
Copy link
 #7

Updated by okurz 8 months ago

  • Related to action #133469: [alert] Salt states don't apply sometimes on individual workers size:M added
Actions
Copy link

Also available in: Atom PDF