Project

General

Profile

Actions
Copy link

action #181829

open

monitor.qe.nue2.suse.org not responding to salt size:S

Added by livdywan 1 day ago. Updated about 15 hours ago.

Status:
Feedback
Priority:
Normal
Assignee:
gpathak
Category:
Regressions/Crashes
Target version:
openQA Project (public) - Ready
Start date:
Due date:
% Done:

0%

Estimated time:
Tags:
alert, osd-deployment, infra, reactive work, gitlab CI, pipeline

Description

Observation

monitor.qe.nue2.suse.org: 
    Minion did not return. [Not connected]

Acceptance criteria

  • AC1: monitor.qe.nue2.suse.org is responsive to salt

Suggestions

  • DONE Verify that monitor.qe.nue2.suse.org is responsive on ssh -> works fine
  • Try to ping salt nodes i.e. retry ssh openqa.suse.de 'sudo salt \* test.ping'
  • tumblesle.qe.nue2.suse.org was also temporarily unresponsive, maybe this isn't about monitor at all?
  • maybe something changed again regarding MTU (lookup related tickets)
  • Look into /var/log/salt/minion
  • sudo journalctl -u salt-minion.service

Related issues 2 (2 open — 0 closed)

Related to openQA Infrastructure (public) - action #181850: osd-deployment | Failed pipeline for master (worker35.oqa.prg2.suse.org and monitor.qe.nue2.suse.org)Blockedokurz2024-03-18

Actions
Copied to openQA Infrastructure (public) - action #181853: worker35.oqa.prg2.suse.org does not respond over salt "Minion did not return. [Not connected]" size:SWorkabledheidler2025-05-06

Actions
Actions
Copy link
 #1

Updated by okurz 1 day ago

  • Copied to action #181853: worker35.oqa.prg2.suse.org does not respond over salt "Minion did not return. [Not connected]" size:S added
Actions
Copy link
 #2

Updated by okurz 1 day ago

  • Related to action #181850: osd-deployment | Failed pipeline for master (worker35.oqa.prg2.suse.org and monitor.qe.nue2.suse.org) added
Actions
Copy link
 #3

Updated by okurz 1 day ago

  • Subject changed from osd-deployment | Failed pipeline for master: monitor.qe.nue2.suse.org not responding to salt to monitor.qe.nue2.suse.org not responding to salt size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #4

Updated by gpathak about 17 hours ago

  • Status changed from Workable to In Progress
  • Assignee set to gpathak
  • Priority changed from High to Normal

The issue found in journal log was:

May 05 06:18:42 monitor salt-minion[9525]: /usr/lib/python3.6/site-packages/salt/states/x509.py:214: DeprecationWarning: The x509 modules are deprecated. Please migrate to the replacement modules (x509_v2). They are the default from Salt 3008 (Argon) onwards.

Found an upstream bug: https://github.com/saltstack/salt/issues/66508
Suggestion is to migrate to x509_v2, so I just followed the explicit-activation and restarted the salt-minion on monitor.

Checked from OSD that the monitor is now responding to salt ping:

gpathak@openqa:~> sudo salt monitor.qe.nue2.suse.org test.ping
monitor.qe.nue2.suse.org:
    True
gpathak@openqa:~>

Reducing the priority as the host has responded to salt ping req.

Actions
Copy link
 #5

Updated by gpathak about 15 hours ago ยท Edited

  • Status changed from In Progress to Feedback

moving this to feedback, to keep the machine under observation
will change the status accordingly based on observation

Actions
Copy link

Also available in: Atom PDF