action #181829
closed
monitor.qe.nue2.suse.org not responding to salt size:S
Added by livdywan 5 days ago.
Updated 3 days ago.
Category:
Regressions/Crashes
Description
Observation¶
monitor.qe.nue2.suse.org:
Minion did not return. [Not connected]
Acceptance criteria¶
-
AC1: monitor.qe.nue2.suse.org is responsive to salt
Suggestions¶
-
DONE Verify that monitor.qe.nue2.suse.org is responsive on ssh -> works fine
- Try to ping salt nodes i.e.
retry ssh openqa.suse.de 'sudo salt \* test.ping'
- tumblesle.qe.nue2.suse.org was also temporarily unresponsive, maybe this isn't about monitor at all?
- maybe something changed again regarding MTU (lookup related tickets)
- Look into /var/log/salt/minion
sudo journalctl -u salt-minion.service
- Copied to action #181853: worker35.oqa.prg2.suse.org does not respond over salt "Minion did not return. [Not connected]" size:S added
- Related to action #181850: osd-deployment | Failed pipeline for master (worker35.oqa.prg2.suse.org and monitor.qe.nue2.suse.org) added
- Subject changed from osd-deployment | Failed pipeline for master: monitor.qe.nue2.suse.org not responding to salt to monitor.qe.nue2.suse.org not responding to salt size:S
- Description updated (diff)
- Status changed from New to Workable
- Status changed from Workable to In Progress
- Assignee set to gpathak
- Priority changed from High to Normal
The issue found in journal log was:
May 05 06:18:42 monitor salt-minion[9525]: /usr/lib/python3.6/site-packages/salt/states/x509.py:214: DeprecationWarning: The x509 modules are deprecated. Please migrate to the replacement modules (x509_v2). They are the default from Salt 3008 (Argon) onwards.
Found an upstream bug: https://github.com/saltstack/salt/issues/66508
Suggestion is to migrate to x509_v2, so I just followed the explicit-activation and restarted the salt-minion on monitor.
Checked from OSD that the monitor is now responding to salt ping:
gpathak@openqa:~> sudo salt monitor.qe.nue2.suse.org test.ping
monitor.qe.nue2.suse.org:
True
gpathak@openqa:~>
Reducing the priority as the host has responded to salt ping req.
- Status changed from In Progress to Feedback
moving this to feedback, to keep the machine under observation
will change the status accordingly based on observation
- Status changed from Feedback to Resolved
This pipeline shows that monitor is now up and responding to salt master ping.
AC is covered, resolving this ticket
- Status changed from Resolved to Feedback
What was the reason for the host not responding? I doubt the deprecation warning was the reason and if it was why should we only apply a manual change on a single machine and not all?
okurz wrote in #note-7:
What was the reason for the host not responding? I doubt the deprecation warning was the reason and if it was why should we only apply a manual change on a single machine and not all?
Right, I thought of creating an MR in our salt repo for the change, but then it slipped from my mind.
I will add this change to other machines as well
- Status changed from Feedback to Resolved
Also available in: Atom
PDF