action #161423
closedcoordination #161414: [epic] Improved salt based infrastructure management
[timeboxed:10h] Incomplete config files on OSD due to salt - Improve salt state application from remotely accessible salt master size:S
0%
Description
Motivation¶
See #161324 . Why did the salt states pipelines end with success when the salt high state was never reported to be successfully applied to the openqa.suse.de salt minion (openqa.suse.de is not mentioned in the list of minions where the state was applied but the pipeline still ended)? We do not know yet but this should help us in the future to spot errors quicker in case similar problems return. Maybe the problem is related to how we run salt over ssh from that minion openqa.suse.de and potentially the exit code from salt was never propagated but the command in bash just ended prematurely? Research about best practices how to apply a high state from a remotely accessible master upstream and investigate this
Acceptance criteria¶
- AC1: We know the best practice how to apply a salt high state on a remotely accessible salt master while avoiding loosing the ssh session in the process
Suggestions¶
- Just do a web research or vague look around if there are any best practices, known problems, instructions for running salt on a remote ssh-reachable host
- Look into how the salt states CI pipelines originally behaved in #161309 and how results of the state application are missing for openqa.suse.de. Maybe we lost connection to the salt master while the high state was applied and then the CI pipeline ended with "success" even though we never received a response from openqa.suse.de?