action #162734
Updated by livdywan 5 months ago
## Motivation
See #161735-4 where nicksinger explains how the salt mine seems to be empty sometimes causing to end up with /etc/wicked/scripts/gre_tunnel_preup.sh being "empty" again (only containing `options:remote_ip=`, e.g. worker36 (offline at point of file generation) lines).
## Acceptance criteria
* **AC1:** gre_tunnel_preup.sh scripts are ensured to have at least one valid `remote_ip=` statement
* **AC2:** All remote_ip= statements represent relevant peers, e.g. current online TAP worker hosts of same architecture
## Suggestions
* Look into https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch.sls#L122
* Start with `sudo salt --no-color -C 'G@roles:worker' cmd.run 'test -e /etc/wicked/scripts/gre_tunnel_preup.sh && grep remote_ip /etc/wicked/scripts/gre_tunnel_preup.sh'`
* The task can be solved by ensuring non-empty entries during generation or also retroactively as part of the CI pipeline execution in a post-deploy monitoring step: Something like find currently online salt connected workers, use that as filter against https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls in a separate script
* Consider the case of an island cluster where actually no peers are expected
* Let this new script make a diff between the old and new version of `gre_tunnel_preup.sh` and do a sanity check on the diff (e.g. if too many lines have been removed reject the change)
* Check if nicksinger already disabled the grain-cache and if that helped