QA » openQA Project » openQA Infrastructure

Motivation¶

See #161735-4 where nicksinger explains how the salt mine seems to be empty sometimes causing to end up with /etc/wicked/scripts/gre_tunnel_preup.sh being "empty" again (only containing options:remote_ip=, e.g. worker36 (offline at point of file generation) lines).

Acceptance criteria¶

• AC1: gre_tunnel_preup.sh scripts are ensured to have at least one valid remote_ip= statement
• AC2: All remote_ip= statements represent relevant peers, e.g. current online TAP worker hosts of same architecture

Suggestions¶

• Look into https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch.sls#L122
• Start with sudo salt --no-color -C 'G@roles:worker' cmd.run 'test -e /etc/wicked/scripts/gre_tunnel_preup.sh && grep remote_ip /etc/wicked/scripts/gre_tunnel_preup.sh'
• The task can be solved by ensuring non-empty entries during generation or also retroactively as part of the CI pipeline execution in a post-deploy monitoring step: Something like find currently online salt connected workers, use that as filter against https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls in a separate script
• Consider the case of an island cluster where actually no peers are expected
• Let this new script make a diff between the old and new version of gre_tunnel_preup.sh and do a sanity check on the diff (e.g. if too many lines have been removed reject the change)
• Check if nicksinger already disabled the grain-cache and if that helped