action #136013
closedcoordination #112862: [saga][epic] Future ideas for easy multi-machine handling: MM-tests as first-class citizens
coordination #111929: [epic] Stable multi-machine tests covering multiple physical workers
Ensure IP forwarding is persistent for multi-machine tests also in our salt recipes size:M
0%
Description
Motivation¶
See #134282 . So far we always relied on the installation to use the "hypervisor role" hence enabling persistent IP forwarding but we never ensured IP forwarding to be properly enabled by https://gitlab.suse.de/openqa/salt-states-openqa/ until https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/987 . We still don't know for sure if that is enough or correct.
Acceptance criteria¶
- AC1: We know with good confidence that our salt-states ensure IP forwarding to be properly enabled by https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/987 or an alternative solution
Suggestions¶
- the situation got way better after forwarding was enabled in salt/firewalld on each bridge with https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/987. net.ipv4.ip_forward = 1 might still be required to be covered in salt but we need to understand what the -directive in firewalld does first. this is possible by e.g. reading firewalld documentation or just set it back to 0, run salt and see if this changes it back to 1.
- https://progress.opensuse.org/issues/135524#note-15
- sudo salt -C 'worker3*' --out=text cmd.run 'sysctl -a | grep net.ipv..conf.br..f orwarding | grep -v v6' is/was a way to verify the settings - note that right now this is set, despite not having been set by our salt config!
- Consider taking out of salt and re-installing a single worker to confirm
- 1. what the defaults are
- 2. what's set after applying our salt states
- 3. what's missing
- Check that what was done in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1004
- Use https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=ovs-client&version=15-SP5 to verify that jobs successfully work
Updated by okurz 8 months ago
- Copied from action #134282: [tools] network protocols failures on multimachine tests on HA/SAP size:S auto_review:"no candidate.*iscsi-target-overview-service-tab|yast2.+firewall.+services.+add.+zone":retry added
Updated by livdywan 7 months ago
See #134282 . So far we always relied on the installation to use the "hypervisor role" hence enabling persistent IP forwarding but we never ensured IP forwarding to be properly enabled by https://gitlab.suse.de/openqa/salt-states-openqa/ until https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/987 . We still don't know for sure if that is enough or correct.
Apparently it's not enough. See #135407-27 and the according MR. The changes in trusted.xml are not enabling forwarding on all interfaces. It's only known to be enabled when something or someone else did.
Updated by livdywan 7 months ago
- Related to action #135407: [tools] Measure to mitigate websockets overload by workers and revert it size:M added
Updated by livdywan 6 months ago
- Related to action #136130: test fails in iscsi_client due to salt 'host'/'nodename' confusion size:M added
Updated by dheidler 6 months ago
- Status changed from Workable to Resolved
The already applied salt rules seem to work fine:
salt -C 'G@roles:worker' cmd.run 'grep forward /etc/sysctl.d/99-salt.conf'
openqaworker17.qa.suse.cz:
net.ipv4.ip_forward = 1
net.ipv4.conf.br1.forwarding = 1
net.ipv4.conf.eth0.forwarding = 1
[...]
salt -C 'G@roles:worker' cmd.run 'sysctl net.ipv4.ip_forward'
openqaworker16.qa.suse.cz:
net.ipv4.ip_forward = 1
openqaworker17.qa.suse.cz:
net.ipv4.ip_forward = 1
openqaworker18.qa.suse.cz:
net.ipv4.ip_forward = 1
[...]