action #152095
closedopenQA Project (public) - coordination #112862: [saga][epic] Future ideas for easy multi-machine handling: MM-tests as first-class citizens
openQA Project (public) - coordination #111929: [epic] Stable multi-machine tests covering multiple physical workers
[spike solution][timeboxed:8h] Ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes size:S
0%
Description
Motivation¶
See lessons learned meeting #139136. We would again benefit from an easier reproducer. Related to #135818 . Come up with a way to ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes.
Acceptance criteria¶
- AC1: We know how to ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes
Suggestions¶
Research upstream about pinging over specific interfaces, GRE tunnels, TAP devices, openvswitch, etc.
- Like
ping -I<interface>
orping X.X.X.X%tap0
? - Checkout network namespaces and if they could be used
- Like
Research about MTU size debugging, tracepath, traceroute, etc.
Experiment in an openQA-environment or openQA-like with the bridges, tap devices, etc.
Demonstrate to the team in written form or interactively
Lookup how the existing check is done via a VM/VNC, and see how this could be simplified
Updated by okurz about 1 year ago
- Copied from action #152092: Handle all package downgrades in OSD infrastructure properly in salt size:M added
Updated by okurz about 1 year ago
- Copied to action #152098: [research][timeboxed:10h] Learn more about openvswitch with experimenting together size:S added
Updated by okurz 12 months ago
- Related to action #152389: significant increase in MM-test failure ratio 2023-12-11: test fails in multipath_iscsi and other multi-machine scenarios due to MTU size auto_review:"ping with packet size 1350 failed, problems with MTU" size:M added
Updated by okurz 11 months ago
- Subject changed from [spike solution][timeboxed:10h] Ping over GRE tunnels and TAP devices and openvswitch outside a SUT with differing packet sizes to [spike solution][timeboxed:8h] Ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes size:S
- Description updated (diff)
- Status changed from New to Workable
Updated by jbaier_cz 11 months ago ยท Edited
I did some initial experiments with the tunnels and openvswitch. The setup should be similar to openQA, but have some (important) differences. I will address them in the next step.
Initial setup for all experiments¶
# Enable ip forwarding
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.all.forwarding=1
# Install and enable openvswitch
zypper in openvswitch3
systemctl enable --now openvswitch
Experiments¶
Host | Network address | Bridge address | Remote IP |
---|---|---|---|
A | 192.0.2.1/24 | 192.168.42.1/24 | 192.0.2.2 |
B | 192.0.2.2/24 | 192.168.43.1/24 | 192.0.2.1 |
Note: instead of having two /24 networks, it is also possible to assign addresses from one bigger network (which have the benefit of not needing explicit route assignment).
Simple scenario¶
Two servers with a single bridge on each side connected with GRE tunnel.
# Create bridge and tunnel
nmcli con add type bridge con.int br0 bridge.stp yes ipv4.method manual ipv4.address "$bridge_address" ipv4.routes 192.168.42.0/23
nmcli con add type ip-tunnel mode gretap con.int gre1 master br0 remote "$remote_ip"
# Test the tunnel with ping
# -M do -- prohibit fragmentation
# -s xxxx -- set packet size
ping -c 3 -M do -s 1300 192.168.42.1
ping -c 3 -M do -s 1300 192.168.43.1
Scenario with openvswitch¶
Two servers with a one virtual bridge connected with GRE tunnel.
# Create bridge, port and interface
nmcli con add type ovs-bridge con.int br0 ovs-bridge.stp-enable yes
nmcli con add type ovs-port con.int br0 con.master br0
nmcli con add type ovs-interface con.int br0 con.master br0 ipv4.method manual ipv4.address "$bridge_address" ipv4.routes 192.168.42.0/23
# Create GRE tunnel
nmcli con add type ovs-port con.int gre1 con.master br0
nmcli con add type ip-tunnel mode gretap con.int gre1 master gre1 remote "$remote_ip"
# Test the tunnel
ping -c 3 -M do -s 1300 192.168.42.1
ping -c 3 -M do -s 1300 192.168.43.1
# ovs-vsctl show
de1f31e9-1b51-4cc3-954a-4e037191ac07
Bridge br0
Port br0
Interface br0
type: internal
Port gre1
Interface gre1
type: system
ovs_version: "3.1.0"
GRE tunnel made in openvswitch¶
openvswitch uses flow-based GRE tunneling, i.e. one interface gre_sys
for all tunnels, the tunnel can be created by ovs-vsctl
. After that, everything works as expected.
# Create bridge, port and interface
nmcli con add type ovs-bridge con.int br0 ovs-bridge.stp-enable yes
nmcli con add type ovs-port con.int br0 con.master br0
nmcli con add type ovs-interface con.int br0 con.master br0 ipv4.method manual ipv4.address "$bridge_address" ipv4.routes 192.168.42.0/23
# Create GRE tunnel
ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre options:remote_ip="$remote_ip"
# Test the tunnel
ping -c 3 -M do -s 1300 192.168.42.1
ping -c 3 -M do -s 1300 192.168.43.1
# ovs-vsctl show
de1f31e9-1b51-4cc3-954a-4e037191ac07
Bridge br0
Port br0
Interface br0
type: internal
Port gre1
Interface gre1
type: gre
options: {remote_ip="192.0.2.2"}
ovs_version: "3.1.0"
openQA-like¶
Each worker has the same 10.0.2.2/15 address set on the bridge interface and some extra openvswitch "magic" (i.e. os-autoinst-openvswitch) which allows the SUT to contact the worker machine via common address. This unfortunately renders the IP unusable for any inter-machine communication (pinging 10.0.2.2 from 10.0.2.2 just can't work).
Maybe the solution here is to just add an extra unique address for the bridge interface, which can be used for network checks. The added address does not even need to be from the same address space as long as we have correct routing tables.
Updated by jbaier_cz 10 months ago
@okurz just to clarify, what would be the best place to put instructions in? A new subsection under https://progress.opensuse.org/projects/openqav3/wiki#Infrastructure-setup-for-o3-openqaopensuseorg-and-osd-openqasusede or is there better more generic place?
Updated by okurz 10 months ago
https://progress.opensuse.org/projects/openqav3/wiki#Infrastructure-setup-for-o3-openqaopensuseorg-and-osd-openqasusede is fine. A better place would likely be the openQA documentation but a little bit more effort to convert markdown to asciidoc. Your choice.