Project

General

Profile

action #60407

[qac][wicked] Investigate how to get multiple NICs in different broadcast-domains when having a MM scenario

Added by cfconrad over 2 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Spike/Research
Target version:
-
Start date:
2019-11-28
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

We discussed with Network team, that we need to have different broadcast domains when having multiple NICs.

Something like this:
http://imagebin.suse.de/2544/img

History

#1 Updated by cfconrad over 2 years ago

  • Status changed from New to In Progress
  • Assignee set to cfconrad

OpenQA offers a NETWORKS= variable. Each network is get's it's own VLAN tag. The order is correlated to the NIC number.

e.g. NETWORKS=net1,net2 will allocate 2 VLANs and assign them to the interfaces.

@see:

openQA/lib/OpenQA/Schema/Result/Jobs.pm::prepare_for_work()

#2 Updated by cfconrad over 2 years ago

So it is possible to create multiple VLAN's and put each NIC in a specific one.
A "problem" might be, that they are not fully independent. As each VLAN belongs from open-vswitch perspective to the same bridge. And we have net.ipv4.ip_forward = 1 enabled on that host.

Example:

Bridge "br1"
  Port "tap0"
    Interface "0"
  Port "tap1"
    Interface "tap64"
  Port "tap41"
    Interface "tap128"
  Port "br1"
    Interface "br1"
      type: internal

#3 Updated by jlausuch over 2 years ago

I would test it as follows:

1) create 2 VMs with 2 NICs in different VLANs
VM1: NIC1 (vlan X) NIC2 (vlan Y)
VM2: NIC1 (vlan X) NIC2 (vlan Y)

2) Assing 192.168.0.100/24 to NIC1 in VM1

3) Assign 192.168.0.101/24 to NIC2 in VM2

4) Check ping from VM1-NIC1 to VM2-NIC2 or viceversa

I think the net.ipv4.ip_forward = 1 option applies from br0 to other phyisical NICs in the host, but I'm not sure.

#4 Updated by jlausuch about 2 years ago

  • Project changed from SUSE QA to openQA Tests
  • Subject changed from [kernel][wicked] Investigate how to get multiple NICs in different broadcast-domains when having a MM scenario to [qac][wicked] Investigate how to get multiple NICs in different broadcast-domains when having a MM scenario
  • Status changed from In Progress to Workable

#5 Updated by jlausuch about 2 years ago

  • Category set to Spike/Research
  • Status changed from Workable to In Progress

#6 Updated by cfconrad about 2 years ago

I did some further tests.

Setup

Running two jobs (wicked_advance) in parallel each with NETWORKS=vnet1,vnet2.

  • SUT: tap43, tap107
  • REF: tap42, tap106

The ovs config looks like:

14:50-autobot#[0] INTERFACES="tap43 tap107 tap42 tap106 br1"
14:52-autobot#[0] for i in $INTERFACES; do
>   echo INTERFACE: $i;
>   echo -n '  vlan_mode='
>   ovs-vsctl get port $i vlan_mode;
>   echo -n '  tag='
>   ovs-vsctl get port $i tag;
>   echo -n '  stp_enable='
>   ovs-vsctl get bridge $i stp_enable
> done
INTERFACE: tap43
  vlan_mode="dot1q-tunnel"
  tag=1
  stp_enable=ovs-vsctl: no row "tap43" in table Bridge
INTERFACE: tap107
  vlan_mode="dot1q-tunnel"
  tag=2
  stp_enable=ovs-vsctl: no row "tap107" in table Bridge
INTERFACE: tap42
  vlan_mode="dot1q-tunnel"
  tag=1
  stp_enable=ovs-vsctl: no row "tap42" in table Bridge
INTERFACE: tap106
  vlan_mode="dot1q-tunnel"
  tag=2
  stp_enable=ovs-vsctl: no row "tap106" in table Bridge
INTERFACE: br1
  vlan_mode=[]
  tag=[]
  stp_enable=false
09:56-autobot#[0] sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0

Check

According to https://progress.opensuse.org/issues/60407#note-3 there is no PING possible.
I also verified it with broadcast packages created with:

echo "FOO" | socat - UDP-DATAGRAM:255.255.255.255:24000,broadcast

and using tshark -i ethX -f 'port !22' on each interface to see if the packages are visible or not.

Summary

We are able to build multi-host scenarios and connect individual NIC's which each other.
As an example I created 3 hosts like:

  1. NETWORKS=vnet1,vnet2
  2. NETWORKS=vnet2,vnet3
  3. NETWORKS=vnet3,vnet4

and configured them to have a real routing scenario like:

             eth1              eth0         eth1              eth0
    +---------+    10.6.0.0/24   +------------+  10.7.0.0/24    +-------+
    |  HOST1  +<---------------->|   Router   |<--------------->+ HOST2 |
    +---------+                  |ip_forward=1|                 +-------+
          10.6.0.2               +------------+            10.7.0.2
                           10.6.0.1      10.7.0.1

Question

In each VLAN is the br1 present. So if an interface takes a ip from 10.0.0.0/15 it can access 10.0.2.2 (which is the default gw running NAT). Is this a problem?

#7 Updated by cfconrad about 2 years ago

  • Status changed from In Progress to Feedback

#8 Updated by jlausuch almost 2 years ago

According to the documentation, the setup should look like this:

            +-------------+
            |             |
            | Test suite  |
            |  (with      |
            |   Jenkins   |
            |   as an     |
            |   option)   |
            |             |
            +-+---------+-+
              |         |
              | serial, |
              | ssh,    +---------------+ serial, ssh,
 *********    | or virtio               | or virtio
 * Outer *    |                         |
 * World *  +-+-----------+           +-+-----------+
 *********  |             |    eth0   |             |
     |      |             +-----------+             |
     | eth2 |  Reference  |    eth1   |   System    |
     +------+  server     +-----------+   Under     |
            |             |    ib0    |   Tests     |
            |             +-----------+             |
            |             |    ...    |             |
            +-------------+           +-------------+

Here, Outerworld would be the worker, connected with OBS bridge to REF using 10.0.2.X
Then, the other 2 interfaces connected to SUT should be different. So, do we need 3 NICs in REF?

#9 Updated by jlausuch almost 2 years ago

Transforming this picture in our setup, it would be something like this:

┌-----------------------------------------------┓
|                                               |
|                      WORKER                   |
|           REF                    SUT          |
|      +----------+           +----------+      |
|      |          |           |          |      |
|      |     eth0 +-----------+ eth0     |      |
|      |          |           |          |      |
|      |     eth1 +-----------+ eth1     |      |
|      |          |           |          |      |
|      |          |           |          |      |
|      |   eth2   |           |   e̶t̶h̶2̶   |      |
|      +----------+           +----------+      |
|           |                       |           |
|           |                       |           |
|           |_________ br0 _________|           |
|                       |                       |
|                                               |
└-----------------------------------------------┘

#10 Updated by jlausuch almost 2 years ago

  • Status changed from Feedback to Closed

I created poo#67774 to follow up this topic with real implementation.

Also available in: Atom PDF