QA » openQA Project » openQA Infrastructure

One major difference I spotted was that e.g. xen5 is using OVS-bridges while ada uses a native linux bridge but not sure if this is really related. I also saw that net.ipv6.conf.eth0.disable_ipv6 is set but on both machines

alright so the bridge causes the issue because it has forwarding enabled on an OS level. This causes Linux to refuse router advertisements because sysctl net.ipv6.conf.br0.accept_ra=1 and forwarding is enabled. If we set accept_ra=2 this disables this behavior and the next RA is applied to the interface:

ada:/etc # ip a s dev br0
19: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:ec:ef:b9:b8:30 brd ff:ff:ff:ff:ff:ff
    inet 10.146.4.1/23 brd 10.146.5.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 2a07:de40:b230:1:a8d8:5bc6:d523:bab6/64 scope global temporary dynamic
       valid_lft 604172sec preferred_lft 85216sec
    inet6 2a07:de40:b230:1:3eec:efff:feb9:b830/64 scope global dynamic mngtmpaddr
       valid_lft 2591754sec preferred_lft 604554sec
    inet6 fe80::3eec:efff:feb9:b830/64 scope link
       valid_lft forever preferred_lft forever

I think we should able to detect this and configure accordingly with salt.

salt '*' cmd.run 'ip l | grep -i master | grep -v tap | grep -v gre_sys | grep -v ovs-system' shows the following hosts potentially "affected" by this:

osiris-1.qe.nue2.suse.org:
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
openqaworker1.qe.nue2.suse.org:
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
qamaster.qe.nue2.suse.org:
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
    12: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    13: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    14: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    15: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    16: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    17: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    18: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    19: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
imagetester.qe.nue2.suse.org:
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
schort-server.qe.nue2.suse.org:
mania.qe.nue2.suse.org:
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
openqa-piworker.qe.nue2.suse.org:
    3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 1000

Just add accept_ra=2 in salt states for all hosts in sysctl.sls

what fixed ada? Was it a transient change of sysctl setting or is it persistent?

me setting net.ipv6.conf.br0.accept_ra = 2 manually so it is transient

I created /etc/sysctl.d/99-poo155824.conf manually to make it persistent on that host for now at least.

ok, I now created
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1115

I think we should try this out and revert if we see problems with that.

We already have

net.ipv6.conf.default.use_tempaddr=1
net.ipv6.conf.default.autoconf=1

on all salt controlled hosts so we shouldn't need this. And the accept_ra should also not matter here. So far it seems only ada needs special treatment for potentially manual configuration. So retracting my MR.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1116 merged

I checked s390zl12 and 13 and it should be easy enough to give them proper AAAA-records as well.
Generating these v6 addresses with only the MAC address available (e.g. if a machine is not already running) requires one to calculate the EUI-64 (see https://www.kwtrain.com/blog/how-to-calculate-an-eui-64-address for an example). I wrote a dirty script to help me with that:

#!/usr/bin/env python3
import sys
if len(sys.argv) < 2:
    mac = input("Enter mac of host: ")
else:
    mac = sys.argv[1]
byte_strings = mac.split(":")
fffe_inserted = byte_strings[:3] + ["ff", "fe"] + byte_strings[3:]
flipped_bit = hex(int(fffe_inserted[0], 16) ^ 0b00000010)
fffe_inserted[0] = flipped_bit.split("x", 1)[1]
v6_notation = list(zip(fffe_inserted[0::2], fffe_inserted[1::2]))
v6_notation = list(map(lambda x: "".join(x), v6_notation))
v6_notation = ":".join(v6_notation)
print(v6_notation)

If the machine is running, we can use the following command to print the mac of the default interface:

ip -j a s dev $(ip -6 -j r s | jq -r "(.[] | select(.dst == \"default\")).dev") | jq -r .[].address

Maybe these two methods can be combined to automate the process for the many machines in nue2