Saw several MM HA tests also failing due to network issues, either by attempting to contact 10.0.2.2, attempting to run yast2 firewall
, name solving, etc.
In the screenshots it could be seen that network was not working properly for the VMs, for example in: https://openqa.suse.de/tests/4238452#step/iscsi_client/5 (could not resolve openqa.suse.de) or https://openqa.suse.de/tests/4238437#step/setup/25 (network 10.0.2.1 unreacheable).
After checking some of the failed jobs, a pattern could be seen in that failed jobs seemed to be limited to openqaworker-arm-1.
Checking on the system, the following could be seen in the status of the os-autoinst-openvswitch service:
openqaworker-arm-1:~ # systemctl status os-autoinst-openvswitch.service
● os-autoinst-openvswitch.service - os-autoinst openvswitch helper
Loaded: loaded (/usr/lib/systemd/system/os-autoinst-openvswitch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/os-autoinst-openvswitch.service.d
└─override.conf
Active: active (running) since Thu 2020-05-14 14:03:00 UTC; 20h ago
Main PID: 3367 (os-autoinst-ope)
Tasks: 1
CGroup: /system.slice/os-autoinst-openvswitch.service
└─3367 /usr/bin/perl /usr/lib/os-autoinst/os-autoinst-openvswitch
May 15 10:54:12 openqaworker-arm-1 ovs-vsctl[21376]: ovs|00001|vsctl|INFO|Called as ovs-vsctl remove port tap2 tag 24
May 15 10:54:24 openqaworker-arm-1 ovs-vsctl[21445]: ovs|00001|db_ctl_base|ERR|no port named tap16
May 15 10:54:24 openqaworker-arm-1 os-autoinst-openvswitch[3367]: ovs-vsctl: no port named tap16
May 15 10:54:24 openqaworker-arm-1 os-autoinst-openvswitch[3367]: 'tap16' is not connected to bridge 'br1'
May 15 10:54:52 openqaworker-arm-1 ovs-vsctl[21859]: ovs|00001|db_ctl_base|ERR|no port named tap15
May 15 10:54:52 openqaworker-arm-1 os-autoinst-openvswitch[3367]: ovs-vsctl: no port named tap15
May 15 10:54:52 openqaworker-arm-1 os-autoinst-openvswitch[3367]: 'tap15' is not connected to bridge 'br1'
May 15 10:55:10 openqaworker-arm-1 ovs-vsctl[22196]: ovs|00001|db_ctl_base|ERR|no port named tap10
May 15 10:55:10 openqaworker-arm-1 os-autoinst-openvswitch[3367]: ovs-vsctl: no port named tap10
May 15 10:55:10 openqaworker-arm-1 os-autoinst-openvswitch[3367]: 'tap10' is not connected to bridge 'br1'
Checking those 3 tap interfaces directly, interfaces were defined in the system, but not listed in the bridge startup script:
openqaworker-arm-1:/etc/sysconfig/network # ip a | egrep 'tap16|tap15|tap10'
15: tap10: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
40: tap15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
41: tap16: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
openqaworker-arm-1:/etc/sysconfig/network # egrep 'tap16|tap15|tap10' ifcfg-br1
Could not continue checking as server was restarted.