QA » openQA Project » openQA Tests

https://openqa.opensuse.org/tests/4018286#step/remote_controller/14 shows

[   63.657904] named[3159]: timed out resolving './DNSKEY/IN': 10.150.1.11#53
[   63.659073] named[3159]: network unreachable resolving './DNSKEY/IN': 2001:7fd::1#53
[   63.660290] named[3159]: network unreachable resolving './DNSKEY/IN': 2001:500:12::d0d#53
[   63.661531] named[3159]: network unreachable resolving './DNSKEY/IN': 2001:500:9f::42#53

We could lower the urgency by working around the problem via https://github.com/os-autoinst/openQA/pull/5536.

Looks like the latest run worked again and it ran across worker21 and 22. Runs from before across worker22 and 24 were passing as well. Do we know what set of workers is affected?

EDIT: I tried to figure out good vs. problematic workers via the job history:

good: 21, 22, 24
bad: 23, 26
unknown: 25

Although I'm not sure how much sense that makes considering 7 days ago even jobs across 23 and 26 passed (https://openqa.opensuse.org/tests/4025897). So this might not even be specific to certain hosts.

• /etc/wicked/scripts/gre_tunnel_preup.sh looks good on all workers. (w20 seems non-existent; w27, w28, arm21 and arm22 are not connected to the gre network at all; this leaves w21 to w26 as valid workers in accordance with how the tap worker class is assigned)
• enough tap devices are configured on all hosts (30 slots, so the highest number would be tap158) and in the trusted zone according to firewall-cmd --get-active-zones - except on w22 where the public is also active and containing the tap interfaces
• cat /proc/sys/net/ipv4/ip_forward also returns 1 on all hosts
• for i in $hosts; do echo $i && ssh root@$i "firewall-cmd --query-forward --zone=trusted" ; done returns "no" on all hosts; I would have expected a "yes" but it looks like we just don't configure forwarding via firewalld here.

I changed the zoning on w22 via for i in {0..178}; do firewall-cmd --remove-interface=tap$i --zone=public ; done and for i in {0..178}; do firewall-cmd --add-interface=tap$i --zone=trusted ; done. Now firewall-cmd --get-active-zones only shows trusted anymore (with all tap devices). Unfortunately I was distracted so there was a delay of 2 hours between those two commands. It doesn't look like this caused much disruption, though. I also ran the same commands with --permanent but got warnings like Warning: ALREADY_ENABLED: tap94 so this was likely not necessary.

The Next & Previous tab in the intial (arm) scenario looks very good. Recent jobs in the (x86_64) wireguard scenario (https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=wireguard_client&version=Tumbleweed) are also passing. I'll keep observing the situation but would not workaround the problem right now (by forcing jobs to run on a single worker) as it doesn't seem very problematic right now.

I suggest to reboot multiple machines and see if that changes the settings

On w22 the zone setting still revered after I've just rebooted the machine. So I ran for i in {0..178}; do firewall-cmd --permanent --remove-interface=tap$i --zone=public ; firewall-cmd --permanent --add-interface=tap$i --zone=trusted ; done and rebooted the machine. Now the zone setting is in fact persistent. (That means previously the "permanent" setting was wrong on w22.)

I also rebooted on of the other hosts to see whether that changes anything but it seems not the case. So I assume only w22 was wrongly configured.

Note that w22 being problematic could even explain failures seen in clusters that only ran across other hosts due to our way of connecting each host with each other host.

Still broken: https://openqa.opensuse.org/tests/4070107#step/networking/15

I now also updated the runtime and permanent configuration (as it looks like only the permanent configuration was updated by @jbaier_cz). The persistent configuration looked good except that on arm21 the eth0 interface was in two zones at once which I fixed (leading to firewalld complaining about it: Error: INVALID_ZONE: public trusted (ERROR: interface 'eth0' is in 2 zone XML files, can be only in one)).

The config only appeared broken on the arm workers and on openqaworker23. Not sure why eth0 was moved again to the public zone there.

I restarted the failing job, let's see whether it worked: https://openqa.opensuse.org/tests/4071829#dependencies

EDIT: The tests are now passing again. So it was really just the firewalld settings again. Last time I only considered x86_64 hosts (as @jbaier_cz had already taken care of arm hosts before) and also checked only one other host's persistent config by rebooting. So probably I just missed updating the persistent config of some hosts.

So I've now just run for i in $hosts; do echo $i && ssh root@$i "firewall-cmd --permanent --zone=trusted --change-interface=eth0" ; done to make sure the persistent config on all hosts is correct (and for i in $hosts; do echo $i && ssh root@$i "firewall-cmd --zone=trusted --change-interface=eth0" ; done to see whether it actually worked).

for i in $hosts; do echo $i && ssh root@$i "firewall-cmd --get-zone-of-interface=eth0"; done shows

openqaworker21
trusted
openqaworker22
trusted
openqaworker23
public
openqaworker24
trusted
openqaworker25
trusted
openqaworker26
trusted
openqaworker-arm21
public
openqaworker-arm22
public

for i in $hosts; do echo $i && sshpass -p $password ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password root@$i "firewall-cmd --permanent --zone=trusted --change-interface=eth0"; done
for i in $hosts; do echo $i && sshpass -p $password ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password root@$i "firewall-cmd --get-zone-of-interface=eth0"; done

So that should resolve the zone for now, see https://progress.opensuse.org/projects/openqav3/wiki/Wiki#Manual-command-execution-on-o3-workers for the $hosts

Btw I also checked cat /etc/sysconfig/network/ifcfg-eth0 and it includes trusted for all hosts

livdywan wrote in #note-40:

ggardet_arm wrote in #note-39:

Still failing, see:
https://openqa.opensuse.org/tests/4098745#step/ovs_server/19

Okay. Maybe this is a difference issue then. I'm taking another look.

Looks like openqaworker-arm21 vs openqaworker-arm22

ggardet_arm wrote in #note-41:

livdywan wrote in #note-40:

ggardet_arm wrote in #note-39:

Still failing, see:
https://openqa.opensuse.org/tests/4098745#step/ovs_server/19

Okay. Maybe this is a difference issue then. I'm taking another look.

Looks like openqaworker-arm21 vs openqaworker-arm22

Good catch! Apparently something once again reverted the zone on arm22 to public? Even though I'd just checked and fixed it... well, I fixed it once more. Let's see if this resolves the error. I'm keeping an eye on the worker as well this time - https://openqa.opensuse.org/tests/4099199#live is on arm22

https://openqa.opensuse.org/tests/latest?arch=aarch64&distri=opensuse&flavor=DVD&machine=aarch64&test=ovs-server&version=Tumbleweed#next_previous and https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=wireguard_client&version=Tumbleweed#next_previous still look good.

I also created https://github.com/os-autoinst/os-autoinst/pull/2491 to tackle the source of the configuration problem.

The scenarios still look good. I'll close this ticket on Monday if it still looks good then.

Please monitor for longer as we overlooked multiple problems and shouldn't need to rely only on ggardet to inform us about related problems.