QA » openQA Project » openQA Infrastructure

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/564

I tried to compare sapworker1 and worker5! and I can find some clue now.

GRE tunnels are not shown up in ip a command on sapworker1. [they can show up after I tried to restart wicked service]. hopefully my finding can help something.

https://openqa.suse.de/tests/11606807#dependencies The test passed on sapworker1 now.

I did below operation on this host:
sudo systemctl restart wicked

One more worker seems hit the same issue:openqaworker1
https://openqa.suse.de/tests/11603529#step/rsync_client/12

I had one test failing on openqaworker1, logged in there, gre tunnels were down, did restart wicked as suggested. Tunnels appeared on "ip a s", but had some tests failing on network issues:

https://openqa.suse.de/tests/11614943

emiura wrote:

I had one test failing on openqaworker1, logged in there, gre tunnels were down, did restart wicked as suggested. Tunnels appeared on "ip a s", but had some tests failing on network issues:

https://openqa.suse.de/tests/11614943

Not sure the command can help:

sudo ip link set gre0 up

There is some difference for firewall configuration:

openqaworker1:

> sudo grep -v ^#  /etc/firewalld/firewalld.conf

DefaultZone=trusted

MinimalMark=100

CleanupOnExit=yes

Lockdown=no

IPv6_rpfilter=yes

IndividualCalls=no

LogDenied=off

FirewallBackend=nftables

FlushAllOnReload=no

RFC3964_IPv4=yes

AllowZoneDrifting=no

worker5:

> sudo grep -v ^#  /etc/firewalld/firewalld.conf

DefaultZone=trusted

MinimalMark=100

CleanupOnExit=yes

Lockdown=no

IPv6_rpfilter=yes

IndividualCalls=no

LogDenied=off

AutomaticHelpers=system

This test ended up with name resolution failure, even with what appears to be a correct network setup:
https://openqa.suse.de/tests/11623555#step/hostname/25

I created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/568 for the SAP workers. I will apply this change manually tomorrow and re-run my test jobs again.

Not sure why openqaworker1 and worker5 were problematic. My SSH connection to openqaworker1 is so slow that I couldn't investigate much. Maybe the slow connection is actually also the source of the problem. On worker5 the correct bridge device is configured so it is not just an obvious mistake as on the SAP workers. However, after reading all the comments is seems that worker5 is actually not problematic anyways and was just mentioned as reference. So the suggestion "for all affected machines openqaworker1, worker5, sapworker{1-7}" from the ticket description is likely just misleading. Thus I will focus on the SAP workers for now.

mkittler wrote:

I created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/568 for the SAP workers. I will apply this change manually tomorrow and re-run my test jobs again.

Not sure why openqaworker1 and worker5 were problematic. My SSH connection to openqaworker1 is so slow that I couldn't investigate much. Maybe the slow connection is actually also the source of the problem. On worker5 the correct bridge device is configured so it is not just an obvious mistake as on the SAP workers. However, after reading all the comments is seems that worker5 is actually not problematic anyways and was just mentioned as reference. So the suggestion "for all affected machines openqaworker1, worker5, sapworker{1-7}" from the ticket description is likely just misleading. Thus I will focus on the SAP workers for now.

@mkittler, so far, I didn't hit any issues on worker5. let me remove it in description part. since it may lead some confustion.

Thanks for the clarification.

I followed-up the Slack chats and apparently there are networking problems (it works but with huge package loss). That would explain the poor SSH connection I had to openqaworker1 perfectly yesterday perfectly. I will only be able to investigate the situation with openqaworker1 once that problem has been resolved.

The same counts unfortunately for the SAP workers as well because those are also in FC basement and at this point also affected equally bad.

As Marius mentioned being a little stuck in the daily, just as an idea #132137 or #132134 might useful to look into as a reference since it's also about multi-machine tests.

Maybe it would be best for now to remove the tap worker class of openqaworker1. I created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/571 for that.

Yesterday I investigated the situation on the SAP workers but could not find out anything useful. Here are nevertheless the findings:

• iptables-save does not show masquerading enabled but that's also how it is on worker8 and 10 and likely ok considering that firewalld is actually using ntf under the hood.
• nft list ruleset shows masquerading-related rules. I computed a diff of the output of that command between sapworker1 worker8 and 10 and besides the name of the ethernet interface there is no difference at all.
• The SUTs can ping each other and the gateway (the bridge interface on sapworker1) but cannot reach any other hosts (which should supposedly work via the masquerading but does not).
• Enable masquerading via iptables so that iptables-save shows -A POSTROUTING -j MASQUERADE followed by COMMIT doesn't change anything.

Same proposal for the SAP workers: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/572

We can wait for #133025 and #132137 where we try to understand better how to do a consistent multi-machine setup and debugging approach

#133025 to be resolved soon. So we can probably unblock? Maybe re-estimate since some time has passed?