QA » openQA Project » openQA Tests

I was able to pause 2 jobs (https://openqa.suse.de/tests/12918352 & https://openqa.suse.de/tests/12918351) running in worker38 & worker39 while they were failing, and could find out the following by testing directly from one of the SUTs:

• SUT can resolve scc.suse.com
• scc.suse.com is not reachable via ICMP, but that's expected
• curl scc.suse.com works
• curl https://scc.suse.com times out
• curl -k https://openqa.suse.de works
• SUT can reach support server
• SUT can reach other node
• SUT can reach updates.suse.com via HTTP and HTTPS
• SUT can reach download.suse.de via HTTP and HTTPS
• SUT can not reach registry.suse.com via HTTPS

I also checked the following in worker39 where that SUT was running:

• curl https://scc.suse.com works
• sysctl -a show that IPv4 forwarding is enabled for eht0, br1 and tap interfaces
• Not sure if this is a problem, but nft list table firewalld in w39 does not show tap interfaces in nat_PREROUTING_ZONES:

        chain nat_PREROUTING_ZONES {
                iifname "eth0" goto nat_PRE_trusted
                iifname "ovs-system" goto nat_PRE_trusted
                iifname "br1" goto nat_PRE_trusted
                iifname "docker0" goto nat_PRE_docker
                goto nat_PRE_trusted
        }

• Nor in nat_POSTROUTING_ZONES:

    chain nat_POSTROUTING_ZONES {
        oifname "eth0" goto nat_POST_trusted
        oifname "ovs-system" goto nat_POST_trusted
        oifname "br1" goto nat_POST_trusted
        oifname "docker0" goto nat_POST_docker
        goto nat_POST_trusted
    }

I added some screenshots from the SUT's tests in https://suse.slack.com/archives/C02CANHLANP/p1701181092050239

okurz wrote in #note-2:

Oh, nice investigation. That looks very specific about the host and port. So I suggest you open an SD ticket mentioning that likely the firewall is blocking traffic there.

looks like it could have been a network hiccups: https://openqa.suse.de/tests/12925142 https://openqa.suse.de/tests/12925143

See: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18221

szarate wrote in #note-3:

looks like it could have been a network hiccups: https://openqa.suse.de/tests/12925142 https://openqa.suse.de/tests/12925143

We saw many failures over the night. Just in case it was network hiccups, I'm restarting the jobs. Will comment here later.

Issue seems to be still present:

• https://openqa.suse.de/tests/12925775#step/register_system/59
• https://openqa.suse.de/tests/12925771#step/register_system/59

Error is not exactly the same as in the jobs I used to create the poo# (those were doing registration via SUSEConnect, these are using yast2), but I figure root cause is the same.

More failures, these ones like the jobs reported yesterday:

• https://openqa.suse.de/tests/12925741#step/suseconnect_scc/20
• https://openqa.suse.de/tests/12925739#step/suseconnect_scc/20
• https://openqa.suse.de/tests/12925735#step/suseconnect_scc/20
• https://openqa.suse.de/tests/12925732#step/suseconnect_scc/20
• https://openqa.suse.de/tests/12925750#step/suseconnect_scc/20

okurz wrote in #note-2:

Oh, nice investigation. That looks very specific about the host and port. So I suggest you open an SD ticket mentioning that likely the firewall is blocking traffic there.

Would an SD ticket actually help?

As reported, https://scc.suse.com is reachable from the worker itself, but not from the VM running in the worker. As I said in the slack thread, I suspect there is something wrong with the NAT rules in the workers.

https://sd.suse.com/servicedesk/customer/portal/1/SD-140538

I created one

IMO it's broken openvswitch

I downgraded openvswitch packages to previous version with reboot on wroker18 and SUSEConnect/SSL works

It was probably just lucky run, maybe not related at all, next runs failed.

openqaworker18:~ # zypper se -s -x libopenvswitch-2_14-0 openvswitch
Loading repository data...
Reading installed packages...

S  | Name                  | Type       | Version               | Arch   | Repository
---+-----------------------+------------+-----------------------+--------+-------------------------------------------------------------
v  | libopenvswitch-2_14-0 | package    | 2.14.2-150400.24.14.2 | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
i+ | libopenvswitch-2_14-0 | package    | 2.14.2-150400.24.9.1  | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
v  | libopenvswitch-2_14-0 | package    | 2.14.2-150400.24.6.1  | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
v  | libopenvswitch-2_14-0 | package    | 2.14.2-150400.24.3.1  | x86_64 | Main Repository
v  | openvswitch           | package    | 2.14.2-150400.24.14.2 | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
i+ | openvswitch           | package    | 2.14.2-150400.24.9.1  | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
v  | openvswitch           | package    | 2.14.2-150400.24.6.1  | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
v  | openvswitch           | package    | 2.14.2-150400.24.3.1  | x86_64 | Main Repository
   | openvswitch           | srcpackage | 2.14.2-150400.24.14.2 | noarch | Update repository with updates from SUSE Linux Enterprise 15
   | openvswitch           | srcpackage | 2.14.2-150400.24.9.1  | noarch | Update repository with updates from SUSE Linux Enterprise 15
   | openvswitch           | srcpackage | 2.14.2-150400.24.6.1  | noarch | Update repository with updates from SUSE Linux Enterprise 15
openqaworker18:~ #

https://openqa.suse.de/tests/12941822#step/smt_server_install/21 smt
https://openqa.suse.de/tests/12941843#step/nfs_server/12 nfs

dzedro wrote in #note-9:

IMO it's broken openvswitch

I downgraded openvswitch packages to previous version with reboot on wroker18 and SUSEConnect/SSL works

Agree, it's not generic time out, but "TLS handshake timeout" so it's something fishy in the SSL, at lest in implementation of it.

I added you to that ticket as well, but nothing happened there so far

To exclude problems with the system/product we run within the VM I tried with a statically linked curl binary from https://github.com/moparisthebest/static-curl. It is also statically linked against OpenSSL and works outside of the VM. Within the VM it also reproduces the TLS handshake timeout. So it is very unlikely that a regression of the product we're testing is responsible for the problem. (It could of course still be a kernel regression but that's not very likely.)

By the way, with --verbose one gets

…
Connected to scc.suse.com (18.194.119.137) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

and then nothing.

I would like to add for the record, that SCC registration seems to work from Yast on sapworker2:14 :
https://openqa.suse.de/tests/12961803#step/scc_registration/43

but not at worker29:39 from SUSEConnect CLI: https://openqa.suse.de/tests/12961827#step/suseconnect_scc/22
where it ends on net/http: TLS handshake timeout

also is broken from SUSEConnect CLI on worker29:39: https://openqa.suse.de/tests/12961829#step/suseconnect_scc/20
but there it throw different error: Net::OpenTimeout

Indeed... MTU issue.

jlausuch@worker38:~> ip a|grep br1
5: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.0.2.2/15 brd 10.1.255.255 scope global br1

After setting the MTU to 1500 in br1, everything works again..

susetest:~ # curl -k https://scc.suse.com|head -10
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
<html lang='en' ng-app='' ng-strict-di='false'>
<head>
<title>SUSE Customer Center</title>
<meta content='initial-scale=1' name='viewport'>
<meta content='all' name='robots'>
<meta content='nWPQ59EF614zzwjOAiG7b1SXCUZKcu7ajpinvshy0xs' name='google-site-verification'>
<link rel="icon" type="image/x-icon" href="https://static.scc.suse.com/assets/favicon-b308f78dd95ea6e03778476d824701284771b296ffd48bfc677e3acc6a2f4db1.ico" />
<link rel="stylesheet" href="https://static.scc.suse.com/assets/application-15bf7ebc1f4a9032f945e051e46e95d63d06ac87d4cff543babe9e2ea4e1592e.css" media="all" data-turbo-track="reload" />
<link href='/humans.txt' rel='author' type='text/plain'>
 91 27229   91 24878    0     0   283k      0 --:--:-- --:--:-- --:--:--  285k

I know this PR is not the culprit as the issue was spotted before this change, but maybe we can force it to 1500 instead of 1450?
The tap devices are set to 1500, so I would assume br1 should be the same.

We've just did a similar test.

The problem with setting 1500 on the bridge is that this will cause problems with gre tunnels. At least I believe this is the case because:

• We have seen connectivity problems with the default setting. VM hosts were not getting an IP at all. This only happened sporadically so it is hard to tell anything for sure but it seemed that lowering the MTU to 1450 helped.
• The FAQ also suggests to lower the MTU like this for our use case (search for 1450 on https://docs.openvswitch.org/en/latest/faq/issues).

When setting the MTU to 1450 in the VM via e.g. ip link set dev eth0 mtu 1450 it works again. Normally the MTU within the VMs is set to 1458 which is a bit too high. Maybe we can set the MTU on the bridge to 1460 as also Dirk Müller suggested. This would be big enough for a MTU of 1458 within the VMs in will hopefully not break traffic via gre tunnels.

EDIT: MR: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1061

mkittler wrote in #note-23:

We've just did a similar test.

The problem with setting 1500 on the bridge is that this will cause problems with gre tunnels. At least I believe this is the case because:

• We have seen connectivity problems with the default setting. VM hosts were not getting an IP at all. This only happened sporadically so it is hard to tell anything for sure but it seemed that lowering the MTU to 1450 helped.
• The FAQ also suggests to lower the MTU like this for our use case (search for 1450 on https://docs.openvswitch.org/en/latest/faq/issues).

When setting the MTU to 1450 in the VM via e.g. ip link set dev eth0 mtu 1450 it works again. Normally the MTU within the VMs is set to 1458 which is a bit too high. Maybe we can set the MTU on the bridge to 1460 as also Dirk Müller suggested. This would be big enough for a MTU of 1458 within the VMs in will hopefully not break traffic via gre tunnels.

EDIT: MR: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1061

Ok, I missed the context and all what you have tried as part of #151310. Thanks for clarification, Marius.

All test jobs look good so far:

• The restarted production job cluster is most likely passed the problematic point.
• There are also no sporadic failures among the other MM jobs. (The cluster failing to to an incomplete job is because I didn't stop the one worker slot it ran on correctly when using its tap device for a test VM.)

I also created a PR for openqa-clone-job to make cloning test jobs easier in the future: https://github.com/os-autoinst/openQA/pull/5385

I also restarted https://openqa.suse.de/tests/12974495. Let's see whether it works as well.

Thanks a lot @mkittler and @jlausuch, very informative. I didn't suspect MTU issues due to the nature of the failure, but good to know that it can cause failures that look like a FW blocking packets. I'm keeping an eye to HA jobs and will report back if I see issues.

https://openqa.suse.de/tests/12980422 which ran today in the morning is still showing "SUSEConnect error: Get "https://scc.suse.com/connect/systems/activations": net/http: TLS handshake timeout" running on worker30. All OSD salt controlled workers have an MTU size of 1460 so we need to think further.

The MTU setting on worker30 is ok and the other production jobs I restarted succeeded. The problem with this particular scenario (https://openqa.suse.de/tests/12980422 / https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=qam-wicked_basic_ref&version=15-SP1) is that the MTU within the SUT is not be set to something <= 1460:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:12:09:f6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.10/15 brd 10.1.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe12:9f6/64 scope link 
       valid_lft forever preferred_lft forever

(from step https://openqa.suse.de/tests/12980422#step/before_test/152 and I could not find any further step that would change the MTU before the SUSEConnect invocation)

I see multiple options:

1. Not lowering the MTU on the bridge anymore and
   1. live with sporadic connectivity issues of the MM setup leaving #151310 unresolved.
   2. try to increase the maximum physical MTU using jumbo frames.
2. Ensure that all MM SUTs have an MTU of <= 1460 configured.

Considering we already do 2. it makes likely most sense to adapt the remaining test scenarios. The good thing is that these problems are always reproducible and considering the manual tinkering with the network these tests already do it should not be hard to set the MTU as well.

This PR will hopefully fix the mentioned wicked test scenario: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18264
EDIT: With the PR merged it now works in production, e.g. https://openqa.suse.de/tests/12982770.

PR to document the steps to reproduce this issue: https://github.com/os-autoinst/openQA/pull/5387