QA » openQA Project

I created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/687 to disable MM tests on worker35 and 36 as these are the most problematic (see query above). Also, after shutting down these machines yesterday "all jobs run without problem except one" which is definitely better than it was with these machines running MM tests.

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/687 merged. Added according rollback step to #139103

I also powered the machines on again and applied the salt states immediately and checked that the tap worker class is not there.

For future reference: One of the problematic scenarios was https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-HA-Updates&machine=64bit&test=qam_alpha_cluster_01&version=15-SP2 - In the error case the hostname module already shows that there's no IP assigned and downloads fail later on (the hostname module itself doesn't fail; supposedly it would be good if this test would fail early).

Just for comparison:

• In passing runs an IP is assigned on eth0 and the mtu is 1458.
• In the failing runs no IP is assigned on eth0 and the mtu is 1500. The interface is shown as up at least.

Considering our discussion with Dirk it is good that the mtu is lowered and maybe the fact that it is not when the test is failing is the culprit. However, it could also just be a symptom. I'm also not sure where the mtu of 1450 that is shown in the good case comes from (it is supposedly either configured automatically or the test code somehow tries to set it).

The MTU is actually lowered manually by tests via https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/data/wicked/dhcp/dhcpd.conf. The configure_static_ip helper for MM tests defined in https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/mm_network.pm does this as well. The support server setup is also taking care of lowering the MTU in https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/support_server/setup.pm.

Now the question is whether any of this code is executed in the problematic test scenarios. If not I'm inclined to blame the tests.

EDIT: It looks like the support server in this scenario (qam_alpha_supportserver@64bit) would execute the configure_static_ip and then the IP and MTU also shows up correctly (also when run as part of the failing cluster), e.g. https://openqa.suse.de/tests/12876164#step/setup/12 and https://openqa.suse.de/tests/12876164#step/setup/26. The other nodes don't seem to make an effort to lower the MTU. Maybe that's the missing bit that would help to stabilize these tests? It would still be strange that it is sometimes not necessary.

(In the problematic cluster the other node that ran on worker29 had a lowered MTU and IP address assigned despite no explicit setup. Strange that it on some nodes/workers it just works nevertheless. Supposedly this node really only parallel_failed because the other node on worker35 failed. This leaves one really wondering what the difference between "good" workers like 29 and "bad" workers like 35 and 36 is.)

based on https://docs.openvswitch.org/en/latest/faq/issues/ "Q: How can I configure the bridge internal interface MTU? Why does Open vSwitch keep changing internal ports MTU?" I just executed ovs-vsctl set int br1 mtu_request=1450 on worker 29,35,39 where @mkittler now triggered a new test cluster. If it does not work it can be reverted with: ovs-vsctl set int br1 mtu_request=[]

Here's the restarted job cluster to see whether @nicksinger's changed made a difference (scheduled so that all jobs run on the same nodes as before where previously the job on worker35 failed): https://openqa.suse.de/tests/12909195

2 more clusters to see whether results are consistent: https://openqa.suse.de/tests/12909617, https://openqa.suse.de/tests/12909619

making mtu size application persistent

• https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1055
• https://github.com/os-autoinst/os-autoinst/pull/2409

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1055 is now merged. Now we should monitor for the impact on tests

@mkittler I suggest you schedule some more multi-machine test clusters.

We enabled the mtu_request setting on all workers via salt. Let's see what impact this has. If everything looks good we can merge https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/689 and monitor the fail ratio of MM jobs by hosts again to see whether the fail ratio of w35 and w36 has dropped.

I created 50 more runs (https://openqa.suse.de/tests/overview?version=15-SP2&distri=sle&build=20231127-1-test-mm-1) to run on any host but only realizing now that none of the jobs will run on w35 or w36 because we disabled tap there.

So I created 25 more runs to run across w35 and w36 (https://openqa.suse.de/tests/overview?build=20231127-1-test-mm-2&distri=sle&version=15-SP2). This way we can now at least compare the fail ratio between w35/w36 and all others.

The pass-rate is a surprising 100 % in both groups (w35/w36, others). So I would say that w35/w36 look good enough to enable them again in production for MM jobs.

Here's the fail/incomplete ratio of jobs with parallel dependencies by worker as of yesterday:

openqa=> select distinct count(jobs.id) as total, sum(case when jobs.result in ('failed', 'incomplete') then 1 else 0 end) * 100. / count(jobs.id) as fail_rate_percent, host from jobs left join job_dependencies on (id = child_job_id or id = parent_job_id) join workers on jobs.assigned_worker_id = workers.id where dependency = 2 and t_started >= '2023-11-29' group by host having count(jobs.id) > 50 order by fail_rate_percent desc;
 total |  fail_rate_percent  |    host     
-------+---------------------+-------------
    51 | 37.2549019607843137 | worker-arm1
   468 | 26.7094017094017094 | worker38
   437 | 24.7139588100686499 | worker29
   552 | 24.4565217391304348 | worker40
   396 | 22.7272727272727273 | worker39
   401 | 22.6932668329177057 | worker37
   326 | 22.0858895705521472 | worker35
   385 | 21.2987012987012987 | worker30
   331 | 18.4290030211480363 | worker36
(9 rows)

So w35 and w36 are not on top (with a gap) anymore (this role has now worker-arm1). That's good for w35/w36 so I'm keeping them enabled for MM jobs. Maybe changing the MTU setting really helped. I'll do this query again later/tomorrow.

Now the fail ratio went a bit down:

openqa=> select distinct count(jobs.id) as total, sum(case when jobs.result in ('failed', 'incomplete') then 1 else 0 end) * 100. / count(jobs.id) as fail_rate_percent, host from jobs left join job_dependencies on (id = child_job_id or id = parent_job_id) join workers on jobs.assigned_worker_id = workers.id where dependency = 2 and t_started >= '2023-11-29' group by host having count(jobs.id) > 50 order by fail_rate_percent desc;
 total |  fail_rate_percent  |    host     
-------+---------------------+-------------
   805 | 22.4844720496894410 | worker38
   138 | 22.4637681159420290 | worker-arm2
   737 | 19.9457259158751696 | worker30
   772 | 19.8186528497409326 | worker29
   691 | 19.3921852387843705 | worker39
   150 | 19.3333333333333333 | worker-arm1
   905 | 18.7845303867403315 | worker40
   644 | 18.0124223602484472 | worker37
   642 | 17.7570093457943925 | worker35
   608 | 15.7894736842105263 | worker36
(10 rows)

w35 and w36 and now even both at the bottom.

Not much has changed except that openqaworker18.qa.suse.cz is not running MM jobs and failing a lot with it:

openqa=> select distinct count(jobs.id) as total, sum(case when jobs.result in ('failed', 'incomplete') then 1 else 0 end) * 100. / count(jobs.id) as fail_rate_percent, host from jobs left join job_dependencies on (id = child_job_id or id = parent_job_id) join workers on jobs.assigned_worker_id = workers.id where dependency = 2 and t_started >= '2023-11-29' group by host having count(jobs.id) > 50 order by fail_rate_percent desc;
 total |  fail_rate_percent  |      host      
-------+---------------------+----------------
    80 | 25.0000000000000000 | openqaworker18
  1162 | 19.4492254733218589 | worker38
   822 | 18.1265206812652068 | worker-arm1
   997 | 17.7532597793380140 | worker39
  1173 | 17.3060528559249787 | worker29
  1276 | 16.9278996865203762 | worker40
   742 | 16.4420485175202156 | worker-arm2
  1069 | 16.1833489242282507 | worker30
   254 | 16.1417322834645669 | mania
   909 | 15.8415841584158416 | worker37
   964 | 15.7676348547717842 | worker35
   936 | 15.0641025641025641 | worker36

EDIT: Looks like openqaworker18.qa.suse.cz has over the last days just executed enough jobs to be on the list at all. I'm filtering out all workers that have only run less than 50 jobs on purpose because with only very few jobs the fail ratio might not be very meaningful. I suppose I should increase the 50 as the time frame since 2023-11-29 increases. So I guess that worker appearing is not a big deal after all.

This all means that the fail rate is still higher than expected. Last time I had a closer look on the failures (on last Friday) it didn't seem to be due to a fundamental issue with the MM setup anymore, though. Likely the higher fail rate is mainly due to #151612. So maybe it makes sense to wait until this issue has been resolved.

Update for os-autoinst: https://github.com/os-autoinst/os-autoinst/pull/2413

I would not go for the biggest possible value to avoid problem "A" and rather go for a round number in the middle to have wiggle room in both directions. If I'd go to an extreme it would actually be the lowest value we can set for the bridge (which would be 1458 for openSUSE tests unless we adjust them to make the MTU even lower) because an error in that direction will show immediately/consistently and not just sporadically and is thus potentially easier to debug.

The graph looks good again for the last 24 hours (AC1): https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&viewPanel=24&from=now-24h&to=now