QA » openQA Project

I know that it does not help much , but I would remove this job https://openqa.suse.de/tests/6588107#step/boot_to_desktop/10 from description . boot to desktop failure hardly likely may be related to MM infra problems

this two also looks like unrelated to MM -
https://openqa.suse.de/tests/6588108#step/2_sw_multipath_s_aa/1
https://openqa.suse.de/tests/6588254#step/installation/18

I think it is important to solve one problem at the time

asmorodskyi wrote:

I know that it does not help much , but I would remove this job https://openqa.suse.de/tests/6588107#step/boot_to_desktop/10 from description . boot to desktop failure hardly likely may be related to MM infra problems

No, it's MM test, one node is preparing PXE second node is booting from it, it can be easily MM or network problem.

asmorodskyi wrote:

this two also looks like unrelated to MM -
https://openqa.suse.de/tests/6588108#step/2_sw_multipath_s_aa/1
https://openqa.suse.de/tests/6588254#step/installation/18

I think it is important to solve one problem at the time

First one can be whatever, second one is not MM, it could be related to network issues, they can be of course removed.
Adding new MM failures every day is no problem at all, unfortunately.
This were just examples of MM or network failures, only MM failures can be used, but when single job tests are failing due to network then MM can fail due to network or MM or both.

Yesterday evening I restarted two MM clusters, node1 & node2 on this cluster run on openqaworker10, I would not care about this detail, I picked this tests as they fail always. I will restart also another cluster running on random worker and collect tcpdump. Here are tcpdumps and some info from workers when the failures happened. ftp://10.100.12.155/MM/
I did do manual retry on qam_ha_rolling_upgrade_migration, the reason test was failing on multiple places is failure in name resolution, this fail happened also on scc registration, unfortunately video is not there to see it, but I had to go to network setup and make installation to re-setup the DHCP setup again, sometimes multiple times. Generally the MM/network/DNS was unable to resolve addresses like scc.suse.com or updates.suse.com. https://openqa.suse.de/tests/6628646#step/register_without_ltss/10 when I retried it passed.

qam_alpha_cluster
https://openqa.suse.de/tests/6628644
https://openqa.suse.de/tests/6628643
https://openqa.suse.de/tests/6628642

qam_ha_rolling_upgrade_migration
https://openqa.suse.de/tests/6628647
https://openqa.suse.de/tests/6628646
https://openqa.suse.de/tests/6628645

Sorry, I did not understand your last comment. Did you want to report about what you did or what you open to do? So do you think it would help to exclude openqaworker10 from all tests (not only multi-machine) and resolve #96260 first?

I tried to find the "fail ratio per worker" with SQL but so far have not found a good approach. Maybe something along the lines of:

select count(jobs.id),workers.host from jobs left join workers on jobs.assigned_worker_id = workers.id join (select count(j.id) as failed_jobs_count,host from jobs j join workers w on j.assigned_worker_id = w.id group by w.host) failed_jobs on workers.host = failed_jobs.host group by workers.host order by count desc;

might work. I guess I need to either conduct a subquery as join or the other way around.

Now did it semi-automatic. All failed per worker host:

select count(jobs.id),host from jobs left join workers on jobs.assigned_worker_id = workers.id where result = 'failed' group by host order by count desc;
 count |        host         
-------+---------------------
 13660 | 
 11749 | openqaworker5
 11173 | grenache-1
  8573 | openqaworker2
  7271 | openqaworker6
  5727 | openqaworker9
  5405 | openqaworker13
  5349 | openqaworker8
  4869 | openqaworker-arm-2
  4619 | openqaworker3
  3200 | openqaworker10
  3016 | openqaworker-arm-1
  3005 | openqaworker-arm-3
  2953 | QA-Power8-4-kvm
  2415 | QA-Power8-5-kvm
  2187 | powerqaworker-qam-1
  1304 | malbec
   360 | automotive-3
(18 rows)

and all

select count(jobs.id),host from jobs left join workers on jobs.assigned_worker_id = workers.id group by host order by count desc;                                                                                                                                  
 count  |        host         
--------+---------------------
 182809 | 
  76185 | openqaworker5
  58381 | openqaworker6
  44507 | openqaworker9
  41977 | openqaworker8
  38853 | grenache-1
  38732 | openqaworker-arm-2
  37838 | openqaworker3
  32992 | openqaworker13
  28221 | openqaworker-arm-3
  27405 | openqaworker2
  21533 | openqaworker10
  20928 | openqaworker-arm-1
  18804 | QA-Power8-5-kvm
  18454 | QA-Power8-4-kvm
  16240 | powerqaworker-qam-1
   9627 | malbec
   1397 | automotive-3
    287 | openqaworker11
(19 rows)

and all failed jobs regardless of worker host:

select count(jobs.id) from jobs where result = 'failed';
 count 
-------
 96835
(1 row)

and all jobs regardless of worker host:

select count(jobs.id) from jobs;                                                                                                                                                                                                                                   
 count  
--------
 715170
(1 row)

so the total fail ratio is 96835/715170 = 13.54%, the openqaworker10 specific fail ratio is 14.86% . An arbitrary other example is openqaworker5 with 15.42%, for grenache-1 28.76% (!). So as conclusion openqaworker10 does not show any significantly different fail ratio itself hence I don't think we need to exclude it.

okurz wrote:

Sorry, I did not understand your last comment. Did you want to report about what you did or what you open to do? So do you think it would help to exclude openqaworker10 from all tests (not only multi-machine) and resolve #96260 first?

I reported what I did, I don't think that only openqaworker10 does have problem. Not sure how does #96260 affect the issue, GRE tunnel is either added or not, if not there is no connection between workers, but failures I see are more like random connection drops.

With #96260 resolved I added #96191 to the backlog now

I tried to install/use ovs-test to debug the openvswitch.
But looks like the package openvswitch-test is broken.

# ovs-test -h
  File "/usr/bin/ovs-test", line 45
    print "Node %s:%u " % (node[0], node[1])
                      ^
SyntaxError: invalid syntax

There is also ovs-tcpdump or other ovs-* tools, but I didn't try if they work.
MM jobs are not failing so frequently anymore when the ticket was created, but there are still MM failures. Maybe it's network, I don't know.

Sorry missed the comment, yes I agree that the failure rate of MM is much better.
Existing or new failures should be handled separately.