QA » openQA Project » openQA Tests

I suspect that the mistake in the reverse PTR that should be fixed by https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/4043 are at fault here so let's wait for that MR. In the meantime I am trying to handle related openQA job failures.

Trying to mitigate with

for i in malbec powerqaworker-qam-1 petrol; do env WORKER=$i result="result='failed'" failed_since=2023-09-18 host=openqa.suse.de bash -ex openqa-advanced-retrigger-jobs; done

not sure if this is the only reason. I ran a state.highstate on malbec and can confirm that after a successful run petrol is missing in the gre_tunnel_preup.sh-script. We just check for "tap" in WORKER_CLASS so even with your changes it still should so up. So I suspect some issue in https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch.sls?ref_type=heads#L36-38

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/620

okurz wrote in #note-4:

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/620

I've fixed the underlying problem in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/997

It makes some aarch64 tests to restart, such as: https://openqa.opensuse.org/tests/3616202#comments

Cloned some tests with openqa-clone-job --within-instance https://openqa.suse.de --apikey D3576CEBF3529E39 --apisecret 63CF71588E55E56B 12230681 _GROUP=0 WORKER_CLASS=tap_poo136130. Unfortunately I messed up and _GROUP=0 still assigned a job-group. https://openqa.suse.de/tests/12376246 is now a validation for "all mm test on petrol-1".

nicksinger wrote in #note-14:

Cloned some tests with openqa-clone-job --within-instance https://openqa.suse.de --apikey D3576CEBF3529E39 --apisecret 63CF71588E55E56B 12230681 _GROUP=0 WORKER_CLASS=tap_poo136130. Unfortunately I messed up and _GROUP=0 still assigned a job-group. https://openqa.suse.de/tests/12376246 is now a validation for "all mm test on petrol-1".

use openqa-clone-job --parental-inheritance --skip-chained-deps --within-instance https://openqa.suse.de 12230681 _GROUP=0 WORKER_CLASS=tap_poo136130 TEST+=-poo136130 BUILD=poo136130

Created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/645 and https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/4160 to get rid of the suffix in the hostname and openqa

Both merged, what's next?

The host diesel can complete jobs while called diesel in theory which can be seen here: https://openqa.suse.de/tests/12443043. But as visible in https://openqa.suse.de/admin/workers/3388 I stumbled over an odd behavior where the host changes its hostname back to diesel-1 after some time (therefore the incomplete, the name changed -> salt changed the config -> worker restarted mid job run to reload changes). I tried to set DHCLIENT_SET_HOSTNAME='no' in /etc/sysconfig/network/ifcfg-eth3 and /etc/sysconfig/network/dhcp but still observed the changing hostname after some longer time period (some hours) which is odd. I found some hints in the wicked man-pages regarding a wicked extension but I'm not sure if this is related to the sysconfig options or where else to configured it so I asked in https://suse.slack.com/archives/C02D92APKNU/p1697550296052419 for some help what I could be missing.

I verified all settings again and rebooted the machine which is running for 1h now without changing back. I started a tcpdump for dhcp packages now to check if I can correlate the hostname change with a dhcp package coming in (e.g. after lease expiration), the command used is: tcpdump -i eth3 port 67 or port 68 -e -n -vvvv -w /root/dhcp_hostname_change

I ran another test by masking systemd-hostnamed but still the contents of /etc/hostname changed so apparently some tool writes to that file directly. Anyhow, guess this completes my experiments and I will just try to get the workers back online by changing our DHCP/DNS entries to refer only to the host itself without any interface enumeration. I will prepare a MR in the ops repo.

https://gitlab.suse.de/openqa/salt-states-openqa/-/pipelines/845191 failed to to petrol not having the correct highstate. I fixed it manually by running an explicit highstate and the pipeline/job succeeded: https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1914672

DNS/DHCP for both diesel and petrol are now changed back to without any number suffix. I tried to validate MM test yesterday and unfortunately they fail cross-machine: https://openqa.suse.de/tests/12639783#step/multipath_iscsi/26 - not sure what this is caused by. Single machine tests however complete successfully: https://openqa.suse.de/tests/12638911 and therefore I will add the normal qemu_ppc64 class into production while keeping MM disabled for now.

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/659 should help the ppc64le queue for single machine tests

My changes in the OPS salt didn't apply, now petrol and diesel get random IPs assigned causing the pipeline to fail: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1929411
I created https://sd.suse.com/servicedesk/customer/portal/1/SD-136220 for help

I had to manually apply the salt-states on walter1 and walter2 by issuing salt-call state.apply over ssh on them. With that I was able to recover both workers and they worked flawlessly on single-machine jobs in OSD. However, diesel seems to be very unstable and experiences random crashes. Petrol seems stable at least.

yes, diesel is running an updated kernel when it shouldn't. running the revert now

zypper rm -u kernel-default-extra-5.14.21-150500.55.31.1.ppc64le kernel-default-5.3.18-150300.59.93.1.ppc64le kernel-default-5.14.21-150500.55.31.1.ppc64le kernel-default-optional-5.14.21-150500.55.31.1.ppc64le && zypper al -m "poo#119008, kernel regression boo#1202138" kernel* && sync && reboot

machine was unable to boot because no kernel and no initrd were installed. I used the petitboot rescue shell to kexec into a leap live system and chrooted into the system. Afterwards I added the Leap15.3 repos with lower prio:

zypper ar -p 105 http://download.opensuse.org/distribution/leap/15.3/repo/oss/ repo-oss-15.3
zypper ref repo-oss-15.3

after that I was able to force install the old kernel from 15.3 with:

zypper in -f kernel-default-5.14.21-150500.55.31.1

which regenerated all necessary files again. After a reboot petitboot was able to find the system again and I could successfully start the normal system which is online again and already assigned all resources to pending jobs. I keep an eye on the stability of that system.

I checked that both diesel+petrol execute tests just fine. https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1 Shows a critically long ppc64le job queue but openQA is also working down the queue.

Related SD-137152

livdywan wrote in #note-36:

Related SD-137152

For that and the general problem of longer ppc64le job queue I created #139010.

Next steps:

1. Update description to not include malbec+powerqaworker-qam-1 anymore as they are offline and will stay offline
2. Ensure multi-machine tests are working on at least one of diesel+petrol and enable multi-machine production use
3. Check multi-machine tests across diesel and petrol and ensure multi-machin tests work using GRE tunnels
4. Continue with the other steps in ticket description suggestion section

I scheduled some test jobs to check the current state: https://openqa.suse.de/tests/12804036#dependencies

Notes from our collaborative session:

• Ticket about ensuring salt-based setup works https://progress.opensuse.org/issues/136013
• First firewall fix https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/987
• Explicit net.ipv4 fix https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1004
• Confirm our docs https://docs.openvswitch.org/en/latest/howto/tunneling/#testing
• Prepare a simplified test e.g.
  • based on wicked involving its own dhcp and clear checking of IPv4 and IPv6 setup
  • extend openqa_worker tests to check that os-autoinst-setup-multi-machine actually created a working network setup, by adding a second worker and checking that one can talk to the other https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/979ea1766de9722a6fa1603c6a2a15c1c108c4ea/tests/install/openqa_worker.pm#L19
  • See how we can test the same case on x86_64 and ppc - in practice many if not most tests are never run on power
• Come up with a verifier based on Open vSwitch troubleshooting guidelines
• Consider double-checking a possible relation to other mm issues #139070 #138707 #139154
• Check if the -1 suffix in workerconf is (ir)relevant afterall
• Verify with https://openqa.suse.de/tests/12188617#step/iscsi_client/10
• Implement openqa-cli support for triggering of jobs on one or multiple specific workers

I cloned the mentioned scenario with worker classes for diesel/petrol: https://openqa.suse.de/tests/12810996

By the way, despite the tap worker class being removed from petrol it looks like the MM configuration is up-to-date enough on that machine. (The IPs of petrol/diesel are configured correctly on either end.)

EDIT: The jobs have finished now. The server ran on petrol and the other client job that ran there as well did not fail in iscsi_client which seems to be the critical module. The other client job that ran on diesel failed in that module (and the other finished as parallel_failed). So the issue is definitely still reproducible.

Just for the record, I cloned the jobs by running sudo openqa-clone-job --skip-download --export-command --skip-chained-deps https://openqa.suse.de/tests/12799476 {TEST,BUILD}+='test-ppc-mm' _GROUP=0 on OSD and then executing the returned command with worker classes replaced accordingly.

I invoked the ovs-vsctl commands on mania/petrol/diesel manually to allow traffic in all directions.

I scheduled one cluster between diesel and mania (https://openqa.suse.de/tests/12816967#dependencies) and one between petrol and mania (https://openqa.suse.de/tests/12816974#dependencies).

The diesel/mania jobs have already finished reproducing the issue. The petrol/mania jobs are already passed the critical module not reproducing the issue.

That means diesel is the problem.

Note that the "roles" of the different hosts were identical. So this could not have made a difference. (I just replaced "diesel" with "petrol" in the worker class assignments when invoking the 2nd API call for petrol.)

Or I messed up the ovs-vctl commands for the connection between diesel/mania. Just for the record, I invoked the following commands:

martchus@mania:~> sudo ovs-vsctl --may-exist add-port br1 gre20 -- set interface gre20 type=gre options:remote_ip=10.168.192.252 # mania -> diesel
martchus@mania:~> sudo ovs-vsctl --may-exist add-port br1 gre21 -- set interface gre21 type=gre options:remote_ip=10.168.192.254 # mania -> petrol
martchus@diesel:~> sudo ovs-vsctl --may-exist add-port br1 gre21 -- set interface gre21 type=gre options:remote_ip=10.168.192.108 # diesel -> mania
martchus@petrol:~> sudo ovs-vsctl --may-exist add-port br1 gre21 -- set interface gre21 type=gre options:remote_ip=10.168.192.108 # petrol -> mania

(The connection between diesel and petrol was already in place. I reused gre20/gre21 for simplicity because we don't need a connection to the x86_64 worker using this gre interface anyways. I have also checked that the IPs show up in the output of ovs-vsctl show.)

Considering I haven't done anything differently on diesel I don't think that's the case, though.

ok, so what's your plan for a next step?

All petrol/mania jobs have now finished successfully. As a first improvement I can create a MR to enable the tap worker class on petrol and mania.

I restarted the petrol/mania cluster to see whether it wasn't just luck: https://openqa.suse.de/tests/12817482

I created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/677 to enable the tap worker class on petrol.

I made a diff of sysctl -a | sort on petrol and diesel. The most interesting output of colordiff -Naur sysctl-petrol sysctl-diesel:

-kernel.hostname = petrol
+kernel.hostname = diesel
-kernel.osrelease = 5.3.18-150300.59.93-default
+kernel.osrelease = 5.3.18-57-default
-net.ipv4.conf.erspan0.arp_notify = 0
+net.ipv4.conf.erspan0.arp_notify = 1
-net.ipv4.tcp_mem = 195231      260310  390462
+net.ipv4.tcp_mem = 195273      260364  390546
-net.netfilter.nf_conntrack_tcp_ignore_invalid_rst = 0

Otherwise the diff mainly shows hardware differences (like a different number of CPU cores) and different UUIDs and a little bit lower limits like:

-kernel.pid_max = 114688
+kernel.pid_max = 65536

The output of ip r, ip a and ovs-vsctl show also doesn't show any relevant differences (but maybe Dirk as more input on that).

mkittler wrote in #note-56:

I made a diff of sysctl -a | sort on petrol and diesel. The most interesting output of colordiff -Naur sysctl-petrol sysctl-diesel:

-kernel.hostname = petrol
+kernel.hostname = diesel
-kernel.osrelease = 5.3.18-150300.59.93-default
+kernel.osrelease = 5.3.18-57-default

the kernel version 5.3.18 is correct as both should be downgraded due to #119008, kernel regression boo#1202138, however the kernel version on diesel looks like its the outdated GM version and should be the same as on petrol. Please install 5.3.18-150300.59.93-default and check multi-machine tests again.

Ok, I installed http://download.opensuse.org/update/leap/15.3/sle/ppc64le/kernel-default-5.3.18-150300.59.93.1.ppc64le.rpm.

After installing the other kernel package and rebooting the diff in sysctl is gone. I scheduled https://openqa.suse.de/tests/12840274#dependencies. Let's see whether it works.

I removed all remaining diesel-1:* slots from OSD. All were offline so I guess the "-1" problem is resolved. salt-key -L and sudo salt openqa.suse.de mine.get roles:worker nodename grain also don't show any names with "-1" anymore.

https://stats.openqa-monitor.qa.suse.de/d/WDdiesel-1 also shows no data (as opposed to https://stats.openqa-monitor.qa.suse.de/d/WDdiesel). Maybe I should cleanup the old dashboard (and check why it doesn't happen automatically).

The new cluster is already at the point where we can call it a success. So the different kernel version might have surprisingly made a difference. Of course I'll retry the jobs a few times to be sure.

About the dashboards: They are not provisioned anymore by salt but one has to remove them still manually as non-provisioned dashboards (which I've just did). I think we actually have already established that this is how it works at some point before (and changing this would be out of scope for this ticket anyways).

It still doesn't work after all. Before the test scenario always failed in iscsi_client and now that module softfailed (which seems generally as good as it gets for this module at this point) and watchdog passed but then ha_cluster_join failed¹. So not so successful after all. I'm restarting the cluster and investigate a little bit as I'm wondering what the difference is now.

¹ test-ppc-mm-ha_ctdb_node02test-ppc-mm (test/VM on diesel) cannot ping test-ppc-mm-ha_ctdb_node01test-ppc-mm (test/VM on mania) via ping -c1 ctdb-node01 due to ping: ctdb-node01: Temporary failure in name resolution.

I've got the same outcome again. I've restarted the cluster once more to be sure the test outcome has changed in a stable way: https://openqa.suse.de/tests/12841664#dependencies