action #152389
Updated by mkittler 5 months ago
## Observation openQA test in scenario sle-15-SP5-Server-DVD-Updates-x86_64-qam_kernel_multipath@64bit fails in [multipath_iscsi](https://openqa.suse.de/tests/13018864/modules/multipath_iscsi/steps/23) ## Test suite description Testsuite maintained at https://gitlab.suse.de/qa-maintenance/qam-openqa-yml. Maintainer: jpupava on 15sp1 is problem missing python-xml package ## Reproducible Fails since (at least) Build [20231210-1](https://openqa.suse.de/tests/13018864) (current job) ## Expected result Last good: [20231208-1](https://openqa.suse.de/tests/13010854) (or more recent) ## Acceptance criteria * **AC1:** failed+parallel_failed on https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&viewPanel=24 is significantly below 20% again * **AC2:** same as AC1 but also after the next weekend and worker host reboots ## Problem Pinging (as of certain sizes via `-s` parameter) and certain traffic (e.g. SSH) hangs when using via GRE tunnels (the MM test setup). TBD ## Suggestions Debug in VMs (using the developer mode or by creating VMs manually) as we have already started in #152389#note-10 an subsequent comments. The mentioned scenario is an easy reproducer but not the only affected scenario. Use e.g. ``` select distinct count(jobs.id), array_agg(jobs.id), (select name from job_groups where id = group_id), (array_agg(test))[1] as example_test from jobs left join job_dependencies on (id = child_job_id or id = parent_job_id) where dependency = 2 and t_finished >= '2023-12-05T18:00' and result in ('failed', 'incomplete') and test not like '%:investigate:%' group by group_id order by count(jobs.id) desc; ``` to find possibly also affected and relevant scenarios. TBD ## Rollback steps 1. Revert https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/693 disabling all tap classes except one x86_64 worker hosts 2. Revert https://gitlab.suse.de/openqa/salt-pillars-openqa/-/commit/4be80b2c720f6023b20355c9f4ac71096dc0aee4 3. Remove silence from https://monitor.qa.suse.de/alerting/silences "alertname=Ratio of multi-machine tests by result alert" ## Further details Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=qam_kernel_multipath&version=15-SP5)