action #160646
Updated by livdywan 7 months ago
## Observation Originally reported in https://suse.slack.com/archives/C02CANHLANP/p1716169544132569 > (Richard Fan) Hello experts, many Multi-machine tests are failed like MM failed jobs on qe-core (edited) After that as visible in https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=1716124476616&to=1716212983933&viewPanel=24 2024-05-19 22:30 there is an increase of parallel_failed. E.g. openQA test in scenario sle-15-SP3-Server-DVD-Updates-x86_64-qam_kernel_multipath@64bit fails in [multipath_iscsi](https://openqa.suse.de/tests/14377269/modules/multipath_iscsi/steps/15) shows that it's not a problem with MTU as the error message is "connect: Network is unreachable" `ssh worker29.oqa.prg2.suse.org "cat /etc/wicked/scripts/gre_tunnel_preup.sh"` shows a problem ``` #!/bin/sh action="$1" bridge="$2" # enable STP for the multihost bridges ovs-vsctl set bridge $bridge stp_enable=false ovs-vsctl set bridge $bridge rstp_enable=true for gre_port in $(ovs-vsctl list-ifaces $bridge | grep gre) ; do ovs-vsctl --if-exists del-port $bridge $gre_port ; done ``` there should be a list of GRE tunnel interface setup calls after the last line between those machines but no GRE tunnels are setup at all ## Reproducible Fails since https://openqa.suse.de/tests/overview?result=parallel_failed&distri=sle&version=15-SP4&build=20240519-1 ## Expected result Last good: [20240519-1](https://openqa.suse.de/tests/14362720) (or more recent) ## Suggestions * *DONE* Mitigate -> https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/814 * Apply workarounds * Retrigger retrigger affected jobs * Investigate error source * Fix the problem or at least abort the generation with error if the section would be completely empty? * Prevent the same and similar problems in the future * Apply rollback steps * Monitor effect carefully ## Rollback steps * Revert https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/814 ## Further details Always latest result in the originally mentioned scenario: [latest](https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=qam_kernel_multipath&version=15-SP3)