QA (public) » openQA Project (public) » openQA Infrastructure (public)

Setting due date based on mean cycle time of SUSE QE Tools

I used sapworker1 as reproducer for the issue at hand while having a terminal open with salt-minion -l debug (stopped the service before).
Logs showed a zeromq connection to OSD TCP/4505 while OSD connects back to the worker at TCP/4506. I used salt sapworker1.qe.nue2.suse.org saltutil.sync_grains on OSD to quickly trigger the issue. Salt looked fine but all of a sudden it stopped working and refused to talk back to OSD answering the "sync_grains"-request. Using wireshark and the remote-ssh-tracing, I found a lot of "TCP retransmissions" on the prg2wg interface. A noticeable pattern was the last packet before retransmissions always showing the max MTU of 1420 (the default of a wg-interface). This hinted to MTU problems. Using ping -I prg2wg -M do -s $((1420-28)) openqa.suse.de I was quickly able to reproduce the problem again - packets go out but no answer comes back (not even an ICMP-message signaling a need for fragmentation). Slowly lowering this 1420 bytes I was able to find the maximum path size to be 1284 with ping/ICMP, so 28 bytes more for TCP, meaning 1312 for the prg2wg-interface. I added my findings to the SD ticket because I still suspect the recent network changes to be the cause of this ("IPSec tunnels got adjusted" -> each tunnel usually lowers MTUs). As a workaround or possible configuration fix, I created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1404 which currently fails pipelines because of #178564

I applied the fix manually on all NUE2 machines with a wg tunnel. With that https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/3964772 passed and salt is able to properly deploy again. I'm keeping the MR (and ticket here) open until #178564 is fixed because I would like to see proper tests first.

Merged but unfortunately running on too many workers and therefore breaking our pipeline: https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/4007282#L154 - trying to come up with a fix.

Deployment of https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1404 failed due to missing needs_wireguard-grains on backup-vm.qe.nue2.suse.org and baremetal-support.qe.nue2.suse.org - I now set it to "False" on these hosts. With a subsequent deployment in a different MR, the change got applied to all machines:

openqa:~ # salt -C '*.nue2.suse.org and not G@needs_wireguard:False' cmd.run 'cat /etc/wireguard/prg2wg.conf | grep -i mtu'
monitor.qe.nue2.suse.org:
    MTU = 1292
petrol.qe.nue2.suse.org:
    MTU = 1292
diesel.qe.nue2.suse.org:
    MTU = 1292
openqa-piworker.qe.nue2.suse.org:
    MTU = 1292
sapworker1.qe.nue2.suse.org:
    MTU = 1292

As discussed with nicksinger reopening. There are multiple pipelines either stalling and running into 2h timeout or weird salt issues: https://gitlab.suse.de/openqa/salt-states-openqa/-/pipelines

I tried with sudo count-fail-ratio salt \* saltutil.sync_grains and found no failure: count-fail-ratio: Run: 20. Fails: 0. Fail ratio 0±0%. No fails, computed failure probability < 15.00%. Probably we need to apply the complete high state.

I tried to manually ensure that the MTU of 1272 is deployed on all relevant machines and partially triggered reboots. Over salt the machines are reachable but over ssh only with a delay of multiple minutes, possibly as the lower MTU is not effective yet. Also couldn't find a stable way to start/restart the wireguard tunnels. Also getting the error "RTNETLINK answers: No such device". Could it be that https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1404/ removed some devices or something?
Anyway, to not constantly fail salt pipelines I have for now removed all NUE2 wg hosts from salt and added a silence. Mind the corresponding rollback actions please

Setting due date based on mean cycle time of SUSE QE Tools

From mmoese in https://suse.slack.com/archives/C02CANHLANP/p1742810337597879 #eng-testing

@qa-tools I see a lot of jobs being abandoned by sapworker1 (like https://openqa.suse.de/tests/17128750 )

I am convinced this is related to this ticket, raising to "Urgent"

So for now most of the machines are online and in salt again without wireguard enabled but with proper MTU settings. It turned out that 1280 is the minimum MTU required for v6 to work. This is the reason why https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1420 broke a lot over night. I run an experiment on sapworker1 with having NFS disabled. With "openqa_share_nfs: False" in /etc/salt/grains we can enable sapworker1 in salt again. https://openqa.suse.de/tests/17138601 is the first test in this setup. https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/4044163#L115 is failing again due to XML-errors but passed sapworker1 quite fast at least.
The same for diesel. I will apply the same on petrol tomorrow after reviewing some openQA jobs running on these machines.