QA (public) » openQA Project (public) » openQA Infrastructure (public)

https://sd.suse.com/servicedesk/customer/portal/1/SD-174272

Apparently we misunderstood enginfra and the wg-tunnel does not allow us back-and-forth communication. Even with the tunnel in place the connection has to be established from within a CC-compliant area. Given our monitoring infra pushes data to the monitoring host, this cannot work.

Together with Robert we found out that "asymmetric routing" is our problem. A CC-system like OSD reaches monitor via the usual network (eth0, 10.145.10.207/24 -> 10.168.192.191) but not over the wg ip of that system (in case of monitor, 10.144.169.6/32). This outgoing connection then "punches" a hole into the firewall allowing the non-CC system to reach OSD just for the answering packet. However, based on how wg sets up routes on non-CC systems (routing all CC traffic trough the tunnel because that is the whole point of the wg tunnel), causes these systems so send back their answer via the tunnel. So OSD sends out a ping via eth0 but receives the answer with a wrong source IP.

The solution is to ensure that non-CC systems always answer packets on the interface they receive them. A way to implement this is https://unix.stackexchange.com/a/23345 - based on that article I did the following on monitor:

echo "240     nowg" >> /etc/iproute2/rt_tables
ip route add default via 10.168.195.254 dev eth0 table nowg #taken from `ip r s default`
ip rule add from 10.168.192.191/22 table nowg prio 1

This appears to work and we have data back in grafana: https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=2024-11-27%2018:03:13&to=now&var-host_disks=$__all&refresh=15m

This is just a temporary workaround and will be gone after a reboot so we need to find a way to make it a) reboot safe and b) deploy it via salt on all machines.

Setting due date based on mean cycle time of SUSE QE Tools

who would have thought that a reboot of monitor.qe.nue2.suse.org caused issues after not all changes are made persistent yet ;) See https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=2024-11-30T18:08:34.881Z&to=2024-12-01T06:24:45.668Z&var-host_disks=$__all&refresh=15m

multiple related failed system service alerts, see https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=2024-12-03T18:04:02.089Z&to=2024-12-04T11:38:01.897Z
I added a silence and according rollback action

okurz wrote in #note-19:

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1316/ is merged.

https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&from=now-7d&to=now&var-host_disks=$__all&refresh=15m&viewPanel=panel-78 looks good. What else is necessary here?

right, I restarted the affected machines and checked with: ip -4 rule show && echo --- && ip -6 rule show if the nowg-rules show up:

monitor.qe.nue2.suse.org:
    0:  from all lookup local
    1:  from 10.168.192.191/22 lookup nowg
    32766:  from all lookup main
    32767:  from all lookup default
    ---
    0:  from all lookup local
    1:  from 2a07:de40:a102:5:5054:ff:fe00:894e/64 lookup nowg
    32766:  from all lookup main

this makes monitor reboot-stable, reduces prio and I can execute all rollback-steps. I will also monitor other affected workers (e.g. sapworker1, petrol, diesel, mania, openqa-piworker) a little closer then usual (e.g. messages in #eng-testing).

The whole network got changed once again, no clue if my change is effective or not but we see monitoring data again.

rollback actions not done yet, multiple failed systemd services still reported so please make sure those are covered and referenced accordingly on https://stats.openqa-monitor.qa.suse.de/alerting/silences

right, had to fix quite some stuff on netboot.qe.prg2.suse.org and baremetal-support.qe.prg2.suse.org for these but after all managed to resolve all of them. Also some loki fixes where needed:

• https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1330
• https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1331