Looks like this can be reproduced even outside of a VM via e.g. curl 'http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-oss-aarch64-Snapshot20240423/repodata/repomd.xml'. But e.g. curl 'http://www.google.de' works so probably specific to reaching o3.
It works with HTTPs, though. Not sure why only HTTP ceased to work running the MM setup script.
Not sure how this worked before. For now I disabled the worker slots on aarch64-o3 by assigning a different worker class in /etc/openqa/workers.ini.
I'm wondering whether this scenario has ever worked on aarch64-o3 since it was moved to the FC basement (because I couldn't find a passing job in that scenario that actually ran on aarch64-o3). And yes, the migration to nginx probably made things worse as well.
And considering jobs like https://openqa.opensuse.org/tests/4103335#step/gnome_window_switcher/9 the network on aarch64-o3 is definitely not completely broken (also not inside VMs).
The output of this network connectivity check looks also good - and the test only fails later trying to access o3: https://openqa.opensuse.org/tests/4104487#step/hostname/25
So I think the problem is not that the entire network is unreachable on aarch64-o3 but only that http traffic to o3 doesn't work.
But it might still be related to the MM setup. The most recent job that still worked (and relied on repo refreshing) is from 5 days ago (before my changes): https://openqa.opensuse.org/tests/4094290#step/zypper_ref/18
It looks like there are jobs that ran after the MM setup but that could refresh repositories just fine, e.g. https://openqa.opensuse.org/tests/4102678#step/zypper_ref/15 (and it really used o3 and plain http, see https://openqa.opensuse.org/tests/4102678#step/zypper_ar/6).
Note that I also cannot reach o3 via http from my laptop (in VPN) or from backup-qam.qe.nue2.suse.org (which is in the neighboring rack). Only https works. I could reach o3 via http only from workers within the o3 network (e.g. arm1).
Looks like the NGINX config was modified around the time issues starting to come up:
Apr 24 09:33 /etc/nginx/vhosts.d/openqa.conf
Considering okurz pts/0 2a07:de40:b2bf:2 Wed Apr 24 09:33 still logged in it was maybe okurz who changed it. EDIT: It was this change: #133358#note-14