Taken from: https://progress.opensuse.org/issues/110683#note-8
Situation with the NFS servers seem to be related not only to the move of the workers to PRG2 (both NFS servers are remote to the workers now, this was not the case in NUE1), but also to the increase of tap capable qemu_x86_64-large-mem workers in osd.
On September 26th, a load of close to 200 MM jobs out of which 117 required NFS access, created a total of 110 established connections on the server causing a drop of bandwidth to the clients below 1MB/s. These jobs with this bandwidth reached their timeout (of 2h15m, increased from 1h30m, but still below the original timeout before the move to PRG2 which was 4h30m) and failed. Later in the day with 45 parallel jobs running, a total of 24 established connections were observed in the NFS server resulting in clients being able to transfer data at speeds of 2 to 6MB/s, allowing these jobs to finish successfully. We're exploring the possibility of further changes to the test code (increasing timeouts, limiting parallel access to the NFS, etc.) while also adding more NFS servers in PRG1 and redistributing the NFS servers per job groups, but if the bottleneck is not in the NFS server itself but in the network links, then this will not help in solving the issue.
We are collecting our debug and results into the NFS issue in https://confluence.suse.com/display/qasle/NFS+disaster+recovery. It's still a WIP.
We are also tracking in https://progress.opensuse.org/issues/135980 and https://progress.opensuse.org/issues/135923 and https://jira.suse.com/browse/TEAM-8573