Actions
action #160985
opennginx on osd returns 503 Service Temporarily Unavailable when accessing assets due to rate limiting
Start date:
2024-05-27
Due date:
% Done:
0%
Estimated time:
Description
Observation¶
After merging https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1182 which added rate limits to nginx, we do see job failures on multiple places.
The nginx log:
2024/05/27 13:20:55 [error] 15939#15939: *654335 limiting requests, excess: 0.880 by zone "defaultlimit", client: 2a07:de40:b203:12:7ec2:55ff:fe24:de9c, server: openqa.suse.de, request: "GET /assets/iso/SLE-15-SP5-Online-ppc64le-Build139.11-Media1.iso HTTP/1.1", host: "openqa.suse.de"
2024/05/27 13:20:55 [error] 15939#15939: *654377 limiting requests, excess: 1.860 by zone "defaultlimit", client: 2a07:de40:b203:12:7ec2:55ff:fe24:de2a, server: openqa.suse.de, request: "GET /assets/hdd/sle-15-SP5-x86_64-Build:34037:freeradius-server-HA-BV.qcow2 HTTP/1.1", host: "openqa.suse.de"
2024/05/27 13:20:55 [error] 15939#15939: *654336 limiting requests, excess: 0.880 by zone "defaultlimit", client: 2a07:de40:b203:12:7ec2:55ff:fe24:de9c, server: openqa.suse.de, request: "GET /assets/hdd/openqa_support_server_sles12sp3.x86_64.qcow2 HTTP/1.1", host: "openqa.suse.de"
2024/05/27 13:20:55 [error] 15939#15939: *654433 limiting requests, excess: 0.850 by zone "defaultlimit", client: 10.100.96.76, server: openqa.suse.de, request: "GET /assets/hdd/SLED-15-SP4-:34041:xdg-desktop-portal-x86_64@64bit_for_regression.qcow2 HTTP/1.1", host: "openqa.suse.de"
2024/05/27 13:20:55 [error] 15939#15939: *654995 limiting requests, excess: 0.830 by zone "defaultlimit", client: 2a07:de40:b203:12:7ec2:55ff:fe24:de2a, server: openqa.suse.de, request: "GET /assets/hdd/sle-15-SP4-x86_64-Build:34037:freeradius-server-HA-BV.qcow2 HTTP/1.1", host: "openqa.suse.de"
Gitlab pipelines: https://gitlab.suse.de/openqa/scripts-ci/-/pipelines/1143088
Jobs: https://openqa.suse.de/tests/14443476
The merge request was reverted (hence normal priority for now) in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1197 until we found out better settings for the limits
Actions