Actions
action #133769
closedhydra.opensuse.org causing excessive load on o3 size:S
Description
Observation¶
While investigating #132167 in the last mob session we noticed that hydra.opensuse.org performs many HTTP requests to the endpoint /admin/influxdb/minion?rc_fail_timespan_minutes=5
(many thousands per hour). User agent is identified as "curl/8.0.1". O3 has no telegraf monitoring, so this is very unexpected.
We double-checked that from o3 itself it is logged with its IP so it's not e.g. munin
Acceptance criteria¶
- AC1: It is known why these requests are performed
- AC2: The request frequency has been reduced (or no more such requests are made at all)
Suggestions¶
- Ask openSUSE heroes who is responsible for hydra.opensuse.org
- If nobody can be identified, use our new nginx rate limiting features to reduce the request frequency :)
Updated by livdywan 10 months ago
- Copied from action #132167: asset uploading failed with http status 502 size:M added
Updated by livdywan 10 months ago
Running tcpdump -A -s 0 -ni any port 80
shows there's no X-Forwarded-For
but it follows the 10.0.0.0/8 via 149.44.161.126 dev shuttle
route. According to Bernhard there's no haproxy rule but it could be a machine NATed by hydra.
Also curl/8.0.1 is installed on hydra which runs SLE 12.
Maybe traffic from monitor or metrics gets misrouted?
Actions