action #133769
closed
hydra.opensuse.org causing excessive load on o3 size:S
Added by livdywan 10 months ago.
Updated 10 months ago.
Category:
Regressions/Crashes
Description
Observation¶
While investigating #132167 in the last mob session we noticed that hydra.opensuse.org performs many HTTP requests to the endpoint /admin/influxdb/minion?rc_fail_timespan_minutes=5 (many thousands per hour). User agent is identified as "curl/8.0.1". O3 has no telegraf monitoring, so this is very unexpected.
We double-checked that from o3 itself it is logged with its IP so it's not e.g. munin
Acceptance criteria¶
- AC1: It is known why these requests are performed
- AC2: The request frequency has been reduced (or no more such requests are made at all)
Suggestions¶
- Ask openSUSE heroes who is responsible for hydra.opensuse.org
- If nobody can be identified, use our new nginx rate limiting features to reduce the request frequency :)
- Copied from action #132167: asset uploading failed with http status 502 size:M added
- Status changed from Workable to In Progress
- Assignee set to livdywan
I'll see what I can find out
Running tcpdump -A -s 0 -ni any port 80 shows there's no X-Forwarded-For but it follows the 10.0.0.0/8 via 149.44.161.126 dev shuttle route. According to Bernhard there's no haproxy rule but it could be a machine NATed by hydra.
Also curl/8.0.1 is installed on hydra which runs SLE 12.
Maybe traffic from monitor or metrics gets misrouted?
- Status changed from In Progress to Feedback
The requests were coming from the old o3. I disabled the openqa_minion plugin there (rm /etc/munin/plugins/openqa_minion_*).
Also, these requests were not actually that many. It was requested only a couple of times every 5 minutes.
Same problem as in #133772 (goaccess dashboard was somehow reporting wrong data).
- Status changed from Feedback to Resolved
So let's assume we're good here
Also available in: Atom
PDF