Project

General

Profile

Actions
Copy link

action #133769

closed

hydra.opensuse.org causing excessive load on o3 size:S

Added by livdywan 9 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
livdywan
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2023-06-29
Due date:
2023-08-04
% Done:

0%

Estimated time:

Description

Observation

While investigating #132167 in the last mob session we noticed that hydra.opensuse.org performs many HTTP requests to the endpoint /admin/influxdb/minion?rc_fail_timespan_minutes=5 (many thousands per hour). User agent is identified as "curl/8.0.1". O3 has no telegraf monitoring, so this is very unexpected.
We double-checked that from o3 itself it is logged with its IP so it's not e.g. munin

Acceptance criteria

  • AC1: It is known why these requests are performed
  • AC2: The request frequency has been reduced (or no more such requests are made at all)

Suggestions

  • Ask openSUSE heroes who is responsible for hydra.opensuse.org
  • If nobody can be identified, use our new nginx rate limiting features to reduce the request frequency :)

Related issues 1 (0 open — 1 closed)

Copied from openQA Project - action #132167: asset uploading failed with http status 502 size:MResolvedkraih2023-06-292023-08-30

Actions
Actions
Copy link
 #1

Updated by livdywan 9 months ago

  • Copied from action #132167: asset uploading failed with http status 502 size:M added
Actions
Copy link
 #2

Updated by livdywan 9 months ago

  • Status changed from Workable to In Progress
  • Assignee set to livdywan

I'll see what I can find out

Actions
Copy link
 #3

Updated by livdywan 9 months ago

Running tcpdump -A -s 0 -ni any port 80 shows there's no X-Forwarded-For but it follows the 10.0.0.0/8 via 149.44.161.126 dev shuttle route. According to Bernhard there's no haproxy rule but it could be a machine NATed by hydra.
Also curl/8.0.1 is installed on hydra which runs SLE 12.
Maybe traffic from monitor or metrics gets misrouted?

Actions
Copy link
 #5

Updated by tinita 9 months ago

  • Status changed from In Progress to Feedback

The requests were coming from the old o3. I disabled the openqa_minion plugin there (rm /etc/munin/plugins/openqa_minion_*).

Actions
Copy link
 #6

Updated by tinita 9 months ago

Also, these requests were not actually that many. It was requested only a couple of times every 5 minutes.
Same problem as in #133772 (goaccess dashboard was somehow reporting wrong data).

Actions
Copy link
 #7

Updated by livdywan 9 months ago

  • Status changed from Feedback to Resolved

So let's assume we're good here

Actions
Copy link

Also available in: Atom PDF