action #159651
closedcoordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances
coordination #108209: [epic] Reduce load on OSD
high response times on osd - nginx with enabled rate limiting features size:S
0%
Description
Motivation¶
Apache in prefork mode uses a lot of resources to provide mediocre performance. After nginx is deployed on OSD we should ensure that nginx helps to improve performance either with our default config or with additional config tweaks
Acceptance criteria¶
- AC1: It is known if Nginx default config or rate limiting features help to prevent OSD unresponsiveness
Suggestions¶
- Follow what was done for Nginx deployment on OSD in #130636 and #159654
- Take a look at what we already do or didn't want to do on o3, see #129490
- Monitor performance over some time, in particular if HTTP response graphs from openQA directly still show unresponsiveness windows or not
- Add changes to salt-states-openqa for any nginx config tweaks
Out of scope¶
- Any major research or big config tweaks for rate limiting, just some very basics if necessary at all
Updated by okurz 6 months ago
- Copied from action #130636: high response times on osd - Try nginx on OSD size:S added
Updated by okurz 6 months ago
- Copied to action #159654: high response times on osd - nginx properly monitored in grafana size:S added
Updated by okurz 6 months ago
- Due date set to 2024-06-09
- Status changed from Workable to Feedback
- Assignee set to okurz
- Priority changed from Normal to Low
- Target version changed from Tools - Next to Ready
awaiting results from https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1182 by fniederwanger
Updated by mkittler 5 months ago · Edited
We ran into the rate limit after merging https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1182 so we had to revert it. Tests ran into the limit on asset downloads (via the cache service). See https://suse.slack.com/archives/C02CANHLANP/p1716807019930009 for details.
Updated by livdywan 5 months ago
- Status changed from Feedback to Rejected
Based on discussing it in Slack it seems we agree this won't work for us.