Project

General

Profile

Actions
Copy link

action #159651

closed

coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances

coordination #108209: [epic] Reduce load on OSD

high response times on osd - nginx with enabled rate limiting features size:S

Added by okurz 3 months ago. Updated about 1 month ago.

Status:
Rejected
Priority:
Low
Assignee:
okurz
Category:
Feature requests
Target version:
Ready
Start date:
2024-04-26
Due date:
2024-06-14
% Done:

0%

Estimated time:
Tags:
infra

Description

Motivation

Apache in prefork mode uses a lot of resources to provide mediocre performance. After nginx is deployed on OSD we should ensure that nginx helps to improve performance either with our default config or with additional config tweaks

Acceptance criteria

  • AC1: It is known if Nginx default config or rate limiting features help to prevent OSD unresponsiveness

Suggestions

  • Follow what was done for Nginx deployment on OSD in #130636 and #159654
  • Take a look at what we already do or didn't want to do on o3, see #129490
  • Monitor performance over some time, in particular if HTTP response graphs from openQA directly still show unresponsiveness windows or not
  • Add changes to salt-states-openqa for any nginx config tweaks

Out of scope

  • Any major research or big config tweaks for rate limiting, just some very basics if necessary at all

Related issues 2 (0 open — 2 closed)

Copied from openQA Project - action #130636: high response times on osd - Try nginx on OSD size:SResolvedmkittler2024-05-17

Actions
Copied to openQA Project - action #159654: high response times on osd - nginx properly monitored in grafana size:SResolvedjbaier_cz2024-04-26

Actions
Actions
Copy link
 #1

Updated by okurz 3 months ago

  • Copied from action #130636: high response times on osd - Try nginx on OSD size:S added
Actions
Copy link
 #2

Updated by okurz 3 months ago

  • Copied to action #159654: high response times on osd - nginx properly monitored in grafana size:S added
Actions
Copy link
 #3

Updated by jbaier_cz 3 months ago

  • Subject changed from high response times on osd - nginx with enabled load limiting or load balancing features to high response times on osd - nginx with enabled rate limiting features size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #4

Updated by okurz about 2 months ago

  • Due date set to 2024-06-09
  • Status changed from Workable to Feedback
  • Assignee set to okurz
  • Priority changed from Normal to Low
  • Target version changed from Tools - Next to Ready
Actions
Copy link
 #5

Updated by mkittler about 2 months ago · Edited

We ran into the rate limit after merging https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1182 so we had to revert it. Tests ran into the limit on asset downloads (via the cache service). See https://suse.slack.com/archives/C02CANHLANP/p1716807019930009 for details.

Actions
Copy link
 #6

Updated by livdywan about 1 month ago · Edited

Are we good then? What and whose feedback is this waiting on?

I guess we're still waiting on @ph03nix here.

Actions
Copy link
 #7

Updated by livdywan about 1 month ago

  • Due date changed from 2024-06-09 to 2024-06-14
Actions
Copy link
 #8

Updated by ph03nix about 1 month ago

livdywan wrote in #note-6:

Are we good then? What and whose feedback is this waiting on?

I guess we're still waiting on @ph03nix here.

I would not know what feedback is required - We figured that having IP-based rate limits for OSD is not working as expected.

Actions
Copy link
 #9

Updated by livdywan about 1 month ago

  • Status changed from Feedback to Rejected

Based on discussing it in Slack it seems we agree this won't work for us.

Actions
Copy link

Also available in: Atom PDF