Project

General

Profile

Actions

action #129490

closed

coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances

coordination #108209: [epic] Reduce load on OSD

high response times on osd - Try nginx on o3 with enabled load limiting or load balancing features

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Apache in prefork mode uses a lot of resources to provide mediocre performance.

Acceptance criteria

  • AC1: It is known if Nginx rate limiting features work for our use cases
  • AC2: Nginx has been deployed successfully on O3

Suggestions

  • See #129619 for results from previous experiments

Files

processes.png (172 KB) processes.png kraih, 2023-06-06 12:16
memory.png (345 KB) memory.png kraih, 2023-06-06 12:16

Related issues 6 (0 open6 closed)

Related to openQA Project (public) - action #129619: high response times on osd - simple limit of jobs running concurrently in openQA size:MResolvedtinita2023-05-20

Actions
Related to openQA Project (public) - action #130477: [O3]http connection to O3 repo is broken sporadically in virtualization tests, likely due to systemd dependencies on apache/nginx size:MResolvedmkittler2023-06-07

Actions
Related to openQA Project (public) - action #132167: asset uploading failed with http status 502 size:MResolvedkraih2023-06-292023-08-30

Actions
Copied from openQA Project (public) - action #129487: high response times on osd - Limit the number of concurrent job upload handling on webUI side. Can we use a semaphore or lock using the database? size:MRejectedokurz

Actions
Copied to openQA Infrastructure (public) - action #129493: high response times on osd - better nice level for velociraptorResolvedokurz

Actions
Copied to openQA Project (public) - action #130636: high response times on osd - Try nginx on OSD size:SResolvedmkittler

Actions
Actions

Also available in: Atom PDF