action #130636
opencoordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances
coordination #108209: [epic] Reduce load on OSD
high response times on osd - Try nginx on osd with enabled load limiting or load balancing features
0%
Description
Motivation¶
Apache in prefork mode uses a lot of resources to provide mediocre performance.
Acceptance criteria¶
- AC1: It is known if Nginx rate limiting features work for our use cases
- AC2: Nginx has been deployed successfully on OSD
Suggestions¶
- Make sure there is an easy way to switch back to Apache in case something goes wrong
- See #129490 for results from O3
- Adapt O3 nginx config for HTTP + HTTPS (O3 only requires HTTP)
- Add changes to salt-states-openqa including monitoring: we have multiple panels regarding apache that need to be adapted for nginx as applicable
- Ensure that we have no alerts regarding "oh no, apache is down" ;)
Updated by livdywan 6 months ago
- Copied from action #129490: high response times on osd - Try nginx on o3 with enabled load limiting or load balancing features added
Updated by kraih 6 months ago
During the openQA weekly we've talked about this ticket and consider it a good candidate for a mob session. Main problems to solve are Salt deployment and SSL configuration. As well as a simple way to rollback the deployment and use Apache again in case something goes wrong.
Updated by okurz 6 months ago
We can prepare the deployment of nginx in parallel to apache, have it deployed and at any time decide when to switch by just disabling/enabling services accordingly. The deployment needs to consider dehydrated+nginx as well. We can switch OSD to nginx to gather realtime data before we suggest to use nginx as default in our openQA documentation and CI infrastructure.