action #157690
closedcoordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances
coordination #157669: websockets+scheduler improvements to support more online worker instances
Simple global limit of registered/online workers size:M
Description
Motivation¶
As observed in #157666 we seem to have a problem when too many openQA workers are connected at the same time. Similar to the global job limit in #129619 we should add a simple, configurable global limit of how many workers can be online at the same time to one openQA instance.
Acceptance criteria¶
- AC1: A KISS configurable for number of online workers exists
- AC2: Rejected openQA workers exceeding the mentioned limit explicitly log or fail that situation
Rollback actions¶
R1: Remove silence
alertname=Broken workers alerthttps://stats.openqa-monitor.qa.suse.de/alerting/silencesalertname=Broken workers alert
Suggestions¶
- In the openQA web API reject openQA worker registration or handling if a global, configurable limit is exceeded. Maybe it makes also sense to allow the registration but prevent the creation of the web socket connection.
- Select a sensible default, e.g. 1k.
- Explicitly log or fail the openQA worker if rejected. A worker could be registered and be tracked as "offline" while rejected and not connected with at best an error message visible in the web UI. If too complicated start with something simpler, e.g. fatal fails of the worker instance.
- Come up with an approach that allows a worker to attempt a connection again at some point, e.g. the web UI could send a waiting period with the rejection. The normal retry on failure would probably just cause more noise.
- Consider to use
OpenQA::Utils::usleep_backoffsame as is done in the chunk uploadingioned limit explicitly log or fail that situation
Updated by okurz 3 months ago
- Related to action #167557: OSD not starting new jobs on 2024-09-28 due to >1k worker instances connected, overloading websocket server added
Updated by okurz 2 months ago
- Copied to action #168178: Limit connected online workers based on websocket+scheduler load size:M added
Updated by okurz 2 months ago · Edited
- Status changed from Resolved to Feedback
I brought up w38+w39 as part of #166802 and now https://openqa.suse.de/admin/workers shows 1038 online workers. Shouldn't that be prevented by the default limit of 1k? I think what could be happening is that while the scheduler tries to limit there are still connection attempts coming in and AFAIK we don't disconnected workers that "made it in". So I now set a number of 900 in OSD locally in /etc/openqa/openqa.ini to see if that has an effect.
Updated by okurz 2 months ago
- Related to action #166802: Recover worker37, worker38, worker39 size:S added
Updated by openqa_review 2 months ago
- Due date set to 2024-10-29
Setting due date based on mean cycle time of SUSE QE Tools
Updated by mkittler 2 months ago · Edited
I could not reproduce the issue that the limit is sometimes not effective by extending the ws server and scalability tests. See https://github.com/os-autoinst/openQA/pull/6009 for my changes.
I actually tried harder than this PR and let the ws restart with a lowered limit - just like what we did in production. You can see the change here (the last commit is the interesting one).
When I run this locally (e.g. SCALABILITY_TEST_WITH_OFFLINE_WEBUI_HOST=0 SCALABILITY_TEST_JOB_COUNT=10 SCALABILITY_TEST_WORKER_COUNT=45 SCALABILITY_TEST_WORKER_LIMIT=50 SCALABILITY_TEST_WORKER_LIMIT_2=25 prove -l -v t/43-scheduling-and-worker-scalability.t) it always works. It makes probably no sense to commit this change as-is (due to the sleep and I'm not sure whether we want to enable the restart subtest in the CI).
Updated by mkittler 2 months ago
- Status changed from In Progress to Feedback
I'm only waiting for an additional review on the PR because otherwise I don't know what else to improve. The online limit is not always working but most of the time at least. We should probably focus on fixing the actual performance problems now.