QA (public) » openQA Project (public)

Motivation¶

As observed in #157666 we seem to have a problem when too many openQA workers are connected at the same time. Similar to the global job limit in #129619 we should add a simple, configurable global limit of how many workers can be online at the same time to one openQA instance.

Acceptance criteria¶

• AC1: A KISS configurable for number of online workers exists
• AC2: Rejected openQA workers exceeding the mentioned limit explicitly log or fail that situation

Rollback actions¶

• R1: Remove silence alertname=Broken workers alert https://stats.openqa-monitor.qa.suse.de/alerting/silences

  alertname=Broken workers alert

Suggestions¶

• In the openQA web API reject openQA worker registration or handling if a global, configurable limit is exceeded. Maybe it makes also sense to allow the registration but prevent the creation of the web socket connection.
• Select a sensible default, e.g. 1k.
• Explicitly log or fail the openQA worker if rejected. A worker could be registered and be tracked as "offline" while rejected and not connected with at best an error message visible in the web UI. If too complicated start with something simpler, e.g. fatal fails of the worker instance.
• Come up with an approach that allows a worker to attempt a connection again at some point, e.g. the web UI could send a waiting period with the rejection. The normal retry on failure would probably just cause more noise.
• Consider to use OpenQA::Utils::usleep_backoff same as is done in the chunk uploadingioned limit explicitly log or fail that situation