coordination #80908
Updated by okurz about 2 years ago
## Motivation We want to upgrade more often but not disrupt openQA jobs on package upgrades as well as re-read configuration whenever a job finishes ## Acceptance criteria * **AC1:** *DONE* openQA worker packages can be upgraded continously without interrupting currently running openQA jobs * **AC2:** *DONE* openQA workers read updated configuration, e.g. WORKER_CLASS, whenever they are ready to pick up new jobs * **AC3:** Both o3 and osd deploy automatically after every change if all relevant checks have passed ## Ideas * Use different git branches, e.g. "dev" or "main" and then "stable" or "tested" or "release" and create automatic merges by bots based on checks * Switch o3 workers to either deploy from worker containers which we update continuously or change the worker to allow non-transactional updates ## Further details One could try what apache does with `apache2ctl graceful` or `systemctl reload apache2`, e.g. see https://elearning.wsldp.com/pcmagazine/apache-graceful-restart-centos-7/ The restart of openQA workers could be simply prevented or delayed, e.g. with `SendSIGKILL=` in the openQA worker systemd service definitions which every openQA user is free to do, but then we could potentially wait hours until the service restarts if ever. Maybe we can still add a "graceful-stop" mode, wait a useful time for all jobs to finish and then restart (or even reboot the host).