QA » openQA Project

Motivation¶

If jobs run into MAX_SETUP_TIME (like we've seen in #96557) or are otherwise cancelled the Minion jobs for asset downloads are not cancelled. That means the worker is unlikely to get out of the situation of being overloaded with too many asset download tasks on its own. Stopping inactive or even active Minion jobs for asset downloads when the related openQA jobs have been cancelled would help with that situation.

Acceptance criteria¶

• AC1: Inactive (or even active) Minion jobs are cancelled if the related openQA job is cancelled.
• AC2: A Minion job can be responsible for multiple openQA jobs (if they share the same assets). This should still work so the cancellation (AC1) should only happen if no other openQA job requires the Minion job.
• AC3: No partial files (or even stale data in the database) are left behind.

Suggestions¶

• Cache service downloads are deduplicated, so make sure no downloads are cancelled that are still required by other openQA jobs on the same worker (might require a new sqlite table to keep track of cancelled jobs)
• Increase or remove the cache service backlog limit once download cancellation is implemented
• Use Minion feature to terminate active jobs properly
• Ensure terminating jobs shut down gracefully
• Make sure not to corrupt the SQLite database by terminating the job too aggressively
• Maybe use USR1/USR2 signals to run custom termination code in the job