coordination #103944: [saga][epic] Scale up: More robust handling of diverse infrastructure with varying performance
coordination #98463: [epic] Avoid too slow asset downloads leading to jobs exceeding the timeout with or run into auto_review:"(timeout: setup exceeded MAX_SETUP_TIME|Cache service queue already full)":retry
Abort asset download via the cache service when related job runs into a timeout (or is otherwise cancelled) size:M
If jobs run into
MAX_SETUP_TIME (like we've seen in #96557) or are otherwise cancelled the Minion jobs for asset downloads are not cancelled. That means the worker is unlikely to get out of the situation of being overloaded with too many asset download tasks on its own. Stopping inactive or even active Minion jobs for asset downloads when the related openQA jobs have been cancelled would help with that situation.
- AC1: Inactive (or even active) Minion jobs are cancelled if the related openQA job is cancelled.
- AC2: A Minion job can be responsible for multiple openQA jobs (if they share the same assets). This should still work so the cancellation (AC1) should only happen if no other openQA job requires the Minion job.
- AC3: No partial files (or even stale data in the database) are left behind.
- Cache service downloads are deduplicated, so make sure no downloads are cancelled that are still required by other openQA jobs on the same worker (might require a new sqlite table to keep track of cancelled jobs)
- Increase or remove the cache service backlog limit once download cancellation is implemented
- Use Minion feature to terminate active jobs properly
- Ensure terminating jobs shut down gracefully
- Make sure not to corrupt the SQLite database by terminating the job too aggressively
- Maybe use USR1/USR2 signals to run custom termination code in the job
- Subject changed from Abort asset download via the cache service when related job runs into a timeout (or is otherwise cancelled) to Abort asset download via the cache service when related job runs into a timeout (or is otherwise cancelled) size:M
- Description updated (diff)
- Status changed from New to Workable