Actions
action #179885
opencoordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
coordination #96263: [epic] Exclude certain Minion tasks from "Too many Minion job failures alert" alert
coordination #99831: [epic] Better handle minion tasks failing with "Job terminated unexpectedly"
Better handle minion tasks failing with "Job terminated unexpectedly" - OpenQA::Task::Git::Clone git_clone
Status:
New
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2025-04-02
Due date:
% Done:
0%
Estimated time:
Description
Acceptance criteria¶
- AC1: minion job "git_clone" has a sigterm handler to decide how to shut down in a clean way in a reasonable time
- AC2: Our minion job list on OSD and O3 do not show any "Job terminated unexpectedly" over multiple deployments for "git_clone"
Suggestions¶
- Implement sigterm handler for "download_asset" like it has already been done for other jobs (e.g. #103416)
- Test on o3 and osd either by manually restarting openqa-gru multiple times or awaiting the result from multiple deployments and checking the minion dashboard, e.g. openqa.suse.de/minion/jobs?state=failed
- (Note: I don't know who suggested this but restarting openqa-gru multiple times in production can result in the actual problem, and then we are left with incomplete jobs, so try to do it in a local instance or at least check and repair things if necessary)
Actions