action #108980: Better handle minion tasks failing with "Job terminated unexpectedly" - OpenQA::Task::Asset::Download - openQA Project - openSUSE Project Management Tool

Actions

action #108980

open

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

coordination #96263: [epic] Exclude certain Minion tasks from "Too many Minion job failures alert" alert

coordination #99831: [epic] Better handle minion tasks failing with "Job terminated unexpectedly"

Status:

New

Priority:

Normal

Assignee:

Category:

Feature requests

Target version:

Start date:

2022-03-25

Due date:

% Done:

Estimated time:

Description

AC1: minion job "download_asset" has a sigterm handler to decide how to shut down in a clean way in a reasonable time
AC2: Our minion job list on OSD and O3 do not show any "Job terminated unexpectedly" over multiple deployments for "download_asset"

Implement sigterm handler for "download_asset" like it has already been done for other jobs (e.g. #103416)
Test on o3 and osd either by manually restarting openqa-gru multiple times or awaiting the result from multiple deployments and checking the minion dashboard, e.g. openqa.suse.de/minion/jobs?state=failed