action #155743
Updated by livdywan 3 months ago
## Observation
Several emails with the subject **Munin - minion Minion Jobs** and content like this:
```
opensuse.org :: openqa.opensuse.org :: Minion Jobs - see https://openqa.opensuse.org/minion/jobs?state=failed
WARNINGs: failed is 452.00 (outside range [:400]).
```
Looking at https://openqa.opensuse.org/minion/jobs?state=failed a lot of [obs_run_run jobs fail](https://openqa.opensuse.org/minion/jobs?id=3440404) reveals failed jobs as recent as 2024-02-11T10:08:17.307669Z:
```
---
args:
- project: openSUSE:Factory:PowerPC:ToTest
url: https://api.opensuse.org/public/build/openSUSE:Factory:PowerPC:ToTest/_result?package=000product
attempts: 1
children: []
created: 2024-02-11T10:06:07.856414Z
delayed: 2024-02-11T10:06:07.856414Z
expires: ~
finished: 2024-02-11T10:08:17.307669Z
id: 3412364
lax: 0
notes:
gru_id: 19905665
project_lock: 1
parents: []
priority: 100
queue: default
result:
code: 512
message: |-
openSUSE:Factory:PowerPC:ToTest/base/ exit code: 1 (1 failures total so far)
openSUSE:Factory:PowerPC:ToTest/microos/ exit code: 1 (2 failures total so far)
retried: ~
retries: 0
started: 2024-02-11T10:06:07.858866Z
state: failed
task: obs_rsync_run
time: 2024-02-21T12:07:01.731854Z
worker: 1952
```
and
```
---
args:
- project: openSUSE:Factory:LegacyX86:ToTest
url: https://api.opensuse.org/public/build/openSUSE:Factory:LegacyX86:ToTest/_result?package=000product
attempts: 1
children: []
created: 2024-02-09T13:33:44.131117Z
delayed: 2024-02-09T13:33:44.131117Z
expires: ~
finished: 2024-02-09T13:35:39.515968Z
id: 3407299
lax: 0
notes:
gru_id: 19902081
project_lock: 1
parents: []
priority: 100
queue: default
result: 'Job terminated unexpectedly (exit code: 0, signal: 15)'
retried: ~
retries: 0
started: 2024-02-09T13:33:44.133221Z
state: failed
task: obs_rsync_run
time: 2024-02-21T12:07:01.731854Z
worker: 1950
```
as well as
```
---
args:
- project: openSUSE:Leap:15.6:ToTest
url: https://api.opensuse.org/public/build/openSUSE:Leap:15.6:ToTest/_result?package=000product
attempts: 1
children: []
created: 2024-02-09T01:21:47.455035Z
delayed: 2024-02-09T01:21:47.455035Z
expires: ~
finished: 2024-02-09T01:25:08.329909Z
id: 3404260
lax: 0
notes:
gru_id: 19899816
project_lock: 1
parents: []
priority: 100
queue: default
result:
code: 256
message: No message
retried: ~
retries: 0
started: 2024-02-09T01:21:47.456660Z
state: failed
task: obs_rsync_run
time: 2024-02-21T12:07:01.731854Z
worker: 1950
```
## Suggestions
*