QA » openQA Project » openQA Infrastructure

Observation¶

Several emails with the subject Munin - minion Minion Jobs and content like this:

opensuse.org :: openqa.opensuse.org :: Minion Jobs - see https://openqa.opensuse.org/minion/jobs?state=failed
    WARNINGs: failed is 452.00 (outside range [:400]).

Looking at https://openqa.opensuse.org/minion/jobs?state=failed a lot of obs_run_run jobs fail reveals failed jobs as recent as 2024-02-11T10:08:17.307669Z:

---
args:
- project: openSUSE:Factory:PowerPC:ToTest
  url: https://api.opensuse.org/public/build/openSUSE:Factory:PowerPC:ToTest/_result?package=000product
attempts: 1
children: []
created: 2024-02-11T10:06:07.856414Z
delayed: 2024-02-11T10:06:07.856414Z
expires: ~
finished: 2024-02-11T10:08:17.307669Z
id: 3412364
lax: 0
notes:
  gru_id: 19905665
  project_lock: 1
parents: []
priority: 100
queue: default
result:
  code: 512
  message: |-
    openSUSE:Factory:PowerPC:ToTest/base/ exit code: 1 (1 failures total so far)
    openSUSE:Factory:PowerPC:ToTest/microos/ exit code: 1 (2 failures total so far)
retried: ~
retries: 0
started: 2024-02-11T10:06:07.858866Z
state: failed
task: obs_rsync_run
time: 2024-02-21T12:07:01.731854Z
worker: 1952

and

---
args:
- project: openSUSE:Factory:LegacyX86:ToTest
  url: https://api.opensuse.org/public/build/openSUSE:Factory:LegacyX86:ToTest/_result?package=000product
attempts: 1
children: []
created: 2024-02-09T13:33:44.131117Z
delayed: 2024-02-09T13:33:44.131117Z
expires: ~
finished: 2024-02-09T13:35:39.515968Z
id: 3407299
lax: 0
notes:
  gru_id: 19902081
  project_lock: 1
parents: []
priority: 100
queue: default
result: 'Job terminated unexpectedly (exit code: 0, signal: 15)'
retried: ~
retries: 0
started: 2024-02-09T13:33:44.133221Z
state: failed
task: obs_rsync_run
time: 2024-02-21T12:07:01.731854Z
worker: 1950

as well as

---
args:
- project: openSUSE:Leap:15.6:ToTest
  url: https://api.opensuse.org/public/build/openSUSE:Leap:15.6:ToTest/_result?package=000product
attempts: 1
children: []
created: 2024-02-09T01:21:47.455035Z
delayed: 2024-02-09T01:21:47.455035Z
expires: ~
finished: 2024-02-09T01:25:08.329909Z
id: 3404260
lax: 0
notes:
  gru_id: 19899816
  project_lock: 1
parents: []
priority: 100
queue: default
result:
  code: 256
  message: No message
retried: ~
retries: 0
started: 2024-02-09T01:21:47.456660Z
state: failed
task: obs_rsync_run
time: 2024-02-21T12:07:01.731854Z
worker: 1950

Suggestions¶