Project

General

Profile

Actions

action #137300

closed

[FIRING:1] (Incomplete jobs (not restarted) of last 24h alert Salt size:M

Added by tinita 7 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2023-10-02
Due date:
% Done:

0%

Estimated time:

Description

Observation

Firing [stats.openqa-monitor.qa.suse.de]
Incomplete jobs (not restarted) of last 24h alert
View alert [stats.openqa-monitor.qa.suse.de]
Values
B0=314 
Labels
alertname
Incomplete jobs (not restarted) of last 24h alert

http://stats.openqa-monitor.qa.suse.de/alerting/grafana/cXo2cmBVk/view
http://stats.openqa-monitor.qa.suse.de/alerting/grafana/cXo2cmBVk/view

Acceptance criteria

  • AC1: Alert is not triggered anymore
  • AC2: It is known what triggered the alert originally

Suggestions

  • Investigate what happened on September 28th
  • run select id,test,reason from jobs where result='incomplete' and t_created >= '2023-09-27' and t_created <= '2023-09-29' limit 30; to find anything obvious by reason. I think we can group by shortened reason

Related issues 1 (0 open1 closed)

Related to openQA Project - action #96684: Abort asset download via the cache service when related job runs into a timeout (or is otherwise cancelled) size:MRejectedmkittler2021-08-09

Actions
Actions #1

Updated by okurz 7 months ago

  • Tags set to infra, alert, incomplete, osd
  • Priority changed from Normal to High
  • Target version set to Tools - Next
Actions #2

Updated by okurz 7 months ago

  • Target version changed from Tools - Next to Ready
Actions #3

Updated by livdywan 7 months ago

  • Subject changed from [FIRING:1] (Incomplete jobs (not restarted) of last 24h alert Salt to [FIRING:1] (Incomplete jobs (not restarted) of last 24h alert Salt size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #4

Updated by okurz 6 months ago

  • Related to action #96684: Abort asset download via the cache service when related job runs into a timeout (or is otherwise cancelled) size:M added
Actions #5

Updated by okurz 6 months ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz
openqa=> select count(id),left(reason,80) as r from jobs where result='incomplete' and t_created >= '2023-09-27' and t_created <= '2023-09-29' group by r order by count desc limit 30;
 count |                                        r                                         
-------+----------------------------------------------------------------------------------
   565 | asset failure: Failed to download dev_tools.dud to /var/lib/openqa/cache/openqa.
   221 | cache failure: Cache service queue already full (10)
    55 | asset failure: Failed to download SLES-15-SP4-x86_64-mru-install-desktop-with-ad
    50 | asset failure: Failed to download SLES-15-SP5-x86_64-mru-install-desktop-with-ad
    36 | asset failure: Failed to download SLES-15-SP5-x86_64-mru-install-minimal-with-ad
    35 | asset failure: Failed to download SLES-15-SP4-x86_64-mru-install-minimal-with-ad
    31 | backend died: QEMU exited unexpectedly, see log for details
    29 | cache failure: Failed to download dev_tools.dud to /var/lib/openqa/cache/openqa.
    26 | backend died: runcmd '/usr/bin/qemu-img create -f qcow2 -F qcow2 -b /var/lib/ope
    25 | asset failure: Failed to download SLES-15-SP5-aarch64-mru-install-minimal-with-a
    19 | asset failure: Failed to download SLES-15-SP3-x86_64-mru-install-minimal-with-ad

since then this did not reappear again. For the first entry, well, I think this is clearly within the scope of testers so we don't need to care about it. For the second issue "Cache service queue already full" we have #96684 which is already in our backlog so good enough.

Actions

Also available in: Atom PDF