action #94237: No alert about too many scheduled tests size:S - openQA Infrastructure - openSUSE Project Management Tool

Actions

Copy link

action #94237

closed

No alert about too many scheduled tests size:S

Added by okurz over 3 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

okurz

Category:

Target version:

openQA Project - Ready

Start date:

2021-06-18

Due date:

2021-08-30

% Done:

Estimated time:

Description

Observation¶

https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?tab=alert&editPanel=9&viewPanel=9&orgId=1&from=1623621600000&to=1623967199000 shows that for a longer time there had been more than 3k scheduled tests but grafana does not show that it would have sent alert messages.

Acceptance criteria¶

AC1: Alert messages are sent when scheduled jobs exceed a defined alert threshold

Suggestions¶

Investigate why there was this sudden surge in nearly 9k blocked jobs on 2021-06-14. Was that because openQABot was offline in the days before that? -> yes, we assume that was the case
Investigate if there was an alert message which maybe just does not show in grafana but was still sent -> according to http://mailman.suse.de/mlarch/SuSE/osd-admins/2021/osd-admins.2021.06/maillist.html there was no such email received by osd-admins@suse.de
Crosscheck system logs on the grafana instance
Ensure that alert messages are sent

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by jbaier_cz over 3 years ago

The openQABot was offline since Jun 10th without anyone noticing. It was re-enabled on Monday and all the missing jobs were scheduled during the afternoon.

Actions

Copy link